Would need to know the 5mp camera model you are using? Maybe it is SDI, Not supported. As said before could be a toggle on the wires where you can change format, Might be that it don't like the format of the camera using Auto in the DVRs setup so could try and set that channel for the type of camera it is rather then leaving it on Auto.. If your DVR has that option.. IF the camera don't have toggle, and don't like the format of AHD, or CVI and can't change to meet that format in the DVR for that channel and the camera isn't able to be changed from the native format that it was built at.. Hikvision is TVI, Dahua is CVI and so on..
Sometimes there is 2 BNC connectors on the wires. If there is the other one normally is CVBS and I would connect to that and test that it is at least working.. If you have a CVBS connection that you have to connect to on the PCB like some of my older Analog cameras, Test with CVBS, If there is video out you at least know the camera is working.. Checking the IR only says that the camera is powering the IR, In some cameras the IR and Power to the Camera are on the same wires and don't require the camera for the IR to come on just power and the LDR to detect there is no light for IR to come on even when camera is dead..
Model of Camera? Or Brand?