PoE cameras' video occasionally freezes & slow pings via NVR

[aesterling]

I'm trying to determine why my cameras' video occasionally freezes/drops frames like in these two examples, both on the NVR recordings and Blue Iris recordings.




Hardware:

• Hikvision NVR DS-7616NI-I2 (built in PoE switch, latest firmware)
• Hikvision DS-2CD2385FWD-I Turrets (connected to NVR, also latest firmware)
• Blue Iris PC with two network cards (one plugged into one of the NVR's unused PoE ports, the other to the internet. Pulling feed from cameras directly.)

Ping tests:

If I ping any of the PoE cameras on the NVR's subnet using the Blue Iris PC, I get slow ping times like this:

Pinging 192.168.254.5 with 32 bytes of data:
Reply from 192.168.254.5: bytes=32 time=128ms TTL=64
Reply from 192.168.254.5: bytes=32 time=151ms TTL=64
Reply from 192.168.254.5: bytes=32 time=167ms TTL=64
Reply from 192.168.254.5: bytes=32 time=168ms TTL=64
​

If I ping the NVR itself (on the same subnet) I get slightly faster speeds:

Pinging 192.168.254.1 with 32 bytes of data:
Reply from 192.168.254.1: bytes=32 time=62ms TTL=64
Reply from 192.168.254.1: bytes=32 time=63ms TTL=64
Reply from 192.168.254.1: bytes=32 time=60ms TTL=64
Reply from 192.168.254.1: bytes=32 time=65ms TTL=64
​

If I ping the NVR's gigabit interface (which is separate from the PoE ports) I get the following:

Pinging 192.168.86.60 with 32 bytes of data:
Reply from 192.168.86.60: bytes=32 time<1ms TTL=128
Reply from 192.168.86.60: bytes=32 time<1ms TTL=128
Reply from 192.168.86.60: bytes=32 time<1ms TTL=128
Reply from 192.168.86.60: bytes=32 time<1ms TTL=128
​

If I unplug one of the cameras, bypass the NVR, use a PoE injector, put the camera on my regular network, and then ping the camera I get normal 2-3ms times. It also appears to fix the video-freezing problem in Blue Iris.

If I plug the camera back into one of the NVR's PoE ports, disable PoE power to the port, and power the camera with an in-line PoE injector, the video still freezes occasionally and pings are slow. (Point being, it doesn't appear to be power related.)

I've searched and read a ton of threads here on similar issues but would appreciate any insight you can give on this one. Maybe I'm missing something simple and obvious

Obviously, I could buy a PoE switch and bypass the NVR, but I'd love to get it working with existing hardware. Thanks in advance!

Finally, here's a screenshot from one of the camera's network config page for reference:

 

[alastairstevenson]

A long shot - for no real technical reason - try adding 192.168.254.1 as the default gateway for one of the cameras, see if the ping times change.

Though I suspect that treating the NVR PoE ports as a regular switch is a bit optimistic - it may be that the inter-POE-port traffic is being handled by the NVR kernel as opposed to switch hardware.

 

[SouthernYankee]

Move the cameras to a Poe switch. Then record them on bi and the NVR. Passing the video through the NVR is a bad idea. Make sure none of the camera traffic moves through the router.

 

[aesterling]

Thanks, alastairstevenson! I added 192.168.254.1 as the default gateway on one camera and am getting erratic ping times now. (Starts out fine and then slows.)

Pinging 192.168.254.5 with 32 bytes of data:
Reply from 192.168.254.5: bytes=32 time<1ms TTL=64
Reply from 192.168.254.5: bytes=32 time=41ms TTL=64
Reply from 192.168.254.5: bytes=32 time<1ms TTL=64
Reply from 192.168.254.5: bytes=32 time=3ms TTL=64
Reply from 192.168.254.5: bytes=32 time=1ms TTL=64
Reply from 192.168.254.5: bytes=32 time<1ms TTL=64
Reply from 192.168.254.5: bytes=32 time<1ms TTL=64
Reply from 192.168.254.5: bytes=32 time<1ms TTL=64
Reply from 192.168.254.5: bytes=32 time<1ms TTL=64
Reply from 192.168.254.5: bytes=32 time=88ms TTL=64
Reply from 192.168.254.5: bytes=32 time=114ms TTL=64
Reply from 192.168.254.5: bytes=32 time=69ms TTL=64
Reply from 192.168.254.5: bytes=32 time=119ms TTL=64
Reply from 192.168.254.5: bytes=32 time=80ms TTL=64
Reply from 192.168.254.5: bytes=32 time=119ms TTL=64
Reply from 192.168.254.5: bytes=32 time=98ms TTL=64
Reply from 192.168.254.5: bytes=32 time=143ms TTL=64
​

SouthernYankee, thanks as well!
I hoped to avoid buying more hardware but I know a PoE switch will be worth the stability it provides.

 

[pozzello]

so it helps a little to set the default gateway on one cam.
please try setting ALL your cams to use the NVR's IP as the default gateway.

Is your BI pulling streams from the NVR's virtual host ports?

without a default gateway, the cams don't know where to address packets destined for the main LAN IP's,
which is probably resulting in some sort of broadcast storm overloading the PoE switch side of the NVR...
packet captures on the PoE switch side might confirm this, but may not be needed if setting default gateway
on ALL cams clears things up...

 

[aesterling]

pozzello, thanks! I only have two cams set up on the NVR currently and now filled in the default gateway field on each to 192.168.254.1. Interestingly, I left a ping running to one cam while I rebooted the other and saw the ping time drop from 150ms down to <1ms. After both were back up, ping time is still erratic jumping from 2ms up to 200ms and back. The second cam has noticeably slower ping times (and slower web interface) so I could spend more time looking into that camera.

No, BI isn't pulling streams from the NVR's virtual host ports but the feature IS enabled on the NVR. (No improvement when I disable it, btw.) I can't get BI to load that stream even though it plays fine in VLC using this URL rtsp://admin:myPW@192.168.86.55:10554/Streaming/Channels/401. With the config shown below, BI displays a full-res still from the camera every few minutes, then says no signal, then an updated still again. But I assume I likely have something configured incorrectly in BI.

You mentioned packet captures on the PoE switch and it looks like this page on NVR's webpage has the option, although I haven't used it before.

Despite all this, everything is working great 90% of the time. I have smooth, clear video in BI and the NVR but only occasionally does one cam or the other freeze for a couple seconds like in the example videos above. It usually happens when there's motion in the frame. I'd spend my time troubleshooting codec, bitrate, etc settings except the ping results caught my attention first and problems went away when I bypassed the NVR.

 

[pozzello]

edit: i see you have 8MP Hik turret cams.

maybe show us the video config page for one of the cams?

 

[aesterling]

Thanks again! I'm running the cams at full res and frame rate, which obviously isn't necessary, but I'd love to continue if I can solve the occasional freezing issue. I might just need to buy a PoE switch since bypassing the NVR appeared to fix the problem during testing.

Hik video settings: (Alley cam)

BI settings for above cam:

I'm wondering why the Driveway cam and Alley cam show such different info in the status window despite being the same model camera, set up the same, and added to BI the same. (AFAIK)

 

[SouthernYankee]

cut the alley and driveway cameras to 12 frames and set the iframe to 12 also. Not sure that this will improve much. But there is not a need for more than 15 frames per second on a security system. Set the video quality to medium. Set receive buffer to 25 MB on the 8 mp cameras.

You are trying to push a lot for bits through the NVR with 8 mp cameras.

 

[aesterling]

Thanks, SouthernYankee! You're definitely right I'll make these changes and see how it behaves for a while.

I may re-terminate the cat6 to the alley cam as well, just to rule out any problems there. That said, the cable tester showed it was fine at installation.