Need assistance with network throughput

[p1morris]

Hello all,

I searched the forums and did not see any threads directly related to my concern. My apologies if I missed anything already existing.

At home, I'm currently running 5 x Hikvision ColorVu DS-2CD2347G1-LU 4MP turret cameras and 5 x Reolink cameras (these are about 5 years old; I'm replacing with Hikvision cameras going forward as these die). I've had to significantly reduce the video quality for all cameras due to very choppy video streaming on home network (at 1920x1080 4 fps the stream is acceptable; however, audio still cuts in and out). I would like to run my cameras at or near max settings to enjoy the image quality that these can provide.

My home network setup:

• When I remodeled my house several years ago, we ran Cat 6 cables for all cameras and throughout my house for our network. All cameras are setup as POE (none are wifi).
• All cables terminate on 2 patch panels.
• I have a Unifi Security Gateway Router and one Ubiquiti Networks Unifi 802.11ac Dual-Radio PRO Access Point (UAP-AC-PRO-US). NOTE: IPS/IDS turned OFF (in case that would affect anything).
• For the cameras, they are all connected to 3 x Hikvision DS-3E0105P-E2 4-Port 100 Mb/s PoE-Compliant Unmanaged Network Switch to provide power and manage traffic.
• These switches (along with all other traffic on my network) are then connected to a NETGEAR ProSAFE JGS524NA 24-Port Gigabit Rackmount Switch 10/100/1000 Mbps (JGS524NA).
• I use Surveillance Station on my Synology DS918+ NAS to record and view the streams (NOTE: I upgraded the RAM on my NAS to 8 gb). My NAS is also setup to run Plex and 2 x Minecraft servers using Docker.

Camera setup:

• For the Hikvision cameras, they are all set at 1920x1080 at 4 fps using H.265. I tried adjusted to H.265 hoping for an improvement but did not seem to make much of a difference).
• For the Reolink cameras, they are set to 1920x1080 (I don't have an options to change the fps) using H.264.

On a side note, my internet quality is unaffected. I'm getting the max download/upload rates promised by my ISP (200 down/10 up).

The video is choppy when viewing through Surveillance Station and directly using the Hikvision software while on my network. Viewing remotely, I'm not concerned about the quality; I just want to have a high quality video saved for later viewing, if needed.

There are so many different items that might affect my network throughput, I'm not sure where to start. I was using my USG for years; switched to a Bitdefender Box for awhile to try it out; then switched back to the USG for continuity in devices. The quality remained the same using both devices.

I've considered that maybe my NAS cannot handle this much traffic. Watching the resource monitor, CPU and memory utilization is generally ~ 40-50% (while I watch my son play Minecraft with security cameras recording continuously), Network down ~ 4 MB/S, up 100 kb/s, volume utilization 12%.

A few questions:

• Do I need to temper my expectations? Is it unreasonable to hope for max quality with all of my cameras on my network?
• Is it possible that the switch is an issue? Would switching to a managed switch make any difference?
• Any ideas on what might be the limiting factor in my network?

I really appreciate any input that you could provide. Thank you for reading.

Paul

 

[alastairstevenson]

Any ideas on what might be the limiting factor in my network?

From what you've described, provided the cabling is good, the network itself should be fine.
Plenty of headroom for a few cameras at say up to 16Mbps each with a gigabit core switch and gigabit uplinks from your PoE switches.
What error counts if any show on the relevant ports of the core switch?

directly using the Hikvision software while on my network.

What Hikvision software?
That, and the NAS performance, may be the common weak link here.

What's the video like when viewed directly on the camera web GUI Live View with a browser?
That would eliminate some variables.

 

[biggen]

I'd say its not your network. My guess would be the NAS which is running the surveillance station. Looking at its specs its a 4-core Celeron which, if we are honest, is pretty low end by today's standard. Its fine for file sharing but asking it to transcode video on the fly may be too much to ask. I'm just guessing but you may not be seeing 100% CPU usage on the NAS because it is hitting a thermal limit which pulls the clock speeds on the CPU back. Again, this is a guess but you have a lot running on that NAS.

Another possibility is you are asking too many IOPS for the drives in the NAS. Are those drives in the NAS also being used to store your VMs/containers? Consumer level mechanical drives have pretty poor performance to begin with and now you are reading/writing to them all the time.

It sounds like you have to IT "know how". Have you considered building out a Blue Iris machine? I run Proxmox at my home and run Blue Iris in a Win 10 VM. Works great. I can use the same host to run other VMs/containers as well.

 

[catcamstar]

From your description, I see you have a large amount of "small" network components. I think it would be worthwhile to "measure" the longest (critical?) path. How to do this? When I had to debug networks, I used 2 laptops (any kind is OK), you put a direct UTP cable between them, and run iperf (iPerf - The TCP, UDP and SCTP network bandwidth measurement tool) hence you know the theoretical bandwidth possible between both laptops. This is your "baseline".

Then put one of the pc's in the network slot of your NAS. The other to the first camera. And rerun the test. If you get +- the same results, it's okay. If it's not (and it won't be, because of 100mbit POE gear inbetween), you'll continue until you finished all of your camera slots. If all is +- "the same", then you know your wiring is OK.

Next point is indeed to have a look at the NAS. You pimped it a lot with memory (which is good), but did you "tweak" your network accordingly? I've seen massive improvements on network bandwidth (file sharing, not surveillance in this case) when messing around with the MTU. But you need to do this on both sides (and your intermediate networking gear needs to work with it too). So it's a bit of "trial and error", ample people swear with MTU of 9k, but this to be tested. However changing/optimising this on the NAS will not solve the choppiness when watching the streams directly from the camera.

So start by physically testing your infrastructure, it may (or may not) just be a bad twist in one of the patch cables between your two panels...

Good luck!
CC

 

[SouthernYankee]

Provide a network drawing, include make, model number and local IP address. Make sure to include wifi and hardwired connections.

1) make sure NONE of the camera traffic pass through the route on the way to the Surveillance Station .
2) make sure that none of the cameras are WIFI, most 2.4 GHZ wifi will support only 3 cameras at 2MP 15fps.
3) for a quick test turn off the reolink cameras.

 

[p1morris]

Thank you so much for reviewing this and for the additional questions. Please see my responses below in bold.

From what you've described, provided the cabling is good, the network itself should be fine.
Plenty of headroom for a few cameras at say up to 16Mbps each with a gigabit core switch and gigabit uplinks from your PoE switches.
What error counts if any show on the relevant ports of the core switch?

I'm using an unmanaged switch and am not sure how to determine this. I don't see the switch listed as a client on my network and don't see anyway to connect to it.


What Hikvision software?
That, and the NAS performance, may be the common weak link here.

What's the video like when viewed directly on the camera web GUI Live View with a browser?
That would eliminate some variables.

When I connect directly to the camera web GUI and adjust to max settings, I see the same choppy video experience that I witness using the Hikvision iVMS-4200 software and through my Synology Surveillance Station.

 

[reflection]

1) When did this "poor quality" begin? Was it always bad or did it just recently become bad? If it was recent, can you think of what might have changed?
2) Are you viewing the video over your laptop or phone which is connected over WiFi? If so, how does it look when you view it over a hardwired computer?
3) Does the quality improve when you only have one camera?
4) You could try unplugging one device at a time and see if things improve with that device disconnected. A bad device/nic could cause a broadcast storm - although this might affect your internet too depending on your topology.

If you want to try iperf, you can load it as a Docker container on your Synology.
I assume your switches are connected in a hub-spoke topology as oppose to a ring? Unmanaged switches don't support spanning-tree and you will get a loop.

 

[p1morris]

I'd say its not your network. My guess would be the NAS which is running the surveillance station. Looking at its specs its a 4-core Celeron which, if we are honest, is pretty low end by today's standard. Its fine for file sharing but asking it to transcode video on the fly may be too much to ask. I'm just guessing but you may not be seeing 100% CPU usage on the NAS because it is hitting a thermal limit which pulls the clock speeds on the CPU back. Again, this is a guess but you have a lot running on that NAS.

Another possibility is you are asking too many IOPS for the drives in the NAS. Are those drives in the NAS also being used to store your VMs/containers? Consumer level mechanical drives have pretty poor performance to begin with and now you are reading/writing to them all the time.

It sounds like you have to IT "know how". Have you considered building out a Blue Iris machine? I run Proxmox at my home and run Blue Iris in a Win 10 VM. Works great. I can use the same host to run other VMs/containers as well.

Yes, I'm using the same drives in my NAS to store my containers. As I think about this, though, I was running these cameras with the same choppy video issues before I installed our Minecraft servers (a week ago).

I've seen a lot of comments re: Blue Iris and it looks like a great option. I was really hoping to use the devices that I already have to be cost conscientious. If I need to go that route, I can though. I've built other servers in the past and have run FreeNAS and OpenMediaVault. Never tried Promox but looks interesting.

 

[p1morris]

From your description, I see you have a large amount of "small" network components. I think it would be worthwhile to "measure" the longest (critical?) path. How to do this? When I had to debug networks, I used 2 laptops (any kind is OK), you put a direct UTP cable between them, and run iperf (iPerf - The TCP, UDP and SCTP network bandwidth measurement tool) hence you know the theoretical bandwidth possible between both laptops. This is your "baseline".

Then put one of the pc's in the network slot of your NAS. The other to the first camera. And rerun the test. If you get +- the same results, it's okay. If it's not (and it won't be, because of 100mbit POE gear inbetween), you'll continue until you finished all of your camera slots. If all is +- "the same", then you know your wiring is OK.

Next point is indeed to have a look at the NAS. You pimped it a lot with memory (which is good), but did you "tweak" your network accordingly? I've seen massive improvements on network bandwidth (file sharing, not surveillance in this case) when messing around with the MTU. But you need to do this on both sides (and your intermediate networking gear needs to work with it too). So it's a bit of "trial and error", ample people swear with MTU of 9k, but this to be tested. However changing/optimising this on the NAS will not solve the choppiness when watching the streams directly from the camera.

So start by physically testing your infrastructure, it may (or may not) just be a bad twist in one of the patch cables between your two panels...

Good luck!
CC

Excellent ideas - I really appreciate it. I'll have some free time this weekend. I'll try troubleshooting the lines and use iPerf to see. I'll report back what I find.

 

[p1morris]

Provide a network drawing, include make, model number and local IP address. Make sure to include wifi and hardwired connections.

1) make sure NONE of the camera traffic pass through the route on the way to the Surveillance Station .
2) make sure that none of the cameras are WIFI, most 2.4 GHZ wifi will support only 3 cameras at 2MP 15fps.
3) for a quick test turn off the reolink cameras.

I'll work on the network drawing; don't have one readily available.

2) All cameras are POE.
3) Easy to do. I'll try this weekend.

For 1), not sure how to do this. Would you be able to explain more?

 

[p1morris]

1) When did this "poor quality" begin? Was it always bad or did it just recently become bad? If it was recent, can you think of what might have changed?
2) Are you viewing the video over your laptop or phone which is connected over WiFi? If so, how does it look when you view it over a hardwired computer?
3) Does the quality improve when you only have one camera?
4) You could try unplugging one device at a time and see if things improve with that device disconnected. A bad device/nic could cause a broadcast storm - although this might affect your internet too depending on your topology.

If you want to try iperf, you can load it as a Docker container on your Synology.
I assume your switches are connected in a hub-spoke topology as oppose to a ring? Unmanaged switches don't support spanning-tree and you will get a loop.

Thanks for the additional questions.

1) The poor quality has existed since I installed the cameras. Even when I had the Reolink cameras using their NVR, I noticed issues with playback and viewing. I just didn't have the time or resources to look into it at that time.
2) Hardwired and to my laptop over wifi has the same issue.
3) I haven't tried this yet; but good point. I'll disconnect all but one and see. I'll report back if that helps.
4) For this - unplug each device one at a time from the switch? I haven't tried but I'll definitely try.

 

[SouthernYankee]

no traffic through the router
1) the router connects to a gigabit switch(1).
2) the Switch(1) connects to all the poe switches.
3) the switch(1) connects to Surveillance Station .


In my normal home network I connect the router to a switch. The router connects to the modem. The router has only two network connections. The only local traffic that pass through the router is my local Wifi traffic.

 

[p1morris]

Hi all,

Thank you all for the help! I think I figured out the issue - and it looks like one of my own doing.

I tried several fixes applied above with, unfortunately, no change. The key to fixing this, for me, was suggested by @alastairstevenson. When I tried connecting directly to the cameras through a browser (entering the IP address of the camera), I had the same issue. However, when I changed the configuration settings for the camera through the browser and changed to live view, the video was clear and seamless. I think the issue arose because I utilized the Synology Surveillance Station software to adjust the camera settings. I'm guessing (though someone might know more) that the Synology software is able to adjust certain settings but not all (?). When I adjusted the settings using Synology, it adjusted the resolution and fps but the max/average bitrate was set low.

In any case, while viewing directly or through Surveillance Station, my video is set to max settings and it is clear. Looking back, I thought it was convenient that I could adjust video settings through Synology but should have really looked at all settings directly on the camera itself.

Thank you all again for the support.

Paul