Cameras keep dropping-off - New DHI-NVR5864

[Akshunhiro]

Hi all, need some insight as to why this may be happening.

I'm working with a vendor to upgrade a system and we're finding that the cameras keep dropping-off after being added.

At first it appeared to be a network issue so a new VLAN was created and setup on the secondary NIC of the NVR5864.

I slowly went through and updated the static IP of each camera but issue continues to happen.

Cameras will disappear for 20-30 seconds before coming back. Ping does timeout when this happens.

These are connected to 9200 series Cisco switches.

Kind of stuck as to what to try next.

 

[TonyR]

Have you tried putting the VLAN aside for a moment until you can insure there's no other issue?

Connect 1 of the NVR LAN ports to the POE switch (uplink if there's one) then connect POE switch to all cameras. NVR should also have unique static IP and cams should, of course, all be on the same subnet as the NVR.

If there's router in the setup insure all those static IP's you assigned the NVR and the cams are unique and OUTSIDE of the router's DHCP pool.

 

[bigredfish]

Assuming a non-PoE NVR, all cameras using external switches, why are some showing port 37777 and some port 80?

My intuition tells me its exceeding max incoming bandwidth. Is that NVR AI-capable? and if so is AI on the NVR enabled? That will reduce allowable incoming bandwidth by 40%

 

[steve1225]

Assuming a non-PoE NVR, all cameras using external switches, why are some showing port 37777 and some port 80?

When You use non Dahua cams (over ONVIF) port 80 is used..

 

[steve1225]

At first it appeared to be a network issue so a new VLAN was created and setup on the secondary NIC of the NVR5864.

What exact model You use (4ks2, I, EI)?
How many cams You have connected?

5xxx line never should have 64 cam models.. This line have to limited processing power..

 

[Akshunhiro]

5xxx line never should have 64 cam models.. This line have to limited processing power..

Hmm, good to know. There are 35 connected currently. Model is mostly HDW4431EM-ASE.

This system is to replace several NVRs in various buildings (linked by fibre). Old system was working fine as the NVRs had the cameras connected directly so really only 7 connections needed to the main network for access.

Will check whether AI is enabled and turn it off if so.

 

[Akshunhiro]

Thanks for all the responses too.

AI features aren't enabled, and, it's only 20 cameras connected currently. Highest channel was 35 so assumed there were 35 cameras connected.

 

[bigredfish]

Depending on the NVR model it will handle max 384 or 320 Mbps incoming bandwidth. (NVR AI on recorder disabled)

My older 5216 is rated at 320 Mbps and there is no way (even if it were possible) that it would handle 35 4MP cameras at any decent bitrate

 

[Akshunhiro]

To try rule this out, I fitted a second NIC to a spare PC and setup BlueIris.

Added 6 cameras (barely 20Mbps coming up with all of the substreams on screen) and still seeing a few drop out.

Seems to be network related so will get another check to check the switch configs,

 

[Flintstone61]

Dumb question , not knowing the Hikvision models... and not knowing cable lengths,
i started to wonder if the Cisco 9200 was an 802.3af or 802.3at rated poe switch.
You lose some power over distance......af 15.4 watts & "at" is 30watts.
I think your cams are probably 802.3af
they have a 1440 watt poe Budget on the 48 port switch.

Cisco Catalyst 9200 Series Switch Architecture White Paper

Enterprise campus networks are undergoing profound changes to support ever-increasing bandwidth demands on the access layer, heightened by the introduction of 802.11ac and Wi-Fi 6 and the rapid growth of powerful endpoints requiring speeds from 10 Mbps to 10 Gbps. These networks are in dire need...

www.cisco.com

.

 

[steve1225]

Hmm, good to know. There are 35 connected currently. Model is mostly HDW4431EM-ASE.

This system is to replace several NVRs in various buildings (linked by fibre). Old system was working fine as the NVRs had the cameras connected directly so really only 7 connections needed to the main network for access.

Will check whether AI is enabled and turn it off if so.

If you login local (over mouse & monitor connected to HDMI) to NVR You can check how much bandwidth is allocated to cameras (on camera list) and how much bandwidth is transferred from cams (maintain->systeinfo->bps) and what is actual CPU usage on NVR... Local (HDMI) interface shows many stats, which aren't available over Web interface...

 

[steve1225]

Thanks for all the responses too.

AI features aren't enabled, and, it's only 20 cameras connected currently. Highest channel was 35 so assumed there were 35 cameras connected.

Hmm, what happened to rest 15 cameras?

if there is such a configuration mess, maybe it is worth defaulting the configuration to the factory one and starting the NVR configuration from scratch?

You didn't answer what exactly model You have (5864-xxx - 4KS2, EI?). If it is 5864-EI model, go to AI -> parameters -> Smart Plan -> local tab and change AcuPick setting to AI mode (like in picture).. this will restore a lot of bandwidth for you:



check the total bandwidth available on NVR.. it should be 320 Mbps (if it is -EI model)..

 

[steve1225]

These are connected to 9200 series Cisco switches.

How long the cable runs are between POE switch and cams?
If it is something like 100m or more You should replace Cisco switch with Dahua ePOE switch, which work even up to 800m..

 

[steve1225]

Cameras will disappear for 20-30 seconds before coming back. Ping does timeout when this happens.

Login to cameras after disappear and check uptime on cam.. If it is short, that it means that camera rebooted and then You known that this is power-over-POE / switch / bad or long cable problem..

 

[Akshunhiro]

Hmm, I can't even get in quick enough to check, Camera goes offline shortly after. Ping does continue for ~30 seconds after the web UI stops responding.

When it comes back up, it does ask for the password again so assuming it has rebooted.

Some cameras are on their own PoE switch though so those ones should be okay for power.

 

[Akshunhiro]

I just took a packet capture and seeing a lot of resets. Not sure how to fix this though.

10.2.30.100 is the static IP of the VMS NIC2.

Oh and sorry, model is DHI-NVR5864-AI/ANZ

 

[steve1225]

Are the problem for all cams? or specific cam(s)?
If specific one - then this can be camera problem (take-off and check locally), cable problem, switch port problem..

If all cameras are missing - don't respond over Web - this is switch / network problem.

Can You describe/draw network architecture / connections (NVR, switches, VLAN's)?

You have Cisco switches - they have very high levels of logging.
You can check POE logs, You can check ethernet ports logs, you can check Spanning tree logs.
Good Cisco network admin will help You a lot - they known how to debug things...

Big question - is the separate VLAN isolated?
So only second port of NVR and all cams connected to this VLAN?

No router / firewall any kind connected to this VLAN?
No loops with other VLAN's / ethernet ports etc?

If You can't isolate what is happening on L2 network (or don't have knowledge), then You can try to isolate this system from existing network...

Get smaller EMPTY POE switch, connect only second NVR port and some (not even all) cameras and check is this working?

PS. To help You more, please answer for each question asked...

 

[Akshunhiro]

Are the problem for all cams? or specific cam(s)?
If specific one - then this can be camera problem (take-off and check locally), cable problem, switch port problem.

90% of them. They just drop in and out randomly but are down more often than not.

The previous setup had NVRs in each building that the cameras connected to directly. The NVRs were then all connected to the main admin data network for clients to access.

Issue started when the cameras were setup on the main admin data network as the NVRs were removed for the upgrade to the new VMS.

Can You describe/draw network architecture / connections (NVR, switches, VLAN's)?

This is part of a university network with various VLANs for different device types. I had wanted to isolate the CCTV network anyway but it's still odd that they're not playing nice on the main admin data network.

To further complicate things, it's a /25 network but they're X.1 (rather than X.129) I did find a couple of cameras which had an incorrect subnet mask set but didn't make a difference after they were updated.

You have Cisco switches - they have very high levels of logging.
You can check POE logs, You can check ethernet ports logs, you can check Spanning tree logs.
Good Cisco network admin will help You a lot - they known how to debug things...

I've reached out to a network engineer, just waiting to hear back.

Big question - is the separate VLAN isolated?
So only second port of NVR and all cams connected to this VLAN?

Yes, this new one is completely isolated. I had initially tested with the default 1 VLAN (as it wasn't in use) and saw the same behaviour so had a new VLAN created to try rule it out. This isn't setup on the router, just the edge and access switches to go back to the secondary NIC on the VMS. Everything is set statically and pings to the cameras from the VMS are successful (when they're up).

No router / firewall any kind connected to this VLAN?
No loops with other VLAN's / ethernet ports etc?

No, and no, not from what I can see. The VLAN was automatically created when specifying it on the access ports, and automatically included in the allowed VLANs on the trunk ports.

If You can't isolate what is happening on L2 network (or don't have knowledge), then You can try to isolate this system from existing network...

This has now been done with the new VLAN 30

Get smaller EMPTY POE switch, connect only second NVR port and some (not even all) cameras and check is this working?

One building is setup this way due to limited free ports. Those cameras are affected too. EDIT: Sorry re-read and that's something we haven't tested. All the cameras are installed so would have to be removed to try this. The vendor/installer has also recommended going a step further by setting the cameras up on their own fibre links between the buildings.

 

[steve1225]

90% of them. They just drop in and out randomly but are down more often than not.

The previous setup had NVRs in each building that the cameras connected to directly. The NVRs were then all connected to the main admin data network for clients to access.

Issue started when the cameras were setup on the main admin data network as the NVRs were removed for the upgrade to the new VMS.

This is part of a university network with various VLANs for different device types. I had wanted to isolate the CCTV network anyway but it's still odd that they're not playing nice on the main admin data network.

This is why...
when cameras were connected to NVR internal switch (or separate switch connected to second port on NVR), cameras were MANAGED by NVR..
NVR worked as router / firewall / NAT / DHCP server / time server for them... All network configuration on cameras were managed by NVR..
On your network You had only NVR visible - cameras were hidden behind NVR firewall...

When You connected cameras to main VLAN - this probably was disaster at first.
You must manually set ip / network parameters on each camera using dahua config tool..

If there was any DHCP server for normal clients on this network segment / VLAN - You must use static IP's from outside DHCP scope/range (but inside network routing range).. No duplicates of static IP and DHCP IP range..

I don't known how big IP segment network was - it was enough for all normal clients and cameras.. DHCP range and static IP's...


You need separate & isolated VLAN..
only second port of NVR and cameras there...

On all switch ports where cameras (or NVR) are connected, You must setup Native VLAN for this ethernet port to CCTV VLAN number.. And disable tagged VLANs...

The best solution is allow back NVR to fully manage the cams (IP's etc)..
Allow NVR works as router / firewall / NAT / DHCP server for cameras..
But first You must have separate isolated VLAN.. and revert configuration on cameras to no static IP (use DHCP)..

Yes, this new one is completely isolated. I had initially tested with the default 1 VLAN (as it wasn't in use) and saw the same behaviour so had a new VLAN created to try rule it out.

Never use VLAN 1 on multi VLAN / switch networks

No, and no, not from what I can see. The VLAN was automatically created when specifying it on the access ports, and automatically included in the allowed VLANs on the trunk ports.

ask Cisco admin is everything ok there..

One building is setup this way due to limited free ports. Those cameras are affected too. EDIT: Sorry re-read and that's something we haven't tested. All the cameras are installed so would have to be removed to try this. The vendor/installer has also recommended going a step further by setting the cameras up on their own fibre links between the buildings.

Yes - but between local POE switch and NVR there is campus network. Which can create problems (we don't known why at this moment)..

Get smaller POE switch and put in building where You have NVR (or move NVR to the building where You have local POE switch)..
connect to that POE switch second port NVR and some cameras from the building...
no campus network connected to this local POE switch...
primary NVR port connect to main VLAN in campus network, where should work...

before connecting cams reset them to use DHCP (no static IP)..
allow NVR to manage them using camera list interface on NVR...

This local installation (NVR + totally separate POE switch and some cams managed by NVR) SHOULD WORK...
If You need you can connect second NIC of VMS computer (or some netbook with tools) to this local POE switch - to check network, pings etc..

if this will work then you known that NVR is ok and those cams (and settings) are ok..
then You can try connect more cameras...

first all direct..
second the rest using campus network (on ethernet port setup for CCTV VLAN)...

remember about reseting camera settings to DHCP (no static IP)..

if this will fail You will known that something is not OK in campus network configuration..

 

[bigredfish]

I’m surprised and happy that I understood and agree with 90% of that.!

* The other 10% is over my head.