Frame Bottleneck

[Zuboloma]

Hi this is my 1st time here.
I have been using BI for a while now, but I am have trouble with a Frame Bottleneck error on cameras.

BI is running on a windows 2022 datacenter Hyper-V instance with these resources allocated to it,

Intel Xeon silver 4310 CPU 2.1ghz 2 socket 48 Virtual cores (30% used)
64gb of ram (30% used)
21TB of all SSD storage (90% used)
10gb fiber network connection to the main switch

38 axis cameras (different makes)

Network topology

Main switch(hp 5406rzl2) - 25 feeds are directly connected to this switch POE
2nd switch (hp 1930) connected to the main using 10gb fiber - 13 feeds connected

BI version 5.8.8.0 x64


Story

I just added the 2nd switch and the 13 feeds and started seeing the bottleneck problem. I thought it had to do with the network connection going to the Hyper-V environment which is using about 140mbps.

to test this limit, I am able to copy files from a different source on to that server and see the network usage go up to 800mbps, so the limit isn't there.

I updated the firmware on most of my cameras to see if that would help anything, it didn't.

I am using main stream on most of the cameras in BI, (I am thinking I might need to use sub streams, but I am not sure)
/axis-media/media.amp?videocodec=h264&size=1&camera={CAMNO}
video compression is direct-to-disk H.264

I out of ideas on how to fix this problem. any suggestions?

Thanks!

 

[bp2008]

Hello!

What is a "Frame Bottleneck" error? Can you share a screenshot? Also a screenshot of the Blue Iris Status window, Cameras tab.

 

[bp2008]

Your server CPU is only 12 cores but you've allocated 48 vCPUs to the machine, which is an entire two sockets (I'm assuming the host hardware has 4 CPUs???). Anyway, I am pretty sure Blue Iris is not NUMA-aware at all so I would not want to assign more than one physical CPU's worth of cores or memory to the VM in order to help avoid efficiency problems. And ideally I would not assign even that much. Maybe start with 20 cores and 24-28 GB of RAM.

38 cameras without using sub streams can be a pretty resource-intensive load depending on resolution, frame rate, and other configuration. And those Xeon Silver 4310 cores, while they're decent for a high density server CPU, they are not particularly fast individually which can be an issue.

 

[Zuboloma]

Thanks for the Replys

I never considered that the CPU might be the bottle neck. after looking at the main host, your right 24 cores are in use while the 2nd 24 cores are doing nothing.

I'll change the allocation to 24 cores. Though what can I change in the cameras to reduce the CPU load? should I add a GPU to the server? will that help?

here is some screen shots of one of the cameras and the task manager of the main host.

Thanks!

 

[wittaj]

Substreams are your friend and bring down CPU tremendously. It allows the CPU to handle activities that can be handled with the substreams.

Using the mainstreams for everything is why folks with a 13th Gen is maxing out and why someone using substreams can run 50 cams on a 4th Gen at under 30% CPU.

If substreams were bad none of us would use it.

 

[Zuboloma]

Can anyone tell me how to setup sub streams then?

 

[bp2008]

Interestingly, Blue Iris's help file doesn't have the word "bottleneck" in it anywhere! If you mouse over the "Frame bottleneck" text or the icon to its right, do you get a tooltip?

You did not post a Blue Iris Status window screenshot, so there's a lot of useful information missing. For example it would show total bit rate and megapixels per second, resolutions and frame rates, along with an indicator if you've inappropriately got hardware acceleration enabled on a system where that is not supported.

I never considered that the CPU might be the bottle neck. after looking at the main host, your right 24 cores are in use while the 2nd 24 cores are doing nothing.
I'll change the allocation to 24 cores.

Right, and like I said I would even try fewer than 24 cores just to help the hypervisor find more time slots to let the VM execute. If your VM has to spend significant time not executing at all because it can only execute when an entire processor is idle, that will be very inefficient. I'm not sure how Hyper-V does scheduling or if that will matter. If your VM gets flopped back and forth between different physical CPUs, that will also be inefficient. I'd like to see this be able to run on just 12 virtual cores (and certainly with sub streams configured, it will be able to do that easily).

Likewise a larger memory allocation could cause your VM to be using memory that is not able to be accessed as efficiently as possible by the CPU it is executing on. A smaller allocation should help the hypervisor handle it as efficiently as possible.

Though what can I change in the cameras to reduce the CPU load?

Sub streams. Here is the sub stream setup guide that I wrote: Sub Stream Guide

should I add a GPU to the server? will that help?

No, it likely won't help at all unless you were going to use the GPU for AI.

 

[wittaj]

By default BI doesn't use substreams.

Simply go to this page in the BI camera setup and select the substream from the pulldown

Sub Stream Guide

.substreamGuide td { border: 1px solid #000000; } .substreamGuide a { color: #0080ff; } .substreamGuide code.inline { margin: 0; background: #f6f6f6; border: 1px solid #e0e0e0; border-radius: 0px; padding-top: 1px; padding-right: 3px; padding-bottom: 1px...

ipcamtalk.com

 

[Zuboloma]

Thanks everyone!

I ended up having to add this line into the sub stream and make a profile to match that resolution in the interface for my axis cameras.

/axis-media/media.amp?videocodec=h264&resolution=640x360&camera={CAMNO}

now the bottleneck error has stopped, the CPU usage is down.

but I'm getting a New error.

"Check FPS and key Rates"

 

[wittaj]

That is indicative of a network bottleneck OR your cameras are arbitrarily changing FPS and iframes, but with Axis that probably isn't happening.

Are your cameras connected to the router or are they isolated via Dual NIC or VLAN?

You probably have a bad switch or cheap switch that can't keep up. Just because it is a GB switch doesn't mean it can actually push it, especially the cheap no name.

 

[Zuboloma]

Main switch(hp 5406rzl2) - 25 feeds are directly connected to this switch POE
2nd switch (hp 1930) connected to the main using 10gb fiber - 13 feeds connected - most of the cameras with the problem are on this switch

but that switch is sitting at 8% usage and 3% ram.

I don't have them connected to a router or in a VLAN, I could move them to a VLAN if I need to.

settings I'm using in the camera

 

[bp2008]

It was all spelled out in the first post. The switches are not cheap noname. They are the "we hear you have a lot of money, let us help you spend it" kind from HP enterprise.

Blue Iris is indicating it is only decoding half a frame per second from both streams.

What I'd do is set the actual frame rate you want instead of letting the camera decide like that. And set the GOP length equal to that frame rate (that appears to be the i-frame interval setting). Did you actually read the sub stream guide?

 

[Zuboloma]

It was all spelled out in the first post. The switches are not cheap noname. They are the "we hear you have a lot of money, let us help you spend it" kind from HP enterprise.

Blue Iris is indicating it is only decoding half a frame per second from both streams.

What I'd do is set the actual frame rate you want instead of letting the camera decide like that. And set the GOP length equal to that frame rate (that appears to be the i-frame interval setting). Did you actually read the sub stream guide?

I will read it now, sorry this is all very new to me.
Thanks!

 

[bp2008]

Also your RTSP URL is effectively /axis-media/media.amp?videocodec=h264&camera=2 due to the {CAMNO} and . I'd take out the {CAMNO} part for readability's sake and just replace it with the proper number. That Cam# field and {CAMNO} string are meant to be used for NVRs and multi-sensor cams where you are required to differentiate between channel numbers by using this field. It doesn't matter for a typical single sensor IP cam that has its own IP address.

 

[bp2008]

Oh also it is recommended to turn off Blue Iris's custom timestamp overlay and have the camera provide that so that a timestamp exists in the raw recordings. I see you have overlays disabled in your camera's encoding settings. configuration.

 

[Zuboloma]

Also your RTSP URL is effectively /axis-media/media.amp?videocodec=h264&camera=2 due to the {CAMNO} and View attachment 188723. I'd take out the {CAMNO} part for readability's sake and just replace it with the proper number. That Cam# field and {CAMNO} string are meant to be used for NVRs and multi-sensor cams where you are required to differentiate between channel numbers by using this field. It doesn't matter for a typical single sensor IP cam that has its own IP address.

Most of the cameras I have are multi sensor cameras. so I thought I needed the {CAMNO}

 

[Zuboloma]

Oh also it is recommended to turn off Blue Iris's custom timestamp overlay and have the camera provide that so that a timestamp exists in the raw recordings. I see you have overlays disabled in your camera's encoding settings. configuration.

Ok I'll take a look into that thanks!

 

[bp2008]

Sure in the case of actual multi-sensor cameras, it is fine to use {CAMNO}. I still prefer to swap out that text macro for the raw number so that the RTSP URL is shorter, but that is just my preference.

 

[Zuboloma]

ok after alot of playing I have learned that
Main
/axis-media/media.amp?streamprofile=0&camera={CAMNO}
Sub
/axis-media/media.amp?streamprofile=1&camera={CAMNO}

in the camera I need to make a stream profile high and low resolution

I can set these settings
I know that BI is using the settings in the camera, I can can the resolution in the profile and it changes in the status of the camera.



when I change the bit rate or frame rate it doesn't seem to change the FPS/Key for main or the sub



what am I missing?

Thanks!

 

[Zuboloma]

Also I am understanding that the FPS/ket should be 15/1 30/1 something like that? right?