High CPU load - on FAST system

thestapler

thestapler

n3wb
Jul 13, 2016
29
0
First off... here are my server specs:
i7-4790K (not overclocked)
Asus Z97-A motherboard
32GB ram
Samsung 850 Pro 500g OS/BI database drive
4x 6TB WD RED drives -- currently set to single drives, save on the first one, move to next, and then the next, and then the last... and it tails off that one -- I have a new raid card to install next week, and going to put them in a raid5 with parity, hopefully changing the I/O load will reduce a lot (I know that there is a lot of I/O with moving the files around all the time)
OnBoard Intel HD graphics (with QuickSync) -- Since there is no console viewing (unless I use RDP - which is VERY far and few between), there is no external video card

Currently, there are 40 cameras, all of them are 1080P. 32 of them are at 6FPS, the other 8 are 12FPS (set from within the camera itself). They are set as Direct to Disk recording (24x7 continuously).
The CPU is constantly spiked at 80-90%, and there is 10 more cameras to add in the next 2 weeks, and about 10 more within the next 6 months.

Right now, there is a computer set up to display 1 of the streams from BI, and I noticed it adds a slight load to the system. I am going to be changing this use VLC and pull an RTSP stream directly from the camera instead, to reduce the load from this (I thought it was weird that it added a slight load with just pulling 1 feed from the server).

With only 62% of the total cameras used (40/64), and it is at 80%+ CPU load, how is it possible to have 64 cameras on a single server? Is there something to set differently? Does anyone have any tips on how to reduce the load further?

We are going to add another server within the next year, but there will be more cameras added as well... are we going to have to build more, and split the load over 3, possibly 4 servers? I thought having the fastest i7 processor, with a high quality motherboard, it would have been able to handle the loads.

Please, if you have any ideas on what can be done, I would be very grateful!

Thanks!
 
Ok, I am running Windows7Pro 64-bit, but it is running as a service though.

Do you think running Windows10pro would be a better option?
 
thestapler said:
Ok, I am running Windows7Pro 64-bit, but it is running as a service though.

Do you think running Windows10pro would be a better option?
Click to expand...
Disable the service just to test the hardware acceleration...
Windows 10 is a better option...you can upgrade for free. Windows 10 upgrade for assistive technology users
I would suggest that after you confirm the upgrade was successful and windows is activated, do a clean install with the microsoft media creation tool via usb.
 
By my estimation your current load is 576 megapixels per second. I am running 678 megapixels per second right now on an i7-3770K (not overclocked) at 40-50% usage. You are using 40 cameras however, while I am using 24. There may be a bit of a performance penalty for each incoming stream. Or maybe there isn't. I just don't know. One thing is for certain, your settings are not optimal otherwise your CPU load would be much lower.

Outgoing streaming from Blue Iris always requires expensive transcoding, even if you are streaming out h.264, which is why you see a noticeable increase in load from it. This is one of the reasons you should strive to keep usage well below 80% during normal running.

Anyway, regarding the settings, I'll expand on what fenderman said.

First of all, make sure you are streaming ordinary H.264. Not "H.264+" or "H.265". Blue Iris can't deal with those as efficiently as H.264.

Go here and note the highlighted settings.

9NWI64e.png


You want hardware acceleration set to "Yes (H.264)". Do not choose the VideoPostProc option, as that will make things much worse. Restart Blue Iris for this change to take effect. Also set a live preview rate limit as I have shown. This only affects the local Blue Iris GUI which you have said you do not use much, so it should be fine to set it to 1 FPS for the best performance.

Now, each of your cameras should be set up for direct to disc recording in Blue Iris DVR format as shown by the highlighted settings here:

qcnGsKt.png


The last things I will mention are here:

i3V6tp6.png


The Max rate setting (which is maddening and should not be necessary) needs to be set a few FPS higher than the camera's actual FPS, because you get visual corruption in direct to disk recordings if the frame rate temporarily fluctuates higher than the Max rate setting.

Now, this is more personal choice than anything, but I uncheck the "Also use for BVR playback" box, which forces Blue Iris to use H.264 software decoding for clips instead of hardware decoding. This box is checked by default, but it causes some minor problems with clip playback so I turn it off for all my cameras. It is important to note that if you play multiple clips simultaneously using the timeline then having this box unchecked will make that require more CPU.

Lastly, since you use direct to disk, you don't need Blue Iris's timestamp overlay. You can delete that overlay via the Edit button at the bottom of the Video tab, as shown in the screenshot above. You should make sure your cameras keep accurate time via NTP (see NetTime - Network Time Synchronization Tool for a free NTP server you can run) and have the cameras embed their own timestamps so they always appear in recordings.
 
  • Like
Reactions: DGMagic, nocalmike, Jrh and 6 others
Ok, currently, we're at ~760 MP/s according to the BI Status page, that's with 40 cameras... (I had my numbers for what cameras were set at what FPS...)

I have them set in "groups", and some cameras are part of multiple groups... is that a performance hit as well? Basically, we have 3 locations, and all feeds go back to the 1 central location... So camera groups would be like:
Building 1
--- Roof
--- Main Floor
--- Basement
and all the cameras would belong to the "Building 1" group, as well as their respective locations. -- should I change this?

moving from running as a service, to just "running", reduced the CPU from being pegged at 100% (I just added 2 cameras, it was actually 38 before, and 2 were not in use). It also reduced the memory from 7G in use, down to 600M (this might change when running over time, no?)

Enabling the hardware encoding, didn't do anything for my CPU load, except there are a bunch of cameras that aren't working... so I went back to having it disabled for now.


All of the cameras are set to direct to disk... that is 1 thing I was told about before.
 
thestapler said:
Ok, currently, we're at ~760 MP/s according to the BI Status page, that's with 40 cameras... (I had my numbers for what cameras were set at what FPS...)

I have them set in "groups", and some cameras are part of multiple groups... is that a performance hit as well? Basically, we have 3 locations, and all feeds go back to the 1 central location... So camera groups would be like:
Building 1
--- Roof
--- Main Floor
--- Basement
and all the cameras would belong to the "Building 1" group, as well as their respective locations. -- should I change this?

moving from running as a service, to just "running", reduced the CPU from being pegged at 100% (I just added 2 cameras, it was actually 38 before, and 2 were not in use). It also reduced the memory from 7G in use, down to 600M (this might change when running over time, no?)

Enabling the hardware encoding, didn't do anything for my CPU load, except there are a bunch of cameras that aren't working... so I went back to having it disabled for now.


All of the cameras are set to direct to disk... that is 1 thing I was told about before.
Click to expand...
you must restart the system after enabling hardware acceleration...HA is the only way you will get this to work...also if you are remoting in it will peg the cpu, you need to use the mobile app to see the cpu consumption or have something track it on the pc when you are not remotely logged in. groups have no effect on cpu
 
I just re-read your message as well @bp2008 -- you suggest putting the timestamp on the camera feed itself, rather than do it from within BI's overlay? Does this reduce load? We do have our own stratus2 time server here at work... so that isn't an issue.
 
I have a lot of groups too and haven't noticed any performance impact from that.

You do have to restart Blue Iris for hardware acceleration to be enabled. It is possible though that you have already exceeded the capacity of Intel's hardware accelerated h.264 decoding (e.g. too many cameras). Your memory usage at only 600 MB with 40x 2mp cameras is surprising. Mine runs a pretty steady 5 GB. If you see memory usage continuously and steadily rise as Blue Iris runs, then you are being affected by a memory leak in the Intel graphics driver and you should downgrade to Download Intel® Graphics Driver for Windows® 10 and Windows 7*/8.1* [15.33]
 
I don't know how much it will reduce load to not have Blue Iris embed the timestamp (certainly a tiny bit at least). The important thing is that direct to disk won't let Blue Iris overlay the timestamp into the video stream when recording.
 
ok, I disconnected from RDP, and the server is still at 82%, with 554.5M in use... (according to the android app).

Seems to be holding steady... load hasn't changed in 20 minutes.
 
Ok... went though all the cameras, and their settings... they are mostly all set to 12FPS @2mb... and in BI, they are set to MAX FPS of 15, they are all set to Direct To Disk, and there are currently 40cameras being continuously recorded.

In the Status page, i show ~15750KB/s - 730 MP/s for the cameras -- which is lower than before, and I upped a few cameras to 12... weird?

I have minimised the BI page (still not running as a service, but hardware acceleration is still off for now), and I am logged out of RDP.

When I log into my phone (android), it shows CPU of 90%, mem 7.5G, with 2hrs of uptime.

I guess my next option is to install the raid card, and reduce the I/O load (moving files from disk to disk is not very efficient), and to upgrade to Win10... then I can see if enabling the Hardware Acceleration makes a difference.

We use BigBrother (xymon/devmon) to monitor servers and stuff, and when I check it... it shows:

CPU states:
total 100%
cpu00 100.0%
cpu01 99.8%
cpu02 99.6%
cpu03 99.6%
cpu04 99.8%
cpu05 99.2%
cpu06 99.9%
cpu07 98.9%

CPU PID Image Name Pri Time Owner MemUsage
95.7% 7064 BlueIris.exe 8 13:59:27 BLUEIRIS-01\blueiris 4194303k
00.9% 4 System 8 7:21:51 NT AUTHORITY\SYSTEM 13396k
 
You can manually disable hardware acceleration for individual cameras, in case a lower number of them (like 20?) can be accelerated while the others use non-accelerated decoding. For me, going beyond about 15 or 16 hardware accelerated cameras resulted in Blue Iris taking a very long time to shut down, and pegging the CPU while doing it.
 
  • Like
Reactions: fenderman
Well... to continue my saga... I have changed ALL the cameras to be 8FPS (which isn't REALLY ideal, but for now... )

I am at 57% CPU (phew!)

When the raid card arrives, I will be reformatting with Windows10, and we'll go from there! I enabled the NTP and the OSD now shows the time on most of the cameras (we have 4 different types, and 1 of them i can't get into, 2 of them require me to use a plugin that I just haven't enabled/installed yet).

I set it as a service again (just to see), and the CPU load drops to about 45%, and when I open the console, we're back up to 80%... sometimes, I need to use the console... maybe that is where having the "nvidea graphics card" would help? no big deal... just thought I would see what it would do.

Anyways... we'll see what happens the end of the week, or early next week with Win10 and the Raid card.

Thanks for your help guys... it is greatly appreciated!!
 
OH... just to put 1 more thing out there... according to Windows "Resource Monitor", I have about 100Mbps of network I/O, and a fluctuation between 40 and 110 MB/sec Disk I/O for blueiris :)

And, double checking the CPU in Resource Monitor, with the console open while running as a service, BI is reporting wrong... it didn't change much... but the bottom of the console in BI shows wrong.
 
Nvidia cards make it slightly more efficient to render the console, but it is more effective and a lot cheaper to just limit the live preview frame rate to something low like 1FPS as I suggested above. That will eliminate almost all the CPU penalty of having the console open.

Did you get hardware acceleration working? You can run BI not-in-service-mode and just minimize the console to have more or less the same effect on CPU usage as running as service would give you.
 
No, I didn't enable the hardware acceleration again... as fenderman suggested/said, i would have to restart the machine... and I didn't have a chance to do it as our tech's were supposed to be installing more cameras... so... it will have to wait til tomorrow to restart it, and try that - and I will let you know :)
 
You must log in or register to reply here.
Top Bottom