Major issues with playback of clips

[AndrewEPowell]

Hi All,

Really hoping you can help me. First post here but i've been reading these threads for years.
We run a CCTV system at our secondary school that was originally only meant to be a few cams. Its grown horrendously to over 100 camera's, and as we've added camera's, we have added new 'servers' to accommodate.
Recently we set about streamlining these as we had 5 servers each only able to cope with around 20 cameras. I read this forum and others and learnt about substreams, D2D recording and more, which has helped me get everything running much better.
However
We still have one major issue which is a huge problem. Trying to watch back clips is almost impossible. The footage just won't play, and skips frames constantly.
I've run some troubleshooting, and the camera's are recording fine. I set the recordings to MP4, and the footage is perfect, no dropping, smooth playback and great to watch. This suggests the issue isn't with the actual footage. We record continuous 24/7, with no motion detection, no alerts, no processing at all. We have one machine dialled in via webinterface just to monitor for camera drop outs etc.
When using BVR, on the BI interface, the same footage is useless. Its skippy, jumpy and just unwatchable.
What am I missing? Or what can be done to solve this?

For information;
Blue Iris 5.4.4.3
CPU usage 54%
RAM usage 8GB

Machine;
Windows 10
CPU i7-9700k
RAM 16GB

Camera's; 39 total.
All Foscam, Mixture of models including FI9961EP (1080p), FI9853EP (720p) & C1/C2
All camera's wired, on fully enterprise level network.
Substreams enabled on most. FPS max 15 on all.

Please help! Its driving me round the bend. Finally got the server running smoothly with all the camera's, however if the footage is unwatchable then we'd have to go back to 5 servers with less camera's each. The plan was to have 2 running 40 each (identical 9700k's w/ 16gb ram) but that doesn't look possible.

 

[sebastiantombs]

Your CPU utilization seems kind of high for the number of cameras and the processor. You list memory utilization at 8GB and the amount of installed memory is 16GB. That seems kind of high as well. I run a mix of 2MP and 4MP cameras at relatively high bit rates on an i7-12700. CPU rarely gets over 6% and memory sits between 3 and 4GB all the time.

What is your storage system? SATA platter drives are the best way to go. In your case I'd guess you have 12GB or more of disk capacity.

Look at the cameras under the cameras tab in the BI console (screen cap below). Access from the graph with lightning bolt icon on the upper left of the screen. I suspect your problem is camera related and you will find the frame and/or key frame rates are not quite what you think they are. Foscam, like Reolink and many other consumer level cameras, play a lot of games in their firmware and reduce frame, iframe and bit rates as they see fit no matter what you may set it to, It's also important to have the frame and iframe match for both the main stream and sub stream.

BI works best with a key frame, iframe, rate set to the same number, IE 15/15. This results in a key frame of one. In other words a full frame is sent once per second and partial frames, those containing motion differences are sent as they occur. Since BI starts motion detection on the key frame, iframe or full frame, if that rate drops as it does with consumer grade cameras, full frames may not be sent for three, four or even more seconds. That can result in all kinds of problems.

I know that Reolink does not use encoding that is standards compliant even though they label it as H264 or H265. I would suspect the same is true of most other consumer grade cameras. That also results in erratic playback and erratic motion detection. Actually, I'm surprised that Foscam can supply both the main stream and sub stream concurrently. That could also cause problems because the CPUs used in cameras are notoriously low powered and will struggle when under heavy loads, like supplying both a main and sub stream. That's another source of stuttering, jerky, skipping video.

 

[Flintstone61]

I had that on my 1st Blue Iris Machine. Until I found out some people were using 2 storage drives to split the load. Then playback improved.
Try toggling Hardware Acceleration off Globally, or on...and see if that changes anything with playback.

 

[Flintstone61]

Imagine asking a device to write 30 cams. then ask it to also take one data stream, and play it back, while also recording that stream in the background.
It may be too much to ask of 1 hard drive. I'm running about 18 Cams on 2 Western digital hard drives. I also noticed that playback improves for me when I deployed Surveilance hard drives.

 

[AndrewEPowell]

Thanks for your reply seba

Your CPU utilization seems kind of high for the number of cameras and the processor. You list memory utilization at 8GB and the amount of installed memory is 16GB. That seems kind of high as well. I run a mix of 2MP and 4MP cameras at relatively high bit rates on an i7-12700. CPU rarely gets over 6% and memory sits between 3 and 4GB all the time.

What is your storage system? SATA platter drives are the best way to go. In your case I'd guess you have 12GB or more of disk capacity.

Look at the cameras under the cameras tab in the BI console (screen cap below). Access from the graph with lightning bolt icon on the upper left of the screen. I suspect your problem is camera related and you will find the frame and/or key frame rates are not quite what you think they are. Foscam, like Reolink and many other consumer level cameras, play a lot of games in their firmware and reduce frame, iframe and bit rates as they see fit no matter what you may set it to, It's also important to have the frame and iframe match for both the main stream and sub stream.

BI works best with a key frame, iframe, rate set to the same number, IE 15/15. This results in a key frame of one. In other words a full frame is sent once per second and partial frames, those containing motion differences are sent as they occur. Since BI starts motion detection on the key frame, iframe or full frame, if that rate drops as it does with consumer grade cameras, full frames may not be sent for three, four or even more seconds. That can result in all kinds of problems.

I know that Reolink does not use encoding that is standards compliant even though they label it as H264 or H265. I would suspect the same is true of most other consumer grade cameras. That also results in erratic playback and erratic motion detection. Actually, I'm surprised that Foscam can supply both the main stream and sub stream concurrently. That could also cause problems because the CPUs used in cameras are notoriously low powered and will struggle when under heavy loads, like supplying both a main and sub stream. That's another source of stuttering, jerky, skipping video.

View attachment 132746

Thanks for your reply sebastiantombs

Couple of points; Yes i've absolutely noticed that the Foscam camera's do not stick to the frame rate set. I access the camera's and set the frame rate to 15fps, but 10 minutes later, its back up to 30, which obviously the server doesn't handle. This happens on a number of the slightly older models, but happens occasionally across the board. At this point, we simply can't afford new camera's all round unless you know where I can buy 80 wide angle (120degrees ish) POE camera's for cheap lol. Is there any way to fix this? Can blue iris limit the frames, or is this purely set by the camera?

One other thing however, is that the video footage itself is fine when recorded in MP4, or when recorded as BVR and played back on a seperate machine. This would suggest to me that it isn't the camera's themselves, but simply that blue iris can't cope? The machine itself is very snappy, I can open task manager/windows explorer etc almost instantly, and it responds very quickly. Its just blue iris that struggles.

For disks, we have 12tb of HDD setup in RAID5 config

 

[sebastiantombs]

BI can only receive what the cameras send. So if they wander off of what is set BI is stuck with that. The key/iframe ratio is the big key though. Unfortunately, there is no really inexpensive decent camera but it might be an idea to think seriously about replacing those Foscams gradually as finances and time permit. I have a Dahua K42, 4MP, and it's wide angle and fully controllable. I also have an Amcrest 841B, 2MP, which also has pan/tilt (no zoom) and is completely controllable. The Dahua is about $75 and the Amcrest is in the $50 range and is actually a Dahua OEM from Amcrest.

How is your RAID5 array attached? That can easily be your bottleneck. I'd suggest getting a pair of 8TB, Western Digital Purples or Seagate Skyhawks or other surveillance related platter drives, and not use a RAID array, especially if it's an external array. RAID5 is nice in theory, but in practice if a drive fails it can take days and days to rebuild the array. During that time you'll really see problems with video.

 

[AndrewEPowell]

Would i not still need to raid the two drives? Therefore causing the same bottleneck.
The current RAID5is inside the machine, so attached by SATA

I think, unfortunately, you do have the answer there. Now just to look at pricing that up as i think we're looking at around £80 - £100 per camera.

Do you think this would solve the issue with BI BVR playback? Or just reduce the CPU usage?

 

[sebastiantombs]

There is no need to RAID two drives. They can be formatted and run individually and the camera load can be split between them to balance the load. My system is using a 4TB, an 8TB and a 500MB and all are discrete drives, not RAID. Just create the same file structure on each drive that BI uses, especially the New, Alerts and Stored directories. Allocate the full drive, minus about 10% in BI then allocate the cameras to each drive. The AUX directories can be renamed in BI to provide easily identifiable names for a second, third, fourth or however many drives are needed. I renamed AUX1 to "Y-New" to id it as the 8TB "Y" drive in the machine. The 4TB is "New". Cameras are split between them by setting the target drive in the record tab of each camera. The 500GB is simply for alerts and actually gets little use.

If your RAID was external, you could move video off of the discrete internal drives, based on drive space to the RAID array. That would allow more archival footage. Since the RAID is internal that's not an option though.

Incidentally, you can easily save each camera to its own subdirectory under "New" by adding a &CAM\ in front of the path. The down side of that is that UI3 can't see those directories so they can't be accessed directly but clips can still be played back by using the timeline or preview in UI3.

I'm not sure about CPU utilization but it could certainly get rid of your playback problems. Since your RAID is created and managed by the OS, yes it can also lower CPU utilization.

 

[AndrewEPowell]

Out of interest, and slightly seperate, but does having camera's in more than one group affect CPU? i've realised i have camera's in multiple groups, as they show up multiple times in the list you posted in your first reply?

BTW thank you for you responses, they are incredibly helpful

 

[sebastiantombs]

No, not at all. Even if a camera is in multiple groups there is still only one instance of the camera video being processed. The same is true of clone cameras. I use clones to do AI detection. You can easily see those in the screen cap I posted because there is no bit rate for them, just the frame and iframe rates. Showing a camera n multiple groups has no impact at all since there is no additional processing at all.

You may be fine with the Foscams once you get off the RAID system. When you open resource monitor do a sort by average CPU utilization and see what it topping out the list. Do the same with memory utilization and see what's topping the list. Also look at disk utilization that way. It may shed some more light on things.

 

[looney2ns]

Along with the above advice, Make certain you have performed every single step here: Optimizing Blue Iris's CPU Usage | IP Cam Talk
Be certain you have excluded Bi folders from any Antivirus software, including Windows Defender, as outlined in the BI help file.
If you haven't properly excluded BI, this will cause playback problems among other issues.

 

[looney2ns]

This as well.
Storage configuration | IP Cam Talk