Out of memory error?

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,428
Reaction score
47,541
Location
USA
Have you overallocated the storage in clips and archive or allocated too much storage to BI and there isn't enough room to write-over or not run a compact/repair in a while?
 
Last edited:

erkme73

BIT Beta Team
Joined
Nov 9, 2014
Messages
1,540
Reaction score
1,412
Not knowingly. It's been humming along quite nicely, and I haven't made any changes. Suddenly I'm getting tons of these errors intermittently.

Allocation shows 90%. DB is set to compact/repair at 2a daily.
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,428
Reaction score
47,541
Location
USA
Strange - is the drive full and maybe it needs more than the 10% free?

I don't know if you took advantage of the new capability or not, but I am wondering if now with the continuous+trigger option that will allow many to go to continuous recording instead of triggered if more than 10% is needed to be free?

What is the RAM - maybe it is a RAM spike during motion?
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
Screen shots
1) windows task manager process tab sorted by memory (most at the top), Must contain, memory, disk, network, GPU, GPU engine columns
2) Blue iris status (lighting bolt graph,upper left corner) clip storage tab
3) blue Iris status cameras tab
 

erkme73

BIT Beta Team
Joined
Nov 9, 2014
Messages
1,540
Reaction score
1,412
Screen shots
1) windows task manager process tab sorted by memory (most at the top), Must contain, memory, disk, network, GPU, GPU engine columns
2) Blue iris status (lighting bolt graph,upper left corner) clip storage tab
3) blue Iris status cameras tab

Thanks guys. I apologize for not providing that right out the gate. I was in a really bad cell area and barely got the post sent. I did a remote reboot of the machine and it hasn't happened for the rest of the day - though I am now getting fairly regular BI restarts. If it happens again tomorrow, I'll grab the required info.
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,428
Reaction score
47,541
Location
USA
Thanks guys. I apologize for not providing that right out the gate. I was in a really bad cell area and barely got the post sent. I did a remote reboot of the machine and it hasn't happened for the rest of the day - though I am now getting fairly regular BI restarts. If it happens again tomorrow, I'll grab the required info.
Restarts sound like a memory leak!
 

erkme73

BIT Beta Team
Joined
Nov 9, 2014
Messages
1,540
Reaction score
1,412
Actually, I may as well start posting the requested info now...

Restarts sound like a memory leak!

@wittaj - is there a utility that will log memory spikes and show if that's happening? Unless I happen to be logged in to watch, the odds of me catching it after the fact is probably pretty slim.



Looking at the log, I see that even now I'm getting non-descript errors (though no push notifications for the error) on my "alerts":

1623637851248.png


Under clips, alerts, it's set up identically to the new and storage:

New tab:
1623637937787.png

Alert:

1623637979350.png



The drive is 14TB, and reports 10% free I did disable the "do not monitor free space" TODAY during diagnostics. Don't know if that's what stopped the out of memory errors.

1623638111464.png

Windows explorer shows a comparable amount of free space:

1623638142943.png

Now I know there is some debate as to whether a USB drive is even capable of handling the bandwidth and write activity of so many cameras. However, this drive has been on this dedicated machine (and the one prior to it) for quite some time - and I have not had any issues like this before. The drive is dedicated. No other writing/reading operations other BI work on this machine or this drive.


Task manager showing current memory usage (16GB actual). While I did have some issues with runaway memory use early on a couple weeks ago when I built this machine - presumably due to driver issues on the video card - it has all settled out, and I've not seen memory exceed 5GB used.

1623638310560.png

Camera status window:

1623638490602.png

I guess it would be helpful to know a bit more detail with the out-of-memory-error.

Using notepad++ I have compiled a list of the memory errors for the month of June. They started yesterday, and have been prolific:

1623638699530.png

Drilling into the log where these errors occur, there isn't any consistent event occurring just prior to the entry. For example, with this one, the camera detected motion as it should. The next entry for the camera was the error:

1623638865900.png

Some of the other errors were preceded by a "no signal" or other issue that might indicate a drop in communication with the camera - but it's not consistent. Some have that, others don't.

Given that have made zero changes to the configuration in over a week, it makes me think maybe it's something that my system doesn't like with the latest builds?

And for the record:

1623638976643.png

Thanks again for letting me bounce these ideas off of you guys.

Much appreciated. The irony is, I moved to a new clean system to try and wrangle up some of the gremlins...
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
Your system is overloaded !!!!!!!!!!!!!!!
You need 16GB of memory on the computer.
What is your CPU ?
Are you running headless ?
Get an good WD purple internal disk drive for the NEW folder.
Lower your frame rate to 15 FPS on all cameras !

====================================
My Standard allocation post.

1) Do not use time (limit clip age)to determine when BI video files are moved or deleted, only use space. Using time wastes disk space.
2) If New and stored are on the same disk drive do not used stored, set the stored size to zero, set the new folder to delete, not move. All it does is waste CPU time and increase the number of disk writes. You can leave the stored folder on the drive just do not use it.
3) Never allocate over 90% of the total disk drive to BI.
4) if using continuous recording on the BI camera settings, record tab, set the combine and cut video to 1 hour or 3 GB. Really big files are difficult to transfer.
5) it is recommend to NOT store video on an SSD (the C: drive).
6) Do not run the disk defragmenter on the video storage disk drives.
7) Do not run virus scanners on BI folders
8) an alternate way to allocate space on multiple drives is to assign different cameras to different drives, so there is no file movement between new and stored.
9) Never use an External USB drive for the NEW folder. Never use a network drive for the NEW folder.


Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.
Hint:
On the Blue iris status (lighting bolt graph) clip storage tab, if there is any red on the bars you have a allocation problem. If there is no Green, you have no free space, this is bad.
======================================
Private ip addresses. Local IP addresses. These addresses are NOT used by the internet. They are for your local home/business network.
10.0.0.0 to 10.255.255.255
172.16.0.0 to 172.31.255.255
192.168.0.0 to 192.168.255.255

Note there is no reason to redact local ip addresses when posting.
-------------------------------------
 

biggen

Known around here
Joined
May 6, 2018
Messages
2,539
Reaction score
2,765
It’s not a debate. USB drives are terrible for constant writes because the USB bus isn’t designed for that activity. I’d definitely install an internal drive and get off the USB one and see if that rids the problem.

BI is surprisingly light on memory usage. I only assign 6GB of memory for all my BI VMs.
 

IAmATeaf

Known around here
Joined
Jan 13, 2019
Messages
3,287
Reaction score
3,252
Location
United Kingdom
I would also change the compact/repair time, I have mine set to run only on a Sunday at lunchtime,a time when I’ll most likely be home. During the compact/repair BI won’t be recording so 2am, although the default isn’t really a good time.
 

erkme73

BIT Beta Team
Joined
Nov 9, 2014
Messages
1,540
Reaction score
1,412
i7 6700, headless. Most cameras are 15fps. Some are higher (those at front catching vehicle activity).


I'm curious why it is showing 50% men usage when I do have 16 gb of physical ram....

Screenshot_20210614-070521.png

Screenshot_20210614-071844.pngScreenshot_20210614-071844.png

This is 10 pro. And all 16 ought to utilized. Incidentally coun an see the USB drive is utilizing no more than 2% of it's bandwidth. It's not continuously writing - at least not according to the performance tab on task manager.

Aside from reserving 500gb for storage clips (which are never generated since I delete new clips), my configuration matches your recommended settings. I do have my clip lengths set to 8hr, not 2-3, but again it's been that way for years without issue.

Cpu is generally well below 80% (typically between 30 and 50 %), even with so many cameras.

none of the cameras are using continuous + trigger. All are trigger only.

I can shuck the external USB and use it as an internal, and will do that to remove it as a source of problems, but, given that this issue started two days ago after making no config changes, I don't want to jump to that conclusion without first addressing all other possibilities. I still have warranty on that drive, which I will forgo once I shuck it.
 

IAmATeaf

Known around here
Joined
Jan 13, 2019
Messages
3,287
Reaction score
3,252
Location
United Kingdom
Just rip the USB a drive open, take out the disk and install it internally, is that not an option?

In the example screenshot you posted above I sort of guesstimate that there are 6-7 events/actions on the USB, maybe BI is making request to read/write and is simply timing out?

Don’t know if you’ve ever tried to perform multiple reads/writes to a USB but the total capabilities/bandwidth will simply be shared, so each concurrent task directly impacts any existing action/task.
 

erkme73

BIT Beta Team
Joined
Nov 9, 2014
Messages
1,540
Reaction score
1,412
That is an option obviously, but I prefer to use that as the la st diagnostic point but since it's going to void the warranty on the drive. Again I don't doubuy what you're saying COULD be the issue. However given that this drive has performed flawlessly both in my old system and my new system until 2 days ago I have hard time believing that this issue has been causing me problems. Ultimately I will remove it and put it into the computer, but I'd like to exhaust all other options first. Once I remove it any remaining warranty on that drive is gone.
 

IAmATeaf

Known around here
Joined
Jan 13, 2019
Messages
3,287
Reaction score
3,252
Location
United Kingdom
Could it be that you’ve started seeing issues as the drive has filled up?

In terms or warranty, be careful when opening it up, plenty of vids on how to attack opening it up without causing damage.
 

erkme73

BIT Beta Team
Joined
Nov 9, 2014
Messages
1,540
Reaction score
1,412
Could it be that you’ve started seeing issues as the drive has filled up?

In terms or warranty, be careful when opening it up, plenty of vids on how to attack opening it up without causing damage.
That's true, but I have fingers like sledgehammers. It's been 90% for months as I'm able to get about 6 weeks on my 60 cameras.
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
Something is not correct. For the number of cameras, the CPU amount is low. The 284 MP/s looks low for a 12153 Kb/s
How many files are in the BI new folder ?
If all you are doing is motion, then set 15 fps on all cameras.

i would try.
1) increase the receive buffer size to 10 MB on each camera that is more than 15 FPS.
2) for a headless computer. use hdmi dummy plug 3) set windows
a) Power setting Sleep: never
b) Power Display screen: never
4) if you have space move the alert folder to the C drive.

The BI data base may be corrupt. If so then delete all the files in the database and reboot the BI PC. I have done this on multiple systems that are continuously recording, but never on a system that is recording only on motion. You will not loss any clips, but you may loose the motion indicators on the time line.

Also look in the Widows BI NEW folder. look for files that are there that are older than they should be, floaters. Manually delete them. I would do this before deleting the DB.

Look at the windows event log for error that are occurring when the memory error occurs.
 
Last edited:
As an Amazon Associate IPCamTalk earns from qualifying purchases.

erkme73

BIT Beta Team
Joined
Nov 9, 2014
Messages
1,540
Reaction score
1,412
CPU was in the upper teens when I first got into all the tweaking - even with 50 cameras at the time. The sub-stream, trigger only, D2D, and all the other tricks really saved the CPU. I moved to the 6th gen i7 (from a 4th gen) to take advantage of the H.265, but have as of yet not changed any of the cameras to 265.

I don't know how the total bit rate and MP count handles the sub-stream calculations. All but a handful of the cameras are sub-stream when not triggered or pulled up in full screen.

New folder has currently has 6683 files - and about a half dozen in then "clipboard" folder nested under new.

1) That's gonna take some doing... I'll need some more "me-time" when kids are asleep.
2) pardon my ignorance, but what is the benefit of the dummy hdmi plug?
3) All power settings set to never/disabled/max power.

4) OS/C-drive is a 256GB SSD with about 50% remaining. Moving existing alert jpgs (22k of them) from the external to the SSD now.
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
The HDMI PLUG allows for GPU operations when there is no monitor. Some software disables the GPU when there is no monitor plugged in.

I currently believe that your performance problem and memory problem are caused by the number of files in your folders. My understand is that windows 10 will need to cache the contents of the folder directory. As it adds files it needs to read the folder directory to verify that the file is not there, that is a lot of traffic over a USB also it is a lot of memory, and a lot of CPU to search the directory.

I do not use alert jpg, so i do not known where to set the file naming.
 
Top