Proper way do delete files for more space?

nbstl68

Getting comfortable
Joined
Dec 15, 2015
Messages
1,399
Reaction score
321
My BI is doing odd things and hitting 79-99% CPU and close to 4GB memory usage. It is also choking on processing video or something like that. Two cameras covering the same area, ( a 5231 2mp and a 4 x 2mp 180 deg Dahua cam.) but when watching live in BI ,the "live" movement doesn't even start in the other camera until almost two minutes later?? Wierd huh?
The computer specs are good enough that it should in theory be able to handle many more cameras than I have I think, (Win 10, I7-6700 CPU, 24GB ram, 128GB SSD and 8TB 7,200 WD Purple BI W 10 cams...1 of them is the 4 x 2mp so count that as four?).

I have BI using the majority of my WD purple 8TB hard drive to store video.

BI folders:
New - c:\BlueIris\New (The SSD, 8.2GB free)
Stored - D:\BlueIris\Stored (The 8TB WD Purple, BI set to use all 8GB, 1.2GB free)

I assumed leaving a couple GB free on both drives would be more than enough, (a GB is a lot of space!) for everything to run efficiently but I'm guessing this may be the culprit or at least a part of the issue.

So, I would like to start by maybe reducing the max use of the WD by a TB and see what happens. To do that I need to delete the oldest BI video files.

All this for my main question... What is the "proper" way \ order to delete the video files and then restrict the max BI use?
Can I just delete the oldest ones from file explorer without messing up BI's databases and tracking of the files or is there a way I should do it all within BI?

If freeing up more space does not help, hopefully I have provided enough additional information so someone can provide some other reasons for the weird time warp recording delay between the 1x 2mp image sensor camera and the 4 x 2mp image sensor camera and why BI is staying in the 80-100% CPU w 4GB memory usage.
Thanks for any thoughts or direction!
 

holiday

Pulling my weight
Joined
Sep 12, 2018
Messages
273
Reaction score
182
Location
Having a holiday
I dont know what is wrong.. but maybe increase or upgrade your ram size.
mine running cpu 36% Ram 5.06G with 39 cameras

cpu = i7-6700
ram = 24gb
 

sebastiantombs

Known around here
Joined
Dec 28, 2019
Messages
11,511
Reaction score
27,690
Location
New Jersey
A couple of comments. First, there is no point in having a New and a Stored folder, especially on different drives. BI has to move the files from one to the other which is CPU and drive overhead. Write everything to the 8TB and let BI handle space. Set the space, not a number of days, for BI to delete files. It'll handle that with no problem. It's also not such a great idea to write video to a SSD drive. Video is written constantly and SSD drive life is dependent on write, not read, operations.

Next is have you tuned BI and your PC? Look in the Wiki, in the blue bar at the top of the page. There are a number of tips in there that will lower CPU utilization such as writing directly to disk and using hardware acceleration.

Also have a look at using sub streams for the cameras. I'm running an i7-600K with 17 cameras and the CPU utilization is under 20% between hardware acceleration, direct to disk recording and, the biggest savings, was enabling sub streams. I have 32GB of RAM and BI typically uses anywhere from four to six GB depending on the time of day.

Optimize Blue Iris

Substreams in BI
 
Last edited:

nbstl68

Getting comfortable
Joined
Dec 15, 2015
Messages
1,399
Reaction score
321
This a lot of goodnl info.
O have 24GB ram so that should be way more than enough.
Also do d2d and 8n"think" I use hardware acceleration...is there a setting for that somewhere I guess or on by default?

"Use substreams"?
I'll read through the links on this. Whenever I set up the cameras I just let BI scan for the settings and never selected substream or main...sonof BI does it by default then I'd say it is using it. If it has to be manually selected then I'm not. I know IN the camera settings via the web interface to each camera I always see main and substreams are enabled...don't know which one BI uses but for recording to the HD to have the best video evidence to play back later, shouldn't that be the main stream with the highest quality video settings, not a substream?


...and just to be clear, changing the space allotted on BI is all I need to do to reduce how many files it stores. Does BI delete 1G)TB of files say if I change it from 8TB to 7TB or do I still need do delete them manually too?
 
Last edited:

sebastiantombs

Known around here
Joined
Dec 28, 2019
Messages
11,511
Reaction score
27,690
Location
New Jersey
Sub streams need to be configured in each camera in BI and is not enabled by default. The easiest way is to click on "find/inspect" in the video configuration tab, then select the sub stream feed. Say "OK" and it should be good to go. Using sub streams will reduce CPU utilization by a factor of three or four. Incidentally, sub streams are available in version 5.2.x.x or higher. Previous versions don't support it.

Hardware acceleration needs to be set manually as well. It can be set for each camera in the video configuration tab or globally in the video tab.
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,428
Reaction score
47,541
Location
USA
BI records both unless you tell it to only record one. But you also have to tell it to pull the substream for each camera.

Post the camera status and what does it show:

1618366719068.png

In this example, it shows one camera is pulling 4MP mainstream and 0.3MP substream and the others are 2MP mainstream.

If you are looking at more than two cameras on the screen, it will be showing the substream. Only when you pull up one camera will it go to mainstream.

Confirm on the playback that you are viewing the mainstream - left click on the mouse and it will give you the option to see which stream you are watching.
 

nbstl68

Getting comfortable
Joined
Dec 15, 2015
Messages
1,399
Reaction score
321
A couple of comments. First, there is no point in having a New and a Stored folder, especially on different drives. BI has to move the files from one to the other which is CPU and drive overhead. Write everything to the 8TB and let BI handle space. Set the space, not a number of days, for BI to delete files. It'll handle that with no problem. It's also not such a great idea to write video to a SSD drive. Video is written constantly and SSD drive life is dependent on write, not read, operations.
I put the db and the "NEW" folder on the main (SSD C:) drive per the BI note. Am I misunderstanding that? Will it really perform better if the New is on the WD drive?
"No point having a New and a Stored folder"...What would l I have then? I thought I read in the help file somewhere it says I have to have both a New and a Stored folder...so would it be better to point both to the D; and leave only the db on the C: SSD and clean up other files off to free up space on the C:? I just read elsewhere windows "likes" to have somewhere between 10 and 25% on the primary drive empty if possible for best performance, (not sure why though)...which seems like a huge waste of space.


1618374601362.png

I am limiting by storage size only. I assumed maybe allowing BI to use the whole drive may be causing some of the issue. I just changed it to max 7.5TB but disc use still shows it full which is why I am thinking changing it in BI just changes the allotment and maybe does not delete anything already there and I may have to do a manual delete of some files. I just didn't want doing that to possibly confuse BI's db.

1618374857631.png

I know storage sizes don't always make sense.
1618374945059.png
 

nbstl68

Getting comfortable
Joined
Dec 15, 2015
Messages
1,399
Reaction score
321
With regard to "Hardware Accelerated Decode", the per camera settings are all "Default", (which is why I assumed it was on) but the global setting is "NO"...so I assume that means the default by camera is then also , "NO"?
And if so, the next question is what do I pick? I was thinking this was an "on\off" setting.
The i7-6700 chip is Intel. My graphics "card" shows as NVidia GeForce GTX 745.
So would I pick Intel or the Nvidia option?

Th BI help file just states, "there are limits to the number of cameras that may be assigned to decode..." but it does not say how many cameras and just that, "actual FPS will be seen to decline..." if it is getting overloaded.

1618376417024.png
 
Joined
Aug 8, 2018
Messages
7,386
Reaction score
25,889
Location
Spring, Texas
Yes, the BI help file does state to put the DB and NEW folder on the fastest drive. But that is fairly old info I think. Most everyone here puts the OS and the BI DB on the fastest drive, which in your case is the SSD. I put my NEW folder on my WD10TB Purple drive. I put the STORED folder on my SSD but never actually use it. Yes, it does need to be somewhere, but you do not need to use it. As pointed out above, moving the data from NEW to STORED takes system resources and REALLY slows down your system. I did this in the beginning and had all sorts of issues. Once I stopped moving those files, everything moved smoothly.

1618384198520.png

1618384212276.png

As far as the HA, pick INTEL. You could set some cams to Nvidia if you want. I split mine up between the two.
 

sebastiantombs

Known around here
Joined
Dec 28, 2019
Messages
11,511
Reaction score
27,690
Location
New Jersey
I suspect that if you click on the graph with the lightning bolt, in the upper left of the BI console, and look at the drives you'll have an over allocation error on that 8TB drive. It is 8T but that's raw space. Just formatting the drive uses space to hold the index. You should leave about 20 or 30GB unallocated so BI has room for small overages as files come and go.

That help file is outdated and was written when everything was a platter drive. The DB should be on the C: drive for best performance, but everything else should be written to platter drives.

@samplenhold gave you the skinny on hardware acceleration. I'm using NVidia on all my cameras, but have a fairly powerful NVidia graphic card. It isn't even being tickled by hardware acceleration, an RTX2070.
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
a little late on the post
====================================
My Standard allocation post.

1) Do not use time (limit clip age)to determine when BI video files are moved or deleted, only use space. Using time wastes disk space.
2) If New and stored are on the same disk drive do not used stored, set the stored size to zero, set the new folder to delete, not move. All it does is waste CPU time and increase the number of disk writes. You can leave the stored folder on the drive just do not use it.
3) Never allocate over 90% of the total disk drive to BI.
4) if using continuous recording on the BI camera settings, record tab, set the combine and cut video to 1 hour or 3 GB. Really big files are difficult to transfer.
5) it is recommend to NOT store video on an SSD (the C: drive).
6) Do not run the disk defragmenter on the video storage disk drives.
7) Do not run virus scanners on BI folders
8) an alternate way to allocate space on multiple drives is to assign different cameras to different drives, so there is no file movement between new and stored.
9) Never use an External USB drive for the NEW folder. Never use a network drive for the NEW folder.


Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.
Hint:
On the Blue iris status (lighting bolt graph) clip storage tab, if there is any red on the bars you have a allocation problem. If there is no Green, you have no free space, this is bad.
======================================
 

nbstl68

Getting comfortable
Joined
Dec 15, 2015
Messages
1,399
Reaction score
321
So, def seeing some red...I should probably start with this. How do I fix an overallocation error?
Once that is corrected I'll move the NEW to D: drive and increase it to 7 of the 8 TB allocation.
Reduce the max amount of Stored to 0 but leave the Stored folder is the suggestion then? Will this cause BI to delete all the existing 7+TB of stored video files, about a month's worth, as it builds to the New folder? I'm not too concerned with losing the older files but just curious.
Then remove a chunk windows\work files off the C: SSD to bring its max use down and see what happens.


1618422963919.png


Regarding @wittaj post #6, it appears mine only shows main stream. I see nothing indicating a sub stream is being pulled\recorded. I still am unclear how adding more of any streams would help?


Regarding HA,
Interesting, I tried both "Intel" and "Nvidia NVDEC" as global selection and left all camera's individual selection to "default" and shut down\restarted BI each time.
When Intel was selected there was 0 difference in CPU\Memory GPU (0) use shown in the task manager. Does that indicate for whatever reason the Intel selection or my chipset, which is supp to support\use Quicksync, is not compatible with this setting?

When I tried NVidia, the GPU in task manger jumps from 0% to around 38-39%. CPU decreases but only to around 60-80% down from 90-100%, ( about the same with or without the BI window maxed or minimized). RAM stays around low 4GB range.

1618423483436.png
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,428
Reaction score
47,541
Location
USA
You have given BI too much GB memory in the folders settings above. Cannot use the whole drive. Need to leave like 10%.

Using the substreams, which is in the camera tab where you set up the camera, allows BI to use the substreams which will bring the CPU down tremendously. Right now, with mainstream only, every operation like motion detection, etc. is using the mainstream and that is CPU intensive. You lose nothing by using the substreams, except maybe a little storage if you record both.

Regarding HA, are you using H264 or H265? Is Smart Codec off?

A member here is running a i7-4790 4th generation CPU with 50 cameras and 30% CPU....you have a much better machine and a lot less cameras and your CPU is way higher...See the wiki to optimize your system and do everything it says. Changing something you think will be better usually ends up creating a larger CPU utilization:

 

nbstl68

Getting comfortable
Joined
Dec 15, 2015
Messages
1,399
Reaction score
321
OK I'm surprised, since it sounds like using sub streams will help reduce CPU a lot, that BI does not automatically config to use the sub streams for some things. I have no clue how to write out for a sub stream so I need to read the link that was provided above then to set up sub streams individually by camera.

I believe I have most cameras on the default H264 except for the 4 x 2mp 180 deg cam which I switched to H265. Most all of these I have record 24/7 and changing that one brought the file size down by half from 4GB\hour to 2GB.
I could probably look at not running them all at 25-30 fps too but I like how that looks in replay esp for a fast moving object.

I've been working through @SouthernYankee list above and already doing most of that. I'll take a look at the full wiki and optimizing link tonight.
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,428
Reaction score
47,541
Location
USA
With the newer BI versions, it is simply a dropdown to select (unless you have an offbrand camera).

The reasons BI doesn't automatically pull it (I suspect) is because some cameras may offer 2 or 3 substreams and this allows the user to select the one that they want. Plus, you have to enable the substream from within the camera GUI, and if you do not have substream checked within the camera itself, then BI may get wonky trying to pull a substream that doesn't exist. So it is off by default.

Keep in mind that movies on the big screen are shot at 24 FPS, so I don't think we need 30FPS for our tablets, phones, and computer screens LOL. 15FPS is sufficient for surveillance cameras.
 

nbstl68

Getting comfortable
Joined
Dec 15, 2015
Messages
1,399
Reaction score
321
I am currently on Ver 5.2.9.2. Does it have to be newer than that? I have mostly Dahua cameras and there are only a couple to choose from. I typically have just run the Find\Inspect and it puts them either on Generic or or I select one at random until the camera functions work. Always wondered why the connection db in BI isn't a lot larger selection.
I do not see a drop down selection for sub stream selection. Just the blank "Sub Streams (+ params) which I know nothing about finding the right info for that.
I do have sub streams enabled in all the camera GUIs themselves.


My selection...
1618435542555.png

What BI always chooses when running Find\ Inspect
1618435583481.png
 

sebastiantombs

Known around here
Joined
Dec 28, 2019
Messages
11,511
Reaction score
27,690
Location
New Jersey
There are two ways to enter the sub stream.

Copy the information in the main stream box and paste it into the sub stream box. Then change the "subtype=0" to "subtype=1".

Alternately, update to a newer version, run find inspect, the select the sub stream from the dropdown menu on the configuration page.
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,428
Reaction score
47,541
Location
USA
Yeah, whatever BI pulls under find/inspect is usually good.

OK, yeah the newer versions has pull downs, so in your case, simply copy what is in the mainstream into the substream:

So copy /cam/realmonitor and then add ? and then copy what is under params and change the 0 for subtype to 1, so something like:

/cam/realmonitor?channel=1&subtype=1...except copy over everything in yours.
 

looney2ns

IPCT Contributor
Joined
Sep 25, 2016
Messages
15,521
Reaction score
22,657
Location
Evansville, In. USA
I put the db and the "NEW" folder on the main (SSD C:) drive per the BI note. Am I misunderstanding that? Will it really perform better if the New is on the WD drive?
"No point having a New and a Stored folder"...What would l I have then? I thought I read in the help file somewhere it says I have to have both a New and a Stored folder...so would it be better to point both to the D; and leave only the db on the C: SSD and clean up other files off to free up space on the C:? I just read elsewhere windows "likes" to have somewhere between 10 and 25% on the primary drive empty if possible for best performance, (not sure why though)...which seems like a huge waste of space.


View attachment 86809

I am limiting by storage size only. I assumed maybe allowing BI to use the whole drive may be causing some of the issue. I just changed it to max 7.5TB but disc use still shows it full which is why I am thinking changing it in BI just changes the allotment and maybe does not delete anything already there and I may have to do a manual delete of some files. I just didn't want doing that to possibly confuse BI's db.

View attachment 86811

I know storage sizes don't always make sense.
View attachment 86812
Study this attachment.
 

Attachments

Top