DeepStack only analyzes 3 images each time regardless of setting

[Dixit]

I looked at the Webinar BlueIris had on this and I have it saving the DeepStack analysis details (.dat file). I also have this setup as a GPU based using my GTX 1080 card so got more than enough headroom I feel.

I was playing with the settings and I have tried 10 images every 500ms, 20 images every 250ms and it seems like it still only analyzes 3 images and says nothing is found. Happening across all my 16cameras even though I had it set to 20images at 250ms

Example below:


Ive seen recommendations of using say 5 images per 1secs or similar, but maybe Im not getting it, but why wouldn't I want to crank it up as much as I can like say 20images every 250ms to give DeepStack as many chances as I have to get it to see if there is an object in that time frame? 20images at 250ms is basically 5secs from the way I see it. I feel the 5 images every 1sec isn't ideal at least for me cause if someone goes out frame in less than 5secs then it may not have captured anything (again thats my thinking).

When it does analyze Ive seen the Python process use about 2-3% GPU load which is nothing so I feel we got plenty of headroom.

 

[wittaj]

What CPU are you running and what does CPU and GUI % look like when motion is happening and DS is triggered and running?

With BI/DS integration, it will only "queue" up so many photos and any photos above what can be handled "spill" out and are not analyzed.

Based on your analysis time, I wouldn't think that is happening as that is usually after about 4,000ms, but I would expect with a GPU that your analysis ms should be in the 50ms range, whereas yours are up over 150-300ms.

How many cameras are triggering and sending to DS at the same time? Maybe collectively it is exceeding the time.

Based on what you are describing, it sounds like there is a bottleneck somewhere or a setting is off somewhere else. What is your break time to trigger when triggered? If it is under 5 seconds, then the "trigger" has ended before it gets through your cycles and DS stops. Bump that up to 30 seconds for testing and see if it helps.

Some people have had success adding giraffe to the items to search for as a way to keep it going longer, so try that as well (unless giraffes are common where you are at LOL).

 

[Dixit]

Running an i7-6700K with 32gb RAM. Like right now my landscapers are here and its setting it off nonstop and the highest my CPU hits is 30% on BlueIris and thats with the app open. GPU never goes above 5%. Like right now Im seeing about 4-5 cameras triggering at the same time with the Landscapers. I do see at times DS sitting in the 50ms timeframe but sometimes sitting in the 100-150 potentially cause Im throwing the 20images at it. I tried even one at 50images at 250ms and noticed that all I saw on that .DAT file was 4 images. Only things Im asking it to look for on the outdoor cams are: person,car,truck,unknown,bird,cat,dog

My break time is sitting at the 10sec default with a 60sec max duration per trigger/alert. I'll keep playing with the break time and images. I just remember on the webinar I think Sam had like 7-8 images showing on his DAT file analysis, so was trying to see how he got that many.

Giraffes? Interesting....... Definitely not common here in rural ATL area lol.

 

[Dixit]

Figured it out. It was the use Main Stream if available setting under AI for each camera. I had it checked based on what was mentioned in the webinar about turning it off for trouble shooting then turning it back on if your computer can handle it and mine didnt seem like it had an issue. But I turned it back off and now Im noticing a ton of analysis and doing it in 50ms or less. Here's one now that just happened and did 15 images.


Guess pulling full frames on main steam off a 4K camera is just not happening right now even though I wasn't seeing technically any CPU or GPU usage to indicate it was getting bogged down. Im going to guess might have to simply do with the high res image size and something on the queuing side of the house as those images are about 4-5MB each vs sub stream where its barely 500KB each.

 

[Shap]

I have a similar experience. I still do not understand why there is a problem pulling 4-5MB files each second on a very powerful machine. Should not be an issue at all.

Anyway, pay attention that you have the same problem I had - you have T-562 and then twice T+2108. And you do not have T+0 (that is actually important).
This is because you have "Hi-Res JPEG" option. Set it to DB Only to solve this problem.

 

[Dixit]

I have a similar experience. I still do not understand why there is a problem pulling 4-5MB files each second on a very powerful machine. Should not be an issue at all.

Anyway, pay attention that you have the same problem I had - you have T-562 and then twice T+2108. And you do not have T+0 (that is actually important).
This is because you have "Hi-Res JPEG" option. Set it to DB Only to solve this problem.

If I change this to DB Im assuming I won't get any files in the Alerts folder right? If so, this may not work since Im using that Alerts folder to drive a Telegram notification using Curl.exe. So it sends me the exact alert image with the AI highlight box around the object as a notification into Telegram.

 

[wittaj]

If you tell it to burn the deepstack rectangle or whatever it is called it will put them in the alerts folder.

 

[Dixit]

I have a similar experience. I still do not understand why there is a problem pulling 4-5MB files each second on a very powerful machine. Should not be an issue at all.

Anyway, pay attention that you have the same problem I had - you have T-562 and then twice T+2108. And you do not have T+0 (that is actually important).
This is because you have "Hi-Res JPEG" option. Set it to DB Only to solve this problem.

This made it a little better, didnt see a T+0 but saw it at +66msec or lower vs +2000 or more we seen before.


But when setting it back to use main stream if available even on DB Only, it still cant handle doing more than 3 images on the same camera. So just set it back to sub stream on AI.

This was with it set to use main stream.

 

[Shap]

Well, it is much better - now you recovered 2 sec(!) of possible identifications!

Same behavior here - so there are 4 bugs (I already open tickets for that, but support used 2MP cameras that do not trigger this problem. They need to use 4Mp+ cameras for that):
------
1. When Hi-Res JPEG option is used, 2 identical frames are sent to DS. Should be only one. (regardless if substream or main stream is used for DS). (BI indeed sends 2 frames, as you can see DS processing time is different for each one of them.)
2. When Hi-Res JPEG option is used, there is a delay for T+.., and there is no T+0 (or close to 0). Motion coordinates are also not in sync - i.e. along with image T+X that is sent to DS, motion coordinates that are sent are from image T+0. This creates a problem with object identification. (regardless if substream or main stream is used for DS)
3. When DB Only option is used, I still do not see T+0. Usually T+60. Based on the motion rectangle coordinates sent to DS, it seems that the motion coordinates are for T+0 but the image is T+60ms. This is much more visible for objects in motion. (I tested it with using substream for DS).
4. When main stream is used,BI seems to not be able to send Hi-Res images to DS fast enough. It is not clear why, as DS processing for each image shows <200ms, but the delay between images that BI is sending is often 1-2 sec. So selecting an option to send 5 images every 500ms will not work when using main stream for DS. Even on a very powerful machine.
I would also expect BI to be able to queue the images for DS processing if needed.

P.S. It is important to mention that all tests were done with just 1 camera active - so there should not be any issues with performance.

 

[jaydeel]

so there are 4 bugs

Just to be clear... these issues are observed with:
1. DeepStack CPU version
2. a 4MP+ camera

Correct?

If so, this explains why I was unable to reproduce you observation. I'm using DeepStack GPU and 2MP cams.

 

[Shap]

Just to be clear... these issues are observed with:
1. DeepStack CPU version
2. a 4MP+ camera

Correct?

If so, this explains why I was unable to reproduce you observation. I'm using DeepStack GPU and 2MP cams.

Correct on both items. (I am using DS CPU version and 4MP camera).

For the camera, I used: Loryta IPC-T5442TM-AS-LED , 4MP camera.
I also tried with Amcrest FCM-IP8M-T2499EW28MM, 8MP camera.

 

[Dixit]

If you tell it to burn the deepstack rectangle or whatever it is called it will put them in the alerts folder.

For some reason this didnt work. I thought it did but this morning noticed Alerts in my BI, but no alerts folder was created (I have a custom folder setup to create one per date). I just turned this back on for one camera to set it back to Hi Res JPEG and walked in front of the camera and bam created the folder and Telegram alert came in (thats really how I knew something was wrong since no Telegram alert came in which normally happens when kids go to school in the morning).

What’s even more odd is even the .DAT analysis files wasn’t even created. I know there had to be a ton a motion on my cameras for DS to analyze.

 

[Shap]

For some reason this didnt work. I thought it did but this morning noticed Alerts in my BI, but no alerts folder was created (I have a custom folder setup to create one per date). I just turned this back on for one camera to set it back to Hi Res JPEG and walked in front of the camera and bam created the folder and Telegram alert came in (thats really how I knew something was wrong since no Telegram alert came in which normally happens when kids go to school in the morning).

What’s even more odd is even the .DAT analysis files wasn’t even created. I know there had to be a ton a motion on my cameras for DS to analyze.

I have only one Alerts folder and JPG and DAT files are created there w/o a problem with DB Only option set. (the resolution is low, but that is expected).

 

[Dixit]

I have only one Alerts folder and JPG files are created there w/o a problem with DB Only option set. (the resolution is low, but that is expected).

Odd for sure. I know DS was running since the alerts are there.

All I did as a test was turn one camera back to JPG and walk-in front of it and within seconds got my Telegram alert and now the folder was created for that alert for todays date.

Last night I swear it was also working but somehow seemed to stop doing anything on alerts folder around 1130p even though no issues with DS.

 

[Shap]

Odd for sure. I know DS was running since the alerts are there.

All I did as a test was turn one camera back to JPG and walk-in front of it and within seconds got my Telegram alert and now the folder was created for that alert for todays date.

Last night I swear it was also working but somehow seemed to stop doing anything on alerts folder around 1130p even though no issues with DS.

One of the things I wish we had in BI is a detailed debug log on what is happening... The current log file does not provide such a level of details ...

 

[Dixit]

One of the things I wish we had in BI is a detailed debug log on what is happening... The current log file does not provide such a level of details ...

Ok ive tested this a bit further and there is no rhyme or reason on how this works now for me. I set it to DB only rebooted the entire system, it created the alert and image and sent it to telegram. But then other cameras don't send alerts to telegram but alerts show up in BI like they should, but looking at Alerts folder, nothing was created for them. Move it to JPG it works.

And I agree, wish there was a better debug to see what its got issues with or why it couldn't create say here a folder or the alert file. I need to walk away from this for a bit before I smash something lol.