DeepStack "Nothing Found"

[Wen]

Running Deepstack with BI 5.4.7.8. Running much better than I expected.

But I get anomalys like this snapshot. Deepstack has recognized that there are two trucks in the FOV. The silver chevy is parked and there is no motion trigger on it. The Waste Management truck is moving at it caused the alert. The confidence level for the cam is set at 50, Real time image is set at 1. And the To Confirm is set at . Make time on the cam is .5

person,car,truck,bus,bicycle,boat,dog,cat

The clip shows "Nothing found". But the Waste Management truck show a hit of 84%. It pulls into to the culdesac and then slowly backs out so the cam gets a consistent reading over the 50% requirement.

This happens fairly frequently, and I can't figure out how to remedy this issue.

Would updating BI help? I don't update without a basis to grab the latest file.

Thanks in advance!

 

[wittaj]

Is this picture from the tuning option? I suspect so since you said the clip shows "nothing found"

If so, when ran in tuning, it will show everything DS looks for.

If this was live, DS should at some point recognize that vehicle is not moving and not send you an alert for it. Mine don't.

Do you have face recognition turned on? Some of us have found that is causing an issue.

What do your BI logs look like - Is it saying DeepStack restarted after every trigger? I suspect it does if you are getting nothing found.

Do your iframes and FPS match in the camera?

And some instances like dusk/dawn, DS may miss and struggle.

 

[Wen]

Yes, it's from the "tuning" option, I played the clip and grabbed that frame that showed the WM truck at 84%. Seems like the clip should have shown as "confirmed" Truck.

The iframes as FPS are both set at 15.

I'm not using Facial Recognition.

Here's an excerpt from the log file. Deepstack is working, it's just not resolving the trigger alert correctly.

6/15/2021 6:23:03.844 AM Cam18WiFi_NG DeepStack: person:86% [1269,204 1435,618] 415ms
10 6/15/2021 6:23:03.987 AM Server ::: Connected
0 6/15/2021 6:23:04.102 AM Cam18WiFi_NG Push: OK to Pixel
3 6/15/2021 6:27:10.028 AM Cam18WiFi_NG MOTION
0 6/15/2021 6:27:11.141 AM Cam18WiFi_NG DeepStack: Alert cancelled (occupied)
10 6/15/2021 6:28:49.292 AM Server ::ffff:192.168.100.154: Connected
10 6/15/2021 6:28:49.637 AM admin ::ffff:192.168.100.154: Login
0 6/15/2021 6:28:53.214 AM App Current profile: SEE THE ROAD 8, 19, 17, 18
3 6/15/2021 6:29:20.505 AM Cam19 MOTION
3 6/15/2021 6:29:20.844 AM Cam8_Wifi_Up MOTION
0 6/15/2021 6:29:22.202 AM Cam8_Wifi_Up DeepStack: Alert cancelled (nothing found)
0 6/15/2021 6:29:22.554 AM Cam19 DeepStack: Alert cancelled (nothing found)
10 6/15/2021 6:29:33.829 AM admin ::ffff:192.168.100.154: Logout, 0:44
10 6/15/2021 6:29:36.176 AM Server ::ffff:192.168.100.154: Connected
10 6/15/2021 6:29:36.207 AM admin ::ffff:192.168.100.154: Login
3 6/15/2021 6:29:53.867 AM Cam8_Wifi_Up MOTION
0 6/15/2021 6:29:54.995 AM Cam8_Wifi_Up DeepStack: Alert cancelled (nothing found)
3 6/15/2021 6:29:56.789 AM Cam19 MOTION
0 6/15/2021 6:29:58.492 AM Cam19 DeepStack: Alert cancelled (nothing found)


Thank you
wittaj

 

[wittaj]

OK, so the tuning option is going to always show more than the live stream for the alert and triggering and AI identification.

Looks like yours is working correctly.

Some of had the option working where in tuning playback, the border around the image turns blue to show which image was sent to DeepStack. Can you see if yours does that as then you would know which images are sent.

So what could be happening is in certain scenarios, the shadow triggers the DS, but the object isn't in the frame so it comes back not triggered.

How many additional pictures are you having BI send to Deepstack? The default is zero. Maybe sending 5 or so might improve it, but that is narrow field of view at the street.

I read in one of the threads that the developer is working on an update that would allow the first image sent to be delayed just a smidge and that may take care of some of the first image issues.

And then of course DeepStack isn't going to be 100% accurate and certain field of views or time of day may be an issue.

I suspect that early morning time and the green truck filling in with the green trees/shrubs and live it suffers. Maybe try lowering the confidence? I run all of mine lower at 10% until I feel comfortable at what % confidence each field of view needs.

 

[Wen]

Thank you wittaj.

The confidence setting is/was 50% and DS is sending 1 real time images. That's a setting of one, so it should be sending three....

I've been trying to dry lab and debug the DS settings. The software usually captures the trigger event and properly categorizes the clip.

I was very reluctant to take the Deep Stack Dive, but it's been fairly painless.

For the other users, make sure to get the correct version of Deep Stack, I spent two days messing with DS because I downloaded the GPU version.

If you're getting a Server Error 100 that may be your issue.

 

[looney2ns]

I would update to 5.4.7.11.

 

[JL-F1]

My BI setup works fine with the GPU version of DS. I do not get any server error 100

 

[wittaj]

I have the CPU version and was getting server error 100 when custom models and faces.

 

[kklee]

I have the CPU version and was getting server error 100 when custom models and faces.

I have the same problem and on top of that, Windows crashed a couple of times due to a memory leak causing it to run out of resources, the two programs identified in the event log as the cause are BI and DS. I've turned off faces for now to see if that helps. The first crash happened a few hours after upgrading to 5.4.7.11 and the second crash didn't happen for a couple of days.

 

[wittaj]

I have the same problem and on top of that, Windows crashed a couple of times due to a memory leak causing it to run out of resources, the two programs identified in the event log as the cause are BI and DS. I've turned off faces for now to see if that helps. The first crash happened a few hours after upgrading to 5.4.7.11 and the second crash didn't happen for a couple of days.

Yep, mine crashed on 5.4.7.11 so I rolled back to 5.4.7.10

 

[JerryCooke]

I'm running 5.4.7.11 and having major 'Nothing Found' issues.

I've got confidence at 40%, real time images at +5, am only looking for people, cats, or dogs. Yet despite my going and standing outside for 10s and it having 80-90% confidence in me, the alert gets flagged as 'Nothing Found'. Very confused.

 

[wittaj]

Open up the log and see if it says DeepStack restarted.

You may need to open the actual log file in a text editor to see all of the logs.

I suspect you have a Server Error 100 going on.

Are you trying faces as well? Many of us have found issues with that running as well.

Are you using custom models? Many have found that only one custom model works.

 

[Dave Lonsdale]

Open up the log and see if it says DeepStack restarted.

You may need to open the actual log file in a text editor to see all of the logs.

I suspect you have a Server Error 100 going on.

Are you trying faces as well? Many of us have found issues with that running as well.

Are you using custom models? Many have found that only one custom model works.

Hello wittaj. It’s a little embarrassing for me to confess that I have not examined the detail in my logs to perhaps gain an insight into my falsely cancelled alerts. Unfortunately I don’t have access to my system this week, but I’m sure it would be helpful for a number of us if you were good enough, please, to post relevant examples from your log together with the an explanation of the key entries.

A couple of related points:-
1. A query I’ve put the forum twice but not answered is does BI wait (up to 1 second) for the next key frame to send to DeepStack after a trigger instead of sending the image we can view that was used to create the trigger? Do you know?
2. DeepStack’s minimum confidence is 40% regardless of the setting (10% is the same as 40%). I did ask KP if he chose this lower limit. He has not but suggested that “to action lower confidence levels than 50% makes no sense”. Personally I disagree - DeepStack does not return false positives but does falsely cancel. Whilst for sure I need to work more on my other settings, I think a lower percentage eg for wanted wild life would be beneficial - I could tolerate a few false positives.

 

[wittaj]

So I just updated to the newest version and the DeepStack error is happening again, so here is what it looks like in the log:

1 6/16/2021 6:45:38.212 PM Front DeepStack: Server error 100
0 6/16/2021 6:45:38.212 PM Front DeepStack: Alert cancelled (nothing found)
0 6/16/2021 6:45:40.8915 PM App DeepStack has been restarted

So I know there is an issue with DeepStack - since I am using the custom model, I will start there and turn that off and see if it gets going.

But the Server Error 100 does not show up within the log screen in BI for me, I have to open the actual log.txt file to see more details.

 

[Dave Lonsdale]

OK wittaj, thank you. Are you thinking of reporting this problem to support? May I ask you to report back how it develops please via this thread? As I said, I don’t have access to my system this week, how will I find my log.txt file?

 

[tward392]

I was getting those errors in my log from 01-June-2021 up until 10-June-2021, haven't seen one since. But I do still get a lot of the "DeepStack: Alert cancelled (nothing found)" and if I run the clip through the test/tuning option it picks it up. I hadn't reported it because I just figured since this stuff was still a work in progress it would get figured out. I just scan through the cancelled alerts to make sure I don't miss anything.

 

[wittaj]

Yes I have reported issues to BI as I come across them. As others have said, it is so new that they probably haven't been able to test every type of situation that people may try this with.

In my instances, turning off Face recognition and rebooting the machine got it going again. Or if I tried to run two custom models at the same time I would get that error.

To view the total log file, you need to have first have told BI to save log files. If you have done that, then you can open up the log file in BI and at the bottom it says "Open File" and you select that and the full log opens up.

The tuning feature is kinda cool, but I do not believe there is a real correlation between it and the actual live DS yet.

In tuning, DS has the ability to slow the frames down to check it (and the help file does mention the AI boxes may be delayed); whereas live it doesn't give it that much time. So it is neat to see, but not yet there on correlation. Now if everyone could get the blue frame around the entire image to know which picture was sent to DS, then it would start to provide some use. Some people get a blue box outline around the outside of the entire picture, but many of us have not seen that.

So I set all of mine to 10% until I see what is the minimum confidence I see in my field of view for all conditions - rain, dark, light, etc. and then set it. Now of course Dave has indicated that DS doesn't look at below 40%, so it would need at least that to ID.

 

[kc8tmv]

Seeing a lot of "DeepStack Sever Error 100" even after last nights update to 5.4.8.0. Also after running a clip through DS analysis with the new DeepStack Analyzer window open, I go to close the clip window and get the below error every time.


Below are the log entries just for today. Note the number of "DeepStack: Server error 100" entries.
Only custom model is Dark.
I am using Faces and that is not working "too bad".

Great progress. Anyone have any thoughts on the below?

0​	6/17/21 12:00 AM​	App	Today's sunrise: 6:43 AM, sunset: 10:27 PM
3​	6/17/21 12:08 AM​	Porch	MOTION
3​	6/17/21 12:08 AM​	Shed	MOTION
1​	6/17/21 12:08 AM​	Porch	DeepStack: Server error 100
0​	6/17/21 12:08 AM​	Porch	DeepStack: Alert cancelled (nothing found)
1​	6/17/21 12:08 AM​	Shed	DeepStack: Server error 100
0​	6/17/21 12:08 AM​	Shed	DeepStack: Alert cancelled (nothing found)
0​	6/17/21 12:09 AM​	App	DeepStack has been restarted
3​	6/17/21 12:22 AM​	Porch	MOTION
0​	6/17/21 12:23 AM​	Porch	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 1:40 AM​	FrontPorch	MOTION
0​	6/17/21 1:40 AM​	FrontPorch	DeepStack: Alert cancelled (nothing found)
0​	6/17/21 2:00 AM​	Clips	DBCompact: started
0​	6/17/21 2:00 AM​	Clips	DBCompact: finished
10​	6/17/21 2:39 AM​	Server	::ffff:45.146.165.123: Connected
3​	6/17/21 2:47 AM​	FrontPorch	MOTION
0​	6/17/21 2:48 AM​	FrontPorch	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 3:22 AM​	Shed	MOTION
0​	6/17/21 3:23 AM​	Shed	DeepStack: Alert cancelled (nothing found)
0​	6/17/21 3:41 AM​	App	Console: disconnected
10​	6/17/21 3:41 AM​	local_console	::: Logout, 5031:15
0​	6/17/21 3:41 AM​	App	Stopped
0​	6/17/21 3:41 AM​	MQTT	Disconnected
0​	6/17/21 3:41 AM​	App	Started version 5.4.8.0
0​	6/17/21 3:41 AM​	App	Updated from version 5.4.7.11
0​	6/17/21 3:41 AM​	MQTT	Connect: OK
0​	6/17/21 3:41 AM​	Server	Started on port 8081
1​	6/17/21 3:41 AM​	App	Service is running as LocalSystem; see Administration in Help
1​	6/17/21 3:41 AM​	App	Re-install service to remove BlueIrisService.exe dependency
4​	6/17/21 3:41 AM​	Clips	C:: 7352, 50.5G/551.0G (+308.3G)
0​	6/17/21 3:41 AM​	App	Shield: Yellow
0​	6/17/21 3:41 AM​	App	DeepStack has been started
0​	6/17/21 3:41 AM​	App	DeepStack responding at 127.0.0.1:8082
0​	6/17/21 3:41 AM​	App	Today's sunrise: 6:43 AM, sunset: 10:27 PM
10​	6/17/21 3:41 AM​	Server	::: Connected
10​	6/17/21 3:41 AM​	local_console	::: Login
0​	6/17/21 3:41 AM​	App	Console: connected 1
0​	6/17/21 3:42 AM​	App	Shield: Green
3​	6/17/21 4:00 AM​	Shed	MOTION
0​	6/17/21 4:01 AM​	Shed	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 8:35 AM​	Drive	MOTION
0​	6/17/21 8:36 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 8:44 AM​	FrontPorch	MOTION
0​	6/17/21 8:45 AM​	FrontPorch	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 8:57 AM​	Drive	MOTION
0​	6/17/21 8:58 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 9:06 AM​	Drive	MOTION
0​	6/17/21 9:07 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 9:14 AM​	Drive	MOTION
0​	6/17/21 9:15 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 10:37 AM​	Drive	MOTION
0​	6/17/21 10:38 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 11:10 AM​	Drive	MOTION
1​	6/17/21 11:12 AM​	Drive	DeepStack: Server error 100
0​	6/17/21 11:12 AM​	Drive	DeepStack: Alert cancelled (nothing found)
0​	6/17/21 11:12 AM​	App	DeepStack has been restarted
3​	6/17/21 11:32 AM​	Drive	MOTION
1​	6/17/21 11:32 AM​	Drive	DeepStack: Server error 100
0​	6/17/21 11:32 AM​	Drive	DeepStack: Alert cancelled (nothing found)
0​	6/17/21 11:32 AM​	App	DeepStack has been restarted
3​	6/17/21 11:46 AM​	Drive	MOTION
0​	6/17/21 11:47 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 11:48 AM​	Drive	MOTION
0​	6/17/21 11:49 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 11:51 AM​	Drive	MOTION
0​	6/17/21 11:51 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 11:52 AM​	Drive	MOTION
0​	6/17/21 11:53 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 11:58 AM​	Drive	MOTION
0​	6/17/21 11:59 AM​	Drive	DeepStack: Alert cancelled (nothing found)
3​	6/17/21 12:00 PM​	Drive	MOTION
1​	6/17/21 12:02 PM​	Drive	DeepStack: Server error 100
0​	6/17/21 12:02 PM​	Drive	DeepStack: Alert cancelled (nothing found)
0​	6/17/21 12:02 PM​	App	DeepStack has been restarted
10​	6/17/21 12:03 PM​	Server	::ffff:192.24.142.210: Connected
10​	6/17/21 12:03 PM​	xxxxxxxxxxx	::ffff:192.24.142.210: Login
10​	6/17/21 12:04 PM​	Server	::ffff:192.168.200.145: Connected
10​	6/17/21 12:04 PM​	xxxxxxxxxxxx	::ffff:192.168.200.145: Login
10​	6/17/21 12:05 PM​	xxxxxxxxxxxx	::ffff:192.168.200.145: Logout, 1:16
3​	6/17/21 12:05 PM​	Drive	MOTION
3​	6/17/21 12:05 PM​	FrontPorch	MOTION
10​	6/17/21 12:05 PM​	xxxxxxxxxx	::ffff:192.168.200.145: Logout, 1:33
1​	6/17/21 12:05 PM​	FrontPorch	DeepStack: Server error 100
0​	6/17/21 12:05 PM​	FrontPorch	DeepStack: Alert cancelled (nothing found)
1​	6/17/21 12:05 PM​	Drive	DeepStack: Server error 100
0​	6/17/21 12:05 PM​	Drive	DeepStack: Alert cancelled (nothing found)
0​	6/17/21 12:05 PM​	App	DeepStack has been restarted
3​	6/17/21 12:06 PM​	Drive	MOTION
0​	6/17/21 12:06 PM​	Drive	DeepStack: People:86% [730,240 874,489] 5411ms
0​	6/17/21 12:06 PM​	Drive	DeepStack: person:80% [755,253 851,479] 5411ms
0​	6/17/21 12:06 PM​	Drive	SMS: xxxxxxxx@mms.att.net with 1 attachment/s
10​	6/17/21 12:06 PM​	Server	::: Connected
0​	6/17/21 12:06 PM​	Drive	Push: OK to sgooch’s iPhone
3​	6/17/21 12:07 PM​	Shed	MOTION
0​	6/17/21 12:07 PM​	Shed	DeepStack: person:75% [282,276 447,575] 4423ms
0​	6/17/21 12:07 PM​	Shed	DeepStack: unknown:0% [326,274 374,336] 4423ms
0​	6/17/21 12:07 PM​	Shed	SMS: xxxxxxx@mms.att.net with 1 attachment/s
10​	6/17/21 12:07 PM​	Server	::: Connected
0​	6/17/21 12:07 PM​	Shed	Push: OK to sgooch’s iPhone
1​	6/17/21 12:07 PM​	Porch	PTZ: 12002: /PSIA/PTZ/channels/1/presets/4/goto
3​	6/17/21 12:08 PM​	Drive	MOTION
0​	6/17/21 12:08 PM​	Drive	DeepStack: person:53% [1942,425 2048,803] 5058ms
0​	6/17/21 12:08 PM​	Drive	SMS: xxxxxx@mms.att.net with 1 attachment/s
10​	6/17/21 12:08 PM​	Server	::: Connected
0​	6/17/21 12:08 PM​	Drive	Push: OK to sgooch’s iPhone
1​	6/17/21 12:09 PM​	Porch	PTZ: 12002: /PSIA/PTZ/channels/1/presets/4/goto
3​	6/17/21 12:10 PM​	FrontPorch	MOTION
10​	6/17/21 12:10 PM​	Server	::ffff:192.168.200.145: Connected
10​	6/17/21 12:10 PM​	xxxxxxxx	::ffff:192.168.200.145: Login
0​	6/17/21 12:10 PM​	FrontPorch	DeepStack: People:90% [1267,316 1653,950] 4181ms
0​	6/17/21 12:10 PM​	FrontPorch	DeepStack: person:88% [1288,329 1634,981] 4181ms
0​	6/17/21 12:10 PM​	FrontPorch	SMS: xxxxxxx@mms.att.net with 1 attachment/s
10​	6/17/21 12:10 PM​	Server	::: Connected
0​	6/17/21 12:10 PM​	FrontPorch	Push: OK to sgooch’s iPhone
3​	6/17/21 12:13 PM​	Upper_Basement	MOTION
0​	6/17/21 12:13 PM​	Upper_Basement	DeepStack: People:92% [462,366 1046,1522] 4412ms
0​	6/17/21 12:13 PM​	Upper_Basement	DeepStack: person:89% [454,405 1047,1502] 4412ms
0​	6/17/21 12:13 PM​	Upper_Basement	DeepStack: Steve24:68% [697,409 805,543] 4412ms
10​	6/17/21 12:13 PM​	Server	::: Connected
0​	6/17/21 12:13 PM​	Upper_Basement	Push: OK to sgooch’s iPhone
0​	6/17/21 12:13 PM​	Upper_Basement	SMS: xxxxxxx@mms.att.net with 1 attachment/s
10​	6/17/21 12:13 PM​	xxxxxxxxxx	::ffff:192.168.200.145: Logout, 2:45
3​	6/17/21 12:16 PM​	Drive	MOTION
0​	6/17/21 12:17 PM​	Drive	DeepStack: Alert cancelled (nothing found)

 

[tward392]

Ken knows and thinks he found the Error when closing clips, so I would assume the next release will fix that.

For some reason the DeepStack: Server error 100 stop for me, last one I saw was 10-Jun-2021 02:34:59 AM....

That would have been between 5.4.7.10 and 5.4.7.11....

 

[kc8tmv]

For some reason the DeepStack: Server error 100 stop for me, last one I saw was 10-Jun-2021 02:34:59 AM....

Are you using facial recognition and / or the custom "Dark" model?