idea / slightly OT / feature request: 3D tracking with multiple camera's

[vertigo]

Not too sure where to put this, so I'll post it here and maybe mods can move it if its not appropriate.

Ive been doing some photogrametry experiments with drones lately, and Im pretty amazed how accurately you can reconstruct 3D meshes from 2D aerial photographs. If you're interested in seeing an example I recently made:


That got me thinking, why dont we use multiple camera's to reduce or eliminate false positives from shadows, light, well, anything that has no 3D volume ? In theory, with multiple camera's (or stereo cameras) you should be able to extract enough 3D information to eliminate 2D false positives. It wont help for 3D objects, like tree branches that actually move in the wind, but it would solve the vast majority of problems with motion triggering. Not only would changing lights or shadows be solved, you could do many new things, like trigger for objects with a certain volumetric size, regardless if they are close or far from the camera. You could differentiate insects from birds, from cats, from dogs from people.

Its probably not trivial to implement, Im sure its also compute intense (photogrametry certainly is), but I cant help but think it would be absolutely fantastic and open up so many possibilities.

 

[bp2008]

Good idea, but as slow as the video security industry moves, I doubt this is going to happen soon in a convenient package. If you want to eliminate false positives, I believe thermal cameras are really good at that. FLIR T4350BN is the cheapest I am aware of right now, at about $450, and of course it is super low resolution, but I think in the next 5 years there will be a lot of better options.

 

[nayr]

you didnt accomplish that in realtime, and thats the trick.. how long did your computer churn away making that video? I suspect alot longer than 1m18s

then you have technology patents that really hurt this industry.. as Security camera companies are not the ones doing the research and development on awesome new image processing techniques.. going to have to wait for all that stuff to expire before anyone would even consider adopting it.

 

[vertigo]

What I did is far from real time, but its also a composite of many 100's of different perspectives of high resolution photo's. Most of the computing time is spent figuring out exactly from what perspective and place each photo was shot and which parts of it match any of all the other photo's. If you have static cams, you'd need to figure that out only once, and create a "3D background". After that, processing should be relatively light.

Some illustrations to prove that; there is already experimental software that does photogrametry in real time on a cell phone:

And thats arguably harder than with multiple static feeds. Also consider the xbox kinect, thats real time 3D analysis and it requires only a relatively minor fraction of the processing power of what is now a fairly old gaming console.

 

[nayr]

I suspect 3d cameras will be the next big thing after 4k has taken a foothold, stereo optics is all you need.. not even multiple cameras.. a bit of distance information would be really good for advanced processing techniques I agree..

but thats not the modus operandi for this industry, for drones and phones and all other video markets, both consumer and professional, move at a much higher pace than security industry.. i mean shit were still trying to convince people to stop installing analouge cameras.

 

[vertigo]

Well, at least there are -or where until very recently- arguments to made for analog video. As an aside, I certainly hope that segment doesnt die too soon, as FPV flyers, we still depend on security camera CCD chips since I have yet to see a good CMOS product for our application. Wireless digital video transmission also is simply not good enough yet for us either (bandwidth requirements, latency, handling of poor rf links, ..).

As for 3D, stereo has very limited appeal, not much info you can extract from it. It may help in some cases, but I doubt its worth the hassle. If you are going to use special camera's, then you probably want a "time of flight" camera, as is used in automotive, machine vision, game controllers etc. They usually have limited resolution for now, but even QVGA should be good enough to estimate object sizes, and you could still record the video in high res. A ToF camera also takes away the high processing burden of computing 3D info from 2D imagery.

But the thing here is, we dont need anything new from the industry; this could be done today with existing cams, all it requires is some clever software.

 

[nayr]

But the thing here is, we dont need anything new from the industry; this could be done today with existing cams, all it requires is some clever software.

and licenses to the numerous patents involved with this stuff.. which will mean some really expensive clever software for limited market.

stereo can roughly give you rough distance and depth, now 3d output would not be very useful I completely agree, but software can do some clever stuff with that data.. as the kinect has shown.. and there are other tricks that can be done with 2 sensors in the same device, such as pairing a fisheye with a PTZ so you can get some actual tracking that's not distracted by shinny objects or missing things out of frame.

The thing is there are much more efficent ways to detect someone's presence in a dynamic outdoor environment than processing video, I wish it wasent so bad too but I dont care that much.. hdd's are cheap, I record always and use real sensors to tell me when activity occurs.. its 100% reliable, never once have I been told my doorbell rang or my mailbox opened, when infact it had not.

Ive delved into FPV flying, you guys are kinda stuck.. its going to take some impressive technology, short of some sort of scifi hyperspace communications the physics are against you to do realtime HD video wirelessly so we can really put these new Oculus headsets to some badass use..

while near-realtime is obviously a not good enough for the pilot, I see potential for use with autopilot paired with digital video system on a nice gimbal tied to a vr headset.. would need some clever UI software to overlay and command the autopilot or you'd still need someone else flying it.. unfortunately the FPV guys are a small minority of people attaching cameras to aerial platforms so I expect you guys to remain renegades repurposing equipment from other industries for quite a while longer.. it seems kinda a miracle you got this far with it.. but it has definitely hit its limits.

 

[vertigo]

and licenses to the numerous patents involved with this stuff.. which will mean some really expensive clever software for limited market.

Thats unlikely. Photogrammetry is not new, the concept has been around for decades and most of the software is, or originates as academic / opensource. See VisualFSM.
Im not aware of any showstopping patents, if any patents at all.

Ive delved into FPV flying, you guys are kinda stuck.. its going to take some impressive technology, short of some sort of scifi hyperspace communications the physics are against you to do realtime HD video wirelessly so we can really put these new Oculus headsets to some badass use..

"zero latency" digital links do exist, even though it comes at a price:
http://www.getfpv.com/connex-hd-video-downlink.html
And zero is really 28ms best case, but thats good enough for most. The real problem is if the connection becomes less than perfect, digital doesnt cope with that very well at all. An analog signal just adds noise, and the brain has very advanced noise filters that cancel that out. Also reconnecting after a brief signal loss is instantaneous with analog and may take much too long on a digital link.

The other problem really is the camera's. I have yet to try a cmos camera that copes with extreme light changes, like from facing the sun to the ground in a shadowy forest, good en fast enough, and without inducing extra latency like some sony chips do for WDR. Im sure the sensors must exist by now, but not the small (and preferably cheap) cams we need. Sony CCD cams are still the gold standard in our community.

while near-realtime is obviously a not good enough for the pilot, I see potential for use with autopilot paired with digital video system on a nice gimbal tied to a vr headset..

All that exists. A DJI phantom does it that way, as does my parrot bebop which I used for the castle video. I can even connect an ocolus rift to it. And near real time is good enough for some application, like drone photography or just to cruise around, these typical drones mostly fly themselves anyway. I just like mini drone racing and freestyling too, and for that, >200ms really is not acceptable. A video to illustrate:

(not me, Im no where near that good)

would need some clever UI software to overlay and command the autopilot or you'd still need someone else flying it.. unfortunately the FPV guys are a small minority of people attaching cameras to aerial platforms so I expect you guys to remain renegades repurposing equipment from other industries for quite a while longer.

Dont be too sure. The drone market is exploding. FPV racing may remain a niche though

 

[nayr]

I moonlighted a UAV shop online for a good while (uaviation.net) and it was making good money selling antennas, radios and other electronic gadgets I made, and I was a big contributor to the OpenPilot project.. but I am not a business man and I did it just so I could afford to stay in the hobby.. its not cheap and when I had a kid it was just too much to juggle so I shelved my planes and been waiting for him to get older and drag me back into the hobby.. but when I left the scene it was getting overan by assholes with too much money and no brains, and the business side had a gold rush of unscrupulous vendors and I dont think it'll ever go back to the good old days when I grew up at an airfield where everyone took responsibility and safety as more important than having fun.

I was more into fixed wings, ive got a 6ft foam wing behind me on my wall right now.. Ive had a few real cheap multirotors small enough to fly around my back yard safely, but didnt get into it all that much.. mostly cost, but partially due to limited flight times and distance capabilities.. I got back into the hobby for autopilot/software control.. FPV came along and was fucking amazing, the shit I dreamed of as a kid.. If I get back into the hobby its going to focus on this, so me and my son can dogfight some wings at 80mph

 

[vertigo]

IFPV came along and was fucking amazing, the shit I dreamed of as a kid..

Yep. Here's a clip for anyone that grew up with star wars dreaming of dog fighting like Luke Skywalker: