[ptx] autopano: first attempt at semi-automatic panorama stitching

[alexandre jenny]

Hi,

What a beautiful start of some autopano software. Good work sebastian.
Let's step into some remarks :

- About the sift feature detector. I have a matlab working version of the
detector and a not working vigra version (I didn't get time until now to
debug it). Let's face it, it's slow (the matlab version), around 5 minutes
for detecting all the features on a 1 MPixels picture. So when applying that
to every picture of a panorama, you could easely reach one hour in duration.
But anyway. After the detection, it seems that the modified ramsac is quite
real-time and also the fitting process. It's really the detector which takes
time.
I think, some stage of the feature detector could be simplified or skipped.
The first one would to remove the scale invariance stage. We seldom stitch
pictures with different focal length. So this could be one idea (It will
speed up things around 3x to 6x).
>From JD : "Lucas-Kanade optimization on the full images for the final
super-accurate registration". That's definitely the good way to use sift.
Having a bunch of points generated on a low resolution picture with sift.
Get the matrices on low resolution and fine tune with that.
>From Ed : "Anyway, the key thing from Matthew's paper seemed to be to try
and find islands of interest points and correlate those rather than the
whole image ?". In don't believe that's the point. In his version of the
sift detector, he already added some subpixel accuracy to the points. So
after feature detection, you have 2 sets of controls points for your 2
pictures. After the modified bin tree research, you can match the most
nearest features and that gives you directly with quite no calculation the
matrice. After some stats check, you are able to say which points on one set
fits the one on the other set. It's used to refine the matrice. There's no
more pass for a correlation calculation as the sift points are already
subpixel located. That's the real improvement of the approach : no more
pixel calculation after  feature detection (For more pictures (let's says
N), the modified bin tree research (it's called best bin first), will also
provide you the nearest features of every couple of sets. And stats will
says you if it matches or not). The whole process of sorting, matching and
calculating the matrices is quick (My estimation, less than 10 s).

- this paper http://citeseer.nj.nec.com/mikolajczyk02affine.html is cool
also. Mikolajczyk and C.Schmid are working with Lowe on features detectors.
They wrote "A performance evaluation of local descriptors" which compares
every feature detectors and the result was that for accuracy and speed, sift
is the best in both area (Moreover, they are much stable). (You can found
the paper here : http://www.inrialpes.fr/movi/people/Mikolajczyk/ ). (the
funny part is that this guy, and Cordelia Smid is working just 20 km from my
location ...)

- Anyway, the story about releasing my code and hugin integration (And
patent issues). I taken to the guy in charge with the patent for the
canadian university for which Dr.Lowe works. He says it's patented. But he
didn't provide any paper about it. I asked him about the fact I've already
rewrote the algorithm. Here is the exact quote :

>"Just a last question which is related to the sift detector. For testing
purpose, I've rewrote myself this detector following Dr Lowe's papers. Do I
need a license or agreement ? What can I do with the code, could it be made
available in an open source software ?
>The patent covers the process used in SIFT, not the code itse;f.  Even if
the code was rewritten, it would still be covered by the patent.  You could
use the code you wrote to validate the scientific principles disclosed in
Dr. Lowe's publications, but if you wanted to use Dr. Lowe's work for your
benefit, commercial or otherwise, you would need a license.  

So what ? I don't what to think. I really don't want to get the same problem
as Dr.Hersch in the past.

- The last remark : I wrote sift detection to provide what I called a
Panotools Preprocessor. That's a soft which is able to precreate a panotools
script (or ptgui, or ptassembler) with already placed control points. So you
just have to check the optimization stage (which could already be
preoptimized) and then control the blending stage.
My dream is that the hugin project provides the background code for allowing
people to work with modules : we could imaging many blending algorithm
(ptsticher approach, band approach like in matthew's thesis, or anything
else). Let's make the interface clean to allow that. The key point is to
keep the core of the subject : having a commun way of description : the
script which stores controls points (if needed), pictures names, matrices,
etc.
Then any algorithm which makes something on the pictures (extract points,
correlates, etc) should just adjust the description in this script file.
Working this way will allow us to plug things together well.

Bye,
  Alexandre 

> -----Message d'origine-----
> De : ptx-bounces at email-lists.org 
> [mailto:ptx-bounces at email-lists.org] De la part de Ed Wildgoose
> Envoyé : lundi 29 décembre 2003 01:58
> À : PTX mailing list
> Objet : Re: [ptx] autopano: first attempt at semi-automatic 
> panorama stitching
> 
> 
> 
> >Yep, it would be interesting, but I haven't found that 
> patent yet, it 
> >doesn't seem to be in the german patent database and I just made a 5 
> >minute search with the US patent database, but didn't find it.
> >
> >  
> >
> Are you sure that it's patented?  I see a lot of references 
> all over the 
> place to the ideas used, so seems hard to see that it can be patented?
> 
> Looking at the bottom of the web page it says something about "if you 
> want to licence it", which I took to mean the source code rather than 
> the algorithm?  ie try and find a commercial buyer for the finished 
> product. 
>