[semi] automatic registration [was hugin beta release]

[JD Smith]

On Tue, 2003-10-14 at 06:08, Pablo d'Angelo wrote:
> PS: cc'ed to the ptx list. maybe somebody with image/signal is lurking
> around there. There are at least to astro physicists who probably
> learned a lot of math at the university :)

Alright, you've flushed me out.  In large format astronomical imaging we
have a very similar set of problems: detect "objects" in overlapping
images, mosaic the images using unknown rotation/shift, and (typically)
known shear/barrel and other distortions.  We have it easier and harder:
easier because our objects have a high degree of regularity -- often an
image contains hundreds of "point sources" with approximately the same
spatial distribution, scaled to different intensities, but harder
because we also have very strict requirements on "photometry", which is
to say all the photons must be accounted for, and none can be lost or
artificially inserted during the image registration/combination.  On the
plus side, our instrumental distortions are typically very well measured
and included in the meta-data (header material) with the images
themselves, and offsets among images are approximately known (how far
did you move the telescope/spacecraft?).  However, we also demand an
unchanging or at least smoothly changing "point spread function", which
many algorithms which otherwise produce beautiful (to the eye) images
fail to deliver.  All this has led to non-interpolative algorithms like
"Drizzle" (try google), or analogous Fourier methods.

These methods in toto are probably not very useful to you, given that
you'd like to retain as much PanoTools functionality as you can.  In
that case, your best bet is to perform robust object detection, remove
as many clear outliers as possible to begin with, then let PTOptimizer
do its thing, perhaps in a couple of rounds with outlier pruning,
remembering of course you can't simultaneously optimize n variables
without at least n points (and with exactly n, there is one and only one
solution (likely the null), which may not approach reality).  Also
remember that most optimizers like to work with many noisy points vs.
several pristine points and a couple of completely bad ones.  To
quantify this "badness", you'd need to fold estimates of the uncertainty
in a given point match in (i.e. how sure is the edge/object detector of
this particular object), which sets the scale for both the chi^2
computation, and for subsequent rounds of trimming.  This points cannot
be over-emphasized: without reasonable point-match error estimates, you
really have no good means by which to exclude or include points on their
own, and must resort to correlation or complete optimization to make the
cuts, which, though not wrong, will be slow.

I can imagine a pano of a set of buildings with clear window corners
creating a lovely, finite and small set of points for
correlation/optimization.  I can also imagine a pano of a copse of trees
at close range where every leaf is individually imaged, and might
produce several point matches each --- a mess.  For this type of image,
your best bet would be full image cross-correlation, but this presumes
good knowledge of the image distortions, which you don't necessarily
have (and is slow for large images). An intermediate technique is to
cross-correlate the object lists, assigning each point some presumed
error "shape" defining the physical extent of each object's probability
(probably a Gaussian, ideally with a size dictated by its error
estimate).  You can then perform a parabolic fit to the 3-D correlation
function and get sub-pixel offset/rotation accuracy.  This method
requires you to get "close" to begin with, since otherwise the
correlation function has lots of hills and valleys.  Also, the more you
move in this direction, the more you re-invent the PT optimizer.  If you
change the basic character of the input point matches (i.e. from within
one-pixel of a match, to possibly completely wrong), you'll probably
find PT's optimizer somewhat brittle --- unable to accommodate the bogus
inputs.

In short, I'd say this is a very tough problem.  I'd bet that the
automatic stitching packages out there don't optimize nearly as many
variables as PT does, probably either pre-specifying all image
distortions, or just ignoring them (for rectilinears).  And the ones
that do work well probably use a variety of different image registration
techniques, switching among them as necessary.  So whichever direction
you take, be sure to leave the "select points by hand" method in --- no
one has yet designed an image classifier/decomposer better than the
human brain.

JD