Use image stacking instead of gluing slices together

[clonejo]

Currently, trainbot pieces slices of the moved train together. This works well when within the used crop rectangle the view is unobstructed, and the exposure/lighting is similar in the X axis.

Unfortunately, i have to deal with some obstruction from foliage:

One can kinda reduce this by narrowing the crop rectangle, but i am already quite limited.

But since we have each piece of the train exposed at least twice (with the current movement detection code, even three times), we can stack them together. Here is an example using a better camera, manual stacking of three frames in GIMP, and then enfuse for stacking:

compared to a single frame:

We could even pick out the parts that don't change between frames, and create a non-rectangular mask for picking only the unobstructed parts.

Possible implementation: We pretty much have all the pieces already, we'll just have to save each frame placed into a separate otherwise transparent image file, and then run enfuse over all the files.

[clonejo]

This also means we get more exposure for each bit of train, and thus reduce noise.

[jo-m]

Indeed, we actually have enough raw images to get rid of obstructions in your specific case.
IMO this is however not true in the general case.

Anyways, feel free to experiment with this and to submit a PR!
I will not implement it.

[jo-m]

This also means we get more exposure for each bit of train, and thus reduce noise.

Noise yes, blur no.

[natevw]

Hmm, I also have problems with obstructions but rather than a whole bunch of extra DSP my thinking was maybe just somehow use offset boxes that get lined back up based on the appropriate time delay from train speed:

I wonder if that would be a bit easier to work into the existing logic and/or a little bit more generalizable to various situations?

[jo-m]

It should be easy enough to adapt the stitching logic so that it can be chosen from which image area the patches are taken for stitching.

[Rahix]

@clonejo, I see you implemented masking in commit 17f933b. Would you be fine with me preparing that for upstreaming? Or even submit the change yourself? :)

[clonejo]

@Rahix Not sure when i will get to it, feel free to pick it up :)

Some learnings i had with my implementation:

• if you create a precise mask, and then knock the camera slightly, you have to create a new mask
• while stacking works, the picture is sharper when you don't stack
• allowing a vertical rectangle through the mask works best. if you use more irregular shapes, you will get some artifacts when the train's movement/velocity has not been estimated perfectly
• i keep forgetting that the white parts of the mask will be kept and not the transparent parts. (i have been creating the mask in gimp by drawing into a new layer above a still frame taken from trainbot-confighelper)

Even with these caveats, it allows me to get much nicer pictures.

[Rahix]

Yeah, pretty much matches my experience as well. I have a catenary post dead center in my field of view, so I have two vertical slices that now get interleaved to reconstruct the train. Velocity estimation is surprisingly accurate, my main problem is different lighting conditions on the two slices leading to a banding effect on the image.

(Ignore the bad mask....)

One more change I made was placing the unmasked frames behind the masked stack. This ensures that worst case something will be shown instead of black blobs being all over the image where no frame's mask left pixels. This is especially a problem with fast-moving trains.

One more neat trick is using the mask to feather the edges of each frame. This gave me much nicer blending results for a lot of cases.

---

I'll cook up a PR based on your changes with my additions and some documentation on how to use it.

[clonejo]

One more change I made was placing the unmasked frames behind the masked stack. This ensures that worst case something will be shown instead of black blobs being all over the image where no frame's mask left pixels. This is especially a problem with fast-moving trains.

nice!

[jo-m]

For the banding issue, another thing worth investigating might be adjusting the camera autoexposure intertia?

[Rahix]

camera autoexposure intertia

Yeah, that's also on my list. I do see exposure jumping a lot in certain images (notice the wavy background brightness):

But the banding I was talkting about above is mainly caused by uneven brighness across the lens aperture in my setup. I was think of adding some sort of "correction" overlay (additive? multiplicative?) which evens this out...