Automated Motion Tracking in Videos Like Gimbal and GoFast

[Mick West]

There's very little, if any, parallax in the clouds. If there is some then you should be able to demonstrate it - and then demonstrate that it changes.

You seem to be applying some subjective metric. You need to be able to measure it - or at least point it out so other people can see it.

I was experimenting with OpenCV today, and got this velocity field overlay video:

I think others have done similar things before, but I can't immediately find them. Unfortunately a lot of noise, and it's not good at tracking in WHT mode. But it's interesting, and might be worth more work.

 

[Leonardo Cuellar]

But it's interesting, and might be worth more work

Fantastic! Mick, how can I transfer those Little tracks into numerical values? What physical size do they express? Angular velocity? What scale should i use to measure them?

 

[Mick West]

This (OpenCV) is something of a new subject for me. I've generally done motion tracking and data extraction with Adobe After Effects, but that's focussed on tracking an object, and it's fiddly to use for measuring the speed of the background. So I thought I'd experiment with the OpenCV library - which lets your program more exactly what you'd like to look at, but is a bit of a pain to use.

The lines are motion vectors, so they represent the direction and distance a group of pixels moves over a period of time (three frames in this example). OpenVC is calculating a pre-pixel "Optical Flow" map. I started with this tutorial
https://learnopencv.com/optical-flow-in-opencv/
and code
https://github.com/spmallick/learnopencv/tree/master/Optical-Flow-in-OpenCV
(and the usual StackOverflow, etc. code results)

Getting the code to run was no easy task on ARM MacOS, but eventually, I got something working using Python. I modified the example to average the vectors for squares of pixels (hence the spacing) - but it's still noisy as there's noise and a lack of fine structure in the original.

I don't think this is the best approach to using OpenCV, but I have very little experience with it.

Going back to Adobe After effects - If you simply drop a track point on the video, it will track it across the screen. I can then reset to another point and continue:

The data from the track can then be compiled into a spreadsheet. I remove the jumps back and compute the per-frame delta ("delta" = "difference" i.e. how much it moves each frame). This gives a per-frame speed and a per-frame angle. Very noisy data. But using a 20-frame moving average shows something of what is going on:

The jump at the end from the loss of lock distorts the true trendline, I should do it again with the video fully stabilized on the target.

The blue is the raw data (with the resets removed). The red is 20-frame moving average. The yellow is a polynomial trendline - which I would not read too much into for the angle as it's influenced a lot more by the bumps, especially at the end. The data extraction was also semi-manual, and poor choice of regions might have led to misleading results.

 

[sitarzan]

Apologies for the redundant info on OpenCV. I was in the process of drafting the following when you posted your comment @MickWest.

cvCalcOpticalFlowBM
Calculates the optical flow for two images by using the block matching method

void cvCalcOpticalFlowBM(
           const CvArr* prev,
           const CvArr* curr,
           CvSize blockSize,
           CvSize shiftSize,
           CvSize max range,

           int usePrevious,
           CvArr* velx,
           CvArr* vely );

...

The function calculates the optical flow for overlapped blocks blockSize.width×blockSize.height pixels each, thus the velocity fields are smaller than the original images... — OpenCV Reference Manual - pg 373 - Motion Analysis and Object Tracking

If I understand OpenCV's documentation correctly, it sounds like the above cvCalcOpticalFlowBM function makes it possible to use Particle Image Velocimetry techniques; but all in software (i.e., no "laser light sheets" required)...

Particle Image Velocimetry (PIV) is a non-intrusive state-of-the-art technique for flow measurements...The PIV technique is based on image recording of the illuminated flow field using seeding particles ... The light scattered by the particles is recorded on a sequence of image frames ... by applying a spatial cross-correlation function as implemented by the OpenPIV resulting with a two dimensional two component velocity field ... The flow is illuminated twice by means of a laser light sheet forming a plane where the camera is focused on...

...

...to find out the velocity field of the moving fluid...from images of small particles, called tracers. The basic principle is to use two images of
the same particles with a small time delay between them. For that purpose, typically two laser shots are created and two images are taken... — OpenPIV Documentation — ppg 3, 8

...I got something working using Python...

I believe DeepFlow (demoed in this Velocity from animated sequence / Optical Flow forum post) is Python too...

...
AFAIK people generate vectors in external apps (usually After Effects or Nuke with plugins like Twixor or go fancy with openCV libs or hacked ffmpeg builds), and then use that image sequence to make a velocity field. I assume that's how the amazing video below was done.

(edit)

Ah, he says in another video he uses deep flow: https://thoth.inrialpes.fr/src/deepflow/

(/edit)

...


Source: https://youtu.be/N8Sed-c1sJI


...

 

[Edward Current]

I overlaid black lines where I have consistently found "new clouds," and then did a very rough eyeballing calculation of the area under the curve for each segment:

This supports that the curve may be a bit high at the end. (About 1/4 of a field passes in the last segment.)

This is the position curve of my "camera goal," which is the animated line that I use to make sure the camera panning is on track. It's the antiderivative of the speed curve.

 

[Leonardo Cuellar]

I think we are going a little further. How many rad/sec does a given crest of clouds move during the movie? Is it possible to get it?

 

[Mick West]

I think we are going a little further. How many rad/sec does a given crest of clouds move during the movie? Is it possible to get it?

That's what the "speed" graph is showing, except in pixels, going approximately from 7 to 1 pixels per frame, or x30 to 210 and 30 pixels per second. The file used is 480 pixels high so multiply by 0.35/480 to get 0.15 to 0.02 degrees per second. All VERY rough.

 

[Mendel]

get 0.15 to 0.02 degrees per second

that matches the rates I computed in https://www.metabunk.org/threads/gimbal-blender-simulation-with-clouds.12209/page-3#post-266506

 

[sitarzan]

...so multiply by 0.35/480...

I'm clear on x30 being the frame rate of the video.

I'm a little out of the loop on where 0.35 might come from though.

I'm guessing that would be this number...

...a 0.35⁰ FOV...

Or something else? TIA.

 

[Mick West]

I'm a little out of the loop on where 0.35 might come from though.

I'm guessing that would be this number...

Yes, the display in the 2x NAR mode is 0.35° by 0.35°. Since the video is 480 pixels high, then we can divide by 480 and then multiply by 0.35 to get degrees per pixel.

 

[logicbear]

I wrote some OpenCV code to try to reproduce these results. It should be possible to tweak it further but I think the results so far are interesting enough to share. The following is a graph of the cloud motion angles relative to the bank angle:

It's sensitive enough to detect all of the big bumps in the cloud motion during sudden pod rotations. With some more work it should be possible to make the results near the end a bit less noisy, but it's already good enough to show some correlation with the bank angle there.

I'm comparing the current frame with one 3 frames later as the larger displacements make it a bit easier to distinguish the signal from the noise. I apply the same transformation to the white hot region as described here , but only for selecting a sparse set of corners to track. I find the displacement of those corners with Lucas-Kanade, currently with the following parameters:

cv::cuda::createGoodFeaturesToTrackDetector(CV_8UC1, 0, 0.004, 3, 17, false);
cv::cuda::SparsePyrLKOpticalFlow::create({ 25, 25 }, 5, 30);

Finally I do a RANSAC (200 iterations, sample size of 5, max motion error of +/- 5 degrees) to reject outliers in the motion angles detected previously. The green lines in the image show the motion vectors that are part of the consensus set i.e the ones that are considered to be part of the cloud motion. The blue line is the final average cloud motion angle visualized.

 

[Mick West]

wrote some OpenCV code to try to reproduce these results. It should be possible to tweak it further but I think the results so far are interesting enough to share. The following is a graph of the cloud motion angles relative to the bank angle:

Nice, seems very close to what I got here:
https://www.metabunk.org/threads/gi...using-clouds-as-the-horizon.12552/post-275771
Overlaid:

Some divergence at the end, but it's very noisy there. The smoother parts are essentially the same.

 

[logicbear]

I've made some further refinements to my cloud motion estimates, applying some things from here and going further.

For GoFast I just used a minimum magnitude threshold to help differentiate the real motion from the static elements in the display, but here the motion slows down much more near the end, so I added a mask to filter out both static elements (with manually specified rectangles) and everything above the line of the clouds (which I've set to the calculated horizon line offset from the center by a manually specified distance). The optical flow matches pixels in a rectangular window, so to remove noise near the static elements I also eroded the mask with a rectangular structuring element of size 19. It was then enough to use a minimum magnitude of just 1 to avoid filtering out real motion vectors. The mask looks like this for frame 0:

The optical flow is estimated between the current frame and current frame + frame diff. If the frame diff is too small then small displacements can make the result too noisy to see the real trendline. If it is too large then some brief oscillations in the results (that are real, not just noise) can get smoothed over, or the number of inliers can decrease as more of the image goes out of view, causing noise. At the start the clouds are moving faster, so the frame diff can be set lower, while at the end they're moving slower so setting it a bit higher is preferred. So I used a frame diff that varies throughout the video. The speed of the cloud motion is always the average magnitude of the inliers divided by the frame diff.

There are also new technologies, like OpenCV

Not all of OpenCV is new technology. In particular the methods used above for the corner detection and optical flow weren't state of the art even a decade ago. I've considered trying some of the latest and greatest optical flow algorithms from here but so far it seemed more convenient for me to just use OpenCV. On the other hand last year a bunch of new outlier rejection methods were added. In this evaluation a method called MAGSAC was recommended, and indeed it did seem to be a bit less noisy than others I've tried on Gimbal. I used a function to estimate a homography from the optical flow with MAGSAC. The average angle and magnitude are now based on the resulting inliers from that. In principle the clouds further away could be moving slower than the closer ones, so that's why a homography needs to be estimated instead of an affine transform, but more work is needed to detect these differences if they are there.

For a frame diff varying from 4 to 7 I get the following:

The image includes the values for all the parameters that were not set to their default values. I've lowered the GFTT block size and quality level to produce more corners to match and let MAGSAC filter out the errors. It's possible to lower the MAGSAC reprojection threshold a bit further, with similar results just fewer inliers.

I've attached the data for this and here's a video of it as well. The arrow is pointing in the measured direction of the clouds and its length is the cloud speed * 20.

For comparison here's the data for the frame diff varying from 2 to 4:

And the data for the frame diff varying from 6 to 10:

should do it again with the video fully stabilized on the target

I fixed a bug that was needlessly adding some large spikes to my center of mass estimate here. It's still not perfect, in particular I know the center of mass is still off when the object goes behind the trackbars and when it's riddled with interlacing effects, but the results of stabilizing the video on that and the calculated horizon angle are still interesting.

Here's the data and a video of it:

And here's the data with frame diff varying from 6 to 10:

Without this stabilization there is a bump in the cloud speed between 29 and 32 seconds that gets straightened out here. At 32.5 seconds errors in the center of mass estimate are to be expected. Previously at around 33 seconds the cloud speed almost gets to zero and the clouds even turn back in the other direction for a few frames as the pod centers the object again, but here that mostly gets sorted out and the speed only goes down to about 0.8 pixels/frame. At 27 and 29 seconds there are oscillations in the speed here that are due to temporary rotations in the image that are probably an artifact of the pod. Maybe those might get straightened out if the image is stabilized on the measured angles/rotations instead of the calculated one.

 

[TheCholla]

Great work, and to me it seems these bumps in the cloud speed slightly precedes the step rotations in the vid. Also, in Sitrec, the close trajectory shows the most significant changes in acceleration during or just before the last three rotations (see pink areas in the acceleration graph below).

This end section looks quite bumpy for two planes, one chasing the other, at constant speed. There is of course the effect of banking but it's not apparent this is responsible for these bumps in acceleration. Not sure how the pod rotation can affect cloud speed, especially just before it rotates, because with image derotation clouds do not rotate.

Here is the end section at 0.3X speed to check cloud speed versus steps :

 

[logicbear]

it seems these bumps in the cloud speed slightly precedes the step rotations in the vid.

Keep in mind that if for example the frame diff is 10 and there is a sudden bump starting at a given frame, then you will already start to see that bump in the cloud speed/angle graph 10 frames earlier than that. Otherwise the bumps in the cloud speed (here stabilized with 4-7 frame diff) line up well with the keyframed/measured glare angles, bumps in the object's position, error angle steps.

This end section looks quite bumpy

I think the bumpiness at the end is probably a combination of the pod struggling to maintain lock during the large rotations needed around 0 degrees and remaining errors in my stabilization.

Not sure how the pod rotation can affect cloud speed, especially just before it rotates, because with image derotation clouds do not rotate.

The video is not fully derotated, it is stabilized on the measured object center of mass and the calculated horizon angle, not the measured cloud angle/rotation. The clouds do clearly rotate back and forth relative to the calculated horizon angle during these bumps. For example here I've color coded the angles for frame 859 and 868 from the bump at 29 sec (lower peak, upper peak).

 

[TheCholla]

I think the bumpiness at the end is probably a combination of the pod struggling to maintain lock during the large rotations needed around 0 degrees and remaining errors in my stabilization.

Those bumps in cloud speed seem to be due to the clouds stopping for a few frames when bumps happen just before the step rotations.

Here I go frame by frame for the bump at 27s:

That explains the dip in cloud speed just before the step rotations. In your cloud speed graph, the peaks right after the dips indicate that cloud motion gets a bit faster after these stops.

The same stops in cloud motion for a few frames happen before the other two rotations at 29s and 32s.

Maybe it's the pod. I don't know, mostly because of this:

 

[logicbear]

Those bumps in cloud speed seem to be due to the clouds stopping for a few frames when bumps happen just before the step rotations.

Based on some math I've done, the clouds really don't seem to be stopping or suddenly changing speed. There are a some known factors that create an appearance of them doing so:
- The image rotates around its center due to some combination of pod roll / derotation / banking, with sudden oscillating rotations near the bumps. This skews the average motion over the clouds.
- The object, along with the entire image, moves off center near the bumps, so if it moves in the opposite direction of the clouds then it appears to decrease the speed.

I tried to formalize these. It's not clear yet if there's a real difference between the motion of the clouds in the back vs the front, but it's probably small enough that we can model the measured motion of the clouds as one linear displacement V followed by a rotation R around the center C of the next image. I'm assuming it's rotating around the center of the image, not around the center of mass of the object, as the rotations are due to the pod and the jet, not the object. So going from a point P1 in the current image to a point P2 in the next image is given by:
P2 = R * (P1 + V - C) + C
This can be rewritten as:
P2 = R * P1 + (R * (V- C) + C) = R * P1 + T
To find R,T I use a function to estimate a partial affine transform that looks like:
P2 = s * R' * P1 + T'
I've seen it suggested that I can just ignore the scale, use R = R', ignore T' and recalculate the translation as T = P2 - R * P1 . I'm not entirely sure if that's the best solution, but it seems to work here, particularly since abs(s-1.0) < 0.03. So we have:
R * (V- C) + C = P2 - R * P1
V = Rinv * (P2 - C) - P1 + C
Finally, if the object's center of mass moves from point O1 to O2 then V', the real motion of the clouds, is:
V' = V - (O2 - O1)

The average angle and speed from just the inliers of the partial affine transform estimation (with 2-4 frame diff) is similar to the results from the homography estimation, but now the rotation of the clouds (in degrees) can also be shown. (data)

When the angle and speed is adjusted with the math from above the bumps in the speed and angle at 24, 27, 29 seconds almost entirely disappear. Around 33 seconds the angles are less bumpy than before but some remaining errors due to interlacing and the object going behind the trackbar are expected there. (data)

It's not clear what causes the apparent discontinuity when the frame diff changes from 2 to 3 at 11 seconds here, but as before it gets straightened out when higher frame diff is used. Here are the adjusted results for frame diff 4-7. (data unadjusted, data adjusted) Notice how we get the same general trend for the speed as the horizon stabilized versions before, but this time entirely based on the measured rotations, without using the calculated horizon angle.

Here are the adjusted results for frame diff 6-10. (data unadjusted, data adjusted)

Some further work that could refine this:
- There's a question of where the center of the image is exactly. I used (213, 210) as that's what the object generally tends to oscillate around in my graph here (the values shown are (x or y - 210) * 20), although the trend shifts a bit. Maybe that can be refined.
- Although the adjustment effectively does full stabilization already, it could still be better to run the optical flow on the stabilized version of the video as the motion doesn't slow down quite as much there.
- Maybe some algorithm could be used for deinterlacing the video ?
- The function for estimating the partial affine transform doesn't have a version that uses MAGSAC in OpenCV, so perhaps a better estimator could be used, particularly one that doesn't try to estimate a scale.
- I haven't quantified this yet but it seems like the whole image might be flickering in brightness. If so, either normalizing that brightness (not just between black hot / white hot) or just using a better optical flow method that is more invariant to illumination changes could perhaps help.

@Mick West What is the y axis for the cloud speed graph in Sitrec ? If I convert from pixels/frame maybe these results could be used to refine the path there ?

 

[Mendel]

Based on some math I've done, the clouds really don't seem to be stopping or suddenly changing speed. There are a some known factors that create an appearance of them doing so:
- The image rotates around its center due to some combination of pod roll / derotation / banking, with sudden oscillating rotations near the bumps. This skews the average motion over the clouds.
- The object, along with the entire image, moves off center near the bumps, so if it moves in the opposite direction of the clouds then it appears to decrease the speed.

So, basically, by assuming the jet flight and the cloud motion are not bumpy, you get a better cloud track. Does that mean you can graph the bumps of the pod camera now?

 

[logicbear]

So, basically, by assuming the jet flight and the cloud motion are not bumpy, you get a better cloud track. Does that mean you can graph the bumps of the pod camera now?

I don't think I've assumed the cloud motion isn't bumpy ? I've only assumed that the observed motion of cloud pixels is a combination of the real cloud motion, a rotation of the image around the center of the image (which I calculate from the measured rotation/translation of the clouds), and the target going off center, along with the whole image, by the amount that I measured. It could easily have turned out that accounting for those factors preserved the bumpiness of the results, but instead it straightened almost everything out, with remaining bumps only around a part where I know my measurements are still off. The measurements of the object's position and the cloud rotation/translation need to add up very precisely for that to happen.

I've already done various graphs of the bumps related to the camera, not sure what you mean ? For example the red lines above show sudden oscillations in the rotation of the image at times that coincide with all of the bumps at 1.5, 24, 27, 29, 33 seconds.

 

[Mendel]

I've already done various graphs of the bumps related to the camera, not sure what you mean ? For example the red lines above show sudden oscillations in the rotation of the image at times that coincide with all of the bumps at 1.5, 24, 27, 29, 33 seconds.

yes, but they're labeled "cloud rotation", and real clouds don't rotate?

if we think of clouds as unbumpable, then the bumps in the image must be pod bumps: lateral or rotational or both. (I think this is simply a presentation issue?)

 

[logicbear]

yes, but they're labeled "cloud rotation", and real clouds don't rotate?

if we think of clouds as unbumpable, then the bumps in the image must be pod bumps: lateral or rotational or both. (I think this is simply a presentation issue?)

You're right, there's an inconsistency in labeling there, sorry. In the first graph here all three of cloud angle, speed and rotation are a measure of how much the pixels belonging to the clouds are observed/directly measured to move from one frame to the next (current frame vs current frame + frame diff). Of course clouds don't actually rotate, and in the next 3 graphs the angle/speed are adjusted to more closely reflect the "real" motion of the clouds (and clouds don't move at an angle that depends on the plane banking either, so even that is a bit misleading), but the rotation component does not get adjusted, it's always just the angle from the partial affine transform that gets estimated between cloud points. Maybe someone can check the math and find a less confusing way to explain what I've found here.

 

[logicbear]

Here I posted the source code for my cloud motion tracking algorithm. It takes as input the data generated by the notebook I mentioned here. Its results are uploaded here. The final adjustments I mentioned above, to account for the motion/rotation of the background, are done in another notebook here.

This has both a CPU and a GPU (NVidia CUDA specific) optical flow implementation but the latter can be 10X faster. Both Google Colab and Kaggle allow running this code on GPUs online for free, but Kaggle provides a 30 hours/week free GPU quota while Colab has a much more restricted variable quota, currently showing only 3 hours for me. Kaggle's free GPU environment also runs the code faster at the moment. Colab's paid tier should provide much better GPUs, although I haven't tried. In both cases a GPU enabled runtime is off by default and needs to be selected from the UI.

Neither Kaggle nor Colab provides a build of the OpenCV library with CUDA support enabled, so I built one for each of them separately with the notebook here, uploaded the binaries here, and wrote some code to download, install and make sure that they can be used. Over time they will surely change the Colab/Kaggle environments in such a way that a new build will be required, so if it fails to install you can let me know and I can make a new build, or people can also make their own build with the provided notebook.

 

[Mick West]

Here I posted the source code for my cloud motion tracking algorithm. It takes as input the data generated by the notebook I mentioned here. Its results are uploaded here. The final adjustments I mentioned above, to account for the motion/rotation of the background, are done in another notebook here.

How would I translate that to angular speed parallel to the horizon?

 

[logicbear]

How would I translate that to angular speed parallel to the horizon?

Adding a line to the angle_plot function we can see that the V vector is already closely aligned with the horizon, but there's a slight difference so maybe it'd be better to project V onto a vector along the horizon. Then translating the magnitude from pixels/frame to degrees/frame should just be matter of multiplying with 0.35/428 (the exact width that corresponds to 0.35 is still a little unclear).

(-object_data.human_horizon).plot(legend=True)