Gimbal Lock and Derotation in FLIR/ATFLIR systems

[Mick West]

Four-axis gimbaled airborne sensor having a second coelostat mirror to rotate about a third axis substantially perpendicular to both first and second axesSource: https://www.youtube.com/watch?v=4X1PRDbtiF0


The above video is an attempt to demonstrate how a system mounted like the ATFLIR targeting pod:
A) requires a derotation mechanism
B) require major camera movements around 0°

It's a tricky thing to explain, and I anticipate this thread getting somewhat detailed and technical, but unless Raytheon wants to weigh in then some digging will be required to figure out exactly what is going on with this rotating glare.

What I'd like to do is collect as many references as possible that address this issue, to help paint a better picture.

One interesting patent I found is US9121758 - "Four-axis gimbaled airborne sensor having a second coelostat mirror to rotate about a third axis substantially perpendicular to both first and second axes" held by Raytheon, which has this interesting discussion on the need for derotation:
https://patents.google.com/patent/US9121758

However, due to this rotation around the roll axis 242, the image from the far field object or scene is also rotated. In order to correct for the rotation of the image, the derotation device 330 is configured to counter-rotate so that the image output by the derotation device is in the same direction independent of the rotation of roll gimbal around the roll axis 242 or the rotation of the first coelostat minor 220 around axis 244 or the rotation of the second coelostat minor 230 around axis 248. In one embodiment, the derotation device 330 is an optical prism. In other embodiments, the derotation device 330 may include reflective optical elements (e.g., minors) and can be, for example, an all-reflective derotation device. However, as it will be appreciated by those skilled in the art that other types of derotation devices can also be used. Furthermore, the derotation device 330 can be omitted, and the derotation function may be accomplished electronically or through image data processing.

Content from external source

And also the issue near the 0° position, referred to to as "gimbal lock" or "gimbal singularity"

As discussed above, when the orientation of the first coelostat minor 220 is such that the sensor line of sight direction is precisely parallel to the roll axis 242, rotation of the gimbal-mounted optics about the roll axis does not change the line of sight direction. This situation where the roll axis cannot steer the line of sight is referred to as a gimbal singularity or gimbal lock. In certain circumstances where the gimbal singularity cannot be avoided, for example because the gimbal singularity is within the desired field of regard (FOR), a fourth gimbal axis 246 may be provided within the plane of the first coelostat mirror 220 and perpendicular to the first rotation axis 244. In one embodiment, the fourth gimbal axis 246 resides on the first rotation axis 244 of the first coelostat minor 220 in that a rotation of the first coelostat minor 220 around the first rotation axis 244 produces a rotation of axis 246. The fourth gimbal axis 246 may be of small angular travel (for example, less than or equal to 5 degrees). As a result, axis 246 travels around the roll axis 242 and avoids the gimbal singularity.

For example, referring again to FIG. 3, when an object being continuously tracked by moving the first coelostat minor 220 in various directions by rotating around the first rotation axis 244 and/or around roll axis 242and/or optional fourth axis 246 using control system 390 is projected to go close to or through the gimbal singularity, and optional fourth gimbal axis 246 is provided with a range of angles (for example, ±3 degrees), the roll axis 242 is no longer used for tracking the object within the ±3 degree range that surrounds the gimbal singularity. Instead, the first rotation axis 244 and fourth gimbal axis 246 are used to continue to track the object within the ±3 degree angular range. When, on the other hand, the object location exceeds, for example, the ±3 degree singularity, the roll axis 242 is used by control system 390 in the tracking motion. In this case, the fourth axis 246 may be gradually returned to 0 degrees and no longer has involvement in the tracking motion. In other words, control system 390controls the tracking by rotating the first coelostat mirror 220 around the fourth gimbal axis 246 when an object is located closely around the singularity (e.g., within the ±3 degree range). Otherwise, when the object is outside the ±3 degree range around the singularity, control system 390controls the tracking by rotating the roll axis 242 and leaving the fourth axis 246 fixed or returning the third axis to 0 degrees.

Content from external source

The mention of +/- 3° is particularly interesting, as all the major apparent motion of the object happens between -3° and +4°

 

[jarlrmai]

Is it worth mentioning that the jet can also rotate and the the camera has to adjust for that as well?

 

[Mick West]

Copied this from my post here: https://www.metabunk.org/posts/217222/


Another source of info for the ATFLIR obsessed is Raytheon's patents.
https://patents.google.com/?q=gimbal&q=IR&q=rotation&q=visible&assignee=raytheon


https://patents.google.com/patent/US6288381B1/

Ideally, a high resolution imaging and laser designation system in a highly dynamic disturbance environment would typically have, at least, a four gimbal set, with two outer coarse gimbals attenuating most of the platform and aerodynamic loads and the two inner most, flexure suspended gimbals providing fine stabilization, with the inertial measurement unit (IMU), IR and visible imaging sensors, and a designating/ranging laser located on the inner most inertially stabilized gimbal.

To reduce gimbal size, weight and cost, the assignee of the present invention has developed a pseudo inner gimbal set for use on various tactical airborne and airborne surveillance systems. This pseudo inner gimbal set uses miniature two-axis flexure suspended mirrors mounted on the inner gimbal together with the IMU and IR sensor, in a residual inertial position error feedforward scheme. The pseudo inner gimbal set replaces the two innermost fine gimbals, while maintaining equivalent performance. With increasing aperture size and constraints required to maintain the size of existing fielded systems, some tactical airborne IR systems are forced to locate the IR and visible sensors and laser off the gimbals using an optical relay path.

Content from external source

The following diagram is a schematic of an ATFLIR, with the nose (and hence the windows) on the left. It explains that the image is "derotated" by a "a reflective derotation mechanism 25" (and another for the visible light at 35).
View attachment 30892

 

[Mick West]

An interesting analysis along these lines on YCombinator from colanderman, who discusses the gimbal lock issue, and hypothesizes it's some warm bird poop on the camera

https://news.ycombinator.com/item?id=20019375

It's appropriate that the first video is called "gimbal", because that's exactly what it is.

Watch the angle readout at the top of the video. The rotation of the object happens exactly around the time that the angle passes 0°. Why is this?

Have you ever watched a PTZ security camera rotate up and over the vertical axis and down the other side? It will tilt up until it nears the vertical axis, at which point it will rotate around that axis, and then tilt back down, now facing the other way. It does this to avoid gimbal lock [1], a state in which it would lose a degree of freedom of rotation. (In this case, it's not the vertical axis, but the forward axis.)

Why doesn't the image rotate then? [shallow speculation] The video software keeps it oriented so that it matches the plane's orientation. (Note that the feed is square, making it easier to make full use of the sensor regardless of rotation.)

Why does the object rotate? This should give you a clue where the object is. If the background is not rotating while the camera is rotating, but the object is, the object is on the camera. It will appearto rotate as the video software rotates the image to compensate for the camera rotation about the forward axis.

So why is the object moving? Well, it's not moving, not if it's on the camera. But whenever the camera moves, it would look like it's moving relative to the background.

So why is the camera moving? It's tracking the object. But the object isn't moving! Well, the camera doesn't track movement. It tracks position. The object is slightly offset from the center of the frame, so the tracking software slightly moves the camera to compensate. This of course does not change the situation, so the tracking software repeats its compensation. This constant camera movement in a single direction gives the appearance that the object is moving.

Why does the object show up on an infrared camera in the first place? It must be warm.

So… what is this warm object, which is stuck on the camera, slightly off-center, causing the tracking software to follow it, through and around the camera's axis, giving the appearance that the object is moving and then rotating?

Well, it's the same thing as this article in the NY Times, which, in service of securing funding from the UFO & Hitler Channel (as @floatrock astutely noted), decided to lend its gravitas to an easily-explainable video glitch which has been paraded by conspiracy theorists as incontrovertible validation of their deepest-held beliefs that extraterrestrials, against all probability, regularly visit Earth.

Bird shit.

Why am I paying for a NY Times subscription again?

[1] https://en.wikipedia.org/wiki/Gimbal_lock

Content from external source

While the gimbal lock analysis is fine, the bird poop explanation does not really hold up, as there's a distinct size change, which seems to indicate the object is getting closer and/or brighter

 

[Agent K]

I demonstrate gimbal lock by making a "thumbs up" gesture and showing how I can "pan" left and right and roll the first, but not tilt up and down. To point it down, I'd have to roll 90 degrees first.

 

[Mick West]

I demonstrate gimbal lock by making a "thumbs up" gesture and showing how I can "pan" left and right and roll the first, but not tilt up and down. To point it down, I'd have to roll 90 degrees first.

I don't follow, where's the lock?

 

[Mick West]

The mention of +/- 3° is particularly interesting, as all the major apparent motion of the object happens between -3° and +4°

An interesting thing with that range is that it's a good chunk of range that the Nimitz/FLIR/Tic-Tac goes over, from 4° right to 6° Left, and as it goes past 2° left there's a fairly distinct rotation of a light area in the background, then a slightly less distinct one at

Which all is very consistent with the idea of a glare being rotated by the camera.

 

[Agent K]

I don't follow, where's the lock?

It's the same thing you show in the video, just using a fist instead of sticks.

 

[Mick West]

It's the same thing you show in the video, just using a fist instead of sticks.

But a fist has more degrees of freedom. With the thumb up I can rotate around all three axes. The gimbal setup can only rotate around two.

 

[Agent K]

But a fist has more degrees of freedom. With the thumb up I can rotate around all three axes. The gimbal setup can only rotate around two.
View attachment 37557

I know, but pretend it can't tilt up and down.
I also demonstrate the North-East-Down axes with the left hand by extending the index finger forward (North), thumb to the right (East), and middle finger down. Then, I can show the yaw-pitch-roll rotation from NED into image coordinates Depth-Column-Row.

 

[Candy-O]

How would FLIR footage of a disc shaped craft that rotates on it's side DIFFER from what we see in the gimball video? Or an actual tic-tac shaped object differ from what we see on the Nimitz footage? I'm not going down any logical path here, I'm just curious how you feel footage of objects as described would differ from what we see.

 

[Mick West]

How would FLIR footage of a disc shaped craft that rotates on it's side DIFFER from what we see in the gimball video?

I'd expect it to remain the same profile through the video, whereas it actually changes shape quite a lot.

Or an actual tic-tac shaped object differ from what we see on the Nimitz footage?

Similar here, the shape of the "object" changes in both visible light and infrared. which suggests something more like an irregular object.

 

[Gerard]

Similar here, the shape of the "object" changes in both visible light and infrared. which suggests something more like an irregular object.

Can you describe what this image is showing. Is the upper image visible and the lower IR ? If so why aren't the engines showing as hot spots ?

 

[Mick West]

Can you describe what this image is showing. Is the upper image visible and the lower IR ? If so why aren't the engines showing as hot spots ?

It's a simulated back-lit image of a plane, where the lower one is more blurred. See 1:54 here:
Source: https://youtu.be/s1oTg0kxzDs?t=114

 

[Mick West]

This patent is interesting, in that it focuses largely on derotation, and discusses the corrections for gimbal lock.
https://patents.google.com/patent/EP2525235B1/



Discussing different methods of derotation:

However, due to the rotation of portion 11A mounted on roll gimbal and thus to the rotation of c-mirror 12 around axis BB, the image from the far field object or scene is also rotated. In order to correct for the rotation of the image, derotation device 20 is configured to counter-rotate so that the image output by the derotation device is in the same direction independent of the rotation of roll gimbal around axis BB or the rotation of c-mirror around axis AA. In one embodiment, derotation device 20 is an optical prism. In another embodiment, derotation device 20 may include reflective optical elements (e.g., mirrors) and can be, for example, an all-reflective derotation device. However, as it can be appreciated other types of derotation devices can also be used. Furthermore, in one embodiment, derotation device 20 can be omitted. In this case, the derotation function can be accomplished electronically or through image data processing.

Content from external source

Discussing gimbal lock

• In one embodiment, whenever the orientation of c-mirror 12 is such that the sensor line-of-sight direction (LOS) is precisely parallel to roll axis BB, for example as shown in FIG. 4B or as shown schematically in FIG. 1, a rotation of the gimbal or portion 11A of optical system 10 around roll axis BB would not change the LOS direction. On the other hand, when c-mirror 12 is oriented such that the LOS direction is at a certain angle α relative to roll axis BB (α is the angle between roll axis BB and LOS direction), for example as shown in FIGS. 3A-3C, 4A, 4C, 5A-5C (e.g., α equal to about 45 deg, about 60 deg., or about 90 deg.), a rotation of the gimbal or portion 11A of optical system 10 around roll axis BB causes the LOS direction to sweep out a cone centered about roll axis BB (e.g., with an angular diameter equal to about 2x45 deg., about 2x60, or about 2x90). The situation where the LOS is precisely parallel to roll axis BB and thus causing the roll axis BB to not steer the LOS [Line of Sight] is called a "gimbal singularity" or "gimbal lock." A control gain of roll axis BB is proportional to 1/sinα. Hence, the gain of roll axis BB can go to infinity when α is equal to 0, i.e., when the LOS is precisely parallel to roll axis BB. From the standpoint of an automated control system 90 (shown in FIG. 1) for controlling the orientation of c-mirror 12, i.e., controlling the rotation around roll axis BB and rotation around axis AA, this gimbal singularity may be problematic because no amount of rotation around roll axis BB produces any desired effect of steering the LOS.
• [0037]
  As a result, in certain applications, the gimbal singularity is to be avoided. However, in an embodiment, where the gimbal singularity cannot be avoided, for example because the gimbal singularity is within the desired field of regard (FOR), then it may be desirable to provide a third gimbal axis TT (shown in FIG. 1). In one embodiment, third gimbal axis TT is within the plane of c-mirror 12 and is perpendicular to rotation axis AA. In one embodiment, the third gimbal axis TT resides on rotation axis AA of c-mirror 12 in that a rotation of c-mirror 12 around axis AA produces a rotation of axis TT. The third gimbal axis TT can be of small angular travel (for example, less than or equal to 5 deg.). As a result, axis TT travels around roll axis BB and avoids the gimbal singularity.
• [0038]
  For example, when an object being continuously tracked by moving c-mirror 12 in various directions by rotating around rotation axis AA and/or around roll axis BB and/or optional third axis TT using control system 90 is projected to go close to or through the gimbal singularity, and optional third gimbal axis TT is provided with a range of angles α, for example, ±3 deg, roll axis BB is no longer used for tracking the object within the ±3 deg. range that surrounds the gimbal singularity. Instead, rotation axis AA and third gimbal axis TT are used to continue to track the object within the ±3 deg. angular range. When, on the other hand, the object location exceeds, for example, the 3 deg. singularity, roll axis BB is used by control system 90 in the tracking motion. In this case, the third axis can be gradually returned to 0 deg. and no longer has involvement in the tracking motion. In other words, control system 90 controls the tracking by rotating c-mirror 12 around third gimbal axis TT when an object is located closely around the singularity (e.g., within the ±3 deg. range). Otherwise, when the object is outside the ±3 deg. range around the singularity, control system 90 controls the tracking by rotating roll axis BB and leaving the third axis TT fixed or returning third axis TT to 0 deg.

Content from external source