The Nautical Almanac provides approximations to calculate the Dip Correction in Celestial Navigation, like: α = 0.0293 · √h. I show that the number is derived from the radius of the earth, taking refraction into account. I also derive exact formulas and provide a Calculator based on the exact formulas.

Dip Correction in Celestial Navigation

In Celestial Navigation we need the zenith angle of a celestial object to calculate the distance to the GP (GP = Ground Position). The zenith angle can be calculated by subtracting the elevation angle from 90°. The elevation angle is the angle between true horizontal and the celestial object, after correcting for astronomical refraction.

Using a Sextant we can measure the angle between the visible (apparent) refracted horizon and a celestial object. The horizon drops from true horizontal with increasing observer height because the earth is a globe. So there is a Dip Angle between true horizontal and the apparent horizon. To get the elevation angle, spanning from true horizontal to the celestial object, we have to subtract the Dip Angle from the sextant reading. This is called Dip Correction.

The Nautical Almanac provides and uses the following formulas to calculate the dip correction table:

The values in the formulas are derived from earths radius R = 6371 km, taking standard refraction k = 0.167 into account. The observer height h is under a square root because it follows from the spherical geometry of the earth:

This formulas are good approximations if the observer height h is much smaller than the radius of the earth. There are also Exact Dip Angle Correction Formulas.

Dip Angle Calculator

The Calculator uses the Exact Dip Angle Correction formulas. Choose the angle units Deg or DM and observer height 1 to reflect the factors of the approximation functions in the Dip Angle. You can specify the tolerances of the measured values. This yields the accuracy of the Dip Angle.

If you enter measured atmospheric parameters P, T and dT/dh then the corresponding refraction coefficient k is calculated. If you enter a refraction coeffiecient, then the corresponding temperature gradient dT/dh is calculated, keeping P and T fixed. When you enter the obersever height the pressure P for this height is calculated using the model of the International Standard Atmosphere. You can overwrite the pressure anytime later.

Derivation of Dip Angle Correction Approximations

The simple dip angle correction formulas (1) used in the Nautical Almanac are derived from the globe model by making some approximations. In the Graph we can see that the dip angle α is the same as the angle at the center of the earth. Move the slider and watch how this two angles are always the same.

We have a right angle triangle with the sides d, R' and R'+h. Using trigonometry and Pythagoras we can calculate the angle α

If we assume that the oberver height h is much smaller than the radius of the earth, we can make the following approximations:

The red factors ${ R^{\,\prime} }^2$ cancel and because h² is much smaller than $2 \cdot h \cdot R^{\,\prime}$ it can be neglegted.

In the dip angle calculation we can make 2 approximations. Because dip angles are very small the sine of the angle is approximately equal the angle in radian: $\sin(\alpha) \approx \alpha$. And because h is much smaller than R' it can be neglegted in the denominator: d/(R+h) ≈ d/R. So we are left with the simplified equation:

If we know the refraction coefficient k we can calculate the refracted radius of the earth $R^{\,\prime}$ from the geometric radius of the earth:

Explanation: Because light rays bend down under standard refraction conditions, all objects in the distance, including the horizon, appear raised, as if the earth were bigger than it is geometrically. The factor of how much bigger the earth appears is 1/(1−k), where k is the refraction coefficient. For standard refraction k = 0.143 this factor is 7/6, which is often used in surveying.

We can replace $R^{\,\prime}$ in equation (7) by the right hand side of equation (8) to use the known geometric radius of the earth R and the refraction coefficient k. The dip correction formula is then:

Now we are almost done. If we use standard refraction k = 0.167 and R = 6,371,000 we can separate all constant factors using √X·h = √X·√h. To convert the dip angle from radian into degrees we multiply by 180/π:

Calculation of Refraction

If we have measured atmospheric parameters like the pressure, temperature and temperature gradient, we can use the following formula to calculate the curvature of light:

Note: this formula is independent on the shape of the earth. There is no R value in this formula, because atmospheric refraction does not need a curved atmosphere, only a vertical density gradient, caused mainly by the vertical temperature gradient. The formula is derived from Fermats principle, that light takes the path with the shortest time through the variing densities of the atmosphere.

The refraction coefficient is derived from the curvature of light and is defined as follows:

The Nautical Almanac uses the following values to calculate the refraction coefficient:

Note: Because I could not find any source for why the Almanac choose k = 0.167, I reverse engineered the temperature gradient dT/dh assuming the listed values for P and T. I could also choose other values to get the same refraction coefficient. I assume that they choose k as an average of observed values. As the Almanac lists in the Dip Angle Correction Table the dip angle only with 2 significant digits, the value for k can vary in a relatively wide range and still get the listed corrections.

Exact Dip Angle Correction

It is possible to get an exact solution for the dip angle, taking refraction into account. But then we can not write a simple formula like α = x · √ h .

Derivation of the Exact Dip Angle Correction

Figure: Geometry showing the Dip Angle (alpha), Observer Height (h) and a curved Light Ray (r) touching the surface of the Earth

The figure shows the geometric construction from which the exact dip angle correction is derived.

Known: observer height Obs h, Radius of the Earth (R) and radius of curvature of the light ray r.

Unknown: Dip Angle (alpha), so that the circle Earth Surface (s) is tangent to the curve Light Ray (r) and passes through the point B.

Geometry

Lets see what we have: The blue circle Earth Surface (s) has to be tangent to both the horizontal through the point B and the arc Light Ray (r) at the point T. So the center of the earth Z lies on the Perpendicular to (e) at the distance R from point B.

The arc Light Ray (r) has to be tangent to both the Tangent to the Apparent Horizon (t) at point A and tangent to Earth Surface (s) at the point T. So the center of the light ray arc Q lies on the perpendicular to the Tangent to Apparent Horizon (t), labeled as Radius Light Ray (r) at a distance r from point A. The center of the earth Z lies on the line QT.

Solving for the Dip Angle

We know r from the Calculation of Refraction, r = 1/c. The line QT has the length r, because it is the same length as the line AQ. Both are the radius of the light ray r.

Now lets regard the yellow triangle with the sides AQ, QZ, ZA:

The sides are: AQ = r, QZ = (r−R) and ZA = R+h. It can be shown that the angle ZAQ at A is the same as the DipAngle (alpha). So this triangle only contains one unknown, the dip angle α.

Using the Law of cosines on the yellow triangle we get:

Expanding the squares gives:

The red parts cancel. After bringing all terms containing cos(α) to one side and factoring it out we are left with:

Dividing both sides by 2·r gives:

Substituting r = 1/c gives:

Isolating cos(α) and factoring out c:

And finally by taking the arcus cosine we get the dip angle in radian:

If we have the refraction coefficient k = c · R we can use it instead of the light curvature c by substituting c = k/R: