On this page I explain how terrestrial atmospheric refraction is calculated from basic physical laws and how the equations and constants used in refraction equations are derived. The same principle can be applied to derive astronomical refraction, although this is not the topic of this page.

Atmospheric Refraction is the bending of light due to a gradient of the refractive index caused by a density gradient of the air. This light bending causes observed objects in the distance to be displaced (mainly up) or distorted and inverted (mirage). In a layer above the surface refraction can vary considerably. Above this layer refraction varies only little and we can derive a mean standard value, see Measured Refraction Coefficients as a Function of Altitude.

In survey it is important to know how much light gets bent to be able to correct the observed height of a distant object for Refraction effects.

Def Refraction Coefficient Globe Model

The density gradient along a light ray is difficult to measure in practice. Consequently, the path of a light ray can not be known exactly. But as long as the air is not too turbulent, we can use an average gradient. In this case a light ray can be approximated by an arc with the radius r.

Refraction can be expressed by the Refraction Coefficient k, which is the ratio of the radius of the earth to the radius of a bent light ray:

The Refraction Coefficient is defined so, that

• k = 0 no refraction
• k = 1 light is bending down with radius R
• k = −1 light is bending up with radius R.

Note: The phenomenon of refraction is not dependent on the shape of the earth. We could define a refraction coefficient that is not related to the radius of the earth. In fact we could simply use the curvature c = 1/r of the light ray. We can always get this curvature from k by dividing all equations for k through R: c = k/R.

Def Refraction Coefficient Flat Earth Model

We can define a similar refraction coefficient as for the Globe for a Flat Earth. Like for the Globe the FE Refraction Coefficient is defined so, that k = 0 is no refraction, k = 1 is light bending down with radius R and k = −1 is light bending up with radius R.

As the curvature of light can be calculated from the Refraction Coefficient: $c_\mathrm{FE} = k_\mathrm{FE} / R$, we get that for a flat earth to appear as a Globe, k_FE has to be negative which implies a negative curvature of light, light must bend upwards.

Refraction Factor, Apparent Radius of Earth

Many calculations for observations that involve the radius of the earth R, like for hidden height, the distance to the horizon ect., assume a straight light ray. But there is a trick to keep the same equations for bent light. We simply decrease the curvature of the earth and the light ray the same amount, so that the curvature of light gets zero. This results in a changed apparent radius of the earth.

Note: Curvature is the inverse of the radius of curvature.

The changed apparent curvature of the earth 1/R' is the real curvature 1/R minus the curvature of the light rays c = 1/r. So by subtracting the curvature of the light ray from the curvature of the earth, we get a geometry where the light ray is straight and the earth appears the same amount less curved:

The geometry between the earth surface and the light ray is the same in the two situations. So we can simply replace R in all equations containg R by R' to take refraction into account.

Now we sove (3) for the apparent radius of the earth R' simply by inverting both sides of the equation:

The Refraction Factor a = 1/(1−k) tells us how much the apparent radius of the earth R' is different than the real radius R of the earth:

Note: because our eyes assume all light rays are straight and interprets the scene accordingly, the earth appears to us scaled by the Refraction Factor.

The connection between the Refraction Coefficient k and the Refraction Factor a is:

If refraction is such that light follows the curvature of the earth (k = 1), then the Refraction Factor is infinite. That means the apparent earth radius is infinite, the apparent curvature (1/R') is zero, i.e. the earth appears flat.

Curvature of Light from Atmospheric Parameters

If we have measured atmospheric parameters like the pressure, temperature and temperature gradient, we can use the following formula to calculate the the curvature of light:

Note: this formula is independent of the shape of the earth.

Curvature of Light from Refraction Coefficient

If we know the apparent size of the earth R' from measurements and assuming a Globe with radius R, or if we know the Refraction Coefficient k, we can calculate the curvature of the light rays as follows:

From the curvature of the light c or c_FE we can calculate the prevailing temperature gradient using (10) for either model respectively.

Refraction Coefficient from Curvature of Light

If we know the curvature of the light rays (7) or if we know the pressure, temperature and temperature gradient at the observer and assume the same conditions along the line of sight, we can use the following formula to calculate the Refraction Coefficient [1]:

This is the eqation vor light at wavelength 550 nm often pusblished and used in geodesy and celestial navigation, ignoring the influence of the humidity gradient, which can be neglegted for an accuracy of about 1%. I use this equation on my websites and calculators, e.g. my Advanced Earth Curvature Calculator.

Temperature Gradient from Curvature of Light

If we know the curvature of light, e.g. by measuring the apparent size of the earth from observerations and using formula (8), or the refraction coefficient, we can solve the equation (7) for the temperature gradient that is causing the measured refraction:

This formula is not dependent on the shape of the earth, so it works for the Globe and the Flat Earth model. But the calculated curvature of light from observations dependends on the assumed model.

So we can use the different calculations of the curvature of light (8) for each model and then using the formula above to calculate the required temperature gradient for both models. It turns out that, to get actual observations above the ground layer, the required temperature gradient on a Flat Earth is physically impossible, about −170°C/km at standard conditions, all the way up in every altitude.

See Determining the Shape of the Earth with Zenith Angle Measurements for an application of this method.

Converting Refractivity and Temperature Gradients

As the curvature of light rays is proportional to the negative Refractivity Gradient (28) wen can use (7) to convert between temperature gradients and refractivity gradients with the following equations:

Refractive Index

Before we can derive equations for atmospheric refraction we first have to investigate how refraction works. We know that the speed of light in a medium is slower than in a vacuum. This is expressed by the refractive index:

The refractive index depends on the density and composition of the medium and the wavelength of the light. In a homogeneous solid like glas the refractive index is constant. As soon as light, or any electromagnetic wave, encounters a boundary with a sudden change in the refractive index, the light gets bent by an angle that can be calculated by Snell's law if we know the refractive index of both sides of the boundary. Light gets always bent towards the denser medium, that is the medium with a greater refractive index.

A typical value for the refractive index of air at sea level under standard atmospheric conditions is n_o = 1.000278.

Refractivity

Since the Refractive Index (n_o = 1.000278) is so close to 1, the difference between the refractive index of air n and that in a vacuum (n_vac = 1) is of most interest. Since n−1 is very small, it is useful to define Refractivity, which is equal to:

So Refractivity of air at sea level is typically N_o = 278 N-units.

Refractivity from Atmospheric Parameters

Before we can derive the equation for the Rrefraction Coefficient k as a function of atmospheric pressure, temperature and temperature gradient, we have to calculate the Refractivity of the air N dependent on some atmospheric parameters.

The refractivity of the atmosphere at a certain point is a function of many things, including the wavelength of light, the air pressure P, the absolute temperature T, and the humidity e at that point. A commonly used expression derived from the Edlen equation is [3] [4]:

NIST notes that the original Birch–Downs humidity correction is effectively fitted near 19.6°C, and recommends multiplying it by 292.75 K / T to account approximately for the temperature dependence of vapor density. [6]

Simplification: Ignoring Humidity

For an accuracy of less than 5% some simplifying assumptions may be made if we limit the range of certain variables:

• Temperature: −50°C ≤ T ≤ +40°C
• Pressure: 200 hPa ≤ P ≤ 1100 hPa
• Humidity and CO₂ content can be neglegted

So for a large range of atmospheric conditions the equation (15) can be simplified to:

On the rest of this page I neglegt the contribution of humidity. For a derivation of how much the humidity gradient influences atmospheric refraction, see Influence of Humidity on Atmospheric Refraction of Light.

Note: This equation is only valid for visible light. For radio and microwaves the influence of water vapor is much bigger because polar water molecules can respond rotationally to microwave electric fields. This mechanism is absent at visible wavelengths. The standard refraction coefficient used for microwaves is k_radio = 0.25, compared to visible light k_light = 0.17. For more information see Comparing Atmospheric Refraction of Radio Waves and Light.

Deriving Value of K1 Constant

The constant K₁ is calculated as follows:

Since according to the ideal gas law the density of air is proportional to P/T and Refractivity N according to (16) is also proportional to P/T, Refractivity is proportional to the density of the air. So we can write:

Note: The factor 100 is present so we can use the pressure P in hPa instead of Pa (Pascal).

The blue term is our K₁. Now we only have to find the refractivity N_o for light of wavelength λ = 550 nm, standard pressure at sea level P_o = 1013.25 hPa and standard absolute temperature at sea level T_o = 288.15 K = 15°C. Humidity is set to 0 and CO₂ content is set to standard 450 ppm, which is irrelevant for an accuracy of less than 0.5% as can be explored with the Calculator for Refractivity based on Ciddor Equation.

We can use the Calculator for Refractivity based on Ciddor Equation (or even the older Edlén Equation) to find N_o = 278. The Ciddor equation was derived empirically by accurate measurements of the refractive index of air under many different conditions [7].

Now we can calculate K₁:

So we finally have refractivity of the atmosphere as a function of altitude:

We can calculate the factor K₁ also by solving (19) for the constant K₁ using P and T at sea level with the following equation, where the refractivity N_o can be calculated using the Calculator for Refractivity based on Ciddor Equation, for dry air with wavelength 550 nm and CO2 concentration 450 ppm at standard atmospheric conditions at sea level:

Deriving Curvature of Light from Refractive Index Gradient

Refraction in the atmosphere is more complicated than in a solid with constant refractive index. The air density decreases continuously with increasing altitude. This is called a density gradient. Consequently the refractive index also decreases continuously with increasing altitude until it reaches 1 in the vaccum of space. How is light affected by a refractive index gradient? We can't apply Snell's law here because we have no boundaries.

But based on Fermat's principle we can use calculus to derive an equation that relates curvature of a light ray to any refractive index gradient.

Consider a bundle of light rays propagating through the air with a refractive index gradient as shown in Fig. 1. The radius of curvature of the light ray is denoted as r. All rays in the bundle must traverse the air in the same time dt, if they are part of the same bundle. Lets consider two rays in that bundle.

Path 2: The wave on this path has a radius of r and a velocity v and traverses the path in a time dt. The length of the path that is traversed in time dt is r·dθ. Because length is velocity times time, we can write this as:

Path 1: The refractive index n is different since the path is higher in the atmosphere. We denote the refractive index here as n + dn where dn is the difference in the indices. The radius of this path is r + dr. The associated velocity is v + dv and the time interval dt is the same for both rays. Similar to (21) we can write for this ray:

We can get rid of dθ and dt by dividing equations (21) and (22):

Subtracting 1 from both sides we obtain:

The definition of the refractive index is n = c/v, where c is the speed of light in vacuum and v the speed of light in the medium. Differentiating with respect to v gives:

We have now a relation between the refractive index and the speed of light in the medium. But we need a relation between the refractive index and the radius of curvature of the light ray. We can get this by replacing the green term in (25) with the blue term of (24):

The inverse of the radius of curvature r is called curvature. Hence, the relation (26) describes the curvature of the ray as a function of the refractive index n and the refractive index gradient dn/dh.

We can make some simplifications: if we assume that the ray propagates close to parallel to the ground, so that the angle β in the figure is small, then cos(β) ≈ 1. The refractive index n of air is very close to 1, so we can neglegt the term 1/n.

So the curvature of the light ray can be expressed as:

Note: As the refractive index decreases with increasing altitude, the refractive index gradient is negative, but due to the minus sign the curvature is positive. From our derivation follows, that a positive curvature means that light gets bent down toward the surface. This causes objects in the distance to appear higher than they are.

This is true even if the light ray starts initialliy horizontal. This is due to the gradient of the refractive index, not the curvature of the atmosphere. So even on a flat earth with a vertical gradient, horizontal light gets bent down towards the denser part of the atmosphere.

Note: If the refractive index gradient is 0, the curvature 1/r of the light ray is zero, which means the light ray is straight.

Now we can calculate the curvature 1/r of a light ray at any point from the refractive index gradient dn/dh at that point. And as the Refraction Coefficient k is a measure of this curvature, using the relation (1), the Refraction Coefficient can be calculated from this gradient:

I found this derivation on the following PDF:

Note: The end result in the linked document is expressed as 1/K, where K is called the K-value. I call this value the Refraction Factor a.

Deriving Refractivity Gradient from Atmospheric Parameters

As we know how the Refraction Coefficient k can be calculated from the refractive index gradient dn/dh or the refractivity gradient dN/dh (28), we can derive the equation for the curvature of light from atmospheric parameters like pressure P, temperature T and temperature gradient dT/dh as shown in (9).

To be able to calculate the refractivity gradient dN/dh, we need equations to calculate pressure P and temperature T as functions of altitude h. This equations are provided for the model of the International Standard Atmosphere. For the troposphere with a linear temperature gradient α:

We can insert this equations in (16) and get refractivity as a function of altitude:

Because the curvature of light at a certain point in its path does not depend on the value of refractivity at this point, rather on the vertical refractivity gradient, we have to calculate the derivative of refractivity with respect to h:

We have to calculate the derivative of the quotient P(h) / T(h), so we can apply the Quotient Rule:

Executing the derivatives:

We can split the blue term into 2 terms:

Applying this split, simplifying and transfering the minus sign to the left of the equation we get:

Resubstituting the expressions for P and T:

gives:

α is simply the temperature gradient α = dT/dh, so we finally get:

We can insert some values:

and K₁ = 79.0 K/hPa to get finally the Curvature of Light from Refractivity Gradient.

Note: The value 0.0343 K/m is a historical rounded figure from the literature on optical refraction. The exact physical constant is 0.0341626 K/m, which is rounded to 0.0342 K/m in the context of radio refraction. The difference is purely a matter of convention and has no physical impact.

Curvature of Light from Refractivity Gradient

As shown in Deriving Curvature of Light from Refractive Index Gradient (28) the curvature of light c depends on the refractivity gradient dN/dh:

Note: In the famous formula for the refraction coefficient k (41) a value of R · K₁ = 503 is used. To get this 503 I use a slightly smaller value K₁ = 78.95. This is justified, because we dropped a 1/n = 1/1.000278 factor in (27). This way we get a better match between k and the curvature of light c when converting between them using R = 6,371,000 m.

The curvature of light is:

• proportional to the negative Refractive Index Gradient
• proportional to the negative Refractivity Gradient
• proportional to the negative Density Gradient
• most sensible to the Temperature Gradient
• independent on the shape of the earth

Note: the Temperature Gradient dT/dh is the most important factor that determines the curvature of light. At the same pressure and temperature you can get light bending up or down depending on the Temperature Gradient!

Refraction Coefficient from Curvature of Light

We can also calculate the Refraction coefficient for the Globe model:

This is the eqation vor light at wavelength 550 nm often pusblished and used in geodesy and celestial navigation, ignoring the influence of the humidity gradient, which can be neglegted for an accuracy of about 1%. I use this equation on my websites and calculators, e.g. my Advanced Earth Curvature Calculator.

Correcting for Refraction

If we know the Refraction Coefficient, height measurements using a theodolite can be corrected with the following approximation equations, as long as the distance d is much less than the radius of curvature of the light ray:

A positive Refraction Angle means the light ray is bent down and the object appears too high. In a zenith angle measurement we have to add the Refraction Angle to the zenith angle to get the correct angle as measured without refraction.

From the Refraction Angle we can calculate the magnitude l of how much an object at distance d appears to be raised due to Refraction:

As Refraction Angles in radian are numbers much less than 1, this equations are very good approximations. The distance d can be the line of sight distance or the distance along the light ray or the distance along the surface of the earth. They are all essentially the same as long as d is much smaller than R or r respectively.

References