On this page I explain how terrestrial atmospheric refraction is calculated from basic physical laws and how the equations and constants used in refraction equations are derived. The same principle can be applied to derive astronomical refraction, although this is not the topic of this page.

Atmospheric Refraction is the bending of light due to a gradient of the refractive index caused by a density gradient of the air. This light bending causes observed objects in the distance to be displaced (mainly up) or distorted and inverted (mirage). In a layer above the surface refraction can vary considerably. Above this layer refraction varies only little and we can derive a mean standard value, see Refraction Coefficient as a Function of Altitude.

In survey it is important to know how much light gets bent to be able to correct the observed height of a distant object for Refraction effects.

Refraction Coefficient Globe

The density gradient along a light ray is difficult to measure in practice. Consequently, the path of a light ray can not be known exactly. But as long as the air is not too turbulent, we can use an average gradient. In this case a light ray can be approximated by an arc with the radius r.

Refraction can be expressed by the Refraction Coefficient k, which is the ratio of the radius of the earth to the radius of a bent light ray:

The Refraction Coefficient is defined so, that k = 0 is no refraction, k = 1 is light bending down with radius R and k = −1 is light bending up with radius R.

Note: The phenomenon of refraction is not dependent on the shape of the earth. We could define a refraction coefficient that is not related to the radius of the earth. In fact we could simply use the curvature c = 1/r of the light ray. We can always get this curvature from k by dividing all equations for k through R: c = k/R.

Refraction Coefficient Flat Earth

We can define a similar refraction coefficient as for the Globe for a Flat Earth. Like for the Globe the FE Refraction Coefficient is defined so, that k = 0 is no refraction, k = 1 is light bending down with radius R and k = −1 is light bending up with radius R.

As the curvature of light can be calculated from the Refraction Coefficient: $c = k_\mathrm{FE} / R$, we get that for a flat earth to appear as a Globe, k_FE has to be negative which implies a negative curvature of light, light must bend upwards.

Refraction Factor, Apparent Radius of Earth

Many calculations for observations that involve the radius of the earth R, like for hidden height, the distance to the horizon ect., assume a straight light ray. But there is a trick to keep the same equations for bent light, by increasing the radius of the earth in the equations by the Refraction Factor a and making the light ray straight. The geometry of the two situations is the same. So we can simply replace R in this equations by R':

Note: because our eyes assume all light rays are straight and interprets the scene accordingly, the earth appears to us scaled by the Refraction Factor.

The connection between the Refraction Coefficient k and the Refraction Factor a is:

If refraction is such that light follows the curvature of the earth (k = 1), then the Refraction Factor is infinite. That means the apparent earth radius is infinity, i.e. the earth appears flat.

Calculating Curvature of Light

If we have measured atmospheric parameters like the pressure, temperature and temperature gradient, we can use the following formula to calculate the the curvature of light:

Note: this formula is independent on the shape of the earth.

If we know the apparent size of the earth R' from measurements and assuming a Globe with radius R, or if we know the Refraction Coefficient k, we can calculate the curvature of the light rays as follows:

From the curvature of the light c or c_FE we can calculate the prevailing temperature gradient using (8) for either model respectively.

Calculating Refraction Coefficient

If we know the curvature of the light rays (5) or if we know the pressure, temperature and temperature gradient at the observer and assume the same conditions along the line of sight, we can use the following formula to calculate the Refraction Coefficient [1]:

Calculating the Temperature Gradient

If we know the curvature of light, e.g. by measuring the apparent size of the earth from observerations and using formula (6), or the refraction coefficient, we can solve the equation (5) for the temperature gradient that is causing the measured refraction:

This formula is not dependent on the shape of the earth, so it works for the Globe and the Flat Earth model. But the calculated curvature of light from observations dependends on the assumed model.

So we can use the different calculations of the curvature of light (6) for each model and then using the formula above calculate the required temperature gradient for both models. It turns out that, to get actual observations above the ground layer, the required temperature gradient on a Flat Earth is physically impossible, about −170°C/km at standard conditions, all the way up in every altitude.

Converting between Gradients

As the curvature of light rays is proportional to the negative Refractivity Gradient (20) wen can use (5) to convert between temperature gradients and refractivity gradients with the following equations:

How does Refraction work?

Before we can derive equations for atmospheric refraction we first have to investigate how refraction works. We know that the speed of light in a medium is slower than in a vacuum. This is expressed by the refractive index:

The refractive index depends on the density and composition of the medium and the wavelength of the light. In a homogeneous solid like glas the refractive index is constant. As soon as light, or any electromagnetic wave, encounters a boundary with a sudden change in the refractive index, the light gets bent by an angle that can be calculated by Snell's law if we know the refractive index of both sides of the boundary. Light gets always bent towards the denser medium, that is the medium with a greater refractive index.

A typical value for the refractive index of air at sea level under standard atmospheric conditions is n_o = 1.000278. Since it is so close to 1, the difference between the refractive index of air and that in a vacuum (n = 1) is of most interest. Since n−1 is very small, it is useful to define Refractivity, which is equal to:

So Refractivity of air at sea level is typically N = 278 N-units.

Refraction in the Atmosphere

Refraction in the atmosphere is more complicated than in a solid with constant refractive index. The air density decreases continuously with increasing altitude. This is called a density gradient. Consequently the refractive index also decreases continuously with increasing altitude until it reaches 1 in the vaccum of space. How is light affected by a refractive index gradient? We can't apply Snell's law here because we have no boundaries.

But based on Fermat's principle we can use calculus to derive an equation that relates curvature of a light ray to any refractive index gradient.

Consider a bundle of light rays propagating through the air with a refractive index gradient as shown in Fig. 1. The radius of curvature of the light ray is denoted as r. All rays in the bundle must traverse the air in the same time dt, if they are part of the same bundle. Lets consider two rays in that bundle.

Path 2: The wave on this path has a radius of r and a velocity v and traverses the path in a time dt. The length of the path that is traversed in time dt is r·dθ. Because length is velocity times time, we can write this as:

Path 1: The refractive index n is different since the path is higher in the atmosphere. We denote the refractive index here as n + dn where dn is the difference in the indices. The radius of this path is r + dr. The associated velocity is v + dv and the time interval dt is the same for both rays. Similar to (13) we can write for this ray:

We can get rid of dθ and dt by dividing equations (13) and (14):

Subtracting 1 from both sides we obtain:

The definition of the refractive index is n = c/v, where c is the speed of light in vacuum and v the speed of light in the medium. Differentiating with respect to v gives:

We have now a relation between the refractive index and the speed of light in the medium. But we need a relation between the refractive index and the radius of curvature of the light ray. We can get this by replacing the green term in (17) with the blue term of (16):

The inverse of the radius of curvature r is called curvature. Hence, the relation (18) describes the curvature of the ray as a function of the refractive index n and the refractive index gradient dn/dh.

We can make some simplifications: if we assume that the ray propagates close to parallel to the ground, so that the angle β in the figure is small, then cos(β) ≈ 1. The refractive index n of air is very close to 1, so we can neglegt the term 1/n.

So the curvature of the light ray can be expressed as:

Note: As the refractive index decreases with increasing altitude, the refractive index gradient is negative, but due to the minus sign the curvature is positive. From our derivation follows, that a positive curvature means that light gets bent down toward the surface. This causes objects in the distance to appear higher than they are.

This is true even if the light ray starts initialliy horizontal. This is due to the gradient of the refractive index, not the curvature of the atmosphere. So even on a flat earth with a vertical gradient, horizontal light gets bent down towards the denser part of the atmosphere.

Note: If the refractive index gradient is 0, the curvature 1/r of the light ray is zero, which means the light ray is straight.

Now we can calculate the curvature 1/r of a light ray at any point from the refractive index gradient dn/dh at that point. And as the Refraction Coefficient k is a measure of this curvature, using the relation (1), the Refraction Coefficient can be calculated from this gradient:

I found this derivation on the following PDF:

Note: The end result in the linked document is expressed as 1/K, where K is called the K-value. I call this value the Refraction Factor a.

Calculating Refractivity of Air

Before we can derive the equation for the Rrefraction Coefficient k as a function of atmospheric pressure, temperature and temperature gradient, we have to calculate the Refractivity of the air N dependent on some atmospheric parameters.

The refractivity of the atmosphere at a certain point is a function of many things, including the wavelength of light, the air pressure P, the absolute temperature T, and the humidity e at that point. A commonly used expression is [3] [4]:

For an accuracy of less than 0.5% some simplifying assumptions may be made if we limit the range of certain variables [4]:

• Temperature: −50°C ≤ T ≤ +40°C
• Pressure: 200 mbar ≤ P ≤ 1100 mbar
• Humidity and CO₂ content can be neglegted

So equation (21) can be simplified to:

The constant K₁ is calculated as follows: Since according to the gas law the density of air is proportional to P/T and Refractivity N according to (22) is also proportional to P/T, Refractivity is proportional to the density of the air. So we can write:

Note: The factor 100 is present so we can use the pressure P in mbar instead of Pa (Pascal).

The blue term is our K₁. Now we only have to find the refractivity N_o for light of wavelength λ = 550 nm, standard pressure at sea level P_o = 1013.25 mbar and standard absolute temperature at sea level T_o = 288.15 K = 15°C. Humidity is set to 0 and CO₂ content is set to standard 450 ppm, which is irrelevant for an accuracy of less than 0.5% as can be explored with the Calculator for Refractivity based on Ciddor Equation.

We can use the Calculator for Refractivity based on Ciddor Equation (or even the older Edlén Equation) to find N_o = 278. The Ciddor equation was derived empirically by accurate measurements of the refractive index of air under many different conditions [5].

Now we can calculate K₁:

So we finally have refractivity of the atmosphere as a function of altitude:

We can calculate the factor K₁ also with the following equation, where the refractivity N can be calculated using the Calculator for Refractivity based on Ciddor Equation, for dry air with wavelength 550 nm and CO2 concentration 450 ppm at standard atmospheric conditions at sea level:

Deriving Equation for Refraction

As we know how the Refraction Coefficient k can be calculated from the refractive index gradient dn/dh or the refractivity gradient dN/dh (20), we can derive the equation for the Refraction Coefficient from atmospheric parameters like pressure P, temperature T and temperature gradient dT/dh as shown in (7).

To be able to calculate the refractivity gradient dN/dh, we need equations to calculate pressure P and temperature T as functions of altitude h. This equations are provided for the model of the International Standard Atmosphere. For the troposphere with a linear temperature gradient α:

We can insert this equations in (22) and get refractivity as a function of altitude:

We need the refractivity gradient, so we have to calculate the derivative with respect to h:

We can split the blue term into 2 terms:

Applying this split and simplifying we get:

Resubstituting the expressions for P and T:

gives:

α is simply the temperature gradient α = dT/dh, so we finally get:

We can insert some values: g/R_S = 0.0343 K/m, K₁ = 79.0 K/mbar to get finally the curvature of light rays. Note: I used a value g = 9.83 m/s² to get the published constant 0.0343. This is slightly more than average gravitational acceleration. Gravitational acceleration on earth at sea level is in the range of g = 9.7639...9.8337 m/s² [6].

Curvature of Light

Note: In the famous formula for the refraction coefficient k (37) a value of R · K₁ = 503 is used. To get this 503 I use a slightly smaller value K₁ = 78.95. This is justified, because we dropped a 1/n = 1/1.000278 factor in (19). This way we get a better match between k and the curvature of light c when converting between them using R = 6,371,000.

The curvature of light is:

• proportional to the negative Refractive Index Gradient
• proportional to the negative Refractivity Gradient
• proportional to the negative Density Gradient
• most sensible to the Temperature Gradient
• independent on the shape of the earth

Note: the Temperature Gradient dT/dh is the most important factor that determines the curvature of light. At the same pressure and temperature you can get light bending up or down depending on the Temperature Gradient!

Refraction Coefficient Globe

We can also calculate the Refraction coefficient for the Globe model:

Influence of Water Vapor

The absolute value of the refractive index is irrelevant for the bending of light, only the gradient, the amount of change per height, is relevant for how much light gets bent.

If the refractive index gradient is zero, i.e. no change in refractive index with increasing altitude, then the curvature of the light ray is zero, no matter what the absolute value of the refractive index is.

The refractive index is proportional to the density of the air. So if the density is constant, then the refractive index is constant and we have no light bending in a constant density. Because at constant air temperature the density decreases with increasing altitude, the refractive index decreases proportionally and light gets bent down towards the denser part of the atmosphere, i.e. objects appear loomed up at standard conditions.

The density of moist air can be calculated as follows: [7]

The maximal humidity ratio for air at 15°C is x = 0.01062 [8]. The corresponding correction factor K_w due to maximum saturation humidity at 15°C is then:

So saturated moist air density is 0.9936 times the density of dry air. Moist air is maximal 0.64% less dense than dry air at 15°C. Can this small difference in density between dry and saturated air have any visible influence at all?

Lets investigate how water vapor can change the density gradient of air and what effect that has on observations. Some Flat Earthers claim, because the air above water contains more moisture than higher up, which means it has less density over water than higher up, and light bends from less dense to denser regions, the light will bend upwards over water and that's the reason why the flat earth appears like a Globe. Is that a physically possible scenario?

The following graphic shows how water vapor can change the density gradient or refractive index gradient respectively from the red curve (dry air) to the blue curve (moist to dry air). Remember, the slope of the curve is responsible for the bending of light (refraction). To get upwards bending, the curve must raise from the left to the right.

The air cannot have less humidity than 0. This limits the influence of water vapor on the refractive index as follows:

Generally the density gradient depends mostly on the temperature gradient, then on pressure and on absolute temperature and only very little on the humidity gradient. A constant humidity has no influence on refraction at all, a decreasing humidity with altitude makes refraction a tiny bit less, an increasing humidity a tiny bit stronger, very marginal, negligible in practice. Only a very strong vapor gradient over water can invert the density gradient enough to cause inferior mirages.

The surface layer is the only place where refraction can vary considerably due to steep temperature gradients due to heat exchange between ground and air. Above the surface layer density always decreases slowly according to the equations of the International Standard Atmosphere. See also Refraction Coefficient as a Function of Altitude.

So a small water vapor layer directly over water can explain some inferior mirages in cases where they are not present without water vapor. But water vapor can not bend light upwards in such a way that it could explain why the earth appears globe shaped if the earth would be flat. Strong enough water vapor layers dry out very quickly and are therefore restricted to small layers over water. Above this layer light will always bend down.

Correcting for Refraction

If we know the Refraction Coefficient, height measurements using a theodolite can be corrected with the following approximation equations, as long as the distance d is much less than the radius of curvature of the light ray:

A positive Refraction Angle means the light ray is bent down and the object appears too high. In a zenith angle measurement we have to add the Refraction Angle to the zenith angle to get the correct angle as measured without refraction.

From the Refraction Angle we can calculate the magnitude l of how much an object at distance d appears to be raised due to Refraction:

As Refraction Angles in radian are numbers much less than 1, this equations are very good approximations. The distance d can be the line of sight distance or the distance along the light ray or the distance along the surface of the earth. They are all essentially the same as long as d is much smaller than R or r respectively.

References