The Effect Of Atmospheric Refraction On The Observed Elevation Angles Of Peaks

Atmospheric refraction slightly increases the observed elevation angle of a peak relative to the observer. The effect is actually quite complicated, since it depends on the precise atmospheric conditions, including atmospheric pressure, temperature, and water vapor content, and thus varies with time and the altitudes of the observer and the observed peak. Fortunately, the effect of refraction is less than ~15% of the effect due to the curvature of the Earth, and typically only increases the observed elevation angle by less than 0.1°.

Refraction is caused by two effects. First, light likes to travel on the path that gets to the observer in the minimum time. (Light is, after all, the fastest thing in the Universe, so you wouldn't expect it would like to take a longer path than it had to, right?) The speed of light is the speed of light in a vacuum divided by the index of refraction. Second, the index of refraction of the atmosphere depends on atmospheric pressure and amount of water present, which change with height in the atmosphere. Therefore light actually travels on a curved path in the atmosphere from one object to another. The path goes higher than the straight-line distance in order to take advantage of the faster speed higher in the atmosphere. Because the path is so curved, the observer must always look a bit higher to see the light rays coming back down from that higher elevation.

Clearly refraction must depend on some power of the distance. If you are observing something close by, light can't get to you any quicker by travelling very far upward. However, if you are far away from an object, light can take advantage of the faster speed at higher elevation and deviate more significantly from a straight line.

A straightforward calculation gives the following formula for the angular change with distance due to refraction between the observer at elevation Zo (measured in km) who also is observing a peak at elevation Zo:

theta = [ 1.6 * c / { 2*(1+a) } ] * d

where d is the distance in miles, theta is in radians, and

a = 2.9e-4 * exp(-Zo/10 km) / (1 + 2.9 * To / 760)

b = 2.9 * alpha / {760 * (1 + 2.9 * To / 760) }

c = a * (b - 1/10 km)

alpha = 6.5° C. / km

The calculation assumes an atmosphere whose pressure, temperature (To in Celsius) and water content only varies with elevation, and an atmosphere whose temperature varies with elevation with the slope -alpha. (Alpha is defined as the rate of temperature drop with altitude, and so is positive in the above formulae in the normal case where the temperature drops with altitude.) "exp" is the exponential function.

Note that this calculation assumes quite a bit. The real atmosphere can vary markedly horizontally, can have temperature inversions, can change its humidity, and have additional components like dust that change the index of refraction. The observer and observed peak are not always at the same elevation assumed in the derivation of this formula. Hence there are no guarantees that this formula will always give accurate results. However, on average, this formula probably gives the correct average answer. The results of this formula at sea level are given in surveying books as the proper term to use.

The book Elementary Surveying gives the equivalent formula in terms of the "elevation loss" in feet of the observed object with distance:

elevation loss = 0.574 * d^2,

which is said to apply to near horizontal shots. What the book doesn't say is that this formula is only correct near sea level.

The elevation loss formula consists of two terms:

The formula above gives a coefficient of 0.088 at an elevation of 0' and a temperature of 65° F. However, the coefficient varies markedly with temperature and elevation. The following table gives some values as a function of elevation, keeping the temperature constant at 65° F.:

Elevation (')Refraction coefficientRefraction + curvature coefficient

The following table shows the variation with temperature, keeping the elevation constant at 6000':

Temperature (° F.)Refraction coefficientRefraction + curvature coefficient

In the tables, I have included a coefficient of atmospheric refraction of 0.073 corresponding to an altitude of 6000' with a temperature of 70° F. and a normal temperature gradient of -6.5° C./km. No elevation angle was changed by more than 0.1° by this term. Using this nominal value eliminates any bias in the calculated angles on average.

I thank Bob Gonsett for providing me with the formula from the surveying book, and stimulating me to do this calculation to understand where the formula came from.

Go to:

Copyright © 1998-2000 by Tom Chester.
Permission is freely granted to reproduce any or all of this page as long as credit is given to me at this source:
Comments and feedback: Tom Chester
Last update: 11 April 1999 (english typo corrected 14 February 2000; a typo in the equation for the quantity "a" corrected 24 November 2000 thanks to Rich Caruana)