Geometry
Great circles
On Earth, the shortest route between two places follows a great circle. On a flat world map, that route usually appears curved.
Interactive great-circle route on a world map
Drag points A and B anywhere on the map.

For this explanation, Earth is treated as a sphere with a mean radius of 6,371 km.
What is a great circle?
A great circle is the largest circle that can be drawn on a sphere. Its plane passes through the centre of the sphere. The equator is a great circle; a meridian together with its opposite meridian forms one as well.
Except for points that are exactly opposite each other, two places on Earth's surface lie on one unique shortest great-circle arc. On an ideal spherical Earth, that arc is the shortest path along the surface.
From coordinates to distance
For two places, we need their latitudes φ₁ and φ₂ and the difference in longitude Δλ. North and east are positive, south and west negative. So 10° E and 20° W give Δλ = 30°.
For example, with φ₁ = 50° N, φ₂ = 20° N and Δλ = 30°:
The central angle is then obtained with the inverse cosine:
Only then do we convert that angle into a distance along Earth's surface:
For the last calculation, σ must be in radians. The interactive map above uses the same spherical model.