The Tiny Roman Empire

The Roman Empire was huge: 5 million sq. km at its peak, larger than India (3.2 million sq. km), about half the size of the United States (9.8 million sq. km).

And yet by population it was tiny: ~60 million, 5x smaller than today's US (330 million), 22x smaller than today's India (1.3 billion).

I wonder what is the primary reason for this. Probably lower agricultural productivity in ancient times?