HistoryOfPythonPackaging

Notes on a thorough-but-hopefully-brief history of Python Packaging

Motivation: clarify what are (to me) the predominant confusions about the modern Python packaging ecosystem and argue that the system is better-evolved than often given credit for. But also admit the sharp corners left by 20 years of language evolution

Notes

In particular, the proliferation in build backends once that term started to exist, the role/origin of tools that build backends often shove users towards (describing what integration frontends are/do with examples along the way), the general state of confusion in air about when something is necessary and when you are being exposed to the opinions of the particular tool(s) you chose.

I argue that the proliferation is Good, Actually™, but an expansive ecosystem requires the end-user to be discerning: a random selection of tooling is likely to not be a great fit for their aesthetics or problem domain. I further argue that Python is pretty unique in its approach to trying to provide a good first-party package experience. Somewhere in there I may like to nod to the fact that much of this work was done in a time where language governance was changing!

Somewhere in there, I definitely want to drive home the impressive fluidity of interactions across FFI boundaries now that backends can be domain-specific (e.g. you can use maturin to build an extension module written in Python instead of cajoling setuptools into helping with that). Python has a reputation as a "glue language" for a reason but I think what build backends do to help with this problem domain is extremely novel. Discussion of Python.h or other FFI implementation details should be left out of this document except inasmuch as mentioning their existence in the broader packaging context.

Other side-quests

• PyPA guides are idiosyncratic and I think beginners can get lost easily there
• requirements.txt
• virtual environments, tools that manage them
• the troubled romance of Python packaging and system packaging
• the yawning chasm that is the distinction between "module" and "distribution" names (e.g. cv2 vs opencv-python)

Rough (non-exhaustive) sketch of geological strata

Pre-Egg era
Egg/easy_install era
pip, wheels era
manylinux1 et al. era
"future" manylinux, PEP 517 era  << YOU ARE HERE

Resources

Video

• Elana Hashman's talk on wheels (emphasis on bdists)
• Dustin Ingram's history of packaging

Not-video

• Official PyPA history
• Ned's sample pyproject.toml
• PEP 440 (the version identifier spec is very good IMO!)
• PEP 517/518 (others?)
• PEP 600 (as well as preceding manylinux PEPs, musllinux PEP)
• https://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/history.html
• https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/