07-09, 13:55–14:25 (US/Pacific), Room 317
One of the most important aspects of developing scientific software is distribution for others. The Scientific Python Development Guide was developed to provide up-to-date best practices for packaging, linting, and testing, along with a versatile template supporting multiple backends, and a WebAssembly-powered repo-review tool to check a repository directly in the guide. This talk, with the guide for reference, will cover key best practices for project setup, backend selection, packaging metadata, GitHub Actions for testing and deployment, tools for validating code quality. We will even cover tools for packaging compiled components that are simple enough for anyone to use.
Preparing scientific software for distribution is a key part of the software lifecycle, but the tools and landscape in Python packaging seem to be constantly evolving. The Scientific Python Development Guide was written to provide up-to-date best practices for packaging, linting, and testing scientific software. It continues to be an unparalleled resource, offering a template that supports many different build backends, including several for compiled code. A WebAssembly-powered repo-review tool can even validate a GitHub repository against the recommendations in the guide, alerting users to new best practices as they evolve.
We will begin by examining project setup, covering backend selection and how to prepare for distribution. We will go over the structure of a project, explaining why certain decisions were made, such as placing the package inside a “src” directory. Common mistakes and pitfalls will be discussed, along with tips on best practices for typical scenarios, such as loading a data file from the package. We will illustrate these concepts using a specific backend, along with a brief explanation of our choice. Additionally, we will cover setting up packaging metadata, including the recently finalized SPDX license system.
The next portion will focus on tooling to facilitate development and distribution. Using the guide as a foundation, we will demonstrate how to use GitHub Actions to test and deploy your code, including recent enhancements like Trusted Publishing and SigStore-signed artifacts. We will also explore related tools for validating code quality and ensuring high-quality packages are reliably produced. Attendees will learn how to use the guide’s template to quickly set up all the discussed components for a new project. We will also showcase how the repo-review tool can assess an existing project’s adherence to best practices and recent packaging updates.
Next, we will cover compiled components; this is an aspect often left off of introductions to packing, but adding compiled code like C++ to a package has become significantly simper with modern packaging tools. We will explore a collection of tools, including modern backends like scikit-build-core and meson-python, binding tools like pybind11 and nanobind, and creating redistributable wheels with cibuildwheel. Combined, these tools allow packages containing compiled code to be distributed with minimal extra effort that target all major platforms, including WebAssembly via Pyodide.
We will conclude with a brief look at the new and proposed changes in the packaging ecosystem that will benefit the scientific community. Upcoming features in pybind11 3.0, culminating years of development, will enable new interoperability with C++. And cibuildwheel 3.0 will expand support to new targets like iOS wheels. Other topics will include the transition from extras and requirement files to dependency-groups, how to use uv in place of pip and traditional workflows, and new proposals coming from the WheelNext organization, such as default extras. Attendees will leave with a solid understanding of modern Python packaging, practical tools to streamline their workflows, and an optimistic outlook on the future of scientific software distribution.
Henry Schreiner is a Computational Physicist / Research Software Engineer in High Energy Physics at Princeton University. He specializes in the interface between high-performance compiled codes and interactive computation in Python, in software distribution, and in interface design. He has previously worked on computational cosmic-ray tomography for archaeology and high performance GPU model fitting. He is currently a member of the IRIS-HEP project, developing tools for the next era of the Large Hadron Collider (LHC).
He is a maintainer/core developer for pypa/build, scikit-build, cibuildwheel, pybind11, meson-python, nox, and plumbum for Python. He is an admin of Scikit-HEP, and a lead designer on boost-histogram, hist, UHI, vector, uproot-browser, Particle, and DecayLanguage packages there. He is also the lead author of the Scientific-Python Development guide and Scientific-Python/cookie. He is the primary author of CLI11, a C++ library used by Microsoft terminal and many others. He is also the lead web developer for IRIS-HEP. He is also the author of Modern CMake and a variety of CMake, GPU, and Python training courses and classes.