Josh Borrow
I am a Research Software Engineer working on the Simons Observatory at the University of Pennsylvania USA, working on data management and visualisation. I also have interests in numerical galaxy formation simulations
I was previously postdoctoral researcher in astrophysics at the MIT Kavli Institute, in Massachusetts, USA. I did my PhD at the Institute for Computational Cosmology at the University of Durham in the UK.
Sessions
Many tools exist for large-scale data transfer (tens of terabytes or more), but they often don't match the needs of scientific data flows. In this talk, I'll explain how we built the 'librarian' framework with FastAPI, postgres, and Globus to ease this challenge. Designed for the Simons Observatory's petabyte-scale data transfer, I'll cover building reliable web services, flexible development with dependency injection, effective testing with pytest, and deployment using NERSC's Spin. I hope to demystify web and database programming for a scientific audience.