2.0 -//Pentabarf//Schedule//EN PUBLISH DDJTZL@@cfp.scipy.org -DDJTZL Full-stack Machine Learning for Data Scientists en en 20230710T080000 20230710T120000 040000 Full-stack Machine Learning for Data Scientists One of the key questions in modern data science and machine learning, for businesses and practitioners alike, is how do you move machine learning projects from prototype and experiment to production as a repeatable process. In this workshop, we present an introduction to the landscape of production-grade tools, techniques, and workflows that bridge the gap between laptop data science and production ML workflows. We’ll present a high-level overview of the 8 layers of the ML stack: data, compute, versioning, orchestration, software architecture, model operations, feature engineering, and model development. We’ll present a schematic as to which layers data scientists need to be thinking about and working with, and then introduce attendees to the tooling and workflow landscape. In doing so, we’ll present a widely applicable stack that provides the best possible user experience for data scientists, allowing them to focus on parts they like (modeling using their favorite off-the-shelf libraries) while providing robust built-in solutions for the foundational infrastructure. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/DDJTZL/ Classroom 106 Savin Goyal Hugo Bowne-Anderson PUBLISH 8BZN3E@@cfp.scipy.org -8BZN3E Modern Deep Learning with PyTorch en en 20230710T133000 20230710T173000 040000 Modern Deep Learning with PyTorch This tutorial will be aimed at Python programmers new to PyTorch and deep learning. However, even more experienced deep learning practitioners and PyTorch users may be exposed to new concepts and ideas when exploring other open source libraries to extend PyTorch. Throughout this 4-hour tutorial session, attendees will learn how to use PyTorch to train neural networks for image and text classification. We will discuss the individual strengths and weaknesses of deep learning and contrast it with traditional machine learning via libraries such as scikit-learn. We will discuss the PyTorch library in detail, exploring it as a tensor library, automatic differentiation library, and library for implementing deep neural networks. After getting a firm grasp of the PyTorch API, we will introduce additional open source libraries to familiarize attendees with the modern open source stack for deep learning. For instance, we will organize our model training loops using the Lightning Trainer, which will help us reduce boilerplate code and get additional benefits such as model checkpointing, logging, and convenient mixed precision training. Then, we will explore multi-GPU training strategies from DeepSpeed library to accelerate model training if multiple GPUs are available. Note that all model code in this tutorial can be run on a laptop computer, but attendees will also be introduced to free GPU options for this tutorial via Google Colab and Lightning to get the full benefits of this multi-GPU training section. Large language transformers have largely replaced recurrent neural networks for text classification and generation. So, in this tutorial, attendees will learn how to adopt and fine-tune large language models from the HuggingFace transformer library. Lastly, as a bonus, we will also build a deep learning model demo using Gradio and deploy it to the cloud using Lightning. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/8BZN3E/ Classroom 106 Sebastian Raschka PUBLISH CJUYJM@@cfp.scipy.org -CJUYJM Mosaic Magic with Matplotlib en en 20230710T080000 20230710T120000 040000 Mosaic Magic with Matplotlib This tutorial is designed for users of Matplotlib who want to learn more about how to lay out complicated figures. Bring a figure you like that you want to replicate the layout of or one that you'd like to improve. - Introduction (10 mins) - Parts of a figure. What makes up a figure (20 mins) - (Build up to: https://matplotlib.org/stable/gallery/showcase/anatomy.html) - Creating a figure with a single axes (10 mins) - Object oriented model of interacting with axes (20 mins) - e.g. Prefer `ax.plot` over `plt.plot` - Multi axes figures (~1.5 hr): - `subplots` (10 mins) - `subplot_mosaic` (30 mins) - `grid_spec` (20 mins) - `subplot2grid` (5 mins) - `add_axes` (5 mins) - `add_subplot` (5 mins) - Inset and zoomed axes (5 mins) - Layout engines (30 mins) - Introduction (10 mins) - Constrained Layout (10 mins) - Compressed Layout (5 mins) - Tight Layout (5 mins) - Labeling figures (20 mins) - Axis/figure labels (10 mins) - Legends (5 mins) - Colorbars (5 mins) - Subfigures (10 mins) - Conclusions/questions (20 mins) Detailed setup instructions will be provided prior to the event. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/CJUYJM/ Classroom 101 Kyle Sunden PUBLISH LZPDBD@@cfp.scipy.org -LZPDBD 3D Visualization with PyVista en en 20230710T133000 20230710T173000 040000 3D Visualization with PyVista Our tutorial will demonstrate PyVista's latest capabilities and bring a wide range of users to the forefront of 3D visualization in Python. - Use PyVista to create 3D visualizations from a variety of datasets in common formats. - Overview the classes and data structures of PyVista with real-world examples. - Be familiar of the various filters and features of PyVista. - Know which Python libraries are used and can be used by PyVista (meshio, trimesh etc). We see this tutorial catering to anyone who wants to visualize data in any domain, and this ranges from basic Python users to advanced power users. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/LZPDBD/ Classroom 101 Alexander Kaszynski Tetsuo Koyama Bane Sullivan PUBLISH NEUUKG@@cfp.scipy.org -NEUUKG image analysis and visualization in Python with scikit-image, napari, and friends en en 20230710T080000 20230710T120000 040000 image analysis and visualization in Python with scikit-image, napari, and friends Between telescopes and satellite cameras and MRI machines and microscopes, scientists are producing more images than they can realistically look at. They need specialized viewers for multi-dimensional images, and automated tools to help process those images into knowledge. This tutorial is aimed at folks who have some experience in scientific computing with Python, but are new to image analysis. To get the most out of it, they should have done some work with NumPy arrays — no need to be an expert! — but they don't need to know [an image from a pipe](https://en.wikipedia.org/wiki/The_Treachery_of_Images). We will cover the fundamentals of working with images in scientific Python. The tutorial will be split into four parts, of about 45 minutes each, plus breaks: - **Images are just NumPy arrays.** In this section we will cover the basics: how to think of images not as things we can see but numbers we can analyze. - **Changing the structure of images with image filtering.** In this section we will define *filtering*, a fundamental operation on signals (1D), images (2D), and higher-dimensional images (3D+). We will use filtering to find various structures in images, such as *blobs* and *edges*. Putting NumPy, SciPy, scikit-image, and scikit-learn together, we'll show how these fundamental filters are related to modern convolutional neural networks. - **Finding regions in images and measuring their properties.** In this section we will define image segmentation — splitting up images into regions. We will show how segmentation is commonly represented in the scientific Python ecosystem, some basic and advanced methods to do it, and use it to take measurements of segmented objects in our images. We will use scikit-image for some basics, and to make object measurements, but we'll also demonstrate how to use a modern, neural-network-based library to find our imaged objects quickly and get on with our science: measuring the things we've imaged. - **Q&A/Quick tour of advanced features.** This section will be more freestyle and will depend on the audience. We may do a guided tour of other advanced image analysis topics, answer lingering questions about the previous sections, or walk around the room and help people apply what they've learned to their own data of interest. Attendees will leave understanding how to work with images in Python, knowing some of the main libraries that can help them do that, and knowing where to get more help if they need it. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/NEUUKG/ Classroom 202 Juan Nunez-Iglesias Lars Grüter Kira Evans PUBLISH UJBWPQ@@cfp.scipy.org -UJBWPQ Introduction to Numerical Computing With NumPy en en 20230710T133000 20230710T173000 040000 Introduction to Numerical Computing With NumPy NumPy provides Python with a powerful array processing library and an elegant syntax that is well suited to expressing computational algorithms clearly and efficiently. We'll introduce basic array syntax and array indexing, review some of the available mathematical functions in NumPy, and discuss how to write your own routines The tutorial is intended for people new to the scientific Python ecosystem. Previous experience in Python or another programming language is useful but not required. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/UJBWPQ/ Classroom 202 Sandhya Govindaraju PUBLISH B9CHA7@@cfp.scipy.org -B9CHA7 PPML: Machine Learning on data you cannot see en en 20230710T080000 20230710T120000 040000 PPML: Machine Learning on data you cannot see Privacy guarantees are **the** most crucial requirement when it comes to analyse sensitive data. These requirements could be sometimes very stringent, so that it becomes a real barrier for the entire pipeline. Reasons for this are manifold, and involve the fact that data could not be _shared_ nor moved from their silos of resident, let alone analysed in their _raw_ form. As a result, _data anonymisation techniques_ are sometimes used to generate a sanitised version of the original data. However, these techniques alone are not enough to guarantee that privacy will be completely preserved. Moreover, the _memoisation_ effect of Deep learning models could be maliciously exploited to _attack_ the models, and _reconstruct_ sensitive information about samples used in training, even if these information were not originally provided. *Privacy-preserving machine learning* (PPML) methods hold the promise to overcome all those issues, allowing to train machine learning models with full privacy guarantees. This workshop will be mainly organised in **three** main parts. In the first part, we will introduce the main concepts of **differential privacy**: what is it, and how this method differs from more classical _anonymisation_ techniques (e.g. `k-anonymity`). In the second part, we will focus on Machine learning experiments. We will start by demonstrating how DL models could be exploited (i.e. _inference attack_ ) to reconstruct original data solely analysing models predictions; and then we will explore how **differential privacy** can help us protecting the privacy of our model, with _minimum disruption_ to the original pipeline. Finally, we will conclude the tutorial considering more complex ML scenarios to train Deep learning networks on encrypted data, with specialised _distributed federated_ _learning_ strategies. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/B9CHA7/ Classroom 203 Valerio Maggio PUBLISH CQRYUC@@cfp.scipy.org -CQRYUC Introduction to Causal Inference en en 20230710T133000 20230710T173000 040000 Introduction to Causal Inference The tutorial will involve a combination of presentations with open Q&A and hands-on exercises contained in Google Colab notebooks. This session will cover the difference between correlation and causation, the pitfalls of conducting an analysis using observational data, how causal inference can help get around these pitfalls, and examples of common, modern modeling approaches used to conduct causal inference (propensity score matching, estimating causal curves, g-computation, and double ML). After the tutorial, the attendees should have a good foundational understanding of causality and the ability to confidently explore the topic on their own. Causal inference can be a very theory-heavy topic, making it impenetrable to novices. In this tutorial, we'll aim to take a more practical perspective on causal inference, while still occasionally touching on the theory. Tutorial participants are not expected to be familiar with causal inference before attending, but we hope they have an earnest curiosity to learn about it! To get the most out of the session, the participants ought to have experience working with the common python data stack: matplotlib, numpy, pandas, and scikit-learn. Attendees should have some experience conducting classic machine learning modeling using the scikit-learn API, although having advanced machine learning expertise is absolutely not a prerequisite. A very basic understanding of statistics would be helpful (e.g. understanding what a mean is, what confidence intervals represent). PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/CQRYUC/ Classroom 203 Roni Kobrosly PUBLISH CDRJYE@@cfp.scipy.org -CDRJYE Introduction to Python and Programming en en 20230710T080000 20230710T120000 040000 Introduction to Python and Programming To make the most of SciPy it helps to have some basic familiarity with the Python language itself. This beginner level tutorial is designed for folks who are brand-new to Python and may not even have much programming experience. I’ll help you get a working Python installation in which you can launch Jupyter Notebooks, a common tool used in scientific research with Python and in SciPy tutorials. Attendees will learn to work with Python variables, the object interface, loops, conditional statements, function definitions, and the use of basic Python data structures through hands-on exercises inside of Jupyter. Students will use the ipythonblocks library to manipulate an image-like grid of colors for immediate, interactive feedback that makes it easy to tell whether code had the intended effect. My goal is for you to leave the tutorial with a basic familiarity with Python (and a working Python installation) that helps you focus on the scientific libraries you’ll learn about in the other tutorials and throughout SciPy. Familiarity with the usage and features of Jupyter will also help you dive headfirst into other tutorials. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/CDRJYE/ Classroom 103 Matt Davis PUBLISH 7BRY3J@@cfp.scipy.org -7BRY3J Scalable machine learning workloads with Ray AI Runtime en en 20230710T133000 20230710T173000 040000 Scalable machine learning workloads with Ray AI Runtime State-of-the-art machine learning (ML) models require an exponentially increasing amount of compute, making it necessary to utilize the full capacity of your laptop or workstation and beyond to cloud cluster. However, scaling introduces challenges with orchestration, integration, and maintenance. What's more, ML systems change quickly. If you rely on piecemeal solutions to parallelize individual stages of pre-processing, training, inference, and tuning, then stitching these evolving systems together requires a lot of overhead. This context drove the development of [Ray](https://github.com/ray-project/ray): a solution to enable researchers and developers to scale Python code to the full capacity of your laptop or cluster without worrying about implementing complex distributed computing logic. This hands-on tutorial introduces Ray AI Runtime (AIR), an open source, Python-based set of libraries that equip researchers and developers with a toolkit for parallelizing ML workloads. We will use a popular computer vision (CV) use case, image segmentation, to guide participants through common ML workloads, including data pre-processing, model training and fine-tuning, and parallel batch inference. #### Resources - GitHub repository with relevant resources including notebooks, setup instructions, reference implementations to coding exercises, and a README for an overview. - Participants will be able to use a pre-configured compute cluster for the duration of the tutorial. #### Audience - Intermediate-level Python and ML researchers and developers. - Those interested in scaling ML workloads up to full laptop capacity to a cluster. #### Prerequisites - Familiarity with basic ML concepts and workflows. - No prior experience with Ray or distributed computing. - (Optional) [Overview of Ray](https://github.com/ray-project/ray-educational-materials/blob/main/Introductory_modules/Overview_of_Ray.ipynb) notebook as background material. #### Key Takeaways - Understand common challenges and trade-offs when scaling CV pipelines from laptop to cluster. - Hands-on skill in using Ray AIR to scale CV workloads, including model training, fine-tuning, inference. #### Outline Challenges with scaling ML systems (10 min) - Why are distributed systems so important to ML in general and CV pipelines in particular? How does Ray provide the common ML compute scale from laptop to cluster? Hands-on lab 1: Composing CV pipelines (60 min) - Examples introducing Ray Data, Train and Tune libraries. Participants will practice composing components to scale an end-to-end ML workload. - Ray Data - Ingest, shard and preprocess the data. - Ray Train - Train a model on the preprocessed training set. - Ray Tune - Run hyperparameter tuning experiment. - BatchPredictor - Perform batch inference on the test set. (10 minute break) Hands-on lab 2: Model training and fine-tuning (60 min) - Learn about approaches to scaling model training. - Code: Implement transformer model fine-tuning with Ray Train and evaluate performance. (10 minute break) Hands-on lab 3: Batch inference (60 min) - Learn about and evaluate several distributed batch inference design patterns. - Implement distributed batch inference through hands-on coding exercises. - Code: Run batch inference using vision transformer and evaluate performance. Next steps (10 min) - How to get involved with Ray and access further resources. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/7BRY3J/ Classroom 103 Emmy Li Adam Breindel PUBLISH ZCUDYT@@cfp.scipy.org -ZCUDYT Controlling Self-Landing Rockets Using CVXPY en en 20230710T080000 20230710T120000 040000 Controlling Self-Landing Rockets Using CVXPY After giving an introduction to CVXPY at SciPy 2022, we want to follow up and provide an in-depth, worked example about one of the most inquired applications of convex optimization: controlling a self-landing rocket. Indeed, this is also one of the most complex problems to solve and practical usefulness has only recently been achieved. Nevertheless, CVXPY makes it possible to elegantly solve a simplified yet at its core realistic version of the problem. The application serves as a common thread that attendees can work along while being introduced to convex optimization and CVXPY in particular, as well as some of the more advanced features of the library. The tutorial will start by introducing the problem of controlling a self-landing rocket and why it is important. We will then provide an overview of convex optimization and how it can be used to solve this problem. Next, we will dive into the details of CVXPY, starting from a simple hello-world example and gradually moving towards expressing the full problem. Stating the problem should look familiar to anyone who has worked with NumPy before, and only requires high-school level physics knowledge to understand. We have integrated our problem with the Kerbal Space Program, which fits the theme of our tutorial nicely. It allows us to make our problem gradually more realistic by incorporating conditions such as drag, fuel usage, and wind. We will run the scripts written by the attendees to see if it manages to land a rocket safely. As we solve the problem, we will showcase some of the more advanced features of CVXPY, including DPP and CVXPYgen, which can give a significant speedup in practice. By the end of the tutorial, attendees will have a thorough understanding of how to use CVXPY to solve complex optimization problems, and how to apply it to real-world problems such as controlling a self-landing rocket. No prior knowledge of convex optimization is assumed, making this tutorial accessible to beginners in the field. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/ZCUDYT/ Classroom 104 Philipp Schiele Steven Diamond Eric Sager Luxenberg PUBLISH NFWZXD@@cfp.scipy.org -NFWZXD How the Little Jupyter Notebook Became a Web App: Managing Increasing Complexity with nbdev en en 20230710T133000 20230710T173000 040000 How the Little Jupyter Notebook Became a Web App: Managing Increasing Complexity with nbdev Our tutorial should appeal to scientists and software developers alike. We hope to convince you that web applications are excellent tools for improving the accessibility of scientific data and software and provide you with the know-how to develop one that accommodates growth and collaboration. The structure of the tutorial is based on a true story about a little Jupyter Notebook… One day, a research scientist created the Notebook to do some exploratory development. After a while, that Notebook grew into a reusable workflow for creating a helpful visualization the scientist often ran for different parameters. The scientist eventually recognized that their workflow might be worth sharing, so they started working with a software developer to help the little Notebook grow into a web application. At first, the developer replaced hardcoded inputs into interactive ipywidgets, and used Voila` to hide the code cells from users. That was the day the little Notebook became a dashboard, but its journey didn’t stop there. Over time, the researcher had new ideas about features they wanted to add, so the developer transformed the dashboard into a tab-based web application that could accommodate more steps with rich instructions. But there was a problem. The Notebook started experiencing growing pains. It contained more code than was comfortable. The developer made the notebook feel better by offloading some of the code into python modules. This worked well for logic, but as the application grew more complex, it was important to develop nested widget components in the visual Notebook environment. At first, the developer coded views in extra notebooks, and then copy-pasted the code into the python module, but this became laborious and confusing. One day, the developer started to write a tool that would export code cells from the notebooks into python modules. That way, the developer could code entirely in notebooks, and they could leave in all the markdown and code cells that documented what they were thinking as they designed the tool. That was the day that the little Notebook became a literate Notebook family. Not long after, the developer was listening to the Talk Python to Me podcast, and heard someone mention a tool called nbdev. The tool was just like the one the developer had made, except it had many more useful features, like notebook-friendly git commits and merges. Eureka! Finally, the developer could accommodate increasing complexity with simple tools. When the developer gave the Notebook family back, the researchers were able to maintain it themselves, without having to download scary IDEs, extensions, or environments. And they all lived happily ever after. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/NFWZXD/ Classroom 104 Nicole Brewer Ludovico Bianchi PUBLISH VBZ9PN@@cfp.scipy.org -VBZ9PN Building better data structures, APIs and configuration systems for scientific software using Pydantic en en 20230710T080000 20230710T120000 040000 Building better data structures, APIs and configuration systems for scientific software using Pydantic One of the most controversial design choices of Python is the use of dynamic types. Dynamic types of variables can often lead to confusion for beginners, but also for experts it is a common sources of hard-to-find bugs. For this reason the concept of type annotations has been introduced later in the language to allow for static code analysis and more detailed source code documentation. Pydantic is a Python library that makes use of these type annotations to parse and validate types for class based data structures. In the past years Pydantic has gained tremendous popularity among web developers and is now the most widely used data validation library for Python. In this tutorial we show how the use of Pydantic can help to build better data structures, APIs and configuration systems for scientific Python packages as well. In many cases the validated types lead to a more pleasant user experience as well as more robust and easier to maintain code. In the first block we introduce the basics of the library such as the concept of Pydantic models, type annotations and atomic types such as int, float, str etc. We show how types are parsed and how models can be configured to forbid extra attributes. At the end of the block participants will try to implement the first Pydantic model and explore basic configuration settings. We then proceed with the introduction of more complex types, such as typed dicts, Enums and date time objects. We will also cover custom types, which can bet used to build nested models. Then we introduce the basics of type validation for multiple scenarios, such as pre and post init and root validation. At the end of this block we will cover the topic of dynamic model creation. In the following hands-on session participants will implement a more complex Pydantic model representing the response from a weather data API at multiple levels of difficulty. The subsequent block will be dedicated to serialization and deserialization of Pydantic models. We will first motivate the need and then introduce the JSON and YAML data formats. We will show how to support custom types for JSON serialization and give an overview of configuration options related to serialization. We will conclude with performance remarks for serialization a large number of model. In the corresponding hands-on exercise participants will use the weather data structure and build a small configurable data processing pipeline which visually compares the weather forecast data from different models. Finally we will give a summary and key takeaways of the tutorial and recommend additional resources for learning Pydantic. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/VBZ9PN/ Classroom 105 Nick Langellier Axel Donath PUBLISH RKV3PZ@@cfp.scipy.org -RKV3PZ Meet your coding best friend: VS Code💖 - A hands-on tutorial on how to get the most out of the world’s most popular Python editor en en 20230710T133000 20230710T173000 040000 Meet your coding best friend: VS Code💖 - A hands-on tutorial on how to get the most out of the world’s most popular Python editor After this tutorial you will walk away with a fully equipped VS Code editor, ready to work on your next project or contribute to your favorite Scientific Python library. We will also cover tips and tricks for data science and visualization and some advanced features you may not have heard of yet. We will cover the following topics: **The basics: VS Code editor and Python extension overview**. We will show you how to set up your editor, where to find the most useful menus and settings, and how to set up your workspace to start developing. We’ll explain how to find and install our favorite extensions for Python, how to use VS Code with Git. **Scientific Python development tips and tricks**. In the second hour, we will cover how to navigate and test your Python code like a pro. We will also cover some data science tools that will help you run your favorite data analysis projects directly in VS Code, as well as some GitHub features to test and document your code. **Advanced development Part I: Work where you want to**. The third hour of the tutorial explains how to use the remote development extensions pack to hook up VS Code to a remote resource, like a powerful VM in the cloud, a local Linux instance, a Docker instance or GitHub Codespaces. **Advanced development Part II: Tools that make you look like you know magic**. It’s time to have some fun and try out cool features for remote collaboration and code generation that will make you feel like the future is here. **Wrap-up & Epilogue**. We’ll recap what we’ve learned in this tutorial and share reading and learning materials to help you on your VS Code journey. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/RKV3PZ/ Classroom 105 Guen Prawiroatmodjo Sarah Kaiser Leopold Talirz PUBLISH 8TAA7K@@cfp.scipy.org -8TAA7K Idiomatic Pandas en en 20230711T080000 20230711T120000 040000 Idiomatic Pandas Are you confused or frustrated with Pandas? Or maybe your own Pandas code when you come back to it later, you find it confusing or difficult to work with. I've taught Pandas to thousands in Corporate settings, Universities, and Virtually. I've also seen the bad code that my students write and have strong opinions on how to correct it. This workshop assumes you know some Pandas and want to apply idiomatic constructs to existing code. There will be some lecture and then breakout time to apply the constructs on your own: We will cover * Types * Chaining * Mutation * Aggregation * Debugging PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/8TAA7K/ Classroom 106 Matt Harrison PUBLISH MQQJKG@@cfp.scipy.org -MQQJKG Advanced Dask Tutorial en en 20230711T133000 20230711T173000 040000 Advanced Dask Tutorial Dask is a popular Python library for scaling and parallelizing Python code on a single machine or across a cluster. It provides familiar, high-level interfaces to extend the SciPy ecosystem (e.g. NumPy, pandas, scikit-learn) to larger-than-memory or distributed environments, as well as lower-level interfaces for parallelizing custom algorithms and workflows. In this tutorial, we’ll cover advanced features of Dask like applying custom operations to Dask DataFrames and arrays, inspecting the internal state of clusters, debugging distributed computations, diagnosing performance issues, and more. Attendees should walk away with a deeper understanding of Dask’s internals, an introduction to more advanced features, and ideas of how they can apply these features effectively to their own data-intensive workloads. Basic Dask experience is required, though knowledge of Dask’s internals is not. This hands-on tutorial is intended for existing or aspiring Dask users looking to gain a deeper understanding of more intermediate and advanced topics. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/MQQJKG/ Classroom 106 Naty Clementi James Bourbeau Julia Signell Charles Blackmon-Luca PUBLISH XBUC8S@@cfp.scipy.org -XBUC8S Thinking in arrays en en 20230711T080000 20230711T120000 040000 Thinking in arrays Array-oriented programming is a paradigm in its own right, challenging us to think about problems in a different way. From APL in 1966 to NumPy today, most users of array-oriented programming are scientists, analyzing or simulating data. This tutorial focuses on the thought process: all of the problems are to be solved in an imperative way (for loops) and an array-oriented way. Matlab will be used for plotting, but all plotting commands will be given (not prerequisites). We'll alternate between short lectures and small group projects (3‒4 people each), in which tutors will be available for help, followed by a guided tour through solutions, alternatives, and trade-offs. Here is a general outline: **0:00‒0:20 (20 min):** Array-oriented programming as a paradigm: APL, SPEAKEASY, IDL, MATLAB, S, R, NumPy. Overview of basic and advanced slicing, broadcasting, and dimensional reduction. Powerful concept: element indexing is function application and advanced slicing is function composition. **0:20‒0:40 (20 min):** Project 1: Conway's Game of Life. Calculating number of neighbors and updating the board "all at once." **0:40‒0:55 (15 min):** Break **0:55‒1:15 (20 min):** Guided discussion of solutions to Project 1. **1:15‒1:35 (20 min):** Array-oriented programming and the "iteration until converged" problem. How to update arrays in which some elements have converged and others haven't. **1:35‒1:55 (20 min):** Project 2: evaluating a decision tree, by walking over each node individually (as in a computer science class) and by million-ball Plinko! (how Scikit-Learn actually does it). **1:55‒2:10 (15 min):** Break **2:10‒2:30 (20 min):** Solutions to Project 2. **2:30‒2:45 (15 min):** Demo: Mandelbrot (fractal) picture, computed 11 different ways: Python, NumPy, C++ (pybind11), Cython, Numba imperative, Numba vectorized, CuPy, CuPy with custom CUDA, Numba-CUDA, JAX-CPU, and JAX-GPU. Discussion of performance and trade-offs. **2:45‒3:05 (20 min):** Non-rectilinear (ragged) arrays and arrays of arbitrary data structures: Apache Arrow and Awkward Array. **3:05‒3:25 (20 min):** Project 3: a big, ragged dataset: computing lengths of taxi trips from polylines with varying numbers of edges. Since this is a big dataset, we'll also look at ways to scale it up with Dask. **3:25‒3:40 (15 min):** Break **3:40‒4:00 (20 min):** Solutions to Project 3. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/XBUC8S/ Classroom 101 Jim Pivarski PUBLISH LJQPVT@@cfp.scipy.org -LJQPVT SymPy Introductory Tutorial en en 20230711T133000 20230711T173000 040000 SymPy Introductory Tutorial SymPy is a pure Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python. SymPy can be used in a wide array of applications. This includes basic usage as an interactive calculator, symbolically modeling problems in physics and engineering, generating fast numeric code, and use in a Python library representing custom symbolic objects. Anyone interested in learning how to get started using SymPy for any such applications should attend this tutorial. This tutorial is a beginner level tutorial and only requires knowledge of how to use Python. Knowledge of mathematics up to basic calculus is recommended. More advanced mathematical topics will be explained as part of the tutorial. Knowledge of other Python libraries such as NumPy is NOT required. There will be a short section near the end on how to interface SymPy with other libraries such as NumPy, but the majority of the tutorial does not make use of any additional libraries. This tutorial will cover the basics of how to use SymPy, and will also touch on some advanced topics. We will start by discussing the basics of how to build mathematical expressions with SymPy and manipulate them. We will look at how to avoid some of the more common pitfalls and gotchas when using the SymPy. We will then move onto the most common functions in SymPy such as simplification functions, solvers, functions for doing operations from calculus such as differentiation and integration, and matrices. Finally, as time permits, we will look into more advanced topics, such as code generation, extending SymPy, interfacing with other libraries such as NumPy, and additional SymPy submodules. After attending this tutorial, attendees will be able to start using SymPy to solve their own problems. They will also be armed with the knowledge of how to discover additional more specific functionality in SymPy that may be required for their particular use-case. We will expect tutorial attendees to have the tutorial materials installed on their computers prior to the tutorial. This way we will not waste time in the beginning getting things installed. The tutorial will also be available online using either Binder or JupyterLite for those that do not wish to install things locally. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/LJQPVT/ Classroom 101 Sangyub Lee Aaron Meurer Anutosh Bhat PUBLISH 7NLG3F@@cfp.scipy.org -7NLG3F Explore generative models in AI with Keras en en 20230711T080000 20230711T120000 040000 Explore generative models in AI with Keras In this tutorial, we will explore the powerful Keras library and the world of generative models in AI. We will begin with a brief introduction to Keras, its history, and its value in creating neural networks. We will then dive into the Keras training pipeline, exploring sequential, functional, and custom models, optimizers, loss and metrics, and the training API. We will also cover Keras extended modules for NLP, CV, and GNN, and walk through an end-to-end example to create and optimize a model. In the second part of the tutorial, we will specifically focus on image generative model stable diffusion architecture. We will explain stable diffusion, demonstrate a latent space walkthrough, and generate images using a colab example. Additionally, we will focus on image inpainting and teaching stable diffusion new concepts, this is called textual inversion. Finally, we will explore how generative models work in NLP, specifically focusing on GPT structure and GPT 2, BART, and the mobile playbook. We will demonstrate XLA compilation and show how general support for text generation using one API can be achieved. By the end of this tutorial, attendees will have a solid understanding of Keras and generative models and how they can be used to create powerful AI applications. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/7NLG3F/ Classroom 202 Divyashree Shivakumar Sreepathihalli Chen Qian PUBLISH F3HAUQ@@cfp.scipy.org -F3HAUQ Resampling and Monte Carlo Methods in SciPy.stats en en 20230711T133000 20230711T173000 040000 Resampling and Monte Carlo Methods in SciPy.stats Scientists and engineers often seek to answer questions of the following forms. 1. Is my sample drawn from this hypothesized distribution? 2. Are my samples drawn from the *same* distribution? 3. Based on these samples, what can I infer about the populations from which they were drawn? Common statistical procedures used to answer questions of these forms include: 1. the one-sample t-test ("Is my sample drawn from a distribution with population mean `m`?"), 2. the two-sample t-test ("Are my two samples drawn from distributions with the same population mean?"), and 3. the confidence interval of the mean ("Given my sample, what can I say about the true value of the population mean?"). Such procedures are developed under technical assumptions (e.g., the samples were drawn from normally-distributed populations) that make the mathematics tractable, yet in practice, these assumptions can never be met exactly. Fortunately for science, the conclusions drawn from the procedures above are relatively insensitive to deviations from these assumptions… except when they’re not! One solution is to abandon frequentist statistics in favor of another paradigm (Bayesian), but the approach suggested by this tutorial is to remove the assumptions, reduce reliance on the analytical approximations, and instead use computers to approximate (or even exactly calculate) responses to the original questions. This idea will lead us to three techniques: 1. Monte Carlo tests (`scipy.stats.monte_carlo_test`) 2. Permutation tests (`scipy.stats.permutation_test`) 3. The Bootstrap (`scipy.stats.bootstrap`) For many of the same reasons that arithmetic (sums and differences) seems simpler than calculus (integrals and derivatives), these techniques are relatively easy to grasp. Likewise, just as computational methods for integration, equation solving, and optimization can solve a wider variety of problems than analytical approaches, these computational statistical techniques are comparatively flexible and easy to apply. During this tutorial, participants will write their own code to execute fundamental resampling and Monte Carlo algorithms and compare the results of their code against the equivalent functions in SciPy. They will apply their new understanding of SciPy's `monte_carlo_test`, `permutation_test`, and `bootstrap` functions to reproduce and extend the capabilities of SciPy's other statistics functions (e.g. to small samples, to discrete distributions). Through this tutorial, participants will improve their ability to apply existing statistical procedures to a given situation and gain the ability to *create* customized statistical procedures for demanding applications. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/F3HAUQ/ Classroom 202 Matt Haberland Albert Steppi PUBLISH ALSYBR@@cfp.scipy.org -ALSYBR Data of an Unusual Size: A practical guide to analysis and interactive visualization of massive datasets en en 20230711T080000 20230711T120000 040000 Data of an Unusual Size: A practical guide to analysis and interactive visualization of massive datasets "Big data" refers to any data that is too large to handle comfortably with your current tools and infrastructure. As the leading language for data science, Python has many mature options that allow you to work with datasets that are orders of magnitudes larger than what can fit into a typical laptop's memory. This tutorial will help you understand how large-scale analysis differs from local workflows, the unique challenges associated with scale, and some best practices to work productively with your data. By the end, you will be able to answer: - What makes some data formats more efficient at scale? - Why, how, and when (and when not) to leverage parallel and distributed computation (primarily with Dask) for your work? - How to manage cloud storage, resources, and costs effectively? - How interactive visualization can make large and complex data more understandable (primarily with hvPlot)? - How to comfortably collaborate on data science projects with your entire team? The tutorial focuses on the reasoning, intuition, and best practices around big data workflows, while covering the practical details of Python libraries like Dask and hvPlot that are great at handling large data. It includes plenty of exercises to help you build a foundational understanding within three hours. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/ALSYBR/ Classroom 203 Pavithra Eswaramoorthy Dharhas Pothina Christopher Ostrouchov PUBLISH QXAYRM@@cfp.scipy.org -QXAYRM Xarray: Friendly, Interactive, and Scalable Scientific Data Analysis en en 20230711T133000 20230711T173000 040000 Xarray: Friendly, Interactive, and Scalable Scientific Data Analysis Xarray is an open-source Python project that makes working with complex, multi-dimensional arrays elegant, intuitive, and efficient. Real-world datasets are often a collection of many related variables on a common grid rather than raw numbers. Such datasets are common in the disciplines of earth science, astronomy, biology, and finance. These datasets are more than just arrays of values: they have labels which describe how array values map to locations in dimensions such as space and time and metadata that describes how the data was collected and processed. Xarray embraces this complexity and enables users to use dataset metadata such as dimension names and coordinate labels to easily analyze, manipulate, and visualize their datasets. For example, the Pandas-inspired Xarray label-based syntax `temperature.sel(place=”Boston”)` is more intuitive and less error-prone compared to NumPy syntax: `temperature[0]`. This hands-on tutorial will introduce data scientists already familiar with Xarray to more advanced concepts. All material will be presented via Jupyter Notebooks, with participants actively coding and performing exercises to solidify understanding of key concepts. The tutorial intersperses teaching intermediate to advanced Xarray concepts with increasingly complex real-world data analysis tasks. The participant learning goals for the tutorial are to: 1. Effectively use Xarray’s powerful multidimensional indexing operations 2. Become familiar with important parts of Xarray’s computational API 3. Understand how to extend Xarray’s built-in capabilities with custom computation functions 4. Understand how Xarray fits in with other array types in the scientific Python ecosystem The structure of our tutorial is based on our extensive experience teaching Xarray over the past few years, including numerous similar tutorials at international conferences like SciPy, as well as in formal classes taught at the National Center for Atmospheric Research and the University of Washington. The tutorial will be presented using [Nebari](scipy.quansight.dev), which will facilitate interactive computation and a consistent computational environment without requiring participants to install any software. Tutorial material will be available online ([link]( https://tutorial.xarray.dev/workshops/scipy2023/README.html)) and we will ensure that proper environment files are available for participants that prefer running the tutorial locally. Participants are expected to have some familiarity with Jupyter notebooks, NumPy, Pandas, and Xarray. No specific domain knowledge (e.g. geoscience) is required to effectively participate in this tutorial. If you are new to Xarray then please go through last year’s tutorial ([link](https://tutorial.xarray.dev/workshops/scipy2022/README.html#scipy-2022)) prior to attending, as our tutorial will assume attendees have a working understanding of these basic concepts. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/QXAYRM/ Classroom 203 Deepak Cherian Thomas Nicholas Anderson Banihirwe Jessica Scheick Don Setiawan Scott Henderson Negin Sobhani PUBLISH Y8PPAA@@cfp.scipy.org -Y8PPAA SciPy Welcome Reception en en 20230711T183000 20230711T203000 020000 SciPy Welcome Reception PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/Y8PPAA/ Enthought - 200 W Cesar Chavez St 200 W Cesar Chavez PUBLISH NDYWUR@@cfp.scipy.org -NDYWUR Power up your work with compiling and profiling en en 20230711T080000 20230711T120000 040000 Power up your work with compiling and profiling Have you ever heard of Numba? It is (mainly) a JIT (Just-In-Time) compiler to make your math-heavy Python code run faster under certain conditions. Most people found all of it to be like a mystery - It sounds like magic, but how does it work? Under what conditions does it work? And because of it, new users found it hard to start using it and it requires a steep learning curve to get the hang of it. This workshop requires no prior experience. However, it will be most beneficial to those who are working with numerical data, like data scientists and researchers. We also expect participants have no knowledge of how compilers work and not much understanding of how CPython works. Through exercises, we will explore in what situation Numba works, when it does not and the reason why. We will also look at some cases where we can make Numba works by changing a few things in your code. Hopefully, by finishing the workshop, you will have a better understanding of how Numba works before you even start using it. This knowledge can save you some time on try and error, making your experience in using it better. **What Attendees will Learn** By the end of the workshop, you will have some understanding of what Numba is and how it speeds up your Python code. You will also have a better idea about the limitation of Numba and when it does not help. You may also know how to change your code to make it benefit from the speeding up of Numba. You will also learn some troubleshooting skills and where to look for help if got stuck in the future. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/NDYWUR/ Classroom 103 Cheuk Ting Ho PUBLISH PTB7DU@@cfp.scipy.org -PTB7DU Python for answering geospatial questions: exploring social inequity in our communities en en 20230711T133000 20230711T173000 040000 Python for answering geospatial questions: exploring social inequity in our communities Recent advances in geospatial analysis and the availability of digital maps have revealed the importance of urban form and built infrastructure as fundamental to understanding the vulnerabilities and vitality of global and local cities. Learning to write Python scripts (assuming low or no-level prior experience) we will discover morphometrics and what they reveal about the urban form of cities. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/PTB7DU/ Classroom 103 bonny p mcclain PUBLISH GQ7PG3@@cfp.scipy.org -GQ7PG3 An Introduction to Cloud-Based Geospatial Analysis with Earth Engine and Geemap en en 20230711T080000 20230711T120000 040000 An Introduction to Cloud-Based Geospatial Analysis with Earth Engine and Geemap The Earth is constantly changing, which creates significant challenges for the environment and human society. To tackle these challenges on a global scale, the Earth science community relies heavily on geospatial datasets that are collected through various means, such as satellite, aerial, and mobile sensors. However, the explosive growth of geospatial datasets over the past few decades has overwhelmed the Earth science community's capacity for storage, analysis, and visualization. Fortunately, the advent of cloud-computing platforms (e.g., Google Earth Engine) has made it possible to access, manipulate, and analyze large volumes of geospatial data on-the-fly. In recent years, Earth Engine has become increasingly popular in the geospatial community and has enabled numerous Earth science applications at local, regional, and global scales. The geemap Python package is built upon the Earth Engine Python API and open-source mapping libraries. It allows Earth Engine users to interactively manipulate, analyze, and visualize geospatial big data in a Jupyter environment. Since its creation in April 2020, geemap has received over [2,500 GitHub stars](https://github.com/giswqs/geemap/stargazers) and is being used by over [800 projects](https://github.com/giswqs/geemap/network/dependents) on GitHub. More than [130 Jupyter notebook examples](https://geemap.org/tutorials/) and an [open-access book](https://book.geemap.org/) are available for learning geemap. This tutorial consists of seven 30-minute sessions and three 10-minute breaks. During each hands-on session, the attendees will walk through Jupyter notebook examples on Google Colab with the instructors. At the end of each session, they will complete a hands-on exercise to apply the knowledge they have learned. The topics that will be covered in this tutorial include: (1) Introduction to Earth Engine and geemap; (2) Using Earth Engine data; (3) Visualizing Earth Engine data; (4) Analyzing Earth Engine data; (5) Exporting Earth Engine data; (6) Creating satellite timelapse animations; and (7) Developing and deploying interactive Earth Engine web apps. This tutorial is intended for scientific programmers, data scientists, geospatial analysts, and concerned citizens of Earth. Attendees should have a basic understanding of Python and the Jupyter ecosystem. Familiarity with Earth science and geospatial datasets is not necessary, but it will be helpful. For more information about Earth Engine and geemap, visit https://earthengine.google.com and https://geemap.org. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/GQ7PG3/ Classroom 104 Steve Greenberg Qiusheng Wu PUBLISH YHEYVY@@cfp.scipy.org -YHEYVY A Hands-on Introduction to Production-grade Data Science Orchestration with Flyte en en 20230711T133000 20230711T173000 040000 A Hands-on Introduction to Production-grade Data Science Orchestration with Flyte # Background This tutorial interleaves lecture-style content and coding exercises to give data scientists, machine learning engineers, and data engineers hands-on experience with Flyte. Flyte is an open source workflow orchestrator that has a Python SDK for writing and scheduling execution graphs in a type-safe, reproducible manner. The topics and concepts covered in this tutorial are transferable to other similar orchestration tools, or would be useful for anyone who wants to build their own orchestrator. We will anchor the tutorial to five challenges of model development and deployment: scalability, data quality, reproducibility, recoverability, and auditability. Using Flyte, we’ll see how to address these challenges and abstract them out to give you a broader understanding of how to overcome them. # Main Content First I’ll define and describe what these five challenges mean in the context of model development. Then I’ll dive into the ways in which Flyte provides solutions to them, taking you through the reasoning behind Flyte’s data-centric and ML-aware design. We'll cover: - **Flyte tasks and workflows**: the building blocks for expressing execution graphs. - **Dynamic workflows**: for defining execution graphs at runtime. - **Map tasks**: Scale embarrassingly parallel workflows. - **Plugins**: Extend Flyte's core functionality. - **Type System**: See the benefits of static type safety. - **DataFrame Types**: Validate dataframe-like objects at runtime. - **Reproducibility**: Containerize and harden your execution graph. - **Caching**: Don't waste precious compute resources re-running nodes. - **Recovering Executions**: Build fault-tolerant pipelines. - **Checkpointing**: Checkpoint progress within a node. - **Flyte Decks**: Create rich static reports associated with your tasks. Attendees will learn how Flyte distributes and scales computation, enforces static and runtime type safety, leverages Docker to provide strong reproducibility guarantees, implements caching and checkpointing to recover from failed model training runs, and ships with built-in data lineage tracking for full data pipeline auditability. # Wrap-up The end of the tutorial will provide a summary of all the main learnings, point to resources to learn more, and a discussion for attendees to address their questions. # Resources - [Flyte Repo](https://github.com/flyteorg/flyte) - [Flyte Docs](https://docs.flyte.org/en/latest/) - [Scipy 2022 Flyte talk](https://www.youtube.com/watch?v=EykWaiHHDNg) - [Scipy 2020 Pandera talk](https://www.youtube.com/watch?v=PxTLD-ueNd4) PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/YHEYVY/ Classroom 104 Niels Bantilan PUBLISH VKXXNH@@cfp.scipy.org -VKXXNH hvPlot and Panel: Visualize all your data easily, from notebooks to dashboards en en 20230711T080000 20230711T120000 040000 hvPlot and Panel: Visualize all your data easily, from notebooks to dashboards Python offers many powerful visualization tools, each with their own strengths and advantages, but few people have the time and interest to learn all the different APIs required to use these different tools. Luckily, a de-facto standard API for data plotting has emerged in the Pandas .plot() API, which is now supported by many different plotting packages. In this tutorial, you will learn how to use hvPlot, a high-level interactive plotting library that exposes the power of Bokeh, Matplotlib, Plotly, Datashader, and Cartopy using the same .plot API you may already know from using Pandas or Xarray's plotting interface. We'll also show you how to turn nearly any expression you can write with that API into a web app with plots and tables by simply substituting widgets for any parameters you want users to be able to select. Thanks to the HoloViz tools on which hvPlot is built, the resulting apps can easily handle big data (up to billions of rows on an ordinary laptop), remote data (either in Jupyter or in standalone apps), streaming data (using streaming dataframe libraries), geographical data (building on the geoscience software stack), and multidimensional data (using Xarray). hvPlot's high-level interface should be sufficient for nearly all of the common data-exploration and data-analysis tasks you want to do with Pandas or Xarray, but in keeping with the HoloViz philosophy of "shortcuts rather than dead ends", we'll also show you how and when to drop down to lower-level APIs when you need to, such as when building more complex apps using Panel, doing complex graphical data calculations using Datashader, or integrating plotting and interactivity into your own libraries using Param and HoloViews. With the techniques you learn in the hands-on exercises in this tutorial, you'll get the tools and know-how to effectively explore, analyze and visualize simple or complex, small or large, and static or dynamic data easily, concisely, and reproducibly. The resulting visualizations and apps can be shared as static images, simple HTML documents with limited interactivity, HTML+WASM documents with full Python-backed interactivity, or as Python apps deployed on a remote server. We expect participants to have previously used some sort of plotting tool and to be comfortable with Python and at least one array-based library (Numpy, Pandas, Xarray, CuPy, cuDF, Dask, etc.). PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/VKXXNH/ Classroom 105 Sophia Yang James A. Bednar PUBLISH C9QZXU@@cfp.scipy.org -C9QZXU Interactive data visualization with Bokeh en en 20230711T133000 20230711T173000 040000 Interactive data visualization with Bokeh Bokeh is a Python library for creating interactive data visualizations. Bokeh allows you to create plots that can be displayed in a web browser, without needing to write HTML and JavaScript. In development for over 10 years, Bokeh has become a core tool for Python data science workflows, used for both exploratory analysis and in presentations. It is actively used in scientific domains including bioscience, geoscience, and astrophysics. Moreover, other useful libraries in the PyData ecosystem, like Dask, ArViz, and the Holoviz tools, build custom applications and workflows with Bokeh. In this tutorial, you’ll learn everything you need to know to create beautiful and powerful interactive plots from scratch. We’ll start by introducing core Bokeh concepts, creating simple static plots like line and bar charts, and customizing them. We’ll then gradually introduce layers of interactivity, create specialized plots like geographic maps, and discuss new features like contour plots. By the end, you will be able to create a complete interactive dashboard using Bokeh. This tutorial is presented by Bokeh core team members and is fully hands-on with several examples and exercises in every section. We hope to enable more people, especially scientists and tool developers, to create pretty yet powerful visualizations. PUBLIC CONFIRMED Tutorial https://cfp.scipy.org/2023/talk/C9QZXU/ Classroom 105 Pavithra Eswaramoorthy Ian Thomas Bryan Van de Ven Timo Metzger Victoria Adesoba PUBLISH DF8PVV@@cfp.scipy.org -DF8PVV Out-Performing NumPy is Hard: When and How to Try with Your Own C-Extensions en en 20230712T104500 20230712T111500 003000 Out-Performing NumPy is Hard: When and How to Try with Your Own C-Extensions While it is well known that C-extensions can improve the performance of Python programs, writing C-extensions that improve the performance of NumPy array operations is different. Many NumPy functions employ highly optimized C routines, some of which take advantage of low-level processor optimizations. In most cases, just writing Python that calls NumPy is faster than a custom C extension. However, for routines that are sufficiently narrow in scope, there are opportunities for optimization. This presentation offers principles to help determine if a routine, implemented as a C-extension, might outperform related NumPy routines called from Python. Along the way, Python project setup, and the basics of the NumPy C API, will be introduced. A narrow use-case of the ``np.nonzero()`` function will be implemented in C as an example: rather than returning all indices of all non-zero values for all dtypes and dimensionalities (as ``np.nonzero()`` does), this new function, ``first_true_1d()``, will return only the index of the first-encountered non-zero value for one-dimensional Boolean arrays. The performance of this far simpler routine, and why it sometimes cannot out-perform ``np.nonzero()``, will be examined. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/DF8PVV/ Amphitheater 204 Christopher Ariza PUBLISH VUFGS8@@cfp.scipy.org -VUFGS8 Can There Be Too Much Parallelism? en en 20230712T112500 20230712T115500 003000 Can There Be Too Much Parallelism? Numerical Python libraries such as NumPy, SciPy, and PyTorch can run computations on multiple CPU cores. These libraries expose a wide range of programming interfaces to control parallelism. These interfaces include environment variables, library-specific APIs, and context managers such as threadpoolctl. While reviewing the interfaces for controlling parallelism, we will learn about the many parallel primitives used in these libraries. We will cover lower-level primitives such as pthreads or OpenMP and higher-level primitives such as Python's multithreading and multiprocessing modules. Libraries that require lower-level parallel primitives need to go through a compilation step with languages and tools such as Numba, Cython, C++, or Rust. When we use multiple forms of parallelism, controlling how many cores your program uses is essential to prevent oversubscription. We will learn how libraries such as Dask, Ray, and scikit-learn handles mix their parallelism with user-provided parallel routines. Finally, we will zoom out to see the overall landscape for controlling parallelism and highlight possible paths to improve the user and developer experience. This is an intermediate talk for software and machine learning engineers that want to understand and configure parallelism in the PyData stack. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/VUFGS8/ Amphitheater 204 Thomas J. Fan PUBLISH 3NBFHV@@cfp.scipy.org -3NBFHV Scientific Python: from `__init__` to `__call__` en en 20230712T131500 20230712T134500 003000 Scientific Python: from `__init__` to `__call__` The Scientific Python project's vision is to help pave the way toward a vibrant, unified, and collaborative scientific Python community. It focuses its efforts along two primary axes: _(i)_ to create a joint community around scientific Python projects and _(ii)_ to support maintainers by building cross-cutting technical infrastructure and tools. Last year we launched the project with new websites, a Hugo web theme, a social media campaign, and a collaborative coordination process similar to PEPs called SPECs. This year, we are fortunate to have received [funding from CZI](https://scientific-python.org/grants/community_and_communications_infrastructure/) for the continued development, maintenance, and support of web and documentation themes, as well as other community infrastructure, in collaboration with Quansight. With the community and communication infrastructure having support for the next few years, we are able to focus more on technical topics and the SPECs. As a first project, we are [funded to work on improving sparse *array*](https://scientific-python.org/grants/sparse_arrays) (vs matrix) semantics in SciPy with the goal of removing sparse *matrices* and, eventually, also NumPy *matrices* from several ecosystem libraries. In line with our philosophy of continually working with the community and incorporating their feedback, we hosted the first of several [Sparse Summits](https://scientific-python.org/summits/sparse/)—virtual meetings to identify sparse array needs in ecosystem libraries. This project spans multiple core projects, including numpy, scipy, scikit-image, networkx, scikit-learn, and many of the packages built on top of these libraries. In addition to the sparse summit, we have hosted a [domain stack summit](https://scientific-python.org/summits/domain-stacks/), to discuss domain-specific umbrella projects that host several others, as well as the first [annual developer summit](https://scientific-python.org/summits/developer/). This in-person workshop brought together over 30 community members for a week-long, collaborative sprint, and tackled topics including build & testing systems, continuous integration infrastructure, release management tools, and community management. Finally, we will update the community on our progress on the [decadal plan](https://scientific-python.org/grants/planning_next_decade/). Our efforts thus far have already culminated in joint efforts to develop tools and shared infrastructure that will positively impact the whole ecosystem. And, while there is still a long road ahead, we are excited to continue preparing the ecosystem for the next decade of scientific computing in Python. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/3NBFHV/ Amphitheater 204 Juanita Gomez PUBLISH KXWZJY@@cfp.scipy.org -KXWZJY Beyond Bits & Qubits: Effective Open Source Community Management in Quantum Computing en en 20230712T135500 20230712T142500 003000 Beyond Bits & Qubits: Effective Open Source Community Management in Quantum Computing basic outline of proposed talk: ### 1. Context This section will provide a brief introduction to Qiskit (https://qiskit.org) as an open-source package and some of the challenges we’ve faced in maintaining and growing our community. ### 2. The Academic Element One of the unique aspects of maintaining an open-source project in a scientific field is the closer relationship to academia compared to other open-source software. This can pose unique challenges, as researchers often have different goals, mindsets and working culture when it comes to publishing code, which doesn’t always work well with traditional open-source ways of working. We continually face these conflicts in Qiskit, so in this section we will talk through some of the effective ways we’ve found to address these differences through education and the development of of the Qiskit Ecosystem (https://qiskit.org/ecosystem). ### 3. Clearly Defined Spaces Defining the mechanisms for *how* different members of the community interact is a subtle yet crucial aspect of community management that requires careful planning. Whether it’s clearly defined issue templates, organised discussion forums, or actual events, having clearly defined spaces can help contributors and maintainers work together more effectively. So this section will demonstrate specific strategies we’ve used in Qiskit and the underlying principles that make them effective. ### 4. Be a Kind Human This section will focus on the incredibly important aspect of fostering a welcoming culture within your open-source community. We will touch on the importance of a code of conduct, contributing guidelines, issue tagging, using empathetic and accessible language, and other general tips for making the whole contribution experience inclusive. ### 5. Metrics and Automation This section will focus on how to use automations to streamline your contributor experience and collect valuable data along the way. From bots to actions to built-in GitHub features there are a ton of options to choose from, so we’ll highlight the ones we’ve found the most useful and the important insights we’ve gained as a result. ### 6. Development meets DevRel Effective community management requires significant time investment, which can take a toll on project maintainers. This section will make the case for working closely with Developer Relations experts (perhaps even hiring one if you haven’t already!) to offload some of that burden. Developer Advocates are highly specialised in communication for a developer audience, and can become valuable assets when brought into an open-source team. ### 7. The Community Management Graveyard To wrap things up, this section will cover ideas that we have tried and failed during our community management journey in Qiskit. Things that started out with the best intentions that just didn’t work out and what we learned from the process. The tone of this section will demonstrate how experimenting is an important part of the process of finding a community management setup that works for you, and that trying and failing in public is what open source is all about. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/KXWZJY/ Amphitheater 204 Abby Mitchell PUBLISH QUNAY9@@cfp.scipy.org -QUNAY9 Thar Be Dragons - Ethical, Legal, and Policy Challenges when Measuring Open Source en en 20230712T143500 20230712T150500 003000 Thar Be Dragons - Ethical, Legal, and Policy Challenges when Measuring Open Source Challenges to outline can include: __Ethical__ - Academia - quantitative + qualitative open source data is not (usually) subject to IRB - Does anti-aliasing across datasets potentially create opportunities for harm for members of open source communities? __Legal__ - When does information become a dataset? - Can I use this data? Which license for what? __Policy__ - Can umbrella foundations "opt-in" communities and projects into ecosystem scale research? - How can communities and projects create clear boundaries about how and where they want the "data exhaust" they release to be used? PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/QUNAY9/ Amphitheater 204 amanda casari PUBLISH S8NSHT@@cfp.scipy.org -S8NSHT In-Process Analytical Data Management with DuckDB en en 20230712T152500 20230712T155500 003000 In-Process Analytical Data Management with DuckDB Data management systems and data analysts have a troubled relationship: Common systems such as Postgres or Spark are unwieldy, hard to set up and maintain, hard to transfer data in and out, and hard to integrate into complex end-to-end workflows. As a response, analysts have developed their own ecosystem of data wrangling tools such as Pandas or Polars. These tools are much more natural for analysts to use, but are limited in the amount of data they can process or the amount of automatic optimization that is supported. DuckDB is a new analytical data management system that is built for an in-process use case. DuckDB speaks SQL, has no external dependencies, and is deeply integrated into the Python ecosystem. DuckDB is Free and Open Source software under the MIT license. DuckDB uses state-of-the art query processing techniques with vectorized execution, lightweight compression, and morsel-driven automatic parallelism. DuckDB is out-of-core capable, meaning that it is capable of not only reading datasets that are bigger than main memory. This allows for analysis of far greater datasets and in many cases removes the need to run separate infrastructure. The “duckdb” Python package is not a client to the DuckDB system, it provides the entire database engine. DuckDB runs without any external server directly inside the Python process. Once there, DuckDB can run complex SQL queries on data frames in Pandas, Polars or PyArrow formats out-of-the box. DuckDB can also directly ingest files in Parquet, CSV or JSON formats. Because DuckDB runs in the same process, data transfer are virtually instantaneous. Conversely, DuckDB’s query results can be transferred back into data frames very cheaply, allowing direct integration with complex downstream libraries such as PyTorch or TensorFlow. DuckDB enjoys fast-growing popularity, the Python package alone is currently downloaded around one million times a month. DuckDB has recently become the default backend of the Ibis project that offers a consistent interface in Python over a variety of data backends. This talk is aimed at two main groups, data analysts and data engineers. For the analysts, we will explain the user values of DuckDB, and how it can be used to improve their day-to-day lives. For data engineers, we will describe DuckDB’s capabilities to become part of large automated data pipelines. The presenters for the proposed talk, Hannes Mühleisen and Mark Raasveldt are the original creators of DuckDB, they are still leading the project and are deeply familiar with its Python integration. - DuckDB Python API Overview: https://duckdb.org/docs/api/python/overview - DuckDB PyPI Download Statistics: https://pypistats.org/packages/duckdb - DuckDB Ibis Backend: https://ibis-project.org/backends/DuckDB/ - Peer-reviewed paper about the concept behind DuckDB by the presenters https://www.cidrdb.org/cidr2020/papers/p23-raasveldt-cidr20.pdf - Talk about DuckDB at FOSDEM 2020 by Hannes: https://archive.fosdem.org/2020/schedule/event/duckdb/ - Talk about DuckDB at CMU by Mark: https://www.youtube.com/watch?v=PFUZlNQIndo PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/S8NSHT/ Amphitheater 204 Alex Monahan Hannes Mühleisen Mark Raasveldt PUBLISH YESNZB@@cfp.scipy.org -YESNZB GraphBLAS for Sparse Data and Graphs en en 20230712T160500 20230712T163500 003000 GraphBLAS for Sparse Data and Graphs Sparse data and graph problems appear in virtually all science and engineering disciplines. Nevertheless, adoption of sparse and graph techniques has been slow (so opportunity to exploit sparsity are plentiful)--perhaps because it's not always obvious to know when to apply them, or existing libraries are too slow or difficult to use. GraphBLAS can help. By expressing graph algorithms in the language of linear algebra, it can handle larger data in parallel and is versatile enough to express custom analyses and integrate into larger workflows. In this talk, we will cover: - How to recognize sparse or graph problems and when to use GraphBLAS - Representing a graph as a sparse matrix - The equivalence between graph problems and sparse problems - How GraphBLAS extends linear algebra with masking and arbitrary semirings to be more capable and work-efficient than `scipy.sparse` - The underlying sparse data structures and how to efficiently convert to and from them - Examples of graph algorithms written in [`python-graphblas`](https://github.com/python-graphblas/python-graphblas/) - Using GraphBLAS as a backend to NetworkX via dispatching to [`graphblas-algorithms`](https://github.com/python-graphblas/graphblas-algorithms) - Benchmarks comparing GraphBLAS, NetworkX, and `scipy.sparse` Come learn what makes GraphBLAS special--and fast!--and how to use it effectively. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/YESNZB/ Amphitheater 204 Jim Kitchen Erik Welch PUBLISH 333PY7@@cfp.scipy.org -333PY7 vak: a neural network framework for researchers studying animal acoustic communication en en 20230712T104500 20230712T111500 003000 vak: a neural network framework for researchers studying animal acoustic communication Are humans unique among animals? We speak languages, but is speech somehow like other animal behaviors, such as birdsong? Questions like these are answered by studying how animals communicate with sound. This research requires cutting edge computational methods and big team science across a wide range of disciplines, including ecology, ethology, bioacoustics, psychology, neuroscience, linguistics, and genomics. As in many other domains, this research is being revolutionized by deep learning algorithms. Deep neural network models enable answering questions that were previously impossible to address, in part because these models automate analysis of very large datasets. Within the study of animal acoustic communication, multiple models have been proposed for similar tasks, often implemented as research code with different libraries, such as Keras and Pytorch. This situation has created a real need for a framework that allows researchers to easily benchmark models and apply trained models to their own data. To address this need, we developed vak [1], a neural network framework designed for this research community, built with core libraries of the scientific Python stack such as numpy, scipy, pandas and dask. In this talk, we will show how vak makes it easy for researchers to work with neural network models through a simple command-line interface and TOML configuration files. As an example, we will demonstrate how we used vak to benchmark a neural network model, TweetyNet [2], that automates annotation of birdsong by segmenting spectrograms. Using vak allowed us to tune hyperparameters and determine the minimal amount of expensive human-annotated data we needed for accurate model performance. We will show how TweetyNet and vak made it possible to relate the complex syntax of canary song to the hidden states of neural activity in the canary brain, and how these tools are being used by other researchers in neuroscience and bioacoustics. Then we will demonstrate how in version 1.0 of vak we have significantly extended its generality, in large part by adopting the Lightning library as a backend. We will show how we are using version 1.0 of vak to reduce the segment error rate of TweetyNet, minimizing the need to clean up predictions with post-processing. In addition we'll walk through how we're using vak to compare performance of TweetyNet with other neural network architectures proposed for similar tasks. Finally we will show work in progress incorporating other families of neural network models into vak, generative and unsupervised learning algorithms for dimensionality reduction and similarity measurements. Both authors are experienced public speakers [3], and the combination of cutting edge neural network models in Python with studies of birds, their song, and the vocalizations of other charismatic animals are sure to make for an entertaining and informative talk. [1] https://github.com/vocalpy/vak [2] https://elifesciences.org/articles/63853, https://github.com/yardencsGitHub/tweetynet [3] https://nicholdav.info/talks/, https://yardencsgithub.github.io/talks/ PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/333PY7/ Grand Salon C Yarden Cohen David Nicholson PUBLISH AQ3Z3U@@cfp.scipy.org -AQ3Z3U tfmodisco-lite: an attribution-based motif discovery algorithm en en 20230712T112500 20230712T115500 003000 tfmodisco-lite: an attribution-based motif discovery algorithm Understanding the binding of proteins to the genome is crucial for deciphering gene expression programs across cell types. Yet, identifying where and when these proteins bind along the genome is complicated. Most proteins bind to a specific sequence of nucleotides, known as a "motif." But not all proteins are this simple: zinc-finger proteins are comprised of many "fingers" that each bind to short 3-4 basepair motifs. While these short motifs are always found in the same order, variable spacing can be found between these short motifs, and not all are always necessary for binding. Other proteins require the presence of a co-factor to bind to their motifs. Faithfully describing the sequence determinants of protein binding, sometimes called the cis-regulatory logic, for all proteins is a challenging task. Increasingly, people have been using machine learning to understand biology by training neural networks to take in nucleotide sequence and predict a readout of interest, e.g. ATAC-seq, ChIP-seq, CAGE, etc. One can then run a feature attribution algorithm, such as ISM or DeepLIFT, to highlight the nucleotides that drive the predicted readouts. However, summarizing these attributions into repeated patterns has thus far been a missing component of the analysis pipeline. TF-MoDISco is a framework for using attribution-weighted sequence to discover motifs. The approach differs from classic motif finding algorithms in both the input and the output. Rather than operating solely nucleotide sequence, TF-MoDISco also takes in the attributions from a machine learning model using any attribution algorithm. These attributions highlight the nucleotides involved in accurate predictions and so distinguish between driver motifs and passenger motifs. At the end of the procedure, TF-MoDISco returns clusters of "seqlets," or found motif hits. These patterns, aligned to each other to account for spacing, represent the true heterogeneity of protein binding to the genome. By returning clustering of seqlets, TF-MoDISco overcomes many of the problems of position-weight matrices (PWMs), such as the inability to account for variable spacing and linear assumption across nucleotides. This talk will describe the TF-MoDISco procedure at a high level (first 15 minutes) and give a tutorial on how to use the code for discovery in practice (second 15 minutes). Examples will come from models used to predict chromatin accessibility via ATAC-seq as well as protein binding via ChIP-seq readouts. Specifically, the tutorial will cover tfmodisco-lite, a rewrite of the original algorithm that scales significantly better, runs faster, and requires less code. By the end of the talk, one should feel comfortable applying the method to their own data and interpreting the reports that are generated. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/AQ3Z3U/ Grand Salon C Jacob Schreiber PUBLISH RSE89M@@cfp.scipy.org -RSE89M Gammapy: a Python Package for Gamma-Ray Astronomy Version v1.0 en en 20230712T131500 20230712T134500 003000 Gammapy: a Python Package for Gamma-Ray Astronomy Version v1.0 By observing the very high energy (VHE) range of the electromagnetic spectrum we can gain valuable insight into to the extreme universe, including remnants of supernova explosions and surroundings of black holes. In the past VHE gamma-ray astronomy was typically conducted by small, closed collaborations as a subfield of particle physics. However, with hundreds of sources now identified, VHE gamma-ray astronomy has emerged as a new branch of Astronomy. This field provides us with the high energy context for understanding the physical processes occurring throughout the universe, across the entire electromagnetic spectrum. The next generation of ground based gamma-ray instruments, particularly the Cherenkov Telescope Array Observatory (CTA)), is set to revolutionize gamma-ray astronomy. With an anticipated sensitivity ten times greater than current telescopes, it has the potential to attract a community of thousands of gamma-ray astronomers. Furthermore, it will operate as an open observatory, making both data and analysis tools readily available to the public. In this contribution we introduce the inaugural stable version (v1.0) of Gammapy, a Python package for gamma-ray astronomy and the primary library for the future CTAO science tools. Leveraging the scientific Python ecosystem, including Numpy, Scipy, and Astropy, Gammapy offers a comprehensive set of standard data analysis tools, making it an indispensable resource for (not only) gamma-ray astronomers. By utilizing common open data formats, Gammapy also enables existing instruments such as VERITAS, H.E.S.S., or HAWC to export and archive their data, preserving it for future analysis using improved methods. Additionally, it facilitates the combination of data from multiple instruments, resulting in more sensitive analyses with greater statistics and a larger energy range. Gammapy tackles the varied structure of gamma-ray data and science analysis cases by implementing a uniform API for N-dimensional sky maps. This API is independent of the underlying pixelization scheme and supports local WCS, allsky HEALPix, and region-based projections. These data structures prove useful for a broad range of applications as well as astronomers observing in other wavelength. Building on these core data structures, Gammapy features a maximum likelihood fitting framework that enables simultaneous modeling of gamma-ray emission in four dimensions: space, energy, and time. By providing a general likelihood interface, Gammapy enables science users to integrate gamma-ray data with astronomical data from other wavelengths, as well as with neutrino data. Thanks to its straightforward Python API, Gammapy can be paired with other Python-based broadband emission modeling packages, allowing for direct measurement of parameters pertaining to common underlying astrophysical processes. This feature is crucial to realizing the full potential of future multi-wavelength and multi-messenger Astronomy. Lastly, we will discuss the valuable lessons learned during our journey to achieve v1.0 quality for an openly developed package. This will involve addressing concerns regarding maintainability, selection of dependencies, handling high dimensional data structures and API design. We believe that sharing our experience will be helpful to other scientific Python projects in the future. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/RSE89M/ Grand Salon C Axel Donath PUBLISH JTXC9W@@cfp.scipy.org -JTXC9W libyt: a Tool for Parallel In Situ Analysis with yt en en 20230712T135500 20230712T142500 003000 libyt: a Tool for Parallel In Situ Analysis with yt ## Motivation and Aims In the era of exascale computing, storage and analysis of large scale data have become more important and difficult. We present libyt, an open source C++ library, that allows researchers to analyze and visualize data using yt or any other Python packages in parallel during simulation runtime. ## Methods ### Connecting Python and Simulation We use Python C API and NumPy C API to connect variables and arrays in simulation to Python. This includes creating a NumPy array through wrapping an existing C array without additional memory, allocating new arrays and assigning values, and building Python objects and module that contain simulation information. We also create Python C-extension methods for Python to request data from simulations. ### Executing Python Codes and Handling Errors libyt runs in situ analysis using Python interpreter. This is like running Python prompt inside the ongoing simulation with data loaded. libyt checks input Python syntax through compiling it to code object. If error occurs, it parses the error to see if this is caused by input not done yet or a real error. ### In Situ Analysis Under Parallel Computing Each MPI process contains one simulation code and one Python. All Python instances will work together to conduct in situ analysis in parallel using mpi4py (Python bindings for MPI). yt (a Python package for analyzing and visualizing volumetric data) supports MPI parallelism feature. libyt borrows this feature and handles data transition between different MPI processes and between simulation and Python. Since every data is separated in different processes, and we cannot predict how Python decomposes the jobs and asks for data, we use one-sided MPI to deal with data exchanging process between each nodes. ## Applications ### Analyzing Fuzzy Dark Matter Vortices Simulation using GAMER + libyt We use GAMER, a simulation for astrophysics, to simulate the evolution of vortices form from density voids in a Fuzzy Dark Matter halo. Each snapshot takes 116 GB, and a total of 321 snapshots are required to capture them (37 TB disk space). We solve this by using yt in libyt to extract our region of interest, which now consumes only 8 GB in each step. The data size is 15 times smaller. - Animation: https://youtu.be/tUjJYGbWgUc ### Analyzing Core-Collapse Supernova Simulation using GAMER + libyt We use GAMER to simulate core-collapse supernova explosions. We use libyt to call yt and draw slice plot of the entropy distribution. Since entropy is not part of the variable in simulation's iterative process, these entropy data will only be generated through simulation when they are needed by yt. libyt tries to minimize memory usage. - Animation: https://youtu.be/6iwHzN-FsHw ## Discussion and Conclusion - libyt provides a promising solution that binds simulation to Python with minimal memory overhead and no additional time penalty. It makes analyzing large scale simulation feasible. - libyt focuses on using yt as its core analytic method, even though it can call arbitrary Python modules. We will extend to more data structure in the future. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/JTXC9W/ Grand Salon C Shin-Rong Tsai PUBLISH DZBF7K@@cfp.scipy.org -DZBF7K Seeing the Sun through the Clouds: Accelerating the SunPy Data Analysis Ecosystem with Dask en en 20230712T143500 20230712T150500 003000 Seeing the Sun through the Clouds: Accelerating the SunPy Data Analysis Ecosystem with Dask The SunPy ecosystem is a set of community-developed, free and open-source Python packages for solar data analysis. The ecosystem consists of the core sunpy package, which provides general capabilities such as data download, data structures, and coordinate transformations, as well as a growing set of affiliated packages which provide more application-specific functionality such as image processing techniques. The entire SunPy ecosystem depends heavily on the broader scientific Python ecosystem, including numpy, scipy, and scikit-image and especially the astropy package, a community Python package for astronomy. Over the last decade, the SunPy ecosystem has evolved organically to serve the needs of scientists analyzing solar physics data. Analysis of observational solar data has traditionally been carried out on desktop or laptop computers or small compute clusters (see Bobra et al., 2020). This limitation is partly due to the longstanding historical reliance on the proprietary Interactive Data Language (IDL) by the solar physics community which has limited scalability due in part to licensing restrictions. However, modern space- and ground-based solar observatories are producing data volumes in the tens of petabytes, necessitating the need for parallelized and out-of-core computation. The surge in popularity of Python within the broader astronomy community as well as the growing availability of computing resources has led to many solar researchers using Python in cloud environments. All of these factors have propelled the development of HelioCloud. Inspired by similar science platforms for other disciplines like Pangeo, HelioCloud is a NASA-funded, AWS-backed cloud computing environment tailored for heliophysics research. HelioCloud provides both a dashboard for creating custom virtual machines as well as a JupyterLab interface. Using the latter allows for interactive, scalable computation enabled by Dask across many compute nodes. Most importantly, HelioCloud is collocated with nearly 1 petabyte of solar physics data such that researchers can perform their analysis without the added latency of needing to download the data. In this talk, we will demonstrate how the SunPy ecosystem, combined with Dask on HelioCloud, can be used to efficiently process high-resolution solar data. First, we will provide a brief description of the SunPy project with particular emphasis on the ndcube and sunkit-image affiliated packages. Next, we will provide a brief description of the JupyterLab interface of the HelioCloud platform. Finally, we will demonstrate a typical scientific workflow on HelioCloud by efficiently analyzing many hours worth of solar active region evolution using sunpy, ndcube, sunkit-image, and Dask to scale out our computation over many workers. Additionally, we will discuss existing incompatibilities between Dask and the astropy ecosystem and how collaboration with the broader scientific Python community could resolve such frictions. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/DZBF7K/ Grand Salon C Nabil Freij Will Barnes Jack Ireland Stuart Mumford PUBLISH 8MMNUD@@cfp.scipy.org -8MMNUD Open Force Field: next-generation force fields with open data, open software, and open science en en 20230712T152500 20230712T155500 003000 Open Force Field: next-generation force fields with open data, open software, and open science **Background** Molecular dynamics (MD) simulations are now critical components in pharmaceutical and biomolecular research. A potential energy function called a ‘force field’ is used to solve the differential equations that describe the particle motion. A vast number of different force fields have now been released, each fit to experimental or quantum chemistry data to reproduce specific properties in a limited region of chemical space. However, the core of most of these date from work published decades ago, and new force field development has primarily taken the form of incremental improvements guided by human chemical intuition rather than systematic, reproducible methods. **Outline** The [Open Force Field (OpenFF) initiative](https://openforcefield.org/) was formed to produce open and extensible infrastructure to build a new generation of MD force fields. We have now developed many software packages for constructing, applying, and benchmarking force fields. We have also generated several high-quality quantum chemistry datasets. Everything is available freely on [GitHub](https://github.com/openforcefield/), [Zenodo](https://zenodo.org/communities/openforcefield/), and the [MolSSI QCArchive server](https://qcarchive.molssi.org/). This work has been successfully used to investigate potential improvements to force fields, as well as simplify many previously difficult aspects of preparing MD systems. Here we will introduce the [OpenFF-Toolkit](https://github.com/openforcefield/openff-toolkit) and [OpenFF-Interchange](https://github.com/openforcefield/openff-interchange) packages. We can use them to quickly assign force field parameters to arbitrary systems of small molecules, and then write these systems out in common MD formats for simulation. We also introduce the [OpenFF-Bespokefit](https://github.com/openforcefield/openff-bespokefit) package for fitting custom torsion parameters, as well as the [OpenFF-QCSubmit](https://github.com/openforcefield/openff-qcsubmit) package for interacting with QCArchive. We show how to use the datasets we have released on QCArchive. We will finally show some of the advancements enabled by our work. The [OpenFF-Evaluator](https://github.com/openforcefield/openff-evaluator) package was instrumental in investigating the effect of using a custom potential for van der Waals’ parameters. We used [OpenFF-Recharge](https://github.com/openforcefield/openff-recharge) to explore adding off-site charges with virtual sites. Finally, we describe the development of a neural network for quickly assigning conformer-independent partial charges – this also employed OpenFF-Recharge, as well as [OpenFF-NAGL](https://github.com/openforcefield/openff-nagl). We hope these examples give a brief overview of how OpenFF can help both common everyday MD tasks as well as larger scientific investigations. **Previous talks** I've previously given [keynote talks](https://www.youtube.com/watch?v=Jw1iVjHkRPM) at the Open Force Field annual meetings and presented at open science meetings convened by the [NIH](https://datascience.nih.gov/news/nih-odss-to-host-sessions-at-ismb-annual-conference), the NSF, and groups in the [scientific computing](https://www.youtube.com/watch?v=hS87inZupdQ) and molecular simulation communities. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/8MMNUD/ Grand Salon C Jeff Wagner PUBLISH JH9JMV@@cfp.scipy.org -JH9JMV Designing user-friendly APIs for the NIST Interatomic Potentials Repository en en 20230712T160500 20230712T163500 003000 Designing user-friendly APIs for the NIST Interatomic Potentials Repository This presentation outlines the Python APIs developed for the public database of the NIST Interatomic Potentials Repository. The entire framework consists of six different Python packages designed for data interaction and generation: DataModelDict, cdcs, yabadaba, potentials, atomman, and iprPy. These packages have an import hierarchy with each subsequent package incorporating or inheriting the previous. All project data is represented with JSON/XML equivalent data models. Having data that can be equivalently represented in JSON and XML takes advantage of the benefits of both formats while placing only minor limits on schema designs. The “DataModelDict” Python class extends the basic dict to allow for easy transformations between Python, JSON and XML, and includes additional methods for exploring and manipulating individual records. All public potentials data are hosted in a CDCS database accessible at https://potentials.nist.gov. CDCS databases store XML formatted records, they support multiple schemas, and provide both a web-based interface and a REST API for interacting with the data. The “cdcs” package defines Python methods for common database interactions that wrap around the REST API calls. The also provides options to build custom REST calls to the database for features not yet directly supported. The JSON/XML equivalent data models means that all records can also be stored in JSON-based Mongo databases or as local collections of JSON or XML files. The “yabadaba” Python package provides an intermediate abstraction layer allowing users to interact with data stored in all three database infrastructures using common methods. It also provides a framework for interpreting and building data records associated with different schemas. These features make it possible for end users to explore and generate data while remaining agnostic to the infrastructure used to store the data. While the “cdcs” and “yabadaba” packages provide APIs for interacting with an arbitrary CDCS database, the “potentials” package provides APIs specifically focused on interatomic potentials content in potentials.nist.gov. Utilizing the yabadaba features, any user can create their own copy of all interatomic potentials listings and then search and explore from either location. Searches can be performed both using simple Python methods or using Jupyter widget-based GUIs. The potentials package also forms the basis for adding new listings to the repository and for generating the traditional static repository website at https://www.ctcms.nist.gov/potentials/. The ”atomman” package focuses on setting up and analyzing atomic configurations and LAMMPS simulations. On the data side, it extends the “potentials” package functionality to interpreting schemas of atomic configurations. Finally, the “iprPy” package is centered around providing a collection of standard atomistic property calculation methods for characterizing interatomic potentials. The iprPy calculations can be performed individually or in high throughput, and can be executed from the command line, from within Python, or using transparent-box demonstration Jupyter Notebooks. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/JH9JMV/ Grand Salon C Lucas Hale PUBLISH X7YH7A@@cfp.scipy.org -X7YH7A Keynote - Open Source Contributors in Space and Time en en 20230712T091500 20230712T100000 004500 Keynote - Open Source Contributors in Space and Time PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/X7YH7A/ Zlotnik Ballroom Michael Droettboom PUBLISH G9P3AG@@cfp.scipy.org -G9P3AG Fast Exploration of the Milky Way (or any other n-dimensional dataset) en en 20230712T104500 20230712T111500 003000 Fast Exploration of the Milky Way (or any other n-dimensional dataset) Blosc is a high-performance compressor optimized for binary data, such as floating-point numbers, integers, and booleans, although it can also handle string data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed direct memory fetch approach, which uses a memcpy() OS call. Blosc is widely used in popular storage libraries like HDF5 (via h5py or PyTables) or Zarr, and is probably producing many petabytes of compressed data every day around the world. C-Blosc2 (https://github.com/Blosc/c-blosc2) is the latest major version of C-Blosc. It comes with Python-Blosc2 (https://github.com/Blosc/python-blosc2), a lightweight Python wrapper that exposes many of its new features. Some of the most interesting features are: - 64-bit containers: There is no practical limit in dataset sizes. - Frames: Data can be serialized either on-disk or in-memory. - Meta-layers: Meta-data can be added in different layers inside frames. - Blosc2 NDim: N-dimensional datasets can be created, read, and sliced efficiently. - Double partitioning: Data can be split into fine-grained cubes for faster reads of n-dimensional slices. - Parallel reads: When several blocks of a chunk need to be read, this is done in parallel. - Support for special values: Large sequences of repeated values can be represented efficiently. With leveraging these features, Blosc2 provides a powerful, yet flexible tool for data handling. For example, when Blosc2 cooperates with libraries like PyTables/HDF5, it allows to query [100 trillion rows tables in human time frames](https://www.blosc.org/posts/100-trillion-baby/). Furthermore, being able to compress multidimensional data is of great help in handling large multidimensional datasets because 1) it reduces the amount of storage resources and 2) reduces the bandwidth necessary to bring data from storage (disk, memory) to the CPU, allowing to process data more effectively in general. Additionally, compression can represent a wide variety of sparse data without requiring a specific format. Instead, compression works to minimize the number of zeros and keep storage requirements to a minimum. We will address common misconceptions about compressing data, such as: 1) decompressing data takes CPU time, which may slow down computations, and 2) when retrieving a subset of data, all affected partitions must be decompressed, adding overhead. To debunk these myths, we offer the following facts: 1) decompressing data within CPU caches often saves transmission cycles, and 2) Blosc2 features a novel double partitioning schema that minimizes decompression overhead. We will leverage Python-Blosc2 to: - Describe the main features of Blosc2 - Provide useful advice on the best codecs and filters for different types of datasets - Explain how to partition multidimensional datasets for efficient slicing - Compare efficiency and resource savings with other packages, such as h5py, PyTables, and Zarr Finally, we will demonstrate an example of exploring the Milky Way's 3D dataset effectively, using data from the Gaia mission. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/G9P3AG/ Zlotnik Ballroom Francesc Alted PUBLISH LAKM79@@cfp.scipy.org -LAKM79 A Computer Vision (ML) Approach to Classifying Clouds and Aerosols from Satellite Observations en en 20230712T112500 20230712T115500 003000 A Computer Vision (ML) Approach to Classifying Clouds and Aerosols from Satellite Observations The purpose of this talk is to share how to make most efficient use of the existing machine learning (ML) software, such as tensorflow, to implement scientific ML methods. We will first describe the science objectives we are trying to achieve, elaborate on lessons learned, and finally introduce future challenges. Our primary science objective is to identify different cloud types and aerosols from satellite imagery, where the cloud types are indicative of different meteorological conditions. The science objective during the talk will be catered towards the broader scientific community while expecting little to no background in atmospheric science or ML. All of this will be accomplished by presenting visualization of satellite imagery throughout to relate the data to the audience. Subsequently we will introduce the ML techniques we have been using. We employ a pretrained VGG16, a convolutional neural network (CNN), which we fine-tune to identify cloud types and aerosols from satellite imagery. There will be accompanying animations illustrating this process and how the inference is combined into the softmax layer providing the result. The specific lessons learned in using ML software is to consider which part of the code executes in CPU or GPU space. Initially we noticed GPU usage was not consistently 100% during inference. To demonstrate potential, we dumped the data to a 200GB file and streamed that directly to the GPU. This test proved what was possible and allowed us to rewrite our generator using keras Sequence where the init and getitem called tf.device locking the data I/O and preprocessing to the CPU leaving the GPU solely for inference. This approached yielded a 2x performance increase. Since our goal is to add value to existing NASA algorithm methodologies via ML, this goal requires us to have labeled data. We experimented with existing labeling packages but in the end decided to incorporate the labeling tasking into existing software the community uses. Thankfully as part of NASA’s participation in Open-Source Science, one of the primary tools used by the Atmospheric Science community, NASA Worldview, is already open sourced. This allowed us to install their docker images and extend this tool bringing the labeling task directly to the Scientists. Additionally, I will talk about the importance of visualizing the data through the entire process of training a CNN. For example, I have a video file flipping through thousands of images from our training set. And I will use this to emphasize the importance of looking at data throughout the process and the importance of being able to share information. Open-Source Science is great but being able to convey information about how ML works is just as important. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/LAKM79/ Zlotnik Ballroom Steve Dutcher PUBLISH T3XSHX@@cfp.scipy.org -T3XSHX Pandera: Beyond Pandas Data Validation en en 20230712T131500 20230712T134500 003000 Pandera: Beyond Pandas Data Validation # Motivation Data quality remains a core concern for practitioners in machine learning, data science, and data engineering, and many specialized packages have emerged to fulfill the need of validating and monitoring data and models. However, as the open source community creates new data manipulation frameworks – notably, new highly performant entrants such as [Polars](https://www.pola.rs/) – existing data quality frameworks need to catch up to support them, and in some cases, the community creates new data validation libraries for these new data frameworks. # Origins [Pandera](https://github.com/unionai-oss/pandera) started as a small project in 2018 with the goal of providing a lightweight, flexible, and expressive API to validate Pandas DataFrames. This part of the talk provides a short primer on data validation and property-based testing with Pandera, providing insights into how its design facilitates code-first schema authoring and maintenance, which in turn gives rise to safer and more robust data pipelines. This primer will contain content similar to the "Introduction to Pandera" notebook in the pandera documentation: https://pandera.readthedocs.io/en/stable/try_pandera.html # Evolution After gaining traction over the years, the author and community of contributors began to expand Pandera’s scope to support pandas-compliant data frameworks such as GeoPandas, Dask, Modin, and Pyspark Pandas (formerly Koalas). As requests for other libraries increased in frequency, it became clear that Pandera in its existing state was not well-suited for extension beyond Pandas objects. This part of the talk focuses on some of the key design failures that made it difficult to extend to other data frameworks. Rewrites are Fun! (not): Imagine doing a complete internal rewrite of a library while bug reports, feature requests, and pull requests are coming in from contributors: does it sound fun? In the author’s experience, it’s like juggling three balls while playing drums with your feet as someone throws water balloons in your face. This part of the talk outlines the challenges, lessons learned, and things the author would have done differently to anticipate issues related to the separation of concerns, modularity, and extensibility. # Conclusion This talk is about how Pandera has evolved to provide a standard schema interface for easily extending and supporting validation backends for arbitrary statistical data containers. Attendees will learn not only about data testing principles such as run-time validation and property-based testing, they will also learn about the challenges of maintaining and evolving an open source project that many people rely on as a critical piece of their data infrastructure. The high-level goal for this talk is to highlight lessons learned from Pandera’s particular journey from supporting only Pandas as a backend to supporting a whole suite of data objects. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/T3XSHX/ Zlotnik Ballroom Niels Bantilan PUBLISH NPG3NS@@cfp.scipy.org -NPG3NS Disciplined Saddle Programming en en 20230712T135500 20230712T142500 003000 Disciplined Saddle Programming Convex-concave saddle problems are a class of optimization problems that generalize convex optimization and have a wide range of practical applications, including game theory, finance, and machine learning. A technical trick to convert to a single convex problem is called dualization. However, carrying out this conversion by hand can be tedious and error-prone. In this context, we introduced Disciplined Saddle Programming (DSP) in a recent paper, and the accompanying Python package is implemented as an extension of CVXPY. It is available at https://github.com/cvxgrp/dsp. DSP is a domain-specific language (DSL) for specifying saddle problems for which the dualizing trick can be automated. DSP is based on the conic-representable saddle programs developed by Juditsky and Nem irovski, who showed how to carry out the required dualization automatically using conic duality. The DSP language and methods extend Nesterov and Nemirovski's earlier development of conic representable convex problems and can be seen as extending disciplined convex programming (DCP) to saddle problems. There are numerous benefits of using DSP. The language makes it easier for practitioners to specify and solve saddle problems, and it can handle a wide range of optimization problems, including many robust optimization problems, which have recently gained wider attention. Indeed, some argue that most optimization problems should be solved as robust problems instead, as inputs are rarely obtained with absolute certainty. Further, hearing about even more applications from the SciPy community is an intended side effect to make the package easier to integrate for practitioners. Just as DCP, and by extension CVXPY, made it easy for users to formulate and solve complex convex problems, DSP allows users to easily formulate and solve saddle problems. The method is implemented in an open-source Python package, also called DSP. This package provides a way to automate the dualization of saddle problems and provides a simple interface for users to formulate and solve complex problems in a structured and disciplined way. In summary, disciplined saddle programming (DSP) is a new approach that can simplify solving saddle problems in convex optimization. It automates the dualization of saddle problems and provides a simple interface for users to specify and solve complex problems in a structured and disciplined way. DSP is designed to be easy to learn and use, and is compatible with the existing CVXPY framework. DSP has the potential to make saddle problems much easier to solve, which could have a significant impact on a wide range of fields that rely on optimization. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/NPG3NS/ Zlotnik Ballroom Philipp Schiele Eric Sager Luxenberg PUBLISH RJPMGC@@cfp.scipy.org -RJPMGC Emukit: Python toolkit for uncertainty quantification en en 20230712T143500 20230712T150500 003000 Emukit: Python toolkit for uncertainty quantification ## Description Emukit is a highly adaptable Python toolkit for enriching decision making under uncertainty. This is particularly pertinent to model complex systems where data is scarce or difficult to acquire. In these scenarios, propagating well-calibrated uncertainty estimates within a design loop or computational pipeline ensures that constrained resources are used effectively. The main features currently available in Emukit are: * Bayesian optimisation: optimise physical experiments and tune parameters of machine learning algorithms; * Experimental design/Active learning: design the most informative experiments and perform active learning with machine learning models; * Sensitivity analysis: analyse the influence of inputs on the outputs of a given system; * Bayesian quadrature: efficiently compute the integrals of functions that are expensive to evaluate; * Multi-fidelity emulation: build surrogate models when data is obtained from multiple information sources that have different fidelity and/or cost. The package was released in 2019, and since then gained popularity among the research communities of Bayesian optimization, Bayesian quadrature, and multi-fidelity modelling. The aim of this talk is to present Emukit to a wider audience of Python developers. It may be of interest to machine learning practitioners in need of hyper-parameter optimization methods, scientists running complex simulations and looking for emulation and UQ techniques, and everyone interested in approaches for decision making under uncertainty. Hearing about our development experience and lessons learned may also be useful to those who look to develop scientific packages in Python. The first part of the talk will focus on technical details of the package. We will start with a brief introduction into Emukit and the methods it provides. Emukit is a replacement for GPyOpt and the reasons that prompted its development will be discussed. We will go over the key software design principles of Emukit, and see how they lead to a flexible and adaptable toolkit, but also how they may hinder the computational efficiency. Other popular frameworks for Bayesian optimization, Trieste and BoTorch, will be used to highlight strengths and weaknesses of Emukit. The second part will focus on usage and adoption. We will talk about target audience of the toolkit, existing uses for teaching and research, and discuss why anyone who is not an expert in Bayesian active learning methods would want to use Emukit. ## Additional materials Emukit is available on Github: https://github.com/EmuKit/emukit. There is also a website about the package: https://emukit.github.io/. Emukit was first presented at NeurIPS workshop on ML and the Physical Sciences, 2019. Corresponding paper on arXiv: https://arxiv.org/abs/2110.13293. Emukit is used for teaching ML and the Physical World course at the University of Cambridge. The course website can be found at https://mlatcl.github.io/mlphysical/. Emukit was also adopted for the Gaussian Process Summer School 2022: https://gpss.cc/gpss22/. Some of the previous talks given by the speaker can be found on his website: https://paleyes.info/#talks. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/RJPMGC/ Zlotnik Ballroom Andrei Paleyes PUBLISH TVRHYS@@cfp.scipy.org -TVRHYS Bayesian Statistics with Python, No Resampling Necessary en en 20230712T160500 20230712T163500 003000 Bayesian Statistics with Python, No Resampling Necessary This talk is a concise update of a talk delivered previously for PyStan, the Python Interface for STAN, which is software for Bayesian inference. Now we will focus on the TensorFlow Probability library. Here are links for the previous talk: https://github.com/c22hatal/bayes_confidence/tree/main/meetup11aug21/meetup11aug21 https://www.youtube.com/watch?v=-7l5QTq5Hz0&list=PLhbPZ4oC18muuVdH3pjpjGmHkJqxCldYR&index=11&t=1073s We first briefly review the Bayesian concepts of prior and posterior and elaborate on how the posterior distribution of the parameters can be approximated by a normal distribution with large sample sizes. This is the key theoretical point of the talk and is discussed in section 4.1 of Bayesian Data Analysis [1]. Through the talk, we will corroborate the proof by using resampling methods. We show that the normal approximation and resampling methods are equivalent with large data using TensorFlow Probability. After the talk, users can confidently use TensorFlow Probability and SciPy/NumPy to perform Bayesian analysis without resampling if their samples are sufficiently large. After the theoretical discussion, we get into how the posterior distribution can be modeled using TensorFlow Probability’s distribution classes. I will show how you can sample from the distributions and calculate the posterior log probability density. We will focus on a linear regression setting where the Normal and 𝜒2 distributions will be used as priors for the slope and intercept parameters. https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/Chi2 https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/Normal https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/JointDistributionNamed Then I show how the posterior modes can be estimated using TensorFlow or SciPy optimization. The Broyden–Fletcher–Goldfarb–Shanno algorithm (BFGS) will be used. This method doesn’t calculate the full Hessian, the second and cross derivatives of the log posterior function. https://www.tensorflow.org/probability/api_docs/python/tfp/optimizer/lbfgs_minimize https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.html The inverse Hessian gives us the posterior variance under our approximation. So I show how you can take numeric derivatives in NumPy to obtain it. Derivatives are taken according to the method in Numerical Recipes [2]. Vectorized computations will be used where possible Then I finally use resampling, particularly Markov Chain Monte Carlo sampling to show how well the approximation to the posterior distribution works. This is accomplished using TensorFlow Probability functions. I provide a framework for simulation in Python that is used to demonstrate these results as well. https://www.tensorflow.org/probability/api_docs/python/tfp/mcmc References 1. Bayesian Data Analysis (3rd. ed.). A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari and D. B. Rubin, 2013 Boca Raton, Chapman and Hall–CRC 2. Numerical Recipes: The Art of Scientific Computing (3rd. ed.). W. H. Press, S. A. Teukolsky, W T. Vetterling, and Brian P. Flannery. 2007. Cambridge University Press, USA. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/TVRHYS/ Zlotnik Ballroom Charles D Lindsey PUBLISH NUT798@@cfp.scipy.org -NUT798 Lightning Talks en en 20230712T170000 20230712T180000 010000 Lightning Talks PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/NUT798/ Zlotnik Ballroom PUBLISH VUD8HM@@cfp.scipy.org -VUD8HM Poster Session and Job Fair en en 20230712T180000 20230712T190000 010000 Poster Session and Job Fair POSTERS: Title - Authors (Track) 1. RECOIL - Ronchi Evaluator and Classifier of Imperfect Lenses (RECOIL) - Allen S. Harvey Jr., Clare Egan (Astronomy and Physics) 2. Planetary Defense Using Python: Measuring Deflection of the Didymos Binary Asteroid System by the NASA DART Mission - Arushi Nath (Astronomy and Physics) 3. pyro: a python hydrodynamics code for teaching and prototyping - Michael Zingale (Astronomy and Physics) 4. Accessing astronomical data with Python - Brigitta Sipőcz (Astronomy and Physics) 5. Spatial and Single-Cell Analysis of MERFISH Data using the Python Library Cormerant - Nicolas Fernandez (Bioinformatics, Computational Biology & Neuroscience) 6. Cross-language Data Grammar for Single-cell Feature Engineering - Dave Bunten (Bioinformatics, Computational Biology & Neuroscience) 7. Biomolecular crystallographic computing with Jupyter - Blaine Mooers (Bioinformatics, Computational Biology & Neuroscience) 8. MDAKits: A Framework for FAIR-Compliant Molecular Simulation Analysis - Ian Kenney (Bioinformatics, Computational Biology & Neuroscience) 9. Obtain quantitative insights through image registration in python - Matt McCormick, Konstantinos Ntatsis (Bioinformatics, Computational Biology & Neuroscience) 10. EEG-to-fMRI: Neuroimaging Cross Modal Synthesis in Python - David Calhas (Bioinformatics, Computational Biology & Neuroscience) 11. Matchmaker: A Toolkit for Collocating and Combining Satellite-Based Earth Observations - Greg Quinn (Earth, Ocean, Geo, and Atmospheric) 12. Building geospatial workflows for Impact using Leafmap, SageMaker Studio Lab, and Open Data on AWS - Qiusheng Wu, Mike Jeffe (Earth, Ocean, Geo, and Atmospheric) 13. Operational Open Science and Software for the Planet's Largest Climate Observatory - Zachary Sherman (Earth, Ocean, Geo, and Atmospheric) 14. Moving the Earth with thermodynamics and python - Cian Wilson (Earth, Ocean, Geo, and Atmospheric) 15. Bringing automated data analysis and machine learning pipelines directly to end users using Unidata tools - Thomas Martin, Hailey Johnson, Drew Camron (Earth, Ocean, Geo, and Atmospheric) 16. Yori: A New, Highly Customizable Tool for Level-3 Data Production - Paolo Veglio (Earth, Ocean, Geo, and Atmospheric) 17. Intuitive Statistics in SciPy - Matt Haberland, Albert Steppi (General Track) 18. Using MyST Markdown in JupyterLab - Rowan Cockett (General Track) 19. PyVista: A Python Library for Interactive 3D Data Visualization and Analysis - Tetsuo Koyama (General Track) 20. SOSA: The Scalable Open-Source Analysis Stack - James A. Bednar, Martin Durant (General Track) 21. Sensitivity Analysis in Python: `scipy.stats.sobol_indices` - Pamphile Roy (General Track) 22. Improving the SciPy-CuPy compatibility for interpolation and signal processing - Edgar Andrés Margffoy Tuay (General Track) 23. aPhyloGeo-Covid: A Web Interface for Phylogeographic Analysis of SARS-CoV-2 Variation using Neo4j and Snakemake - Nadia Tahiri, Wanlin Li (Machine Learning, Data Science, and Ethics in AI) 24. Quantifying Uncertainty in Time Series Forecasting with Conformal Prediction - Fede Garza Ramirez (Machine Learning, Data Science, and Ethics in AI) 25. Anti-Patterns: How not to do things in Python - Gajendra Deshpande (Machine Learning, Data Science, and Ethics in AI) 26. pomegranate v1.0.0: now with PyTorch - Jacob Schreiber (Machine Learning, Data Science, and Ethics in AI) 27. Data engineering and analytics for photolithography manufacturing process at DuPont ‚Äì a practical approach from lab to fab - Avishek Panigrahi, Sumanth S, Abhishek Shrivastava, stefan caporale (Machine Learning, Data Science, and Ethics in AI) 28. Stochastic Unitary Constraints - Victoria Schneider, Sara Logsdon, Delaney Ott (Machine Learning, Data Science, and Ethics in AI) 29. Hamilton: Scalable, Portable, and Self-Documenting Dataflows in Python - Elijah ben izzy, Stefan Krawczyk (Machine Learning, Data Science, and Ethics in AI) 30. Teaching machine learning in professional education - Nadia Udler (Machine Learning, Data Science, and Ethics in AI) 31. Magic Data Abstractions (for Magic™ data) - Valerio Maggio (Machine Learning, Data Science, and Ethics in AI) 32. Self-Supervised Cilia Segmentation - Meekail Zain, Shannon Quinn (Machine Learning, Data Science, and Ethics in AI) 33. Data-centric ML pipeline for resolving data drift and optimizing data preprocessing - Hongsup Shin (Machine Learning, Data Science, and Ethics in AI) 34. "Clockwork" detection in categorical telemetry data - Benoit Hamelin (Machine Learning, Data Science, and Ethics in AI) 35. Intro to Quantum Computing for Drug Design - Maurice Benson (Machine Learning, Data Science, and Ethics in AI) 36. PyQtGraph - High Performance Visualization for All Platforms - Nathan Jessurun (Machine Learning, Data Science, and Ethics in AI) 37. Accelerating Drug Discovery on the Cloud with Open Source Python - Nathan Knapp (Materials and Chemistry) 38. Modeling Multiphase Multicomponent Precipitate Growth with Phase-Field and Python - Trevor Keller (they/them) (Materials and Chemistry) 39. Materials Project: building an open-source, data-driven platform for materials science - Ruoxi Yang (Materials and Chemistry) 40. Rozha: Supporting and Simplifying Multilingual Natural Language Processing - Ian Goodale (Social Science and the Digital Humanities) 41. Spatial Microsimulation & Activity Allocation in Python: An Update on the Likeness Toolkit - James Gaboardi, Joe Tuccillo (Social Science and the Digital Humanities) 42. Python meta packages - Jorge Martinez, Roberto Pastor (Tending Your Open Source Garden: Maintenance and Community) 43. quartodoc: a tool for quick and easy package documentation - Michael Chow (Tending Your Open Source Garden: Maintenance and Community) 44. TUG-RSE: Pulling Students into Research Software Engineering - Aman Goel (Tending Your Open Source Garden: Maintenance and Community) 45. CI/CD pipelines for scientists - Jorge Martinez (Tending Your Open Source Garden: Maintenance and Community) 46. First steps toward supercharging remote development with Spyder - Carlos Cordoba (Tending Your Open Source Garden: Maintenance and Community) 47. Chalk'it : dataflow and drag-and-drop Python dashboarding - Mongi Ben Gaid (Tending Your Open Source Garden: Maintenance and Community) 48. Accessible documentation for everyone - Jorge Martinez, Revathy Venugopal (Tending Your Open Source Garden: Maintenance and Community) 49. Patterns and Anti-Patterns when Measuring Diversity in Open Source - Amanda Casari (Tending Your Open Source Garden: Maintenance and Community) PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/VUD8HM/ Zlotnik Ballroom PUBLISH 3NH3L8@@cfp.scipy.org -3NH3L8 SciPy Attendee Social Event hosted by Open Source Science (OSSci) en en 20230712T190000 20230712T210000 020000 SciPy Attendee Social Event hosted by Open Source Science (OSSci) PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/3NH3L8/ Zlotnik Ballroom Scholz Garten, 1607 San Jacinto Blvd PUBLISH VVVQRU@@cfp.scipy.org -VVVQRU Using Numba for GPU acceleration of Neutron Beamline Digital Twins en en 20230713T104500 20230713T111500 003000 Using Numba for GPU acceleration of Neutron Beamline Digital Twins Motivation MCViNE is a software package for creating digital twins of neutron scattering experiments using a Monte Carlo ray-tracing approach. These simulations are useful in performing advanced neutron data analysis as well as in designing novel neutron instruments and sample environments. Specifically, it has been used in the initial designs for instruments in the Second Target Station at the Spallation Neutron Source at Oak Ridge National Laboratory. Currently, MCViNE only runs on CPUs which is a bottleneck in large simulations with tens of billions of neutrons, and in modelling complex multiple scattering, with some simulations taking months to complete. Due to the massively parallel nature of Monte Carlo methods, bringing GPU acceleration to these simulations would offer superior performance and scalability. MCViNE is originally implemented in C++ and parallelized using MPI, and it has bindings to Python for user interaction; however, extensibility for the user can be very difficult. Methods To improve performance and to ease user contributions, Python and Numba were chosen to create a new package providing GPU acceleration of MCViNE. Numba is an open-source JIT (just-in-time) compiler for Python using LLVM to generate efficient machine code and supports GPUs using NVIDIA CUDA. Numba is designed for scientific computing and can support NumPy arrays and functions. Currently, we are only using Numba for its GPU capabilities. Using Python and Numba for this application allowed several advantages such as utilizing an extensible object-oriented approach and polymorphism. Each MCViNE instrument is composed of several components, such as a neutron source, a guide, and a monitor. During the simulation, neutrons can travel through each component in the instrument. Each component has a method (“propagate”) defined for propagating the neutron through it. Additionally, sample environments are created using constructive solid geometry (CSG) with each primitive shape defined as a CUDA kernel. To each constructed shape, many CUDA kernels are available, each modeling a different type of scattering physics. Due to different component/scattering-kernel types and geometric shapes, using an object-oriented design was beneficial. Furthermore, this structure allowed for custom on-the-fly CUDA kernel generation for complex instrument/sample geometries and physics. Results and Conclusions Python and Numba was used to create a GPU accelerated version of MCViNE, which has so far achieved speedups of up to 1000x over the original CPU implementation. This performance gain enables more sophisticated data analysis for neutron scattering and impacts neutron scattering science and instrument design. Using Python has helped increase the usability, extensibility, and maintainability of the codebase. Additionally, coupling Python with Numba allowed complex combinations of CUDA kernels to be generated at runtime, which would have been significantly harder to implement in other languages. The techniques used in this project could also be applied to other scientific computing applications. Resources: https://github.com/mcvine/acc https://mcvine.ornl.gov/ https://github.com/mcvine/mcvine PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/VVVQRU/ Amphitheater 204 Coleman Kendrick PUBLISH AXSVZ3@@cfp.scipy.org -AXSVZ3 Interactive Exploration of Large-Scale Datasets with Jupyter-Scatter en en 20230713T112500 20230713T115500 003000 Interactive Exploration of Large-Scale Datasets with Jupyter-Scatter Visualizing datasets as a 2D scatter plot is one of the most popular data visualization methods for understanding the distributions, identifying trends, and discovering correlations. The method is used in any scientific domain. For instance, in biology, machine learning, or digital humanities, high-dimensional datasets are often summarized with dimensionality-reduction methods like PCA, t-SNE, or UMAP, and the results are typically visualized as 2D scatter plots to discover clusters. Unfortunately, many visualization tools are unable to scale or compromise user experience with datasets that grow in size, dimensionality, and quantity. For instance, while datashader can render datasets of almost any size, it offers limited interactions. On the other hand, Plotly provides interactivity but does not extend nearly as well to millions of points. Ideally, we want to be able to render and interactively explore one or more datasets with millions of data points. Jupyter-scatter (https://github.com/flekschas/jupyter-scatter) is a purpose-built widget for Jupyter Notebook, Lab, and Google Colab that supports interactive, interlinked, and scalable exploration of multiple large-scale datasets as scatter plots. It focuses on data-driven visual encodings, offers pan+zoom interactions, and two-way lasso selection. Beyond a single instance, jupyter-scatter can compose multiple scatter plots and synchronize their views and selections. Moreover, points can be connected by spline-interpolated lines. Thanks to the underlying WebGL rendering engine (https://github.com/flekschas/regl-scatterplot), changes in the spatial or color encoding of the points are smoothly transitioned. Finally, the widget API is inspired by seaborn and integrates seamlessly with Pandas DataFrames. As the number of arguments can get overwhelming when many properties are customized, jupyter-scatter provides a functional API that groups properties by type and exposes them via meaningfully-named methods. This functional API additionally allows users to programmatically modify active widgets from Python. To further ease the usability, jupyter-scatter infers sensible default color encodings from the data and dynamically adjusts the point opacity based on the point density in the current field of view. Using examples from single-cell biology and machine learning we demonstrate how jupyter-scatter works, how it enables more efficient exploration of large-scale datasets, and how it can be integrated with other ipywidgets to build bespoke applications. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/AXSVZ3/ Amphitheater 204 Fritz Lekschas PUBLISH VGAUQN@@cfp.scipy.org -VGAUQN Accessibility best practices for authoring Jupyter notebooks en en 20230713T142000 20230713T145000 003000 Accessibility best practices for authoring Jupyter notebooks Accessibility practices are for everyone, but this may be especially important to notebook authors in academic and public settings where it is often legally required. Using [staple accessibility frameworks](https://www.w3.org/WAI/WCAG21/Understanding/intro#understanding-the-four-principles-of-accessibility), this talk will dive into what it means to make your notebook’s content accessible and provide actionable guidance on how you as an author can improve your notebooks. These skills can be applied regardless of preferred notebook interface, author skill set, or prior accessibility knowledge. This talk is best for an audience that is familiar with Jupyter notebooks. Prior accessibility knowledge or any other Jupyter knowledge is not necessary. The content is likely to be most engaging for an audience who regularly authors notebooks. The structure of the talk will be as follows: 1. Background and introduction to accessibility (7 minutes) 1.1 Why this talk? (Hint: community members have requested it) 1.2 Defining accessibility and scoping: what we will and won’t cover in the talk 1.3 Common terms (disability, WCAG, assistive technology) 2. Breaking down the notebook with WCAG (13 minutes) 2.1 Perceivable elements (Labels, colors, alternative forms of content) 2.2 Operable elements (Labeling for interactive areas, keyboard controls) 2.3 Understandable writing and structure (Markdown headings, summaries, plain language) 3. Adopting a notebook accessibility checklist (2 minutes) 4. What you can do next (2 minutes) 5. Questions (6 minutes) At the end of this talk, attendees will * Have an awareness of foundational accessibility principles and how they can appear in Jupyter notebooks. * Be able to identify common accessibility pitfalls (ie. misused Markdown, incomplete visualizations, etc.) in Jupyter notebooks and what to do instead. * Have a checklist for easy reference of accessibility best practices when writing their own notebooks or editing existing ones. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/VGAUQN/ Amphitheater 204 Stephannie Jimenez Gacha Isabela Presedo-Floyd PUBLISH 7ZGCQM@@cfp.scipy.org -7ZGCQM Scientific and technical publishing with Python and Quarto en en 20230713T150000 20230713T153000 003000 Scientific and technical publishing with Python and Quarto Research and data science isn’t just experiments and code, it’s also communicating about our results, creating reports, sharing analyses, and teaching. To communicate effectively, we need to weave together narrative text and code to produce elegantly formatted, interactive output. Not only does it need to look great, but it needs to be reproducible, accessible, easily editable, diffable, version controlled and output in a variety of formats, such as PDF, HTML and MS Word. Jupyter has already made so much of this possible! The open-source publishing platform, Quarto, combines with Jupyter, or is a VS Code extension, so that we can easily use the output format and the styling that’s needed for any situation. You can author documents as plain text markdown or Jupyter notebooks with scientific markdown, including equations (LaTeX support!), citations, cross references, figure panels, callouts, advanced layouts, and more. Quarto (https://quarto.org/) is a markdown format that adds executable Python code blocks and build on top of Pandoc to produce a variety of output documents. This allows you to create fully reproducible documents and reports—the Python code required to produce your output is part of the document itself, and is automatically re-run whenever the document is rendered. This means you can create documents as plain text markdown or Jupyter notebooks that can be easily rendered into presentations, websites and manuscripts in a variety of journal formats. You can also engage readers by adding interactive data exploration to your documents using Jupyter Widgets, htmlwidgets for R, Observable JS, and Shiny. In this talk, I’ll discuss how to author these dynamic, computational documents with Quarto and Python, showing how to get started and highlighting a few of my favorite things. I’ll walk through how to use a single source document to target multiple formats - transforming a simple document into a presentation, a scientific manuscript, a website, a blog, and a book in a variety of formats including HTML, PDF and MS Word. I’ll share workflows for creating and automating reports, an approach to creating online lessons, and finally how to publish Jupyter notebooks within existing content management systems like Hugo, Docusaurus, and Confluence, so that you can get started creating whatever content you need. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/7ZGCQM/ Amphitheater 204 Tracy Teal PUBLISH RVLFPB@@cfp.scipy.org -RVLFPB Taming Black Swans: Long-tailed distributions in the natural and engineered world en en 20230713T155000 20230713T162000 003000 Taming Black Swans: Long-tailed distributions in the natural and engineered world You would think we'd be better prepared for disaster. But events like Hurricane Katrina in 2005, which caused catastrophic flooding in New Orleans, and Hurricane Maria in 2017, which caused damage in Puerto Rico that has still not been repaired, show that large-scale disaster response is often inadequate. Even wealthy countries -- with large government agencies that respond to emergencies and well-funded organizations that provide disaster relief -- have been caught unprepared time and again. The are many reasons for these failures, but one of them is that rare, large events are fundamentally hard to comprehend. Because they are rare, it is hard to get the data we need to estimate their likelihood precisely. And because they are large, they challenge our ability to imagine quantities that are orders of magnitude bigger than what we experience in ordinary life. In terms introduced by Nassim Taleb, a "black swan" is a large, impactful event that was considered extremely unlikely before it happened, based on a model of prior events. If the distribution of event sizes is actually long-tailed and the model is Gaussian, black swans will happen with some regularity. However, black swans can be "tamed'' by using appropriate models, including lognormal, Student t, and Pareto distributions. In this talk, I introduce these distributions and show how they can be used to model measurements from natural and engineered systems -- including earthquakes, craters on the moon, solar flares, file sizes, and stock market crashes. We will use distributions and optimization tools from SciPy to estimate parameters and generate predictions, and Matplotlib to visualize the results. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/RVLFPB/ Amphitheater 204 Allen Downey PUBLISH X8KZ3E@@cfp.scipy.org -X8KZ3E View, annotate, and analyze multi-dimensional images in Python with napari en en 20230713T163000 20230713T170000 003000 View, annotate, and analyze multi-dimensional images in Python with napari napari is an n-dimensional image viewer for Python. If you’ve ever tried `plt.imshow(arr)` and made Matplotlib unhappy because `arr` has more than two dimensions, then napari might be for you! The napari canvas can be 2D or 3D. When you give napari an array with more dimensions than the canvas, it will automatically create sliders for those additional dimensions, allowing you to rapidly explore your full data, rather than a few sampled slices. Image analysis and visualization involves more than images though: feature detection algorithms result in *points*, segmentation results in *label images*, annotation results in *shapes* such as rectangles or polygons, and more. Napari provides *layers* that can be displayed on top of each other or side by side, allowing users of Scientific Python to gain a rapid understanding of the algorithms they’re using — where they work well and where they might go wrong. Sometimes, image analysis algorithms get you *this* far, but not quite far enough. In such cases, it’s useful to manually curate their output, then continue with downstream steps of an analysis. Napari provides editing tools for its layer types, allowing one for example to add missing points to the output of a peak detection algorithm, remove incorrect ones, paint over incorrect parts of a segmentation, or draw polygons around missed objects of interest. The resulting data points are saved in standard Scientific Python data structures, such as NumPy or Zarr arrays. This design makes it easy to seamlessly weave together image exploration, image computation, processing, and analysis, and data annotation, curation, and editing. Napari provides a *plugin interface*, allowing developers to extend napari’s capabilities, providing users with novel ways to interact with their data. Because napari provides both a library accessible within Python, IPython, and Jupyter, *and* a standalone executable script, we have even found that napari plugins can be an effective way to help collaborators run Python image analysis workflows without needing to launch Python. In this talk, I’ll introduce napari’s history, demonstrate all the features described above, and discuss current limitations and where we’re going. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/X8KZ3E/ Amphitheater 204 Juan Nunez-Iglesias PUBLISH MFQQRJ@@cfp.scipy.org -MFQQRJ Interactive Analysis of Satellite Imagery with Earth Engine and Geemap en en 20230713T104500 20230713T111500 003000 Interactive Analysis of Satellite Imagery with Earth Engine and Geemap The Earth is constantly changing, which creates significant challenges for the environment and human society. To tackle these challenges on a global scale, the Earth science community relies heavily on geospatial datasets that are collected through various means, such as satellite, aerial, and mobile sensors. However, the explosive growth of geospatial datasets over the past few decades has overwhelmed the Earth science community's capacity for storage, analysis, and visualization. Fortunately, the advent of cloud-computing platforms, such as Google Earth Engine, has made it possible to access, manipulate, and analyze large volumes of geospatial data on-the-fly. In recent years, Earth Engine has become increasingly popular in the geospatial community and has enabled numerous Earth science applications at local, regional, and global scales. The geemap Python package is built upon the Earth Engine Python API and open-source mapping libraries. It allows Earth Engine users to interactively manipulate, analyze, and visualize geospatial big data in a Jupyter environment. Since its creation in April 2020, geemap has received over [2,500 GitHub stars](https://github.com/giswqs/geemap/stargazers) and is being used by over [800 projects](https://github.com/giswqs/geemap/network/dependents) on GitHub. More than [130 Jupyter notebook examples](https://geemap.org/tutorials/) and an [open-access book](https://book.geemap.org/) are available for learning geemap. This presentation introduces Earth Engine and highlights the key features of geemap for interactive mapping and geospatial analysis with Earth Engine, such as - Searching and loading datasets from the Earth Engine Data Catalog - Visualizing raster and vector datasets interactively - Using Cloud Optimized GeoTIFFs (COG) and SpatioTemporal Asset Catalogs (STAC) - Visualizing the Dynamic World global land cover datasets - Creating satellite timelapse animations This presentation is intended for scientific programmers, data scientists, geospatial analysts, and concerned citizens of Earth. Attendees should have a basic understanding of Python and Jupyter Notebook. Familiarity with Earth science and geospatial datasets is not necessary, but it will be helpful. For more information about Earth Engine and geemap, visit https://earthengine.google.com and https://geemap.org. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/MFQQRJ/ Grand Salon C Steve Greenberg Qiusheng Wu PUBLISH BAD9ZQ@@cfp.scipy.org -BAD9ZQ Accelerating the Use of Public Geophysical Data for Recharging California’s Groundwater en en 20230713T112500 20230713T115500 003000 Accelerating the Use of Public Geophysical Data for Recharging California’s Groundwater California's Central Valley is one of the world's most productive farmland, but the region faces a serious threat to groundwater sustainability due to population growth and climate change. Recharging ground aquifers is essential to address this challenge, however a major data gap exists in the subsurface. Geophysical data can provide crucial information about the subsurface, but neither the data nor the analytic tools required to derive subsurface information is readily accessible to those working on the recharge problem. In this talk, we will present our development of a web-application and companion public database for accelerating groundwater recharge in California, which is a part of the Sustainability Accelerator Project funded by Stanford Doerr School of Sustainability. Our application uses electrical resistivity data obtained from electromagnetic geophysical surveys, as well as ancillary data from driller's logs (containing information about sediment/rock) and water level/quality measurements, to create 2D maps of recharge metrics. These maps guide the location of recharge sites, and the public resistivity and ancillary data are compiled into an online database using Redivis and displayed in a custom web-application. The application provides project partners the ability to utilize research codes without requiring knowledge of Python, and is flexible to allow updates by researchers to support rapid changes and feedback from partners to meet their specific needs for a recharge site location. The development of the web-application was a collaborative effort between academic researchers and software engineers at Curvenote. The application enables direct use of research code by front-facing practitioners tackling the recharge problem in California. We utilized open source Python packages, to create Jupyter Notebooks that can execute each stage of the workflow. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/BAD9ZQ/ Grand Salon C SEOGI KANG PUBLISH XMBALS@@cfp.scipy.org -XMBALS UXarray, a python library for unstructured climate and weather data en en 20230713T142000 20230713T145000 003000 UXarray, a python library for unstructured climate and weather data After less than a year of development, UXarray has already become a popular Python repository with an active community engagement, boasting more than 10 forks and 77 stars on GitHub. The UXarray project aims to bridge the gap between traditional operations on structured grids and modern standards for unstructured grids, such as the UGRID specification. Global climate models have traditionally used rectangular latitude-longitude grids for their data layout, but these grids lead to computational challenges at high resolutions due to the convergence of lines of longitude at the poles. Therefore, modeling centers worldwide have adopted unstructured grids that allow for quasi-uniform distribution of data over the sphere. However, analyzing data on these grids is far more difficult than on latitude-longitude grids, often requiring groups to apply lossy regridding to their data so that traditional tools can be applied. To partly address this problem, groups worldwide have moved towards the adoption of standards for unstructured grid data, such as the UGRID specification developed under the Climate-Forecast (CF) conventions. Most climate models output data in the NetCDF format, and the CF conventions are an important standard for organizing the metadata of these files and includes details on how to describe a rectangular latitude longitude grid. The UGRID specification describes how a NetCDF file can represent an unstructured grid, but it has potential issues. Currently, the UGRID specification is under consideration to be included in the netCDF CF conventions. Our new Python library, UXarray, supports operations directly on unstructured grid data, reducing the need for creating regular-grid copies of unstructured grid output and simplifying the workflow. Unstructured grids can be provided in files following various conventions, such as UGRID, SCRIP, EXODUS, etc. These conventions have different definitions and representations of the attributes and variables used to describe the unstructured grid topology. Moreover, the UGRID convention does not enforce standard variable namings for most of the attributes and variables, except for a few required ones. UXarray unifies all of these conventions at the data loading step by representing grids internally in the UGRID convention, regardless of the original grid file type. Furthermore, it uses a set of standardized names for topology attributes and variables, while still providing the user with the original attribute names and variables from the grid definition file. All of these features lay the foundation for the development of quick and efficient algorithms for climate scientists around the world. Our design for UXarray aims to maintain Xarray interoperability, which allows us to utilize various Xarray-compatible packages. UXarray uses Numba for loop optimizations and faster computation. Additionally, we provide examples and performance metrics showcasing interoperable read/write operations, grid and corresponding data reading, efficiency and optimization built into UXarray, and visualization. Overall, UXarray aims to simplify the workflow for climate and weather scientists working with unstructured grids and allow them to efficiently analyze and visualize their data. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/XMBALS/ Grand Salon C Rajeev Jain PUBLISH QYHD3G@@cfp.scipy.org -QYHD3G Introducing yt_xarray en en 20230713T150000 20230713T153000 003000 Introducing yt_xarray A number of recent efforts within the [*yt*](https://yt-project.org/) community have broadened the scope of scientific domains supported by *yt*. Some of these efforts included improving generic functionality while others focused on adding functionality required for specific domains outside the astrophysics scientific community. For geoscience data in particular, the addition of a geographic coordinate handler and an interface to [*cartopy*](https://scitools.org.uk/cartopy/docs/latest/) for producing maps within the *yt* plotting framework enabled analysis of geographic datasets. Getting the data into *yt*, however, was not as streamlined as it could be; with the exception of some new custom data ingestors (termed "frontends" in yt) for specific geoscience data products, most geoscience data still required manual loading of arrays with generic *yt* loaders. In addition to extra steps for the user, this limitation also required that the data fit entirely within memory. [*yt_xarray*](https://yt-xarray.readthedocs.io/en/latest/) fills this gap in data regularization required for loading geodata in *yt* by leveraging [*xarray*](https://docs.xarray.dev/en/stable/) for reading of data on demand as *yt* needs it. Rather than a traditional *yt* frontend, *yt_xarray* v0.1 introduced an *xarray* `accessor` object that streamlines the creation of *yt* datasets from subsets of fields, simplifying the process of using *yt* with most regularly gridded datasets that *xarray* can load. While the initial release focuses on simply returning a *yt* dataset object for use with any *yt* function, future releases will further simplify access to *yt* functions from *xarray* by providing *yt* function wrappers from within *yt_xarray*. While *yt* and *xarray* have some similarity in that they both load and manpipulate coordinate-referenced arrays, *yt* is inherently is designed primarily for volumetric data while *xarray* supports sets of labeled arrays more generally. This difference informed a number of important design choices in *yt_xarray*, in particular with regards to how chunked arrays are handled. For gridded datasets in *yt*, a physical domain can be subdivided into multiple grid objects so that a single *yt* "chunk" maps to a subdomain of the whole grid. During processing, subdomains are processed sequentially so that data is loaded as needed. In *xarray*, chunks are defined as contiguous index ranges within arrays, with the actual data potentially residing in on-disk files or existing as delayed computations. *yt_xarray* merges these two chunking systems by building *yt* grids that map spatial subdomains to index ranges of *xarray* fields. This allows a 1:1 mapping of *Dask*-*xarray* chunks to yt grid objects but also allows multiple *Dask*-*xarray* chunks to be contained within a yt grid object. In this presentation, we will provide an overview for using *yt_xarray* for loading and analyzing regularly gridded 2D and 3D *xarray* datasets. In addition to the general usage and development plans, we will describe the design of *yt_xarray* with a focus on leveraging the performance benefits of distributed arrays loaded via *xarray*. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/QYHD3G/ Grand Salon C Chris Havlin PUBLISH LCWBBP@@cfp.scipy.org -LCWBBP Tidy Geospatial Cubes en en 20230713T155000 20230713T162000 003000 Tidy Geospatial Cubes The open-source project, Xarray, combines the convenience of labeled data structures inspired by Pandas with NumPy-like multi-dimensional arrays ("cubes") to provide an intuitive and scalable interface for scientific analysis. Xarray is now widely used across many areas of scientific research, with a particularly strong user base in the physical sciences. New users commonly struggle to fit their dataset into the Xarray data model and, in particular, struggle with conceptualizing and constructing an Xarray object that makes subsequent analysis steps easy (“dataset wrangling”). We take inspiration from the “tidy data” concept for dataframes — “datasets structured to facilitate analysis” (Wickham, 2014) — and attempt a definition of tidy data for labeled array objects provided by Xarray. A ‘tidy dataset’ framework will help streamline processing workflows across the physical sciences and provide a set of norms and principles to guide the use and construction of large and complex datasets encountered in these fields. The utility of this exercise is twofold: helping dataset producers construct more useful Analysis-Ready datasets; and developing a set of guidelines that can help users wrangle their datasets into a form that enables convenient analysis with Xarray. In addition, a commonly-defined concept for ‘tidy’ geospatial array data might enable development of ‘tidy’ tools that consume and produce tidy datasets (Wickham, 2014). This presentation will examine three datasets and the processes of ‘tidying’ them. We will demonstrate various ways that a dataset may be ‘untidy’ — not conducive to analysis — and present a useful set of rules to define ‘tidy geospatial cubes.’ The examples we will discuss are: 1) Harmonized Landsat Sentinel-2 (HLS), a dataset of multispectral reflectance measurements, 2) Aquarius, a dataset of remotely sensed sea surface salinity measurements; and 3) ITS_LIVE, a multi-sensor dataset of ice velocity measurements for glaciers and ice sheets based on satellite image pairs. Our presentation will walk through common analytical workflows with these remote sensing datasets and highlight the organizational choices a user must make along the way (related to metadata, variables, coordinates, and dimensions) to efficiently arrive at a computational result with Xarray. Defining a common framework for labeled array objects will ease the learning curve for new users and minimize the time spent on data-wrangling steps. At present, the examples are satellite remote sensing datasets, and we recognize that there might be elements of the ‘tidy Xarray’ definition that are specific to this subdomain. We hope to spark a discussion that will help generalize the presented principles. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/LCWBBP/ Grand Salon C Deepak Cherian Emma Marshall Scott Henderson PUBLISH EPPR7R@@cfp.scipy.org -EPPR7R Climate Model Evaluation Workflow Built on Jupyter Notebooks en en 20230713T163000 20230713T170000 003000 Climate Model Evaluation Workflow Built on Jupyter Notebooks Motivation Within the field of climate modeling, there is a need to run collections of scripts generating plots of common diagnostic metrics of climate model output, for example as models are run with different configurations during development. These scripts often involve manual configuration, and the output is not necessarily well-organized for interpreting and sharing. Jupyter notebooks help address this problem, creating more readable workflows that can be annotated and edited interactively, then easily presented to others as a Jupyter book. However, Jupyter notebooks are not by default parameterizable or runnable in batches. This project addresses this gap by utilizing Papermill to create a package that can run collections of Jupyter notebooks with configurable parameters, cache generated data products, and publish results as a Jupyter book, while continuing to support the interactive development work that Jupyter notebooks enable. This framework is not limited to use within climate modeling; the infrastructure is useful to any data science project that would benefit from a batch-executable, parameterizable, and shareable Jupyter notebook-based workflow. Methods This project uses a number of existing open-source Python tools, building on the Jupyter ecosystem using Papermill as well as Jinja templating, supporting Dask functionality, and publishing a Jupyter book. It brings these tools together to create a powerful workflow that combines their functionality. The project infrastructure will be published as a Python package and on Github, and examples showcasing its functionality will be made available. Results Currently (as of 3/1/23), the project is in the development stage, with several working demos. By the time of the conference, a more complete version will be public on Github with documentation and installable as a Python package, along with examples that can be downloaded and built on. Conclusion We have developed a framework for data analysis using collections of parameterizable Jupyter notebooks, along with infrastructure to support Dask, caching of data products, building a Jupyter book and other features. This is a powerful application of the Jupyter ecosystem and can be applied to a wide range of fields outside of the climate model evaluation use case it was initially developed for. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/EPPR7R/ Grand Salon C Elena Romashkova PUBLISH DQQBWR@@cfp.scipy.org -DQQBWR Keynote - How Open Source Tools Power the Efforts of Biological Data Analysis and Drug Discovery en en 20230713T091500 20230713T100000 004500 Keynote - How Open Source Tools Power the Efforts of Biological Data Analysis and Drug Discovery PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/DQQBWR/ Zlotnik Ballroom Angela Pisco PUBLISH TKQFWU@@cfp.scipy.org -TKQFWU Subpoenas Less Scary en en 20230713T104500 20230713T111500 003000 Subpoenas Less Scary PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/TKQFWU/ Zlotnik Ballroom Rebecca BurWei David Zeber PUBLISH H9FDBV@@cfp.scipy.org -H9FDBV Diversity Luncheon Keynote: How can we protect vulnerable groups while measuring representation in our communities? en en 20230713T121500 20230713T130000 004500 Diversity Luncheon Keynote: How can we protect vulnerable groups while measuring representation in our communities? PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/H9FDBV/ Zlotnik Ballroom Sophia Vargas PUBLISH XSQKSA@@cfp.scipy.org -XSQKSA Using Python to accelerate sustainable aviation fuel research and development en en 20230713T142000 20230713T145000 003000 Using Python to accelerate sustainable aviation fuel research and development Aviation comprises 2-3% of global carbon dioxide emissions and 9-12% of U.S. transportation greenhouse gas emissions. Sustainable aviation fuels have the potential for reducing emissions and environmental impacts; however, due to high costs and high-volume requirements, experimental property testing of bio-based jet fuels is usually conducted years after initial bench-scale experiments are completed. Neglecting to conduct property testing early in the development cycle can lead to wasted investments spent on production of biofuels that do not meet performance expectations. Machine learning has already proven to be a valuable tool for predicting sustainable aviation fuel properties and accelerating research. In 2020, we presented our approach at SciPy (https://www.youtube.com/watch?v=ENOf0IZDla8) to predict high-throughput aviation fuel properties of over 10,000 molecules with molecular descriptors. The correlation analysis and tree-based methods for feature ranking were later published in Fuel (https://doi.org/10.1016/j.fuel.2022.123836). Using the property prediction models, we created the first Python-based, comprehensive, open-source webtool that enables scientists and companies to explore viable bio-based molecules without spending time and money testing in the lab (https://feedstock-to-function.lbl.gov). Because aviation fuels are made of blends of molecules and compounds, our current research focuses on expanding the webtool to predict properties of fuel blends using Fourier transform infrared (FTIR) spectra and experimental property data. Specifically, we use binning and smoothing techniques to reduce experimental noise in more than 6700 FTIR spectra features and use non-negative matrix factorization (NMF) for feature selection to develop models that predict fundamental properties of biofuel blends (e.g., boiling point, flash point, melting point, specific gravity, and kinematic viscosity). The predictive models are also integrated into the webtool to help sustainable aviation fuel research. Our workflow includes using libraries such as Numpy, pandas, scikit-learn to reduce FTIR spectra data into interpretable components to predict properties, and the Tree-based Pipeline Optimization Tool (TPOT) to develop property prediction models with reduced FTIR spectra as features. Specifically, we will discuss methods for coalescing experimental spectra data from different sources, and will present methods for reducing the influence of experimental noise on model performance. We will also discuss using NMF as a dimensionality reduction technique that correctly groups FTIR spectra wavelengths together and results in meaningful features. Additionally, we will address common pitfalls such as defining an applicability domain, and recognizing and limiting the possibility of overfitting. By sharing our experience and lessons learned, we aim to help the community overcome similar challenges when developing models for advancing science, while also demonstrating how a Python-based, open-source webtool can facilitate faster, less expensive bioprocess optimization and scale-up of sustainable aviation fuels. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/XSQKSA/ Zlotnik Ballroom Ana Comesana PUBLISH MEGK33@@cfp.scipy.org -MEGK33 Contributor experience - Why it matters en en 20230713T150000 20230713T153000 003000 Contributor experience - Why it matters Behind every successful open source project is a strong contributor community. Engaging and supporting contributors requires specialized knowledge, experience, and time commitment from project leaders. However, a chronic lack of resources and time often inhibits them to focus on this work. Recognizing these challenges, in late 2021, we created a team of Contributor Experience Leads to support contributors to the four foundational libraries in the Scientific Python ecosystem: NumPy, SciPy, Matplotlib, and pandas. In this presentation, we will share insights from the work of our team and discuss why it is vital for project maintenance and sustainability. We will examine what we have identified as primary goals and priorities for a Contributor Experience team in each project, taking into account project size, structure, and governance model. We will also discuss how this work could be applied to other projects in the SciPy ecosystem. Finally, we will talk about the Contributor Experience Project (https://contributor-experience.org), a community of practice and an open-source community-led project dedicated to developing best practices for onboarding and supporting contributors to open source. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/MEGK33/ Zlotnik Ballroom Noa Tamir Melissa Weber Mendonça Inessa Pawson PUBLISH T3NSL8@@cfp.scipy.org -T3NSL8 Zarr: Community specification of large, cloud-optimised, N-dimensional, typed array storage en en 20230713T155000 20230713T162000 003000 Zarr: Community specification of large, cloud-optimised, N-dimensional, typed array storage Zarr is a data format for storing chunked, compressed N-dimensional arrays and is sponsored by [NumFOCUS](https://numfocus.org/project/zarr) under their umbrella. In this presentation, we will discuss the evolution of Zarr, first introduced at [SciPy 2019](https://youtu.be/qyJXBlrdzBs); the development of the [Zarr Enhancement Process (ZEP)](https://zarr.dev/zeps/) and its use to define the next major version of the [Zarr Specification (V3)](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html); as well as uptake of the format across the research landscape. ### Outline: First, we’ll be talking about: ### Introduction and Working of Zarr (10 mins.) - What is Zarr, and how it works? - The inner workings of Zarr using illustrated graphics - When and Why should you use Zarr? - Extensive pluggable compressors (via [numcodecs](https://github.com/zarr-developers/numcodecs/)) and file-storage systems - What is the [Zarr Specification](https://zarr.readthedocs.io/en/stable/spec/v2.html)? - A summary of the technical specification of Zarr - Adoption of the Zarr specification in various programming languages like Python, C, C++, Java, and Javascript and how all of us form a wonderful community together - Development of Zarr since it was first presented in SciPy 2019 by Alistair Miles - Highlighting some important technical and community milestones since 2019 - Securing grants from [CZI](https://chanzuckerberg.com/eoss/proposals/zarr-a-common-backbone-for-the-scalable-storage-of-annotated-tensor-data/) and getting sponsored by NumFOCUS After this: ### Usage of Zarr across several domains (5 mins.) - Interoperability with Dask, Xarray and Numpy - Adoption of Zarr by various communities like Geospatial, Bio-imaging, Genomics, Data Science/Engineering etc. - Development of convention processes like [GeoZarr](https://github.com/zarr-developers/geozarr-spec) and [OME-Zarr](https://github.com/ome/ome-zarr-py) Then we’ll discuss: ### [ZEP Process](https://zarr.dev/zeps/) (10 mins.) - Need and origin of a community feedback process for the evolution of Zarr specification - How it works? - Transformation from steering council governed to community-owned specification - Learnings when migrating from [Spec V2](https://zarr.readthedocs.io/en/stable/spec/v2.html) → [Spec V3](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html) And finally: ### Conclusion (5 mins.) - Key takeaways - How can you get involved? - QnA This talk aims to address an audience who works with large amounts of data and is looking for a format which is transparent, open-source, reliable, cloud-optimised, and friendly to the environment. Also, we’d like to invite anyone interested in the lessons we learnt by maintaining the project throughout the years. The tone of the talk is set to be informative, story-telling and fun. ### After this talk, you’d: - understand the basics of Zarr and its specification, - know why you should have a process for your project, - have essential takeaways regarding when an OSS project transitions from a young to a mature stage - as well as the pros and cons of a steering council vs a community-owned open-source project PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/T3NSL8/ Zlotnik Ballroom Sanket Verma Josh Moore John Kirkham PUBLISH UT3CUZ@@cfp.scipy.org -UT3CUZ Building MetPy for the Long Term: Working to Keep an Open Source Project Sustainable en en 20230713T163000 20230713T170000 003000 Building MetPy for the Long Term: Working to Keep an Open Source Project Sustainable MetPy is an open-source Python package for meteorological and atmospheric science applications, leveraging significantly many other pieces of the scientific Python stack (e.g. numpy, matplotlib, scipy, etc.). Its goal is to provide tested, reusable components suitable to a wide array of tasks, including scripted data visualization and analysis. The guiding principle is to make MetPy easy to use with any dataset that can be read into Python. MetPy’s general functionality breaks down into: reading data, meteorological calculations, interpolation, and meteorology-specific plotting. MetPy also has significant integration with XArray, as well as extended support for interpreting netCDF Climate and Forecasting Convention metadata. As a scientific software project that has actively solicited users across the research and education spaces, MetPy has placed a heavy emphasis on the sustainability of the project. Too often core scientific libraries fall into disarray, with a heavy toll on the reproducibility of scientific results. Even given our strong institutional support, our goal with the MetPy project is to build the project with an eye to these potential problems and keep the project sustainable as much as possible. One axis of sustainability for us lies on the side of technology and project infrastructure, which has been highly automated. This starts with our unit tests and test coverage, run automatically on GitHub, using its Actions service. These tests are run across a variety of OS, python version, and package manager combinations, as well as covering a wide array of sets of dependencies. This gives us great coverage of potential breakages. This also extends to automated documentation builds and publication, link checking, code quality checks, and, most importantly, making releases. This combination of processes, built heavily on the Github Actions service minimizes the need for humans in the loop of standard software development steps, allowing us to maximize the use of development time elsewhere. Technological automation is important for sustainability, but it’s only one part of the equation; in order to have a truly sustainable open source project, you must also be solving the issue of people. MetPy follows open development practices to drive community participation as much as possible. We use Github issues, pull requests, discussions, and projects extensively to allow input from any interested user. We also hold regular, open developer calls to keep the project moving forward; we have also started holding community calls to try to give the community more of a voice and input into the direction of the project; these are also done with a goal to encourage more of the community to become involved directly with MetPy development. This talk will share our lessons learned, both with technology and people, to help other projects who want to try to improve their overall sustainability. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/UT3CUZ/ Zlotnik Ballroom Ryan May PUBLISH HWW7S7@@cfp.scipy.org -HWW7S7 Lightning Talks en en 20230713T172000 20230713T182000 010000 Lightning Talks PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/HWW7S7/ Zlotnik Ballroom PUBLISH SZP3LA@@cfp.scipy.org -SZP3LA [BoF Room 103] PyArrow in pandas and Dask en en 20230713T131500 20230713T141000 005500 [BoF Room 103] PyArrow in pandas and Dask PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/SZP3LA/ Classroom 103 Matt Harrison James Bourbeau Patrick Hoefler PUBLISH GEDFS7@@cfp.scipy.org -GEDFS7 [BoF Room 103] Python Visualization and App Tools en en 20230713T183000 20230713T192500 005500 [BoF Room 103] Python Visualization and App Tools PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/GEDFS7/ Classroom 103 Sophia Yang Kushal Kolar Bane Sullivan Juan Nunez-Iglesias Elliott Sales de Andrade Jon Mease Nathan Jessurun Hadley Wickham James A. Bednar PUBLISH A9EGX9@@cfp.scipy.org -A9EGX9 [BoF Room 104] Where on Earth is my Pixel? en en 20230713T131500 20230713T141000 005500 [BoF Room 104] Where on Earth is my Pixel? PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/A9EGX9/ Classroom 104 Juan Nunez-Iglesias Josh Moore PUBLISH 7DDDWU@@cfp.scipy.org -7DDDWU [BoF Room 104] Funding Open Source Software en en 20230713T183000 20230713T192500 005500 [BoF Room 104] Funding Open Source Software PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/7DDDWU/ Classroom 104 Demitri Muna Paige Martin PUBLISH 3HXLZV@@cfp.scipy.org -3HXLZV [BoF Room 105] Scientific Python Ecosystem Coordination en en 20230713T131500 20230713T141000 005500 [BoF Room 105] Scientific Python Ecosystem Coordination PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/3HXLZV/ Classroom 105 Jarrod Millman Stéfan van der Walt Juanita Gomez PUBLISH H3GNAT@@cfp.scipy.org -H3GNAT [BoF Room 105] Scientific Python Packaging Summit en en 20230713T183000 20230713T192500 005500 [BoF Room 105] Scientific Python Packaging Summit PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/H3GNAT/ Classroom 105 C.A.M. Gerlach Henry Schreiner III PUBLISH DQR9NU@@cfp.scipy.org -DQR9NU New CUDA Toolkit packages for Conda en en 20230714T104500 20230714T111500 003000 New CUDA Toolkit packages for Conda Based on feedback from package maintainers and end users, we’ve extended and restructured the CUDA Toolkit packages in conda-forge. We’ve added new packages for CUDA components that were requested. Also we’ve more finely split out CUDA toolkit packages by CUDA component to provide package maintainers and end users a light-weight, precise method for including and stating CUDA dependencies. In addition to the CUDA redistributable libraries already available, we have included compilers, debuggers, profilers, etc. Thus providing users of the conda-forge channel a full development suite that they can use in their own projects. Also these greatly simplify the build infrastructure in conda-forge. Finally more libraries are included, which will allow package maintainers to enable additional features in recipe builds. Similarly packages have become more granular. Each component of the CUDA toolkit is separated out. Further components are split into packages used at build time and run time. Maintainers of packages can now select which components they depend on for a build and only depend on the needed shared library at runtime. In terms of the package ecosystem, this makes CUDA component usage legible in downstream recipes and packages, which can make updates more targeted and easier to manage. For end users all of this means quicker downloads, more compact installs, and a smoother upgrade path. To aid package maintainers and users in leveraging this new functionality, we will share the overall package structure and how this is integrated into conda-forge. Also we will share examples from recipes on how these CUDA packages can be used. Similarly we will show how these packages can be integrated into development workflows. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/DQR9NU/ Amphitheater 204 Rick Ratzel John Kirkham Thomson Comer PUBLISH T7DTX8@@cfp.scipy.org -T7DTX8 Python Array API Standard: Toward Array Interoperability in the Scientific Python Ecosystem en en 20230714T112500 20230714T115500 003000 Python Array API Standard: Toward Array Interoperability in the Scientific Python Ecosystem This talk will have the following outline: * A motivating example, adding array API standard usage to a real-world scientific data analysis script so it runs with CuPy and PyTorch in addition to NumPy. * History of the Data APIs Consortium and array API specification. * The scope and general design principles of the specification. * Current status of implementations: * Two versions of the standard have been released, 2021.12 and 2022.12. * The standard includes all important core array functionality and extensions for linear algebra and Fast Fourier Transforms. * NumPy and CuPy have complete reference implementations in submodules (numpy.array_api). * NumPy, CuPy, and PyTorch have near full compliance and have plans to approach full compliance * array-api-compat is a wrapper library designed to be vendored by consuming libraries like scikit-learn that makes NumPy, CuPy, and PyTorch use a uniform API. * The array-api-tests package is a rigorous and complete test suite for testing against the array API and can be used to determine where an array API library follows the specification and where it doesn’t. * Future work * Add full compliance to NumPy, as part of NumPy 2.0. * Focus on improving adoption by consuming libraries, such as SciPy and scikit-learn. * Reporting website that lists array API compliance by library. * Work is being done to create a similar standard for dataframe libraries. This work has already produced a common dataframe interchange API. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/T7DTX8/ Amphitheater 204 Ralf Gommers Stephannie Jimenez Gacha Leo Fang Saul Shanabrook Travis Oliphant Matthew Barber Aaron Meurer Thomas J. Fan John Kirkham Stephan Hoyer Tyler Reddy Andreas Mueller Athan Reines Mario Alexandre Passos PUBLISH A7EZZV@@cfp.scipy.org -A7EZZV What happens when the main maintainer of a project takes a step down? en en 20230714T131500 20230714T134500 003000 What happens when the main maintainer of a project takes a step down? Free and open source software is made by the community, for the community, off the community. The community is made out of amazing people who are human beings. The Python community would not be what it is without people. Some people of these people are maintainers of projects. They devote a significant amount of their time to guarantee the health of a project, review new contributions, solving questions... The community recognizes their effort and usually evolves around them. However, what happens when a maintainer steps down of a project? How does the community react to this situation? What about for tiny projects? This talk presents some key concepts for building a healthy community around a project to guarantee its survival over time. These key concepts include not only good coding practices like documentation but also the creation of community meetings for everyone, promotion of software, financial support, and tons of passion among others. As an example, the case of "poliastro" is used. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/A7EZZV/ Amphitheater 204 Jorge Martínez PUBLISH EDZ9YB@@cfp.scipy.org -EDZ9YB Better (Open Source) Homes and Gardens with Project Pythia en en 20230714T135500 20230714T142500 003000 Better (Open Source) Homes and Gardens with Project Pythia A “community garden” metaphor is particularly apt for a free- and open-source software project and community. Enthusiasm, creativity, and openness work both for the SciPy conference and Albany NY’s Tulip Festival. But a “garden”, be it botanical or cyber, requires nurturing. With regard to free- and open-source software, there are bounteous examples. Pull requests (PRs) are sown and merged; Issues are resolved, and bugs are removed. Yet we also see signs of formerly fruitful repositories that have been left to languish. Issues proliferate like weeds; bugs roam freely, and eventually the repos’ stars fade away. It is incumbent on the SciPy community to ensure that the projects we are invested in take the more fruitful path. One such open source “greenspace” is Project Pythia (hereafter Pythia). Now in its 3rd year, Pythia extends Pangeo by providing an educational and training hub for the geoscientific Python community. It has three key components: 1. Foundations: The core geoscientific Python stack (JupyterBook) 2. Cookbooks: Advanced and domain-specific workflows (JupyterBooks) 3. Resource Gallery of externally-hosted geoscientific Python resources Here we discuss Pythia’s infrastructure, which sustains the above components in a year-round “community garden”. Pythia’s content is built upon an open stack of infrastructure for reproducibility and collaboration that provides for the care and nurturing of the community it serves. We have built a cloud-based publishing system upon Jupyter Book that automates notebook execution in a reproducible, curated environment. Users can interact with notebooks via Binder links, launching directly into an identical environment. The platform provides automated code- and link-checking, ensuring a rapid healing cycle. Collaboration is achieved through PRs that trigger the same execution infrastructure and a rich preview. Our infrastructure relies on GitHub, which encourages open development via PRs. Pythia uses this process extensively for building and maintaining its “garden”, for the core team and community contributions. GitHub’s focus on collaboration provides users a sense of ownership of whatever “garden” they choose to visit, and provides a path for others to visit and contribute. GitHub’s Actions power Pythia’s automation of key steps in the notebook execution/publishing process. We periodically re-run the publication workflow as health checks for on-going maintenance of the materials, as well as for new “plantings” via PRs. Pythia’s web portal displays the updated content, which users can download to try out and build on in their own “backyard gardens”–computing environments. A garden may need more powerful tools. While GitHub Actions may often suffice, real-world scientific workflows have compute and data requirements that exceed GitHub’s free resources. Pythia’s notebooks can also be executed on our dedicated cloud using BinderHub, which provides a way to execute notebooks within custom environments. Pythia’s workflows are able to validate and deploy results directly from execution on its BinderHub. The same BinderHub instance powers interactive user sessions, guaranteeing that users execute code in the same environment in which the rendered web pages were built. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/EDZ9YB/ Amphitheater 204 Kevin Tyle Drew Camron PUBLISH 9JTLCF@@cfp.scipy.org -9JTLCF Community-first open source: An action plan! en en 20230714T143500 20230714T150500 003000 Community-first open source: An action plan! Open source communities come in a lot of different flavors and have many different ways of operating. However, there is a common thread of promoting kindness in communication, improving the contributor and user experience, and working to make the project more inclusive, accessible, and sustainable. We, the presenters, recently worked to transition a company-backed open source project, Nebari (https://nebari.dev/), to be more community-oriented in its development, maintenance, and governance. We focused on creating a community-first foundation that builds on years of learnings from other leading communities, including Jupyter, NumPy, Gatsby JS, and more. In this talk, we want to share our journey and the things we learned along the way. We aim to provide a step-by-step guide for open source projects looking to adopt more community-driven practices. We will discuss everything from repository management, and contributor and maintainer pathways, to documentation and governance principles. This talk will be most helpful for projects in their formative stages and projects transitioning from company-backed models, however we feel everyone can learn something new to implement in their communities. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/9JTLCF/ Amphitheater 204 Pavithra Eswaramoorthy Dharhas Pothina PUBLISH AXPZZG@@cfp.scipy.org -AXPZZG Small Town Police Accountability: A Data Science Toolkit en en 20230714T104500 20230714T111500 003000 Small Town Police Accountability: A Data Science Toolkit Recent years have highlighted the urgent need for transparency and accountability within police departments across the United States. Typically, large cities have access to policing data and the resources to analyze and interrogate such data to hold authority accountable. Small towns face the same injustices at the hands of police, but these issues receive comparatively little attention, in part due to a lack of resources and tools to investigate the data. Additional challenges arise in the clarity and consistency of the data that may be available. Consequently, the public are generally unable to take data-informed action toward social justice in these regions. The overarching goal of the The Small Town Police Accountability (SToPA) Research Lab is to create an adaptable tool that enables small-town residents to analyze police actions to increase transparency and accountability. This talk will introduce the interdisciplinary work of the research group to (1) obtain data through digital portals and records requests, (2) create a flexible, scaffolded software toolkit for organizing and analyzing police data for users with various levels of technical expertise and (3) use data-driven modeling tools to uncover potential patterns and anomalies in select small town data, serving as a template for investigations elsewhere. The SToPA toolkit consists of a range of components including instructions for data gathering; adaptable tools for reading, cleaning, and organizing data; and machine learning applications to analyze and understand patterns in policing. Using case studies of a handful of small towns, the SToPA toolkit provides a broadly applicable methodology for reading and parsing police data. Where data is available online in a somewhat structured format, the SToPA library offers tools for web crawling and scraping. In other cases, data is only available as a printed physical copy, necessitating digitization, text identification using tools such as PyTesseract, word-level data cleaning, and testing for accuracy. This pipeline includes the use of user-defined, non-standard language dictionaries (such as a list of town-specific locations), geometric methods for word location detection, regular expressions, and fuzzy string matching. After data is collected, cleaned, and structured, a second thrust of the SToPA lab is to analyze police interactions with machine learning and statistical tools. A diverse set of policing data, including dates, locations, names, and free text narratives, yields rich opportunity for exploratory analysis and modeling. Explorable maps were created with various mapping and plotting libraries, revealing location-based patterns. Town-specific data from the US Census allows for demographic comparisons between how citizens are distributed vs. how they are policed. This analysis is further refined using statistical sampling and inference tools such as scikit-learn and PyEI. Narrative text data, unstructured language across thousands of reports, was also analyzed with natural language processing techniques such as topic modeling. This talk aims to be accessible to a diverse audience and to empower and inspire others to contribute to the growing SToPA repository: https://qsideinstitute.github.io/SToPA/ PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/AXPZZG/ Grand Salon C Anna Haensch Ariana Mendible PUBLISH LMMPRP@@cfp.scipy.org -LMMPRP Using Linear Tracking Data to Estimate Backcountry Recreation Popularity en en 20230714T112500 20230714T115500 003000 Using Linear Tracking Data to Estimate Backcountry Recreation Popularity Official sources (typically governments) provide the cleanest and most trustworthy data. Decades of established standards and years of archived records provide a framework for reliable data collection making it a strong foundations for research. The drawback to official centralized sources is that they often focus on the macro level, and because of this, the data tends to be lower resolution, leaving broad areas of obscurity at a micro level. With a need to geospatially estimate and represent backcountry recreation habits on a statewide level down to square mile grid, our team needed high-resolution datasets. Social media data is high-resolution, dense and valuable, which often leads companies to limit access to their data. App downloads and active user counts fluctuate with the market and long-term utility of an app's data is not guaranteed to last. Despite these limitations, social media offers significantly higher resolution data than official sources. We will discuss the methods we developed to overcome the unique challenges of processing and standardizing social media data so it complements and informs official datasets. We will cover acquiring data from multiple apps using modular methods that can be applied to new apps as older ones become obsolete. It is rare for apps to offer identical metrics, so we developed a flexible approach that can translate different metrics into a standardized form. It is also important that a model addresses the inherent unknowns that lie beyond the app's userbase. Our specific use case uses linear geolocation data gathered from mobile tracking apps. Data comes in the form of GeoJSON coordinates, Google Earth polylines, and shapefiles. We will discuss the specific packages used to read each and store them in a common format. Linear data brings with it unique challenges relative to point and polygon data. We will describe how we used Python to redistribute points along line segments while maintaining a minimum distance between segment vertices as a first step towards standardizing the linear data. We then needed to "thin" the data to minimize spatial bias caused by overlapping line segments and circuitous routes; this will include our reasoning for not averaging or interpolating new data points between each dataset, and how instead we used a method that preserved the integrity of original geolocation data. We will explain the ways multiprocessing and nonlinear data structures were used to process large numbers of vertices when we aggregated all datasets together and ran the thinning algorithm on the combined points; the goal of which was to create an overall presence dataset that represents recreation across the state with minimal spatial bias. We will also review the resulting data structure: coordinate pairs, aggregated metrics and IDs that point back to rows from the original datasets gathered from each app. The presentation will finish with a summary and conclusion on how the resulting presence data was processed using the MaxEnt ecological model to inform and supplement official data sources and provide the state of Arizona with a clearer picture of recreation. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/LMMPRP/ Grand Salon C Vincent Sutherland David C. Folch PUBLISH BDV3EE@@cfp.scipy.org -BDV3EE Allegro and FLARE: Fast and accurate machine learning potentials for extreme-scale simulations en en 20230714T131500 20230714T134500 003000 Allegro and FLARE: Fast and accurate machine learning potentials for extreme-scale simulations Molecular dynamics is a common method for studying molecules and materials at the atomistic level, in which the dynamics of atoms are simulated directly using Newton’s equations of motion. This requires a model for the forces between the atoms, often referred to as a potential. Traditionally, there have been two approaches to computing the interatomic forces. First, there are empirical potentials, which are based on simple, physically motivated functional forms with a few parameters that are fit to match experimental measurements of material properties. These models are fast, but they have limited accuracy and are hard to transfer between applications. The alternative is quantum mechanical methods, which are highly accurate. In return, they are computationally expensive and have limited scalability. In recent years, machine learning potentials (MLPs) have emerged as a compromise in terms of accuracy and computational efficiency. The idea is to generate a small amount of training data with a quantum mechanical method. The MLP learns to reproduce its forces and energies and can be used for large and long-timescale molecular dynamics simulations with an accuracy approaching that of the quantum mechanical method. Allegro and FLARE are two drastically different MLPs. FLARE approximates the energy of an atom as a sparse Gaussian process (SGP) as a function of the atom’s local environment. The environment is encoded in a rotationally invariant vector with high descriptive power. By using an invariant descriptor, FLARE correctly respects the symmetry of the problem. Allegro, on the other hand, exploits the symmetry of the problem by using an equivariant neural network, i.e., a neural network where tensor product layers force the features to systematically transform with the input. While more computationally demanding, the added symmetry information allows Allegro and other equivariant models to be significantly more accurate and data-efficient than traditional models. For extreme-scale simulations, scalability and performance are of utmost importance. Through its model design of avoiding message passing, Allegro is the only scalable equivariant neural network potential, with excellent performance demonstrated up to 100 million atoms. FLARE, being a simpler model, takes this to the extreme and has achieved record scalability and performance, simulating 0.5 trillion atoms on 27,336 NVIDIA V100 GPUs. On the implementation side, Allegro and FLARE are also very different. Allegro is implemented in Python with PyTorch, which allows for a high-level implementation with excellent GPU performance through the JIT compiler. FLARE has a low-level training backend written in C++ with OpenMP, MPI, and Kokkos. The C++ code is conveniently wrapped for Python use with pybind11. In this talk, we will compare and contrast these two methods, discuss lessons learned, and show spectacular scientific applications. Links: Allegro repository: https://github.com/mir-group/allegro Allegro paper: https://www.nature.com/articles/s41467-023-36329-y FLARE repository: https://github.com/mir-group/flare FLARE LAMMPS active learning tutorial: https://bit.ly/flarelmpotf Preprint on FLARE scalability: https://arxiv.org/abs/2204.12573 PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/BDV3EE/ Grand Salon C Anders Johansson PUBLISH EYHNUV@@cfp.scipy.org -EYHNUV A Graph-Neural Network-Based model for rapid prediction of Thermal Transport in Metal-Organic Frameworks en en 20230714T135500 20230714T142500 003000 A Graph-Neural Network-Based model for rapid prediction of Thermal Transport in Metal-Organic Frameworks Metal-organic frameworks (MOFs) are a promising class of porous materials that have potential applications in various areas, including gas storage and separations. However, effective thermal energy management in MOFs is critical to enhancing their performance in these applications. Unfortunately, there is still a lack of understanding regarding the structure-property relationships that govern thermal transport in MOFs. In order to provide a data-driven perspective on these relationships, a large-scale computational screening study was conducted to investigate the thermal conductivity of MOFs. This study utilized classical molecular dynamics simulations to calculate the thermal conductivities of 10,194 hypothetical MOFs generated using the Topology-Based Crystal Constructor (ToBaCCo) code developed in Python. These MOFs comprised 1,015 different topologies, along with 40 types of organic edge building blocks and 38 inorganic and organic nodular building blocks. The study discovered that high thermal conductivity in MOFs is favored by high densities, small pores (<10 Å), and four-connected metal nodes. Moreover, it identified 36 MOFs with ultra-low thermal conductivity (<0.02 W/mK) primarily due to their extremely large pores (~65 Å). Additionally, the study uncovered six hypothetical MOFs with exceptionally high thermal conductivity (>10 W/mK). To handle a large number of MOFs screened, an algorithm was developed to adaptively determine the appropriate plateaued interval of the thermal conductivity vs. correlation time curve based on a set of criteria. The search strategy utilized for finding the optimal plateaued interval involved iteratively performing linear fitting to data segments of 2 ps in length at 1 ps increments if the data was between 0 and 10 ps, and segments of 10 ps length at 5 ps increments if the data was beyond 10 ps. The normalized slopes and normalized average oscillation amplitudes were then calculated with respect to the average thermal conductivity for each of those data segments. Using the 10,194 MOF-thermal conductivity data, a range of state-of-the-art graph neural network-based models, including CGCNN, iCGCNN, MEGNet, DimeNet++, ALIGNN, and others, were trained for the rapid prediction of thermal conductivity in MOFs. Finally, the model that demonstrated the best performance on the test data was applied to screen the Computation-Ready, Experimental (CoRE) MOF database, resulting in the identification of experimentally viable MOF structures with potentially exceptional thermal transport properties. This talk will discuss the ToBaCCo hypothetical MOF crystal generation algorithm, various state-of-the-art GNN architectures, and their implementation in PyTorch. This presentation will be of interest to the wider material science community, particularly those with a passion for deep learning models. The findings of this study have the potential to enhance our understanding of thermal transport in MOFs, paving the way for the development of more efficient MOFs for gas storage and separation applications. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/EYHNUV/ Grand Salon C Meiirbek Islamov PUBLISH F9P3F3@@cfp.scipy.org -F9P3F3 From Espaloma to SAKE: To brew, distill, and mix force fields with balanced briskness, smoothness, and intricacy. en en 20230714T143500 20230714T150500 003000 From Espaloma to SAKE: To brew, distill, and mix force fields with balanced briskness, smoothness, and intricacy. A force field as accurate as quantum mechanics (QM) and as fast as molecular mechanics (MM), with which one can simulate a biomolecular system efficiently enough and meaningfully enough to get quantitative insights, is among the most ardent dreams of biophysicists. Machine learning force forces have been designed to bring us one step closer to this dream, by fitting simpler functional forms to QM data and extrapolating to chemically and geometrically diverse regions. Nonetheless, current state-of-the-art architectures, though approaching or surpassing the quantum chemical accuracy, are by magnitudes slower than MM and manifest various pathologies when it comes to interpretability, generalizability, and stability. In this talk, we introduce our efforts to approach the lotusland from two angles: by making MM force fields more accurate (using a GNN to replace the atom typing schemes, Espaloma) and making state-of-the-art machine learning force fields faster (maintaining local universal approximative power without employing spherical harmonics, SAKE). Along the way, we show a plethora of useful gadgets, including the first unified force field for joint protein--ligand parametrization, an AM1-BCC surrogate charge model thousands-fold faster with error smaller than discrepancies among backends, and a way to forecast the fate of dynamic systems before the simulation even starts. With these, we identify the opportunities and challenges of machine learning force fields design: What interpretable, stable, simple yet expressive function forms to use? How do we bake domain knowledge in, e.g., forces vanish when particles are far and explode when close? Can we detach sophisticated neural networks during inference? Can force fields be uncertainty-aware? And finally how do we stir these ingredients well to achieve the delicious balance between stability and speed and accuracy? PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/F9P3F3/ Grand Salon C Yuanqing Wang PUBLISH HPFVLT@@cfp.scipy.org -HPFVLT Keynote - Responsible AI in Practice: How far we've come and where we're going en en 20230714T091500 20230714T100000 004500 Keynote - Responsible AI in Practice: How far we've come and where we're going PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/HPFVLT/ Zlotnik Ballroom Dr. Rumman Chowdhury PUBLISH ALEQSL@@cfp.scipy.org -ALEQSL Modern compute stack for scaling large AI/ML workloads en en 20230714T112500 20230714T115500 003000 Modern compute stack for scaling large AI/ML workloads Existing production machine learning systems often suffer from various problems that make them hard to use. For example, data scientists and ML practitioners often spend most of their time stitching and managing bespoke distributed systems to build end-to-end ML applications and push models to production. To address this, the Ray community has built Ray AI Runtime (Ray AIR), an open-source toolkit for building large-scale end-to-end ML applications. Ray is a distributed compute framework, powering large scale machine learning models such as OpenAI's ChatGPT. By leveraging Ray’s distributed compute strata and library ecosystem, the Ray AI Runtime brings scalability and programmability to ML platforms. The main focus of the Ray AI Runtime is to provide the compute layer for Python-based AI/ML workloads and is designed to interoperate with popular ML frameworks and other systems for storage and metadata needs. In this session, we’ll explore and discuss the following: Why and what is Ray How AIR, built atop Ray, allows you to program and scale your machine learning workloads easily AIR’s interoperability and easy integration points with other systems for storage and metadata needs AIR’s cutting-edge features for accelerating the machine learning lifecycle such as data preprocessing, last-mile data ingestion, tuning and training, and serving at scale Key takeaways for attendees are: * Ray as a general purpose framework for distributed computing * Understand how Ray AI Runtime can be used to implement scalable, programmable machine learning workflows. * Learn how to pass and share data across distributed trainers and Ray native libraries: Tune, Serve, Train, RLlib, etc. * How to scale python-based workloads across supported public clouds PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/ALEQSL/ Zlotnik Ballroom Jules S. Damji Amog Kamsetty PUBLISH Q9KTXS@@cfp.scipy.org -Q9KTXS Ultra fast visualization of large datasets using modern graphics APIs in jupyter notebooks en en 20230714T131500 20230714T134500 003000 Ultra fast visualization of large datasets using modern graphics APIs in jupyter notebooks Over the past decade, advanced analyses pipelines have been developed for large neuronal datasets [1][2]. However, fast visualization and live interactivity during data collection is largely unsupported. While current tools within the Python plotting ecosystem (ex. *pyqtgraph, VisPy, napari*) allow for interactive data visualization, they either fail to leverage modern GPUs efficiently, lack intuitive APIs for rapid prototyping, or require users to write their own shaders. Additionally, other popular plotting libraries, such as *bokeh* and *matplotlib*, are not geared towards fast interactive visualization with millions of objects. Given these challenges with current visualization tools, the need for a modern GPU-driven interactive plotting library exists. In this presentation, we will go through the technical details, as well as a brief demo on how *fastplotlib* makes fast interactive visualization of complex neuronal datasets possible. We will also demonstrate the broader applicability of *fastplotlib* as a fast, general-purpose plotting library. *Fastplotlib* is built on top of *pygfx* which is a cutting edge Python rendering engine that utilizes *Vulkan*, which can efficiently leverage modern GPU and CPU hardware. *Vulkan*, released in 2016, is the successor to *OpenGL* and features a low overhead with respect to the amount of code per-draw-per-object allowing for speed even when rendering millions of objects. *Pygfx* is also non-blocking, which allows for interactivity and modification of already drawn objects. *Fastplotlib* utilizes the *pygfx* rendering library for fast visualization with an expressive API for scientific visualization. The benefits of *fastplotlib* are that it reduces boilerplate code which allows users to focus on their data without having to manage the underlying rendering process. Additionally, *fastplotlib* allows for animations as well as high-level interactivity among plots, which can be combined with lazy loading of very large neuronal imaging movies that are hundreds of gigabytes or terabytes in size. Furthermore, *fastplotlib* can be used in jupyter notebooks, allowing it to be used on cloud computing and other remote infrastructures. In total, these unique features and the underlying architecture create a plotting library that is fast, easy to use, and multifaceted. Initially, *fastplotlib* was developed for use in the neuroscience community to aid in the analysis of large neuronal datasets. However, the long term goal of this project is to provide an open source software that serves as a general-purpose scientific plotting library. As we are currently in the early stages of development, we are looking for community involvement and to connect with other developers to further progress our software package. https://github.com/kushalkolar/fastplotlib PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/Q9KTXS/ Zlotnik Ballroom Kushal Kolar Caitlin Lewis PUBLISH SXJFBQ@@cfp.scipy.org -SXJFBQ DataJoint: Bringing databases back into data science en en 20230714T135500 20230714T142500 003000 DataJoint: Bringing databases back into data science Research teams work on complex scientific data with many contributors. They execute quickly evolving and complex computational pipelines around such data. This requires a systematic approach to structuring data with clarity and transparency, linking it with distributed computation. Relational databases solve many of these problems; they support data integrity and facilitate queries in large, collaborative repositories. However, working with relational databases through SQL from Python can be awkward. As a result, many data scientists have dismissed relational databases and missed out on their great capabilities. Enter DataJoint, an open-source framework designed explicitly for managing scientific data. DataJoint uses a relational database system as its backend but utilizes Python programming constructs to define and query the database, similar to object-relational mappers commonly used in web development. It is specifically designed from the ground up for supporting complex data and distributed computations, making it an ideal tool for data scientists. One of the most significant advantages of DataJoint is that it allows you to design complex databases directly from a Jupyter notebook. It provides its own sublanguage for defining database schemas to capture relationships between data elements, including beautiful diagrams for convenient navigation. DataJoint also provides a convenient query language that reduces the complexity of SQL select statements into an algebra of five operators. Data operations are well integrated with other data science tools such as numpy and pandas. Most importantly, DataJoint makes computations a first-class citizen in its data model. Computational dependencies are encoded as part of the database design, so the database schema serves to specify the computational data pipeline and workflow. DataJoint has been in continuous development and use for about 14 years and is currently used in approximately a hundred research labs. A rich collection of standardized workflows, DataJoint Elements, has been in development by the research community. In this talk, we will introduce the basic principles of scientific databases, including how to create a database, how to visualize its structure, how to enter and delete data, and how to define and execute computational dependencies. We will also showcase examples from past and current neuroscience projects. For large-scale computations, DataJoint can be combined with job orchestration tools for scalable computing. Furthermore, we will introduce the new DataJoint SciViz library that provides a low-code approach for creating websites for data visualization to show off your work. DataJoint has become a part of the data science tool stack for working with scientific databases, providing the full rigor of relational databases for maintaining data integrity and consistency, especially in dynamic collaborative projects. Finally, we will share some glimpses of our future developments and invite diverse teams to contribute and collaborate, making DataJoint an even more powerful tool for managing scientific data. With DataJoint, scientists can bring relational databases into the modern era of data science and streamline their data management and computational workflows. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/SXJFBQ/ Zlotnik Ballroom Dimitri Yatsenko PUBLISH CQNJ9Z@@cfp.scipy.org -CQNJ9Z An API for efficient and low-latency access to the largest standardized single-cell data repository by CZ CELLxGENE Discover. en en 20230714T143500 20230714T150500 003000 An API for efficient and low-latency access to the largest standardized single-cell data repository by CZ CELLxGENE Discover. As a part of the CZ CELxGENE Discover suite (cellxgene.cziscience.com) we have deployed Python and R APIs to query the largest aggregation of single-cell data from 50 million cells along >60 thousand genes from the major human and mouse tissues. The data is comprised of more than 700 individual datasets represented as a single gene expression count matrix along with metadata data frames, where all cells have harmonized annotations across 11 variables (e.g. cell type, tissue, sequencing technology, donor id, etc) and all gene IDs and labels have been standardized on GENCODE references (https://github.com/chanzuckerberg/single-cell-curation/blob/main/schema/3.0.0/schema.md) . The APIs are able to perform efficient cell-based queries across all cells regardless of the dataset of origin. The concatenated data presents a unique opportunity to apply machine learning on single-cell gene expression at an unprecedented scale for biological discoveries. More importantly, the data and APIs are built around a recently developed technology, TileDB-SOMA, which allows for cloud-optimized storage and access, low-latency access for larger-than-memory slices of data, querying and filtering under lazy evaluation, and transformers to pandas, pyarrow, anndata and Seurat. The APIs are free to use (https://pypi.org/project/cell-census/) and the data is hosted publicly online, which allows users to fetch slices of data with less than 10 lines of code and under 2 minutes. Our main objective is to accelerate biological discoveries by providing ready-to-use standardized gene expression data from 50 million human and mouse cells in an interoperable manner. We are eager to provide the support necessary to enable researchers to effectively use the data and APIs. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/CQNJ9Z/ Zlotnik Ballroom Pablo Garcia-Nieto PUBLISH RTA7JG@@cfp.scipy.org -RTA7JG Lightning Talks en en 20230714T153000 20230714T163000 010000 Lightning Talks PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/RTA7JG/ Zlotnik Ballroom PUBLISH JXWQPG@@cfp.scipy.org -JXWQPG [BoF Room 103] SciPy 2023 Sprint Prep BoF en en 20230714T164000 20230714T173500 005500 [BoF Room 103] SciPy 2023 Sprint Prep BoF PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/JXWQPG/ Classroom 103 Gil Forsyth Brigitta Sipőcz Madicken Matt Davis PUBLISH XNVLQA@@cfp.scipy.org -XNVLQA [BoF Room 103] CPython performance en en 20230714T174500 20230714T184000 005500 [BoF Room 103] CPython performance PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/XNVLQA/ Classroom 103 Michael Droettboom PUBLISH VA7ENC@@cfp.scipy.org -VA7ENC [BoF Room 104] Future of Python Programming Language in the Artificial Intelligence Era en en 20230714T164000 20230714T173500 005500 [BoF Room 104] Future of Python Programming Language in the Artificial Intelligence Era PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/VA7ENC/ Classroom 104 Gajendra Deshpande PUBLISH LGZUNG@@cfp.scipy.org -LGZUNG [BoF Room 104] Beyond Notebooks: From reproducible to reusable research en en 20230714T174500 20230714T184000 005500 [BoF Room 104] Beyond Notebooks: From reproducible to reusable research PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/LGZUNG/ Classroom 104 C.A.M. Gerlach Juanita Gomez PUBLISH LTDRGY@@cfp.scipy.org -LTDRGY [BoF Room 105] Open Source Project Code of Conduct Management and DEI Support en en 20230714T164000 20230714T173500 005500 [BoF Room 105] Open Source Project Code of Conduct Management and DEI Support PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/LTDRGY/ Classroom 105 Noa Tamir Leah Silen Inessa Pawson PUBLISH 9H9KFM@@cfp.scipy.org -9H9KFM [BoF Room 105] SciPy 2024 en en 20230714T174500 20230714T184000 005500 [BoF Room 105] SciPy 2024 PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/9H9KFM/ Classroom 105 SciPy 2023 Committee PUBLISH NMNJYF@@cfp.scipy.org -NMNJYF Open Source Sprints [Kickoff in Room 204] en en 20230715T090000 20230715T100000 010000 Open Source Sprints [Kickoff in Room 204] Sprints FAQs What will you do as an attendee? There are a variety of ways to contribute during the sprints session including testing code, fixing bugs, adding new features, and improving documentation. You could also contribute to an entirely brand new project that our ecosystem is missing. One of the best parts about the sprints is that you might also have the opportunity to work with authors and core contributors of your favorite open source packages, as well as, the opportunity to work alongside other developers who are just as excited as you are to make the SciPy community even better. What are the benefits of attending a sprint? Make open source Python better! Code alongside package authors/contributors, while learning from them. Become a power user of a core package by gaining a deeper understanding of its inner workings. Improve your github profile. Get to know other SciPy community members at the Sprints dinner. Can I participate? Yes! Sprints are free and open to everyone no matter what your programming level of experience. Sprints are a great way to add your contribution to your favorite Python libraries and packages. Thanks to the generosity of our sponsors, sprints are free of charge for all participants, including the Sprints dinner on Saturday evening. If you aren't sure about how you can contribute to a project, it's not a problem. We'll get you up to speed at the How to Contribute to Open Source BoF on Friday and we have helpers at the beginner friendly sprints. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/NMNJYF/ Amphitheater 204 Dr. Tania Allard Brigitta Sipőcz Alan Braz PUBLISH WTNHTR@@cfp.scipy.org -WTNHTR Open Source Sprints [Kickoff in Room 204] en en 20230716T090000 20230716T100000 010000 Open Source Sprints [Kickoff in Room 204] Sprints FAQs What will you do as an attendee? There are a variety of ways to contribute during the sprints session including testing code, fixing bugs, adding new features, and improving documentation. You could also contribute to an entirely brand new project that our ecosystem is missing. One of the best parts about the sprints is that you might also have the opportunity to work with authors and core contributors of your favorite open source packages, as well as, the opportunity to work alongside other developers who are just as excited as you are to make the SciPy community even better. What are the benefits of attending a sprint? Make open source Python better! Code alongside package authors/contributors, while learning from them. Become a power user of a core package by gaining a deeper understanding of its inner workings. Improve your github profile. Get to know other SciPy community members at the Sprints dinner. Can I participate? Yes! Sprints are free and open to everyone no matter what your programming level of experience. Sprints are a great way to add your contribution to your favorite Python libraries and packages. Thanks to the generosity of our sponsors, sprints are free of charge for all participants, including the Sprints dinner on Saturday evening. If you aren't sure about how you can contribute to a project, it's not a problem. We'll get you up to speed at the How to Contribute to Open Source BoF on Friday and we have helpers at the beginner friendly sprints. PUBLIC CONFIRMED Talk https://cfp.scipy.org/2023/talk/WTNHTR/ Amphitheater 204 Dr. Tania Allard Brigitta Sipőcz Alan Braz