07-11, 11:25–11:55 (US/Pacific), Room 315
Extreme weather events threaten industries and economic stability. NOAA’s National Centers for Environmental Information (NCEI) addresses this through the Industry Proving Grounds (IPG), which modernizes data delivery by collaborating with sectors like re/insurance and retail to develop practical, data-driven solutions. This presentation explores IPG’s technical innovations, including implementing Polars for efficient data processing, AWS for scalability, and CI/CD pipelines for streamlined deployment. These tools enhance data accessibility, reduce latency, and support real-time decision-making. By integrating scientific computing, cloud technology, and DevOps, NCEI improves climate resilience and provides a model for leveraging open-source tools to address global challenges.
Extreme weather events are increasing, posing risks to industries and economic stability. NOAA’s National Centers for Environmental Information (NCEI) supports informed decision-making through the Industry Proving Grounds (IPG), modernizing data delivery and accessibility.
The IPG bridges industry needs with scientific data by engaging sectors like architecture, re/insurance, and retail to co-develop useful data products. This ensures data services are technically robust and practically applicable, helping industries navigate risks and sustain economic growth.
Modernizing NOAA’s product development infrastructure is key to improving reliability, performance, cost-efficiency, maintainability, and ease of future improvements. Efforts focus on streamlining procedures, enhancing capabilities, simplifying the code base, and leveraging cloud-based solutions. This enables NCEI to provide timely insights while reducing maintenance costs and improving system resilience.
This presentation explores the technical innovations behind IPG’s success. We will discuss standardizing coding language, integrating Polars, a fast Python DataFrame library optimized for large-scale data analysis. Polars' multi-threaded execution and lazy evaluation improve performance when processing NOAA’s vast datasets. Additionally, using AWS enhances data accessibility, scalability, and computational power for real-time processing and distribution.
CI/CD pipelines are crucial to IPG’s efficiency, ensuring continuous integration and deployment of software updates. Automated testing and seamless deployment minimize downtime and enhance responsiveness to industry needs. These practices demonstrate how DevOps methodologies improve scientific computing workflows and data-driven decision-making.
Core tools like Polars, AWS, and CI/CD enable streamlined data ingestion, processing, and dissemination, reducing latency and enhancing data quality. Information centers play a key role in open-source software development by providing real-world feedback, identifying bottlenecks, and improving performance.
This talk highlights how scientific computing, cloud technology, and software engineering contribute to climate resilience. NCEI’s approach serves as a model for integrating open-source tools, cloud services, and CI/CD practices to maximize environmental data’s impact.
Participants will gain insights into science, data engineering, and industry resilience, learning how software engineering best practices can enhance real-world decision-making and tackle global challenges.
Meteorologist and Software Engineer with a passion for leveraging technology to enhance environmental data analysis and decision-making.