SciPy 2025

The rise of deep learning over the last 15 years has given rise to complex non-linear models that are much more difficult to interpret than heuristic or linear regression models, but often have better performance. In response, Explainable AI (XAI) algorithms have been developed to make these models' decisions more understandable. Many of these techniques originated in the computer vision community, driven by the need for interpretability in areas such as medical imaging. While effective for niche image-based tasks, these methods are often not well-suited for direct application to Earth system models.

Earth system models present unique challenges for XAI. These models are high-dimensional, deal with large-scale data, and incorporate physically meaningful and autocorrelated spatial and temporal relationships. Most existing XAI approaches struggle to handle these types of inputs effectively. They often violate physical laws and fail to account for uncertainty, making their explanations less reliable and insightful for model developers and end users. We argue that “physical XAI”—interpretable methods that aim to verify that the model has learned physically meaningful relationships—should not simply involve applying standard XAI algorithms to Earth system models. Instead, it should include customized methods that respect the unique characteristics of the Earth systems. Furthermore, we advocate to expand the definition of “physical XAI” to include meaningful data analysis by domain experts throughout the model development process.

To advance this idea of physical XAI, we introduce an approach that adapts and extends existing XAI techniques in three unique ways. First, we develop novel methods for perturbing features in more physically consistent ways. Many XAI techniques, such as partial dependence plots (PDPs), rely on feature perturbation to assess variable importance. However, autocorrelated features can cause standard perturbation methods to overestimate or underestimate a variable's importance. To address this, we propose perturbing groups of variables simultaneously relative to their own distributions, ensuring that the perturbed inputs remain physically realistic. This approach generates more consistent and meaningful explanations of model behavior. Secondly, we demonstrate our method on uncertainty values in addition to raw probabilities to gather more meaningful insight. Lastly, we apply our models and feature groups to global sensitivity analysis methods, such as the Sobol Indices method, which are commonly used for dynamic models but rarely used for data-driven models. This can provide higher-order variable (or group) interactions that have physically meaningfulness.

We believe that XAI does not need to be constrained to an algorithm. Domain experts have invaluable knowledge that can be utilized by examining model training data and output. With expert interrogation of the data during the development process we can get a much better understanding of what the model has learned. We demonstrate this process in a variety of ways: examination of physical properties of local instances, physical examination of composites, and interactive examination of model output with user controlled input. This iterative examination often leads to improvements in the data curation, feature engineering, and/or model tuning process.

These physical XAI methods are demonstrated on two Earth system models: a winter precipitation type classifier, and AI weather prediction models within the CREDIT (Community Research Earth Digital Intelligence Twin) framework both developed at NSF NCAR. In the CREDIT framework, we perform physically realistic perturbations of the atmospheric state in models trained with physical constraints vs those trained without to analyze the effect of the physical constraints on propagating changes through the atmosphere and provide a Jupyter notebook that enables anyone to do so. For the precipitation type model, we provide a jupyter widget that allows a user to interactively adjust the vertical atmospheric profiles and examine how the model probabilities and uncertainties change as a result. Additionally, we provide jupyter notebook demonstrations of physical explainability using the Sobol Indices variance-based approach and our novel extension to partial dependence plots.