07-11, 14:35–15:05 (US/Pacific), Room 317
PhD students, postdocs and independent researchers often struggle when trying to execute code developed locally in the cloud or HPC clusters for better performance. This is even more difficult if they can't count on IT staff to set up the necessary infrastructure for them on the remote machine, which is common in third-world countries. Spyder 6.1 will come with a whole set of improvements to address that limitation, from setting up a server automatically to easily run code remotely on behalf of users, to manage remote Conda environments and the remote file system from the comfort of a local Spyder installation.
The last 10 years have witnessed the rise of the PyData ecosystem to become the lingua franca of scientific computing. Behind that incredible success, no doubt, there has been its open source nature, which allows students and researchers in any part of the world to use it at no cost. However, the fact that the ecosystem is a public good doesn't necessarily make it a democratic one. Although all PyData resources (i.e. its libraries and documentation) are free, the ecosystem still requires a good amount of technical knowledge to set it up and get the most out of it.
This is especially true for remote development. It's not an easy task to move code developed locally to HPC clusters or the cloud in order to access a larger pool of resources (memory, CPUs or GPUs) to execute it. Perhaps the simplest workflow to do that involves logging through SSH to the remote machine, creating a Python environment there, moving your code using the scp command and finally running it. That's certainly not straightforward for many scientists and engineers, but doable.
However, what happens when your code doesn't work because the local and remote environments are slightly different? Or when you need to do some adjustments to make it work remotely? Then the local and remote versions of the code start to diverge and you're more or less forced to work on an SSH terminal for remote development, which is atrocious.
A possible solution to that, using OSS tools, is to set up JupyterHub and JupyterLab on the remote machine and work from the latter. But without IT staff to rely on for help, that could be a daunting endeavor. That's even more difficult in third-world countries because that staff is either small or non-existent. Spyder 6.1 aims to democratize this process by drastically simplifying what is required from users to get started with remote development.
As it'll be shown in the first part of this talk, all users have to do is enter their SSH credentials and decide what packages they'd like to use for their code. Then Spyder will establish a connection to the remote machine, install a server there (called Spyder Remote Services) to handle all facilities required for remote development, and create a Conda environment with the requested packages. Finally, by simply opening an IPython console for the machine, users will be ready to run their code on it and graphically upload/download data sets or other files to/from it.
The second part will describe how the architecture behind these enhancements was carefully designed to make it easily extensible by third-party plugins. Spyder Remote Services is a Jupyter Server extension, so it can be customized by other extensions and also installed in an existing JupyterHub deployment. On the Spyder side, the Remote Client plugin offers a context manager to interact with the server API as needed. This is no small detail because the remote facilities of other IDEs (e.g. VSCode and PyCharm) don't offer any way to do something similar.
I got involved with Spyder in 2010 as a volunteer, and became its maintainer in 2013. I worked for Anaconda from 2015 to 2017. The first two years there I created conda packages for Qt, VTK, Boost, Pandoc, Graphviz, CMake, etc (most of my recipes were used by the Conda-forge team when the project started). In my final year I led a team of three developers working on Spyder. I joined Quansight in 2018 and left in 2022. There I managed a team of five developers (hired by Quansight), also in charge of maintaining Spyder. Since then I've been working on Spyder full time thanks to a CZI grant awarded to the project at the end of 2022.
Python and Spyder core developer, specializing in docs, infra, and UI. Python Docs Team and PEP Editor. Star✦Fleet Commander. Former NASA-funded ML researcher.