SciPy 2025

Flyte is an open source data and machine learning orchestrator built on top of Kubernetes. Although the backend is built with Golang, most developers use the Python SDK, flytekit, for interacting with Flyte. With the core logic written in Golang, we get the benefits of a compiled language while having the developer ergonomics of Python. In this talk, we will learn about the design constraints and implementation details of flytekit’s core features.

• Flytekit enables us to run the same code locally and quickly bring that into a remote cluster by adding a --remote flag. The mechanism for sending our code to the cluster is called “fast registration”, which intelligently finds what our code needs and uploads that to a blob store.

• Flytekit requires the developer to provide types for all functions. Flyte’s “type transformers” use the Python types, so it knows how to serialize data between tasks in a workflow.

• Scientific Python code has many dependencies and building Docker images can be complex. Flytekit’s image builder abstraction enables us to build images without learning Docker and quickly get up and running.

• Flytekit’s Python SDK communicates with the Golang backend using Protobuf. This communication mechanism allows Flyte to interoperate with other languages, such as Javascript or Java.

• Flyte’s plugin system allows us to extend the flytekit with custom code. For example, we can define our own “type transformers”, run Distributed GPU workflows, or use Dask and Ray on Flyte.

By the end of this talk, you will have a deeper understanding of the design decisions and compromises made by flytekit for building a Python SDK used to orchestrate data and machine learning workflows.