SciPy 2024

Building Multi-Agent Generative-AI Applications with AutoGen
07-10, 13:55–14:25 (US/Pacific), Ballroom

Discover the potential of multi-agent generative AI applications with AutoGen, a pioneering framework designed to tackle complex tasks requiring multi-step planning, reasoning, and action. In this talk, we will explore the fundamentals of multi-agent systems, learn how to build applications using AutoGen, and discuss the open challenges associated with this approach, such as control trade-offs, evaluation challenges, and privacy concerns.

With AutoGen's open-source platform and growing ecosystem, developers can harness the power of generative AI to create advanced AI assistants and interfaces for the digital world. This talk is ideal for those with a general understanding of generative AI and Python application development.


Generative AI models have made significant progress on tasks such as summarizing passages, extracting entities and generating code etc. However, they currently struggle to address more complicated tasks that require multi-step planning, reasoning, and action - for example, building a complete Android app for displaying stock prices. To create helpful AI assistants that can seamlessly handle these complex tasks, it is crucial to develop agents that can act, collaborate with other entities (including humans), and serve as interfaces to the digital world.

But how do we define such multi-agent workflows, empower developers to build them, and address the open challenges that arise? AutoGen is a pioneering attempt to answer these questions and offers a generic framework for building multi-agent AI applications.

In this talk, we will cover the following:
1. An overview of multi-agent systems. This section will cover agents, provide a definition of complex tasks, and motivate why a multi-agent approach is well-suited to addressing complex tasks.
2. Building multi-agent applications with AutoGen. This section will offer an introduction to AutoGen, the leading framework for building multi-agent applications, explain key concepts and design principles, and provide concrete examples of how to build simple (generate a chart for stock prices) and complex (build an Android App!!) agent workflows
3. Open Challenges with Multi-Agent applications. In this final section, we will discuss open challenges associated with a multi-agent approach, including control trade-offs (autonomy vs deterministic behavior), evaluation challenges, and addressing privacy, security, and continuous learning within these systems.

AutoGen is an open-source project (MIT License) on GitHub - https://github.com/microsoft/autogen. With over 22k stars, more than 200 contributors, and a growing ecosystem of integrations, AutoGen is a vibrant community of enthusiasts prototyping applications across various industry use cases.

The talk is intended for an intermediate audience with some general familiarity with generative AI and writing applications in Python.

Chi Wang is a principal researcher in Microsoft Research. He has worked on large language model and AI frameworks, automated machine learning, machine learning for systems, scalable solutions for data science and data analytics, and knowledge mining from text data and graph data (with a SIGKDD Data Science/Data Mining PhD Dissertation Award). Chi is the creator of AutoGen, a popular and rapidly growing open-source framework for enabling next-gen AI applications (with an Open100 award and TheSequence’s pick of 5 favorite AI papers in 2023). Chi is the creator of FLAML, a fast open-source library for AutoML & tuning used widely inside and outside Microsoft.

Senior Research Engineer at Microsoft Research