Orchestra-o1: A New Framework for Coordinating AI Agents Across Text, Image, Audio, and Video

Multi-agent systems have become the dominant paradigm for tackling complex AI tasks: instead of a single monolithic model doing everything, a conductor routes subtasks to specialized agents that work in parallel. The trouble is, almost every existing orchestration framework was designed with text - or at most text plus images - in mind. Real-world tasks rarely cooperate with that constraint.

A paper posted to arXiv this week tackles that problem head-on. Orchestra-o1 (arXiv:2606.13707) introduces an omnimodal agent orchestration framework that can coordinate agents across text, image, audio, and video simultaneously 1.

The Core Problem: Existing Frameworks Are Modal Mono-taskers

The authors - Fan Zhang, Vireo Zhang, Shengju Qian, Haoxuan Li, Hao Wu, Jinyang Wu, Donghao Zhou, Zhihong Zhu, Zheng Lian, Xin Wang, and Pheng-Ann Heng - frame the gap clearly in their abstract 1. As the paper states, existing orchestration frameworks "are limited to a narrow set of modalities and struggle to generalize to more complex settings where heterogeneous modalities coexist and interact" 1. This limitation is especially pronounced in omnimodal scenarios, "where tasks require the unified understanding and coordination of diverse inputs such as text, image, audio, and video" 1.

Consider a query about a news broadcast clip: it might simultaneously require parsing on-screen text, analyzing audio speech, and understanding visual scene context. No single specialist agent handles all of that well, and no prior orchestration layer was built to route them together efficiently.

How Orchestra-o1 Works

Orchestra-o1 sits above a pool of specialist sub-agents and introduces a unified orchestration mechanism that enables three interlocking capabilities 1:

1. Modality-aware task decomposition - the orchestrator analyzes incoming tasks and identifies which modality (or combination of modalities) each subtask belongs to before assigning it.
2. Online sub-agent specialization - rather than pre-assigning fixed roles, sub-agents are specialized on-the-fly based on task demands, keeping the system adaptive.
3. Parallel sub-task execution - independent subtasks are dispatched concurrently, reducing end-to-end latency on multi-modal queries.

This scalable design is intended to allow agent systems to tackle complex real-world tasks involving heterogeneous information sources 1.

DA-GRPO: Training the 8B Open-Source Variant

Beyond the framework itself, the paper introduces DA-GRPO - Decision-Aligned Group Relative Policy Optimization - a reinforcement learning method tailored for agentic settings 1. DA-GRPO is used to train Orchestra-o1-8B, an open-source 8-billion-parameter model that the authors report achieves state-of-the-art performance among open-source omnimodal agents 1.

This is significant for the broader community. Closed-source frontier models dominate omnimodal benchmarks today, and a capable, openly trainable 8B model with documented agentic RL gives practitioners a concrete foundation to build and fine-tune from.

Benchmark Results: Performance on OmniGAIA

The paper's primary evaluation target is OmniGAIA - an established community benchmark for omnimodal agentic tasks that predates this paper. OmniGAIA was introduced in a separate work focused on benchmarking omni-modal AI agents, and is designed to stress-test "unified perception over vision, audio, and language, together with long-horizon reasoning and multi-turn tool use in realistic scenarios" 2. Its evaluation data spans video-with-audio and image-plus-audio settings, drawing from sources including FineVideo, LongVideoBench, and COCO 2017 2 - making it a substantive external benchmark, not an in-paper construct.

Orchestra-o1 is reported by the authors to surpass the second-best approach by +10.3% accuracy on OmniGAIA 1. Because OmniGAIA is an externally defined benchmark rather than one introduced by the same paper, this result carries more weight than an in-paper evaluation - though, as with any single-paper claim, independent replication and community validation remain the ultimate standard.

Where This Fits in the Orchestration Landscape

The multi-agent orchestration space has been moving quickly. Several concurrent lines of work illustrate the range of approaches:

• Orchestral AI (arXiv:2601.02577) is a lightweight Python framework that "provides a unified, type-safe interface for building LLM agents across major providers," eliminating manual format translation through a single universal representation for messages, tools, and LLM usage 3.
• MAS-Orchestra (arXiv:2601.14652) takes a training-time approach, formulating "MAS orchestration as a function-calling reinforcement learning problem with holistic orchestration, generating an entire MAS at once," in which sub-agents are abstracted as callable functions to enable global system-level reasoning 4.
• AOrchestra (arXiv:2602.03786) introduces a framework-agnostic abstraction that models any agent as a tuple of Instruction, Context, Tools, and Model, enabling the orchestrator to spawn specialized executors for each task on demand 5. Across three challenging benchmarks - GAIA, SWE-Bench, and Terminal-Bench - AOrchestra achieves a 16.28% relative improvement against the strongest baseline when paired with Gemini-3-Flash 5.

A complementary data point comes from a separate June 2026 paper on sandboxed coding agents (arXiv:2606.00579), which found that "coding agents with only text+image access and a sandboxed tool-use interface can match, and in several settings outperform, SOTA native omnimodal models and predefined multimodal agent scaffolds across multiple audio-video benchmarks" 6. Their strength, the authors argue, comes from writing code and orchestrating tools to extract relevant evidence from transcripts, frames, and other modality signals, "thereby converting omnimodal tasks into retrieval and information-processing problems rather than ingesting entire media streams" 6.

Orchestra-o1 takes a different angle from all of these. Rather than optimizing the orchestrator's planning strategy, the tool-calling interface, or sub-agent spawning mechanics, it targets the modality boundary as the primary bottleneck. The claim is that modal heterogeneity - not task complexity or tool availability - is what breaks existing agent swarms on real-world omnimodal queries.

The sandboxed-coding result is worth holding in tension with Orchestra-o1's claims: it suggests that explicit native modality support is not always necessary to compete on audio-video tasks. How these two strategies trade off in practice - native omnimodal routing vs. code-and-retrieval conversion - remains an open empirical question.

Why It Matters

Three things make Orchestra-o1 worth tracking:

• Scope: Text-only and text+image agents cover a large fraction of today's deployments, but production use cases increasingly involve audio (customer calls, podcasts, voice assistants) and video (surveillance, media, education). A framework that natively orchestrates all four modalities addresses a real gap in the current ecosystem.
• Open model: DA-GRPO + Orchestra-o1-8B gives the research community a trained omnimodal agent model they can actually run, inspect, and improve - not just a closed API to benchmark against.
• RL for agents: Decision-Aligned GRPO is part of a broader trend of applying reinforcement learning directly to agent behavior rather than to next-token prediction. If it generalizes beyond this paper's setup, it could become a standard recipe for agentic fine-tuning.

The shift from single-agent to multi-agent workflows was the last major paradigm change in applied LLM engineering. Orchestra-o1 makes a credible case that the next challenge - coordinating agents not just across tasks but across fundamentally different sensory modalities - is where the field needs to focus next.