Survey · 2026

Orchestrating LLMs with Specialized Models

A Survey on Heterogeneous Multi-Model Agents (HMMAs)

Zheng Zhang* · Nanjie Yao* · Yu Feng · Qi Chai · Liu Liu · Deheng Ye · Peilin Zhao · Xiangxin Zhou · Hao Wang · Hui Xiong

HKUST (Guangzhou) · Tencent · NTU · SJTU  ·  *equal contribution

0Papers
0Venues
2023–26Years
0Interaction Patterns
0Application Domains
0Open-source Repos

Overview

Large language models have become a common center for building intelligent agents, yet many real tasks still need abilities that a single LLM cannot provide. A growing line of work therefore connects LLMs with specialized non-LLM models: object detectors, segmentation models, diffusion generators, and robot policies. We call this emerging paradigm Heterogeneous Multi-Model Agents (HMMAs).

This survey systematically reviews 572 HMMA papers from major AI venues between 2023 and 2026, organizes them into five interaction patterns, and analyzes architectural choices across information flow, interface design, feedback structure, uncertainty handling, and model coupling.

Definition An HMMA is an agentic system in which an LLM orchestrates one or more heterogeneous, probabilistic, task-specialized non-LLM models (for perception, generation, or action) to accomplish tasks beyond any single model's ability.

Collection funnel

0 main-conference papers
2023–2026 · 19 venues
LLM two-stage screening
0 candidates
manual verification
0 HMMA papers
Positioning of HMMAs among adjacent paradigms: compound AI systems, multi-agent systems, multimodal LLMs, and model routing.
HMMAs sit between adjacent paradigms: unlike compound AI systems the components are probabilistic specialists; unlike multi-agent systems they are heterogeneous models rather than peer LLM agents; unlike multimodal LLMs the capabilities remain separate models orchestrated by the LLM.

Paper Browser

All 572 surveyed papers with their taxonomy labels. Search, filter, and click a row for details.

…
Title Venue Year Pattern Domain

Five Interaction Patterns

Systems are organized by which specialist roles the LLM orchestrates: Perception, Generation, Action.

Architectural Analysis

How HMMAs are wired: distributions over five cross-cutting design dimensions (share of 572 papers).

Information flow

Interface type

Feedback structure

Uncertainty handling

Model coupling

Application domains

Reproducible Pipeline

The literature base was built with a fully scripted, resumable LLM-assisted pipeline, reusable for other survey topics.

1

Crawl

Metadata for 19 venues from papers.cool + DBLP, abstracts enriched via Semantic Scholar.

2

Coarse screen

LLM filters title + abstract for topic relevance, optimized for recall.

3

Deep screen

Full-text LLM screening of downloaded PDFs, optimized for precision.

4

Annotate

Structured taxonomy labels extracted per paper; suspects automatically re-annotated.

5

Verify

Authors manually verify all 591 candidates; 19 filtered out, 572 remain.

Citation

@article{zhang2026hmma,
  title   = {Orchestrating LLMs with Specialized Models: A Survey on Heterogeneous Multi-Model Agents},
  author  = {Zhang, Zheng and Yao, Nanjie and Feng, Yu and Chai, Qi and Liu, Liu and Ye, Deheng and Zhao, Peilin and Zhou, Xiangxin and Wang, Hao and Xiong, Hui},
  journal = {Preprints},
  year    = {2026},
  doi     = {10.20944/preprints202607.1041.v1},
  url     = {https://www.preprints.org/manuscript/202607.1041}
}