Soundscape Agent

Project Page

SoundscapeAgent: Agentic Soundscape Construction for Controllable Synthesis and Scalable Audio-Language Supervision

Hao Zhang1,*†, Yiwen Zhao2,*, Yixuan Zhang2, Yiwen Shao2, Steve Yves2

1Wuhan University    2Tencent Hunyuan, Bellevue, WA, USA

*Equal contribution. †Corresponding author. Yiwen Zhao completed this work during an internship at Tencent.

Code Paper (arXiv) Listening Test Results

Abstract

We present an agentic soundscape construction framework for controllable compositional audio generation that makes explicit the scene planning, source selection, temporal layout, and rendering steps typically handled implicitly by single-shot text-to-audio models. An LLM-based agent converts user intent into an executable scene plan, acquires assets through retrieval and on-demand generation, renders controllable multi-event mixtures, and exports aligned scene metadata. The framework also supports human-in-the-loop interaction through user-guided tool selection and editable scene plans. Together, these components provide an inspectable and reusable approach to controllable soundscape synthesis and scalable audio-language data construction. Listener studies and objective metrics demonstrate competitive generation performance against text-to-audio baselines, while models trained with agent-generated data consistently outperform real-only baselines in downstream audio reasoning.

System Overview

Overall pipeline plot

Your browser cannot preview this PDF inline. Open the system overview plot.

Framework formulation from the paper: \[ u=(u^{\text{text}},u^{\text{img}}), \qquad (y,E,m,d)=\mathcal{A}(u,c) \] \[ u \xrightarrow{\text{intent parsing \& executable planning}} E \xrightarrow{\text{retrieval / on-demand generation}} A \xrightarrow{\text{structured rendering \& mixing}} (y,m,d) \] Framework overview from the paper: given user condition u and optional controls c (e.g., duration and output channels), the system outputs (i) waveform y, (ii) structured scene specification E, (iii) metadata m, and (iv) aligned textual description d, through four stages: multimodal intent parsing and executable scene planning; adaptive asset acquisition via retrieval or on-demand generation; structured rendering and safety-aware mixing; metadata-backed export and description generation.

Why not plain text-to-audio?

SoundscapeAgent iis an LLM-driven, agentic pipeline that turns user intent into executable scene plans, combines retrieval and on-demand generation, and renders controllable multi-event soundscapes with aligned metadata to support both high-quality synthesis and scalable audio-language supervision.

  • Acoustic scenes are multi-event, temporal, and layered, but plain T2A maps prompt → one waveform, with scene structure hidden.
  • It gives weak control over which events happen, when they happen, and their foreground/background roles.
  • It is hard to inspect, edit, or reuse intermediate decisions (no explicit plan, limited provenance).
  • For scalable data construction, plain T2A lacks structured outputs needed for metadata-aligned supervision.

Capability Demos (SoundscapeAgent)

Controllability

These examples highlight capabilities that are difficult to express with single-shot text-to-audio: duration, temporal layout, source-level inclusion/exclusion, layer/gain balance, etc.

Duration control (same intent)

Prompt: "A peaceful rainy night."

5s

10s

30s

Source-level control (include / remove events)

Base: "A peaceful rainy night. 10 seconds, mono audio."

Base scene

Remove one source (e.g., no crickets)

Add one source (e.g., add crackling fire)

Interactive Editing (local edits)

Base: "A peaceful rainy night. 10 seconds, mono audio."

Before edit

Edit A: Modify one event (e.g., make the rain louder)

Edit B: Shift onset (e.g., move the thunder onset 1 second earlier.)

Open-ended intent grounding

Open-ended user intents (without explicit sound-event prompts) mapped to coherent, plausible soundscapes to demonstrate intent-to-scene grounding.

Intent case 1

Input intent: "I feel upset, can you generate something calming?"

Listen for: coherent grounding from abstract emotional intent to plausible scene composition.

Intent case 2

Input intent: "I need to focus on studying for a while."

Listen for: how the system derives appropriate ambience and event density for focus.

Niche / unusual prompt stress test

Prompt example: "A zombie apocalypse scene"

Image-conditioned generation

Image-to-soundscape cases.

Image case 1

Input image for image-conditioned generation case 1

Image source: EOS

Image case 2

Input image for image-conditioned generation case 2

Image source: Condé Nast Traveler

Stereo / spatial audio output

Same scene rendered in mono vs stereo (e.g., forest waterfall).

Mono

Stereo

Audio stems

The outputs contain the full mix, as well as the stems for each event.

Layered mix and stems (optional)

Full mix (e.g.,peaceful rainy night)

Stem 1 (e.g., background)

Stem 2 (e.g., wind)

Stem 3 (e.g., crickets)

Stem 4 (e.g., thunder)

UI and reasoning trace

Plan + interface trace (example)

{
Soundscape Agent — session log
Started: 2026-06-19 01:45:40.370809

────────────────────────────────────────────────────────────
user
────────────────────────────────────────────────────────────
A peaceful rainy night. 5 second, mono audio.

────────────────────────────────────────────────────────────
              
────────────────────────────────────────────────────────────
assistant
────────────────────────────────────────────────────────────
All clips retrieved! Now let me mix a peaceful 5-second rainy night soundscape in mono.

────────────────────────────────────────────────────────────
TodoWrite
────────────────────────────────────────────────────────────
[x] List library categories
[x] Retrieve audio clips for rainy night scene
[x] Mix and save 5-second mono soundscape <- mix_and_save

(3/3 completed)

────────────────────────────────────────────────────────────
Waveform Envelope
────────────────────────────────────────────────────────────
┌────────────────────────────────────────────────────────────┐  
│                                              █             │
│                      █          █          █ █ █           │
│                      █         ██          █ █ █           │
│                    █ ██ ██ █  ███       █ ███████          │
│    █  ████   ████ █████████████████  █ ██████████          │
│  ███ ███████ ████ ███████████████████████████████          │
│  ███████████████████████████████████████████████████       │
│██████████████████████████████████████████████████████████  │
└────────────────────────────────────────────────────────────┘
  0s             1s             2s             3s             5s

────────────────────────────────────────────────────────────
Safety Check
────────────────────────────────────────────────────────────
✓ Safety check:
  Loudness: -38.8 LUFS  →  -38.8 LUFS  (ceiling: -14 LUFS)

────────────────────────────────────────────────────────────
mix_and_save result
🌧️ **Peaceful Rainy Night** — done! Here's a summary:

---

### 🎧 Output
- **File:** `peaceful_rainy_night.wav`
- **Path:** `.../soundscapes/peaceful_rainy_night.wav`
- **Duration:** 5 seconds · Mono · 44.1 kHz / 32-bit PCM
- **Meta:** `peaceful_rainy_night_meta.json`
- **Stems:** `peaceful_rainy_night_stems/`

---

### 🎚️ Layer Design

| Layer | Sound | Timing | Role |
|-------|-------|--------|------|
| 🌧️ Deep Background | **Rain** (BBC real recording) | 0–5s | Continuous acoustic floor, looped & stabilized |
| 💨 Mid Background | **Wind** | 0–4.5s | Soft ambient layer, gently filling the space |
| 🦗 Mid Background | **Crickets** | 0–5s | Nocturnal texture; adaptive anti-harsh guard applied |
| ⛈️ Foreground | **Thunder** | ~1.5–4.5s | Distant rumble drifting in, natural onset jitter (21 ms) |

---

### ✨ Mix Details
- Rain provides the **steady, immersive acoustic bed**
- Crickets & wind weave together as **mid-layer ambience**
- A single **soft thunder roll** swells in the middle, keeping it peaceful (not dramatic)
- Hybrid tone shaping + adaptive anti-harsh guard for smooth, natural timbre


}

Metadata + aligned description

Generated clip

Aligned metadata provided below

Metadata snippet

{
  "duration_sec": 5,
  "output_channels": 1,
  "num_events": 3,
  "detailed_caption": "This audio is a 5.0-second mono environmental mixture. A deep background bed is present across most of the scene and acts as the acoustic floor. The mix contains 3 scheduled events with explicit onset, duration, and relative level control. Timeline details: [0.00s-4.50s] wind; layer=mid_bg; snr=5.0dB; pan=0.00; width=1.00 | [0.00s-5.00s] crickets; layer=mid_bg; snr=7.0dB; pan=0.00; width=1.00 | [1.52s-4.52s] thunder; layer=foreground; snr=12.0dB; pan=0.00; width=1.00",
  "timeline_caption": "[0.00s-4.50s] wind; layer=mid_bg; snr=5.0dB; pan=0.00; width=1.00 | [0.00s-5.00s] crickets; layer=mid_bg; snr=7.0dB; pan=0.00; width=1.00 | [1.52s-4.52s] thunder; layer=foreground; snr=12.0dB; pan=0.00; width=1.00",
  "events": [
      {
        "event_index_timeline": 0,
        "tag": "wind",
        "layer": "mid_bg",
        "start_time_s": 0.0,
        "end_time_s": 4.5,
        "duration_s": 4.5,
        "snr_db": 5.0,
        "pan": 0.0,
        "width": 1.0,
        "description": "wind appears from 0.00s to 4.50s as mid_bg with snr 5.0 dB."
      },
      {
        "event_index_timeline": 1,
        "tag": "crickets",
        "layer": "mid_bg",
        "start_time_s": 0.0,
        "end_time_s": 5.0,
        "duration_s": 5.0,
        "snr_db": 7.0,
        "pan": 0.0,
        "width": 1.0,
        "description": "crickets appears from 0.00s to 5.00s as mid_bg with snr 7.0 dB."
      },
      {
        "event_index_timeline": 2,
        "tag": "thunder",
        "layer": "foreground",
        "start_time_s": 1.521,
        "end_time_s": 4.521,
        "duration_s": 3.0,
        "snr_db": 12.0,
        "pan": 0.0,
        "width": 1.0,
        "description": "thunder appears from 1.52s to 4.52s as foreground with snr 12.0 dB."
      }
    ]
    ...
}

Comparisons and Study-Oriented Demos

This section mirrors Track A categories from the paper and provides baseline comparisons.

Agent vs single-shot T2A baselines (five scenario types)

Scenario A — AudioCaps style

Prompt: "A person snoring."

Listen for: semantic alignment and recording naturalness.

SoundscapeAgent

TangoFlux

EzAudio

AudioLDM2

Scenario B — Ambience-heavy

Prompt: "A snowy village street with muffled wind, distant footsteps, and sparse ambient movement."

Listen for: background continuity, texture realism, and spatial coherence.

SoundscapeAgent

TangoFlux

EzAudio

AudioLDM2

Scenario C — Event-rich

Prompt: "A busy city street with passing cars, footsteps, bicycle bells, distant construction, and occasional horns."

Listen for: source separability, compositional richness, and layer balance.

SoundscapeAgent

TangoFlux

EzAudio

AudioLDM2

Scenario D — Temporally specified

Prompt: "Steady rain continues throughout; a dog barks around 3 s; thunder occurs near 8 s."

Listen for: event ordering and timing compliance.

SoundscapeAgent

TangoFlux

EzAudio

AudioLDM2

Scenario E — Affective / abstract

Prompt: "A peaceful meditation garden with a gentle atmosphere."

Listen for: plausible grounding of abstract intent and mood consistency.

SoundscapeAgent

TangoFlux

EzAudio

AudioLDM2

Track A — Listening test results

MOS summary snapshot

Track A MOS comparison dashboard across SoundscapeAgent, TangoFlux, EzAudio, and AudioLDM2

Subjective MOS summary from the Track A listening study (22 participants, 1600 ratings). View detailed Track A results →

Key takeaway

  • SoundscapeAgent achieves the highest overall MOS among the four compared systems.
  • Compositional construction strategy maintains strong performance on relatively straightforward caption-conditioned generation.

Track B - Audio reasoning

MMAU improvement snapshot

MMAU benchmark improvement snapshot for Track B audio reasoning evaluation

Your browser cannot preview this PDF inline. Open the MMAU results (PDF).

Track B downstream audio reasoning results on MMAU.

Key takeaway

  • Agent-generated structured data improves reasoning over real-only training.
  • Largest gains appear on temporally/compositionally complex tasks.
  • Data quality/diversity balance matters more than synthetic volume alone.

Citation

Will be replaced with detailed BibTeX entry after publication.

@inproceedings{soundscape-agent-2026,
  title   = {Soundscape Agent: Structured Sound Synthesis with Retrieval, Generation, and Control},
  author  = {Last, First and Others},
  year    = {2026}
}

Contact: anonymous@institution (will be updated after publication).