Automate audio ducking with Python and the Shotstack API

Audio ducking makes music quieter while someone speaks, then returns it to its normal level between lines. The words stay easy to follow, and the music still carries the gaps.

With Shotstack’s video editing API, you can build those volume changes from your voice-over clips’ timing. This article gives you a Python script that generates the fades, builds the edit and renders a narrated video. It also shows what the duck did to the example mix, measured on the rendered files.

TL;DR

• Put voice-over, music and muted background video on separate tracks.
• Set the music asset’s volume to an array of tweens.
• Build each fade and hold from the voice clips’ start and length.
• Merge nearby lines into one ducked stretch to avoid rapid volume changes.
• In the example, ducking from 0.8 to 0.2 lowered the music by 12 dB. Speech went from up to 7 LU (loudness units) below the music to 5 to 10 LU above it.

What is audio ducking?

Audio ducking temporarily lowers a background sound to make a foreground sound easier to hear. In a narrated video, the voice stays at its chosen level while the music moves down and up around it.

The pattern for each spoken line is:

1. Hold the music at its normal level.
2. Fade down shortly before the voice starts.
3. Hold the low level throughout the line.
4. Fade back up after the voice ends.

When two lines are close together, keep the music low across both. Raising it briefly in a small gap can make the mix sound like it is pumping.

This example uses timeline timing, not speech detection. If you only need to mute footage or replace its sound, see mute video and replace audio with Python.

Lay out the Edit API timeline

Use three tracks in this order:

Tracks are layered in array order, with the first track on top. Clips on the same track must not overlap.

Keep the music in an audio asset clip on its own track. The timeline’s soundtrack field is deprecated.

The example runs for 34 seconds. Both the music and the background video start at zero and cover the full timeline.

The voice clips are three cuts from one speech file:

All times are seconds. trim selects where playback begins inside the source file, while start places the clip on the timeline.

You can also supply separate recordings for each line. For generating those recordings, see AI speech generation.

Control music volume with tweens

An audio or video asset’s volume accepts either a number from 0 to 1 or an array of Tween objects. Zero mutes the asset, and one plays it at full volume.

The number scales the clip’s own level. In the example render, 0.8 played the music about 2 dB below the source file and 0.2 about 14 dB below it, so the duck is 12 dB deep.

Each tween describes one part of the volume envelope:

Tween times are clip-relative, not timeline-relative. They match timeline times here because the music starts at zero. If you move the music clip, subtract its timeline start from the envelope times.

The interpolation options behave differently:

• linear changes volume at a constant rate.
• bezier follows a curve shaped by easing.
• constant jumps instantly from from to to, without a gradual transition.

For a hold, use identical from and to values. The script includes every hold, so the envelope describes the whole music clip.

See the animations and tweening guide for animation concepts and the API reference for field definitions.

Generate audio ducking with Python

Save the following as duck.py. It uses only Python’s standard library.

The settings lower the music from 0.8 to 0.2, with a 0.4-second fade down and a 0.8-second fade up. Lines separated by less than 1.5 seconds share one ducked stretch. FADE sets the shape of every fade.

duck_tweens() also merges stretches when their fades would overlap, keeps fades inside the timeline and rounds tween times to milliseconds.

import json
import os
import sys
import time
import urllib.error
import urllib.request

API = "https://api.shotstack.io/edit/v1/render"
MUSIC_URL = "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/music/unminus/palmtrees.mp3"
BROLL_URL = "https://shotstack-assets.s3.amazonaws.com/footage/drone-city-flyover.mp4"
VOICE_URL = "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/audio/financial-podcast.mp3"
LINES = [
    {"src": VOICE_URL, "trim": 41.46, "length": 9.8, "start": 3},
    {"src": VOICE_URL, "trim": 51.85, "length": 8.1, "start": 15},
    {"src": VOICE_URL, "trim": 60.33, "length": 6.2, "start": 25},
]
TOTAL = 34  # length of the video in seconds
MUSIC_LEVEL, DUCK_LEVEL = 0.8, 0.2  # volume, 0 to 1
FADE_DOWN, FADE_UP = 0.4, 0.8  # seconds
MIN_GAP = 1.5  # lines closer than this share one duck
FADE = {"interpolation": "linear"}


def duck_tweens(lines, total, high, low, fade_down, fade_up, min_gap, fade=FADE):
    spans = []
    for line in sorted(lines, key=lambda x: x["start"]):
        start, end = line["start"], line["start"] + line["length"]
        if spans and start - spans[-1][1] < max(min_gap, fade_down + fade_up):
            spans[-1][1] = max(spans[-1][1], end)
        else:
            spans.append([start, end])
    tweens, cursor = [], 0.0

    def add(a, b, start, length):
        if length > 0:
            shape = fade if a != b else {"interpolation": "constant"}
            tween = {
                "from": a,
                "to": b,
                "start": round(start, 3),
                "length": round(length, 3),
            }
            tweens.append({**tween, **shape})

    for start, end in spans:
        down, up = max(cursor, start - fade_down), min(total, end + fade_up)
        add(high, high, cursor, down - cursor)
        add(high, low, down, start - down)
        add(low, low, start, end - start)
        add(low, high, end, up - end)
        cursor = up
    add(high, high, cursor, total - cursor)
    return tweens


def clip(asset, start, length):
    return {"asset": asset, "start": start, "length": length}


tweens = duck_tweens(LINES, TOTAL, MUSIC_LEVEL, DUCK_LEVEL, FADE_DOWN, FADE_UP, MIN_GAP)
voice = [
    clip({"type": "audio", "src": x["src"], "trim": x["trim"]}, x["start"], x["length"])
    for x in LINES
]
music = clip({"type": "audio", "src": MUSIC_URL, "volume": tweens}, 0, TOTAL)
broll = clip({"type": "video", "src": BROLL_URL, "volume": 0}, 0, TOTAL)
edit = {
    "timeline": {"tracks": [{"clips": voice}, {"clips": [music]}, {"clips": [broll]}]},
    "output": {"format": "mp4", "resolution": "hd"},
}


def call(url, body=None):
    data = json.dumps(body).encode() if body else None
    key = os.environ["SHOTSTACK_API_KEY"]
    request = urllib.request.Request(
        url, data, {"x-api-key": key, "Content-Type": "application/json"}
    )
    try:
        with urllib.request.urlopen(request) as response:
            return json.load(response)["response"]
    except urllib.error.HTTPError as error:
        sys.exit(f"HTTP {error.code}: {error.read().decode()[:300]}")


if "--dry-run" in sys.argv:
    print(json.dumps(music, indent=2))
    sys.exit()
if not os.environ.get("SHOTSTACK_API_KEY"):
    sys.exit("Set SHOTSTACK_API_KEY first.")
render_id = call(API, edit)["id"]
print("Render queued:", render_id)
while True:
    time.sleep(5)
    status = call(f"{API}/{render_id}")
    if status["status"] == "done":
        print(status["url"])
        break
    if status["status"] == "failed":
        sys.exit(f"Render failed: {status.get('error')}")

Keep voice clips non-overlapping and inside TOTAL. Each source needs enough audio or video for its trim and length.

Get an API key, then run:

export SHOTSTACK_API_KEY="your-api-key"
python duck.py

The script posts the edit, checks its status every five seconds and prints the video URL when rendering finishes:

Render queued: 3e8ff368-e7c6-414b-9789-d6685f696f75
https://shotstack-api-v1-output.s3-ap-southeast-2.amazonaws.com/35tqpmb0ya/3e8ff368-e7c6-414b-9789-d6685f696f75.mp4

For the request lifecycle in more detail, see render your first video with the Shotstack API.

If the key is missing, the script stops with Set SHOTSTACK_API_KEY first. If the API rejects the request, it prints the HTTP status and the error message.

This is the rendered example:

Inspect the generated music envelope

Run python duck.py --dry-run to print the music clip without rendering. The example produces 13 tweens: an opening hold, four tweens for each line and a closing hold.

The first line begins at 3 seconds and ends at 12.8. Its duck starts at 2.6, and the music is back at its normal level at 13.6. These are the first four tweens:

[
  {
    "from": 0.8,
    "to": 0.8,
    "start": 0.0,
    "length": 2.6,
    "interpolation": "constant"
  },
  {
    "from": 0.8,
    "to": 0.2,
    "start": 2.6,
    "length": 0.4,
    "interpolation": "linear"
  },
  {
    "from": 0.2,
    "to": 0.2,
    "start": 3,
    "length": 9.8,
    "interpolation": "constant"
  },
  {
    "from": 0.2,
    "to": 0.8,
    "start": 12.8,
    "length": 0.8,
    "interpolation": "linear"
  }
]

The example’s gaps are 2.2 and 1.9 seconds, so all three lines get separate ducked stretches. Changing their timing or adding lines recalculates the envelope without rewriting any tweens by hand.

What the duck does to the mix

To see the effect, we rendered the example three more ways: speech only, ducked music only and music at a flat 0.8. Then we measured each line’s window with FFmpeg’s EBU R128 loudness meter (ebur128):

Without ducking, the music was louder than the speaker in all three lines, by 1.8 to 7 LU. With the duck, the speech sat 10.3, 5.1 and 7.3 LU above the music.

The fades landed where the tweens put them. Each fade down finished at the line’s start, and each fade up reached the normal level 0.8 seconds after the line ended.

Tune the mix by ear

Treat the example values as starting settings, not a universal mix:

• Drop depth: lower DUCK_LEVEL if the music masks words. Halving it lowers the ducked music by another 6 dB. Raise it if the background disappears too much.
• Normal level: adjust MUSIC_LEVEL for the gaps between lines.
• Fade down: increase FADE_DOWN for a more gradual lead-in. The fade always completes before speech begins.
• Fade up: increase FADE_UP if the music returns too abruptly.
• Gap threshold: increase MIN_GAP to keep nearby lines grouped.

Line 2 shows why depth matters: that part of the track is louder, so the same duck left the speech only 5.1 LU clear. A lower DUCK_LEVEL gives a busy passage more room.

A linear volume fade is not linear in loudness. In the render, the first quarter of the fade up restored about 5 of the 12 dB, so the return can sound quick.

For a softer curve, set FADE = {"interpolation": "bezier", "easing": "easeInOut"}. With that setting the same quarter restored about 2.5 dB.

Handle unknown voice-over durations

If a voice-over’s length is unknown, first use the Shotstack Probe endpoint: GET /edit/v1/probe/{url-encoded file URL} returns metadata including format.duration. Use that duration, minus any trim, as the clip length before building the tweens. For files on your own machine, ffprobe reads the same value.

Use steps for a simpler level change

For a single fixed level change, split the music into consecutive clips with different numeric volume values, as in the audio volume levels template. This changes the level in steps, without fades.

Keep narration clear with Shotstack

Build the music envelope from the same timing data that places your voice-over clips. With Shotstack, the ducking stays aligned when your script, line count or pacing changes, and every render applies it the same way.

Sign up for Shotstack to render your narrated videos with generated volume transitions.

Frequently asked questions (FAQs)

Will a pause inside one voice clip bring the music back up?

No. The script treats the entire clip as speech, including internal silence. To lift the music during a long pause, split the speech into separate clips with their own timing.

Can I duck background sounds other than music?

Yes. You can apply the same volume envelope to an audio clip that holds ambience or other background sound. If you have several background audio clips, calculate each envelope in that clip’s local time.

What if two speakers talk at the same time?

Put overlapping voice clips on separate tracks, since clips on one track cannot overlap. Pass both speakers’ timing entries to duck_tweens() so their overlapping speech becomes one continuous ducked stretch, then build their voice tracks separately.

Written by

Peace Aisosa

Peace Aisosa is a Customer Success Specialist at Shotstack. She writes about video automation when she's not troubleshooting customers' video pipelines.

Published November 7, 2026