We Open-Sourced Our FFmpeg Agent Pipeline as ff-toolkit

Every AI video/audio pipeline ends up hand-writing the same FFmpeg subprocess calls and the same JSON tool definitions. We did it four times across four client projects before we stopped and asked: why does this library not exist yet? Today we are open-sourcing ff-toolkit — the FFmpeg-to-LLM bridge we wish we had found on day one.

FFmpeg is an extraordinary tool. But its CLI was designed for a human typing in a terminal, not for an LLM deciding what to do next. When you want an agent to clip a video, it needs a JSON schema that describes the parameters. When it calls that tool, you need a dispatcher that validates input and returns structured results — not raw stderr. And if you want Claude Desktop or Cursor to drive FFmpeg directly, you need an MCP server. None of this exists in FFmpeg itself. So every team builds it from scratch.

ff-toolkit wraps the five FFmpeg operations that cover roughly 80% of real-world media agent workflows: clip (trim by timestamp), merge (concatenate files), extract audio (with codec/sample-rate control for ASR pipelines), add subtitles (burn or embed), and transcode (format, codec, resolution, bitrate). Each operation is a clean Python function with full type hints and docstrings.

The key differentiator is the schema system. Call openai_tools() and you get a list of tool definitions ready to pass to the OpenAI SDK. Call anthropic_tools() and you get the Anthropic format. The tool names are identical across both — so your dispatch logic works regardless of provider. A single dispatch() function routes any LLM tool call to the correct operation and returns a structured FFmpegResult.

from ff_toolkit.schemas.openai import openai_tools
from ff_toolkit.dispatch import dispatch

# Pass to any OpenAI-compatible model
response = client.chat.completions.create(
    model="gpt-4o",
    tools=openai_tools(),
    messages=messages,
)

# Execute whatever the model calls
tc = response.choices[0].message.tool_calls[0]
result = dispatch(tc.function.name, json.loads(tc.function.arguments))

ff-toolkit ships with a built-in MCP server. Add three lines of JSON to your Claude Desktop or Cursor config and Claude can clip, merge, extract audio, add subtitles, and transcode your files directly. No glue code, no middleware. This was the feature that convinced us to open-source it — the MCP ecosystem needs more production-quality tool servers, not more toy demos.

We built ff-toolkit for our MCN video pipeline at InThePond. Our Digital Employee agents needed to autonomously process uploaded videos — clipping highlights, extracting audio for transcription, adding translated subtitles, and transcoding for different platforms. The logic was hand-written across multiple projects. Extracting it into a shared library saved us time internally. Open-sourcing it fills a gap we kept seeing in the agent tooling ecosystem: everyone builds their own FFmpeg wrapper, and nobody shares it.

Install with pip install ff-toolkit (requires FFmpeg on PATH). Run ffkit probe some_video.mp4 to verify it works — no API keys needed. The full source, 37 tests, and four example scripts are on GitHub. We welcome feedback on the API design and PRs for new operations.