videoctx MCP server

videoctx

<!-- mcp-name: io.github.dunfred/videoctx -->

Index a video once. Query it cheaply, forever, with any LLM.

videoctx ("video context") turns any video into an open, portable Video Context index (.vctx, see SPEC.md) using only local, model-free tools: ffmpeg keyframes, PySceneDetect shot boundaries, faster-whisper transcripts, and optional one-line visual captions from any cheap VLM. An agent answers most questions from the index's text layer for fractions of a cent, and pulls individual frames on demand only when a question needs pixels.

videoctx index lecture.mp4          # one-time, local, free
videoctx show  lecture.vctx         # the text view the model reads
videoctx ask   lecture.vctx "What diagram is shown when she explains backprop?"

What people use it for

Ask questions about videos the way you'd ask about a document:

she explain backpropagation, and what's on the slide at that moment?"

• Lectures and courses: "Summarize this 50-minute lecture." "Where does

"Find the moment the client pushed back on pricing."

• Meetings and interviews: "List the action items with timestamps."

the dialog say?"

• Screen recordings: "At what time does the error appear, and what does

"Generate chapter markers."

• Content creation: "Find the three strongest moments for short clips."

question across all of them costs pennies and answers in seconds.

• Video archives: index a folder of talks once, then every future

ability to read video at all.

• AI agents: give any MCP-capable agent (Claude Code, Cursor, ...) the

Who this is for: developers and anyone comfortable running a few terminal commands. You'll need Python, ffmpeg, and an Anthropic API key for question-answering (indexing itself is free and offline). If that's not you yet, star/watch the repo: a no-install version is on the roadmap.

Why

Native video ingestion re-tokenizes the whole video on every question: an hour of footage is ~250-350k tokens, every single time. Proprietary video platforms index into closed representations behind per-minute pricing. videoctx is the third option: an open on-disk format any model can consume, produced by free tools, where the marginal cost of a follow-up question on an hour-long video is about $0.02.

Benchmark results (Video-MME, Oct 2026)

Full protocol and honest cost accounting (including prompt-cache write premiums) in paper/draft.md. Query model: Claude Opus 4.8.

Numbers below are single-touch runs of the frozen configs (v0.2: captions + OCR + evidence rules, Opus 4.8 loop; v0.3, 2026-10-21: adds text-evidence rules, verbatim transcript quotes and a final letter-vs-evidence check) under full cost accounting. Where we measured the same configuration more than once, we report the observed range: on n=150, single runs wobble by 2-3 questions; on n=30, by 2-3 questions (= 7-10 points). Treat sub-3-point deltas on this benchmark as noise.

Subset composition caveat: videos were taken in Video-MME dataset order, which clusters by domain. The medium and long cells are entirely Knowledge-domain videos; the short cells are 76-88% Knowledge (remainder Film & Television). Read these as Knowledge-dominated subsets of Video-MME, not the full benchmark; future splits will be domain-stratified.

Short videos, held-out test half (n=150):

Medium videos, 4-15 min (n=30, wide error bars):

Shortened. The full README is on GitHub.