Sparkforensics MCP server

SparkForensics

Your Spark job is slow. SparkForensics tells you why, and what to do about it.

Drop a Spark event log in and get an evidence-backed breakdown of which stages are slow, why, and how to fix them. Nothing installs on your cluster and nothing leaves your machine.

Try it now → with your own event log. No install, no account, no upload.

Your browser doesn't support inline video. Download the demo instead.

Spark's own History Server gives you raw metrics. It won't tell you that stage 14 spilled 40GB because of a skewed join key, or that cold start is eating a third of the job. SparkForensics reads the same event logs and turns them into findings.

Why SparkForensics

dashboard; nothing is uploaded anywhere. An optional local-server mode adds a small Node companion for when a Spark History Server blocks direct browser fetches with CORS.

• No install: it's a static web app. Drop a log file and read the

so a 240MB+ (or multi-GB) NDJSON log doesn't freeze the tab.

• Event logs are streamed and parsed off the main thread in a Web Worker,

pressure, stragglers, cold start, cache/memory utilization, autoscaling churn, job/stage failures and partition sizing, plus a SQL Plan Advisor (duplicate subtrees, small files, broadcast sizing) and a config audit. Full roster in docs-site/contributor-guide/architecture/detector-contract.md.

• About 25 built-in detectors cover skew, shuffle, spill, GC

not severity alone, so the top row is the fix worth doing first.

• Fix These First orders every finding by its estimated wall-clock impact,

runs of the same job and shows which stages regressed, rather than every difference.

• Run comparison (A/B) mode structurally matches stages across two

you know when to trust a flag and when it's a hint worth checking manually.

• Every finding tracks what it could and couldn't confirm from the log, so

run against budget thresholds (max runtime, skew, spill, failure rate...) and exits non-zero on violation, so a regression can fail a build. The run can come from a local event-log file/directory or, with --shs-base-url, fetched directly from a Spark History Server.

• A headless CLI (sparkforensics-analyze) evaluates a

diagnoserun (thresholded findings + remediation text), getrunsummary (app/stage/job/sql counts and duration), compareruns (categorized findings delta between two runs), evaluatebudgets (evaluate a run, optionally against a second run, against pass/fail budget thresholds), getfindingevidence (raw evidence for one finding), and getfinding_documentation (detection/tuning reference docs for one finding type) so an LLM agent can investigate a run directly.

• An MCP server (sparkforensics-mcp) exposes

by hand, see sparkforensics-operator, an Airflow operator that runs SparkForensics on a job's event log and acts on the result.

• For running the analyze CLI automatically after every Spark job instead of

Unlike tools that need a cluster-side plugin or a running server plus an LLM agent (Spark History Server MCP tools), SparkForensics needs nothing installed anywhere else: the event log and a browser tab are enough.

Get started

Shortened. The full README is on GitHub.