Shadowjoin MCP server

shadowjoin

Measure the matches your exact join is silently dropping.

Your pipeline joins two sources on a name, an email, a SKU, a company. It matches what matches. Everything else just... isn't there. No error, no exception, no alert — the query returns, the dashboard renders, and a quarter of the rows that should have matched quietly don't.

shadowjoin runs a shadow matcher alongside your real join key and tells you how many matches you are losing, which normalization rule would recover each one, and — if you have any ground truth at all — proves the recoveries are real.

npx shadowjoin scan \
  --left  customers.csv   --left-key  company_name \
  --right crm_export.csv  --right-key account_name

Fourteen seconds, no sound. Every figure in it is verbatim CLI output from the committed example below — higher-resolution version.

The progress output above is real: shadowjoin shows what it is doing while it works. All of it is suppressed automatically when stdout is not a terminal, so piping to a file, to jq, or running in CI produces byte-identical output to before — and there is no postinstall script, because a tool you point at a production database has no business running code at install time.

What it found on public data

Joining the S&P 500 constituent list against NYSE + Nasdaq listings on company name. Both files carry ticker symbols, so every recovered match can be checked rather than believed:

left rows 503   right rows 6,709

  exact join matches                1   0.2%
  recovered by shadow join        428   85.09%
  still unexplained                74

  LOWER BOUND ON LOST MATCHES: 85.09% of left rows
  verified against ground truth: 419 correct, 1 wrong, 8 ambiguous  (precision 99.76%)

  rules worth adopting, in order (greedy by marginal recovery):
  1.  strip security/listing boilerplate (common stock, ADS, ...)   +68       68
  2.  strip corporate suffixes (inc, corp, ltd, plc, gmbh ...)     +319      387
  3.  strip punctuation                                             +33      420
  4.  sort tokens (order-insensitive match)                          +4      424
  5.  remove all whitespace                                          +1      425

The exact join found one match. There were four hundred and twenty-eight, and 419 of them are confirmed correct by ticker.

Reproduce it yourself — the data ships with the repo:

git clone https://github.com/harsh-chandak/shadowjoin && cd shadowjoin
npm install && npm run build
node dist/cli.js scan \
  --left  examples/sp500.csv    --left-key  Security --left-id Symbol --left-truth Symbol \
  --right examples/listings.json --right-key name    --right-id symbol --right-truth symbol

A join you would actually ship

The example above is deliberately unflattering — a short index name against a full security description. Here is the realistic case.

Unicode CLDR country names (what your UI displays) joined to UN official country names (what your reference data uses), on the country name, with ISO 3166 alpha-3 as ground truth. That is a join people write every week.

exact join matches              205 / 249   82.33%
  recovered by shadow join          5          2.01%   precision 100%
  still unexplained                39

  rules worth adopting:
  1.  & -> and                 +4
  2.  remove all whitespace    +1

Shortened. The full README is on GitHub.