PySpark MCP server

PySpark MCP Server

SQL migration assistance, AWS Glue job template generation, and Spark code optimization — as an MCP server.

Not the live-Spark pyspark-mcp package. This project is SQL → PySpark /

Glue source generation, published as pyspark-tools.

SemyonSinchenko/pyspark-mcp introspects a

running SparkSession. A deprecated pyspark-mcp console script remains here so

old configs keep working; it prints a warning, then starts this server.

What It Does

• SQL Dialect Transpilation — Convert between PostgreSQL, Oracle, Redshift, MySQL, Snowflake, and Spark SQL using SQLGlot
• PySpark DataFrame API Generation — Generate DataFrame API source text from SQL, with optimization hints
• AWS Glue templates — Job script strings, DynamicFrame conversions, Data Catalog definitions, S3 layout advice
• Batch Processing — Walk SQL files/directories and emit converted modules
• Code Review & Optimization — Pattern-based review of existing PySpark source
• Pattern Detection — Find duplicated snippets and suggest utilities

What It Doesn't Do

• Recursive CTEs → provides Spark SQL equivalent + guidance (PySpark has no native recursive CTE support)
• MERGE/PIVOT/CONNECT BY → transpiles to Spark SQL, provides DataFrame API guidance
• Perfect 1:1 DataFrame API transpilation for all SQL — complex queries get Spark SQL + recommendations
• It does not start a SparkSession, submit Glue jobs, or execute SQL
• optimize(mode="code") returns suggestions; it does not rewrite your code
• glues3 is a path heuristic** (no AWS call, no measured speedups)
• It does not replace SemyonSinchenko/pyspark-mcp for live catalog/plans

Why this vs calling sqlglot yourself

SQLGlot already transpiles dialects. This MCP adds three things around that kernel: DataFrame-API pretty-printing with join/window/cast mappings that the conversion tests lock, Glue job boilerplate strings (bookmarks, DynamicFrames, catalog tables) so an agent can emit a file instead of assembling one, and a 14-tool FastMCP surface so an LLM picks convert / mode=sql instead of wiring sqlglot itself. If you only need sqlglot.transpile(...), use sqlglot.

Quick Start

pip install pyspark-tools
pyspark-tools

Zero-clone alternative: uvx pyspark-tools. run_server.py is a development convenience that inserts sys.path and prints startup banners. Prefer pyspark-tools in configs and production.

Try it

pip install pyspark-tools
python -c "from pathlib import Path; from pyspark_tools.sql_converter import SQLToPySparkConverter as C; from pyspark_tools.consolidated_tools import glue_job; c,s,o=C(),Path('examples'),Path('examples/out'); [(o/f'{n}.py').write_text(c.convert_sql_to_pyspark((s/f'{n}.sql').read_text(), dialect=d).pyspark_code) for n,d in [('postgres_orders','postgres'),('oracle_decode','oracle')]]; (o/'orders_etl_glue.py').write_text(glue_job(mode='template', job_name='orders_etl', sql_query=(s/'postgres_orders.sql').read_text())['template'])"

Writes the same files as examples/out/. MCP stdio CLI: pyspark-tools.

Example: SQL → PySpark

SELECT o.customer_id, c.name, SUM(o.amount) AS total
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.status = 'paid'
GROUP BY o.customer_id, c.name

Call convert with mode=sql. Captured converter output (dialect=spark):

Shortened. The full README is on GitHub.