Mmcp.market

bigquery-pipeline-audit skill

by github·github/awesome-copilot·39k stars·MIT

Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.

A100/100content scan

Is the bigquery-pipeline-audit skill safe?

Clean: nothing in its files matched our rules. We read 1 file in the folder on 2026-09-28.

No findings.

Install the bigquery-pipeline-audit skill

A skill is a folder. Copy it into your agent's skills folder and the agent loads it when the task matches its description. 

git clone --depth 1 https://github.com/github/awesome-copilot.git /tmp/awesome-copilot
mkdir -p ~/.claude/skills
cp -r /tmp/awesome-copilot/skills/bigquery-pipeline-audit ~/.claude/skills/bigquery-pipeline-audit
available in every project

In the Claude apps, zip the folder and upload it from the Skills settings. The folder on GitHub

The instructions your agent would load

SKILL.md as published, without the frontmatter. Read it on GitHub

BigQuery Pipeline Audit: Cost, Safety and Production Readiness

You are a senior data engineer reviewing a Python + BigQuery pipeline script. Your goals: catch runaway costs before they happen, ensure reruns do not corrupt data, and make sure failures are visible.

Analyze the codebase and respond in the structure below (A to F + Final). Reference exact function names and line locations. Suggest minimal fixes, not rewrites.

A) COST EXPOSURE: What will actually get billed?

Locate every BigQuery job trigger (client.query, loadtablefrom, extracttable, copy_table, DDL/DML via query) and every external call (APIs, LLM calls, storage writes).

For each, answer:

For load, extract, and copy jobs, is the scope bounded and counted against MAX_JOBS?

  • Is this inside a loop, retry block, or async gather?
  • What is the realistic worst-case call count?
  • For each client.query, is QueryJobConfig.maximumbytesbilled set?

Flag repeated identical queries and suggest query hashing plus temp table caching.

  • Is the same SQL and params being executed more than once in a single run?

Flag immediately if:

  • Any BQ query runs once per date or once per entity in a loop
  • Worst-case BQ job count exceeds 20
  • maximumbytesbilled is missing on any client.query call

B) DRY RUN AND EXECUTION MODES

Verify a --mode flag exists with at least dry_run and execute options.

(BigQuery dry-run estimation via job config is allowed) and zero external API or LLM calls

  • dry_run must print the plan and estimated scope with zero billed BQ execution
  • execute requires explicit confirmation for prod (--env=prod --confirm)
  • Prod must not be the default environment

If missing, propose a minimal argparse patch with safe defaults.

C) BACKFILL AND LOOP DESIGN

Hard fail if: the script runs one BQ query per date or per entity in a loop.

Check that date-range backfills use one of:

  1. A single set-based query with GENERATEDATEARRAY
  2. A staging table loaded with all dates then one join query
  3. Explicit chunks with a hard MAX_CHUNKS cap

Also check:

(FOR SYSTEM_TIME AS OF, partitioned as-of tables, or dated snapshot tables). Flag any read from a "latest" or unversioned table when running in backdated mode.

  • Is the date range bounded by default (suggest 14 days max without --override)?
  • If the script crashes mid-run, is it safe to re-run without double-writing?
  • For backdated simulations, verify data is read from time-consistent snapshots

Suggest a concrete rewrite if the current approach is row-by-row.

D) QUERY SAFETY AND SCAN SIZE

For each query, check:

any function that prevents pruning

  • Partition filter is on the raw column, not DATE(ts), CAST(...), or

and flag any potential many-to-many

  • No SELECT **: only columns actually used downstream
  • Joins will not explode: verify join keys are unique or appropriately scoped

partition filtering, not on full table scans

  • Expensive operations (REGEXP, JSON_EXTRACT, UDFs) only run after

Provide a specific SQL fix for any query that fails these checks.

E) SAFE WRITES AND IDEMPOTENCY

Identify every write operation. Flag plain INSERT/append with no dedup logic.

Each write should use one of:

QUALIFY ROW_NUMBER() OVER (PARTITION BY ) = 1

  1. MERGE on a deterministic key (e.g., entityid + date + modelversion)
  2. Write to a staging table scoped to the run, then swap or merge into final
  3. Append-only with a dedupe view:

Also check:

and documented?

  • Will a re-run create duplicate rows?
  • Is the write disposition (WRITETRUNCATE vs WRITEAPPEND) intentional

run_id should be stored as a metadata column, not as part of the uniqueness key, unless you explicitly want multi-run history.

  • Is run_id being used as part of the merge or dedupe key? If so, flag it.

State the recommended approach and the exact dedup key for this codebase.

F) OBSERVABILITY: Can you debug a failure?

Verify:

slot milliseconds, and duration

  • Failures raise exceptions and abort with no silent except: pass or warn-only
  • Each BQ job logs: job ID, bytes processed or billed when available,

runid, env, mode, daterange, tables written, total BQ jobs, total bytes

  • A run summary is logged or written at the end containing:
  • run_id is present and consistent across all log lines

If runid is missing, propose a one-line fix: runid = run_id or datetime.utcnow().strftime('%Y%m%dT%H%M%S')

Final

1. PASS / FAIL with specific reasons per section (A to F). 2. Patch list ordered by risk, referencing exact functions to change. 3. If FAIL: Top 3 cost risks with a rough worst-case estimate (e.g., "loop over 90 dates x 3 retries = 270 BQ jobs").

More skills from github/awesome-copilot

  • Aacquire-codebase-knowledgeUse this skill when the user explicitly asks to map, document, or onboard into an existing codebase. Trigger for prompts like "map this codebase", "document this architecture", "onboard me to this repo", or "create codebase docs". Do not trigger for routine feature implementation, bug fixes, or narrow code edits unless the user asks for repository-level discovery.
  • Aacreadiness-assessRun the AgentRC readiness assessment on the current repository and produce a static HTML dashboard at reports/index.html. Wraps `npx github:microsoft/agentrc readiness` and hands off rendering to the @ai-readiness-reporter custom agent. Supports policies (--policy) for org-specific scoring. Use when asked to assess, audit, or score the AI readiness of a repo.
  • Aacreadiness-generate-instructionsGenerate tailored AI agent instruction files via AgentRC instructions command. Produces .github/copilot-instructions.md (default, recommended for Copilot in VS Code) plus optional per-area .instructions.md files with applyTo globs for monorepos. Use after running /acreadiness-assess to close gaps in the AI Tooling pillar.
  • Aacreadiness-policyHelp the user pick, write, or apply an AgentRC policy. Policies customise readiness scoring by disabling irrelevant checks, overriding impact/level, setting pass-rate thresholds, or chaining org baselines with team overrides. Use when the user asks about strict mode, AI-only scoring, custom weights, CI gating, or wants org-wide standardisation.
  • Aad-campaign-analyzerUse this skill when the user shares ad campaign performance data and asks what to cut, scale, or test. Trigger for prompts like "analyze my ad campaigns", "where am I wasting ad spend", "reallocate my ad budget", "which ads are actually working", or "ROAS analysis". Do not trigger for campaign planning or creative generation without performance data.
  • Aadd-educational-commentsAdd educational comments to the file specified, or prompt asking for file to comment if one is not provided.
  • Aadobe-illustrator-scriptingWrite, debug, and optimize Adobe Illustrator automation scripts using ExtendScript (JavaScript/JSX). Use when creating or modifying scripts that manipulate documents, layers, paths, text frames, colors, symbols, artboards, or any Illustrator DOM objects. Covers the complete JavaScript object model, coordinate system, measurement units, export workflows, and scripting best practices.
  • Aagent-architectureDesign AI agent architectures through requirements discovery, or audit and diagnose architectural flaws in existing agents. Architecture only; excludes implementation and general code review.
  • Aagent-governancePatterns and techniques for adding governance, safety, and trust controls to AI agent systems. Use this skill when: - Building AI agents that call external tools (APIs, databases, file systems) - Implementing policy-based access controls for agent tool usage - Adding semantic intent classification to detect dangerous prompts - Creating trust scoring systems for multi-agent workflows - Building audit trails for agent actions and decisions - Enforcing rate limits, content filters, or tool restrictions on agents - Working with any agent framework (PydanticAI, CrewAI, OpenAI Agents, LangChain, AutoGen)
  • Aagent-owasp-complianceCheck any AI agent codebase against the OWASP Agentic Security Initiative (ASI) Top 10 risks. Use this skill when: - Evaluating an agent system's security posture before production deployment - Running a compliance check against OWASP ASI 2026 standards - Mapping existing security controls to the 10 agentic risks - Generating a compliance report for security review or audit - Comparing agent framework security features against the standard - Any request like "is my agent OWASP compliant?", "check ASI compliance", or "agentic security audit"
  • Aagent-skill-stackFind, evaluate, and assemble the smallest compatible set of AI Agent Skills for an end-to-end natural-language goal. Use when a user wants Skills for a multi-step workflow, asks which Skills fit a project, needs an installed-Skill audit or conflict check, has low Skill recall, wants indirect helpers such as humanizers or compliance checks, or wants a project-specific Skill Stack with controlled installation. Search local Skills, registries, GitHub, and OpenCLI; compare adoption, verified fit, safety, and overlap. Do not use for locating one known or common Skill; use the generic find-skills workflow.
  • Aagent-supply-chainVerify supply chain integrity for AI agent plugins, tools, and dependencies. Use this skill when: - Generating SHA-256 integrity manifests for agent plugins or tool packages - Verifying that installed plugins match their published manifests - Detecting tampered, modified, or untracked files in agent tool directories - Auditing dependency pinning and version policies for agent components - Building provenance chains for agent plugin promotion (dev → staging → production) - Any request like "verify plugin integrity", "generate manifest", "check supply chain", or "sign this plugin"

All agent skills → · MCP servers