Placebo: An Active Falsification Layer for Snowflake Data Agents

Executive Summary

The fundamental problem with autonomous AI data agents is exploratory speed without scientific skepticism. When tools like Snowflake Cortex Code (CoCo) analyze enterprise data, they can test dozens of dimensional slices in seconds. However, across large, noisy time-series datasets, testing enough combinations guarantees that random variance will yield statistically significant correlations (p < 0.05). Left unchecked, this produces false discoveries and automated p-hacking.

Placebo is an open-source statistical audit engine built to verify discoveries made on Snowflake before they reach human decision-makers. Rather than accepting correlations at face value, Placebo acts as an automated econometric reviewer. It subjects every candidate finding to pre-trend checks, synthetic in-time placebo dates, leave-one-out sensitivity testing, and multiplicity corrections, using open-weight Gemma models to generate evidence-grounded explanations.

---

Direct Alignment with Challenge Criteria

1. Snowflake CoCo Integration as the Autonomous Analyst

• Exploratory Data Analysis: We use Snowflake Cortex Code (CoCo) to interpret user questions in plain English and formulate analytical SQL queries directly against Snowflake tables.
• Protocol Enforcement via Agent Skills: We developed a custom CoCo skill (skill/placebo-audit/SKILL.md) and a Model Context Protocol (MCP) server. Under this protocol, CoCo must register every candidate hypothesis into an append-only ledger before executing exploratory queries.
• Lineage and Search Space Auditing: CoCo tracks and logs every executed Snowflake query ID and dimensional slice. This provides the exact search space denominator required to penalize multiple-testing bias rather than hiding exploratory attempts.

2. Freely Accessible Snowflake Marketplace Dataset

• Source and Listing: We connected directly to the public, freely accessible Global Weather & Climate Data for BI listing provided by Pelmorex Weather Networks on the Snowflake Marketplace.
• Target Table: PELMOREX_WEATHER.PUBLIC.POINT_HISTORY_DAY.
• Scope and Dimensions: 3,690 daily records across 10 global cities (including Paris, Tokyo, New York, Calgary, Houston, and Sydney) covering the 2025–2026 observation window.
• Telemetry Columns: CITY_NAME, DATE_VALID_STD, AVG_TEMPERATURE_AIR_2M_F, and TOT_PRECIPITATION_DAY_IN.
• Infrastructure: Live database connection handled through snowflake-connector-python using connection pooling, push-down SQL filtering, and secure credential handling.

3. Open-Weight AI Architecture Powered by Gemma

• Substantive Role in Synthesis: Once Placebo computes deterministic mathematical tests (Welch’s t-test, ordinary least squares regressions, and Benjamini-Hochberg FDR adjustments), Google’s open-weight Gemma model (run via Ollama and the Gemini API) synthesizes the executive audit summary.
• Strict Evidence-Grounded Prompt Contract: Gemma is bound by a strict guardrail contract. It only receives structured JSON outputs containing observed slopes, p-values, and test thresholds. Gemma explains the econometric rationale behind the verdict, but is strictly prohibited from inventing statistics or altering the mathematical determination.

---

Step-by-Step System Flow

1. Hypothesis Formulation: A user enters a question into our web studio (for example: “Did average temperature in Paris rise after Jan 1st, 2026?”). CoCo identifies the intervention target, date window, and metric.
2. Ledger Pre-Registration: CoCo logs the candidate hypothesis into Placebo’s SQLite ledger, establishing an immutable audit trail of the session search space.
3. Live Snowflake Execution: CoCo issues push-down SQL queries to the POINT_HISTORY_DAY table in Snowflake, retrieving the exact historical series.
4. Econometric Falsification Battery: Placebo executes four automated tests on the retrieved Snowflake data:
   • Pre-Trend Invariance Check: Fits an OLS regression on pre-intervention data. If the metric was already drifting downward or upward prior to the event date, the claim is rejected.
   • Synthetic Placebo Dates Test: Evaluates 5 counterfactual historical dates before the event. If random dates replicate the effect, the finding is marked spurious.
   • Multiplicity FDR Correction: Applies the Benjamini-Hochberg procedure across all tested queries to control the false discovery rate.
   • Leave-One-Out Sensitivity: Jackknifes individual clusters to ensure the result is not an artifact of an isolated outlier.
5. Grounded Explanation and Verdict: The engine assigns a deterministic grade (REFUTED, SUSPICIOUS, or CONFIRMED). Open-weight Gemma explains the decision in natural language, and the full interactive breakdown appears on the studio dashboard.

---

Empirical Proof: The Paris Warming Anomaly

• The Superficial Finding: A conventional post-event comparison suggests Paris experienced an apparent +11.49°F warming after January 1st, 2026, with an unadjusted nominal p-value of 0.0000. An unassisted AI analyst would routinely report this as a statistically significant discovery.
• The Placebo Verification:
  1. Placebo’s pre-trend regression identifies a statistically significant cooling slope of -0.216°F per day (p = 0.0000) leading up to January 1st. The subsequent rise is simply normal winter seasonal variation rebounding from a trough.
  2. 5 out of 5 synthetic placebo dates before January also produce false positive significance on untouched baseline data.
• The Result: The system issues a REFUTED verdict, preventing an incorrect causal claim from being accepted as authentic business intelligence.

---

Repository Structure and Open-Source Licensing

The project is fully open source under the Apache 2.0 License. The repository is cleanly partitioned into modular components:

• placebo/snowflake/: Snowflake connection management, credentials handling, and push-down query execution.
• placebo/timeseries/: Core econometric tests (pre-trends, placebo dates, leave-one-out sensitivity).
• placebo/statistics/: Two-sample effect sizing, Welch’s t-test, and Benjamini-Hochberg FDR multiplicity control.
• placebo/mcp/: Standardized Model Context Protocol (MCP) server for Snowflake CoCo and remote agent connectivity.
• placebo/llm/: Open-weight Gemma prompt contracts, evidence serializers, and verification guardrails.
• frontend1/: Next.js 16 interactive investigation studio with real-time Snowflake telemetry.