/differential-fuzzer

name: differential-fuzzer
description: Information about the differential fuzzer tool, how to run it and use it catch bugs in Turso. Always load this skill when running this tool

Differential Fuzzer

Always load [Debugging skill for reference](../debugging/)

The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.

Location

`testing/differential-oracle/fuzzer/`

Running the Fuzzer

Single Run

# Basic run (100 statements, random seed)
cargo run --bin differential_fuzzer

# With specific seed for reproducibility
cargo run --bin differential_fuzzer -- --seed 12345

# More statements with verbose output
cargo run --bin differential_fuzzer -- -n 1000 --verbose

# Keep database files after run (for debugging)
cargo run --bin differential_fuzzer -- --seed 12345 --keep-files

# All options
cargo run --bin differential_fuzzer -- \
  --seed <SEED>           # Deterministic seed
  -n <NUM>                # Number of statements (default: 100)
  -t <NUM>                # Number of tables (default: 2)
  -c <NUM>                # Columns per table (default: 5)
  --verbose               # Print each SQL statement
  --keep-files            # Persist .db files to disk

Continuous Fuzzing (Loop Mode)

# Run forever with random seeds
cargo run --bin differential_fuzzer -- loop

# Run 50 iterations
cargo run --bin differential_fuzzer -- loop 50

Docker Runner (CI/Production)

# Build and run from repo root
docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .
docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer

Environment variables for docker-runner:

• `TIME_LIMIT_MINUTES` - Total runtime (default: 1440 = 24h)
• `PER_RUN_TIMEOUT_SECONDS` - Per-run timeout (default: 1200 = 20min)
• `NUM_STATEMENTS` - Statements per run (default: 1000)
• `LOG_TO_STDOUT` - Print fuzzer output (default: false)
• `GITHUB_TOKEN` - For auto-filing issues
• `SLACK_WEBHOOK_URL` - For notifications

Output Files

All output goes to `simulator-output/` directory:

| File | Description | |------|-------------| | `test.sql` | All executed SQL statements. Failed statements prefixed with `-- FAILED:`, errors with `-- ERROR:` | | `schema.json` | Database schema at end of run (or at failure) | | `test.db` | Turso database file (only with `--keep-files`) | | `test-sqlite.db` | SQLite database file (only with `--keep-files`) |

Reproducing Errors

Always follow these steps

1. **Find the seed and profile** in the error output:

   INFO: Starting differential_fuzzer with config: SimConfig { seed: 12345, ..., weight_profile: Writes }

2. **Re-run with that seed and profile** (a seed only replays under the same profile):

   cargo run --bin differential_fuzzer -- --seed 12345 --profile writes --verbose --keep-files

3. **Read the minimized reproduction first.** On an oracle failure the fuzzer writes these files to `simulator-output/`:

• `minimized.sql` - a shrunken state script plus the shrunken failing

statement, produced automatically. Start here.

• `turso-state.sql` / `sqlite-state.sql` - each engine's full state as a

replayable script, when you need more than the minimized version kept.

• `test.sql` - every executed statement (the failing one is marked

`-- FAILED:`). The minimizer falls back to replaying this history when the failure depends on how the state was built, not just its contents.

• `schema.json` - table structure at failure time.

4. **Probe the reproduction with `differential_probe`.** It runs a statement-per-line script on Turso and SQLite side by side, prints both outcomes for every statement, marks divergences, and compares the final table contents. Exit code 1 means something diverged.

   cargo run -q -p differential-fuzzer --bin differential_probe -- \
       simulator-output/minimized.sql

Use it instead of piping SQL into the two shells: the tursodb shell cannot `ATTACH ':memory:' AS aux`, so fuzzer reproductions with an `aux` schema only run correctly through the probe. Reading from stdin also works: `echo "SELECT ~X'96';" | cargo run -q -p differential-fuzzer --bin differential_probe`.

5. **Bisect by editing the script.** Copy `minimized.sql`, simplify one thing at a time (replace an expression with a constant, drop a column, drop a state line), and re-run the probe after each edit. The divergence marker tells you immediately whether the edit kept the bug. This loop usually ends at a one-line kernel you can hand to `EXPLAIN` on both engines.

6. **Create a regression test** in `.sqltest` (preferred) or `.rs` from the kernel. Always load the [Debugging skill for reference](../debugging/).

Understanding Failures

Oracle Failure Types

1. **Row set mismatch** - Turso returned different rows than SQLite 2. **Turso errored but SQLite succeeded** - Turso rejected valid SQL 3. **SQLite errored but Turso succeeded** - Turso accepted invalid SQL 4. **Schema mismatch** - Tables/columns differ after DDL

Warning (non-fatal)

• **Unordered LIMIT mismatch** - LIMIT without ORDER BY may return different valid rows

Key Source Files

| File | Purpose | |------|---------| | `main.rs` | CLI parsing, entry point | | `runner.rs` | Main simulation loop, executes statements on both DBs | | `oracle.rs` | Compares Turso vs SQLite results | | `schema.rs` | Introspects schema from both databases | | `memory/` | In-memory IO for deterministic simulation |

Tracing

Set `RUST_LOG` for more detailed output:

RUST_LOG=debug cargo run --bin differential_fuzzer -- --seed 12345