/differential-fuzzer

name: differential-fuzzer
description: Information about the differential fuzzer tool, how to run it and use it catch bugs in Turso. Always load this skill when running this tool

Differential Fuzzer

Always load [Debugging skill for reference](../debugging/)

The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.

Location

`testing/differential-oracle/fuzzer/`

Running the Fuzzer

Single Run

# Basic run (100 statements, random seed)
cargo run --bin differential_fuzzer

# With specific seed for reproducibility
cargo run --bin differential_fuzzer -- --seed 12345

# More statements with verbose output
cargo run --bin differential_fuzzer -- -n 1000 --verbose

# Keep database files after run (for debugging)
cargo run --bin differential_fuzzer -- --seed 12345 --keep-files

# All options
cargo run --bin differential_fuzzer -- \
  --seed <SEED>           # Deterministic seed
  -n <NUM>                # Number of statements (default: 100)
  -t <NUM>                # Number of tables (default: 2)
  -c <NUM>                # Columns per table (default: 5)
  --verbose               # Print each SQL statement
  --keep-files            # Persist .db files to disk

Continuous Fuzzing (Loop Mode)

# Run forever with random seeds
cargo run --bin differential_fuzzer -- loop

# Run 50 iterations
cargo run --bin differential_fuzzer -- loop 50

Docker Runner (CI/Production)

# Build and run from repo root
docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .
docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer

Environment variables for docker-runner:

• `TIME_LIMIT_MINUTES` - Total runtime (default: 1440 = 24h)
• `PER_RUN_TIMEOUT_SECONDS` - Per-run timeout (default: 1200 = 20min)
• `NUM_STATEMENTS` - Statements per run (default: 1000)
• `LOG_TO_STDOUT` - Print fuzzer output (default: false)
• `GITHUB_TOKEN` - For auto-filing issues
• `SLACK_WEBHOOK_URL` - For notifications

Output Files

All output goes to `simulator-output/` directory:

| File | Description | |------|-------------| | `test.sql` | All executed SQL statements. Failed statements prefixed with `-- FAILED:`, errors with `-- ERROR:` | | `schema.json` | Database schema at end of run (or at failure) | | `test.db` | Turso database file (only with `--keep-files`) | | `test-sqlite.db` | SQLite database file (only with `--keep-files`) |

Reproducing Errors

Always follow these steps

1. **Find the seed and profile** in the error output:

   INFO: Starting differential_fuzzer with config: SimConfig { seed: 12345, ..., weight_profile: Writes }

2. **Re-run with that seed and profile** (a seed only replays under the same profile):

   cargo run --bin differential_fuzzer -- --seed 12345 --profile writes --verbose --keep-files

3. **Read the minimized reproduction first.** On an oracle failure the fuzzer writes these files to `simulator-output/`:

• `minimized.sql` - a shrunken state script plus the shrunken failing

statement, produced automatically. Start here.

• `turso-state.sql` / `sqlite-state.sql` - each engine's full state as a

replayable script, when you need more than the minimized version kept.

• `test.sql` - every executed statement (the failing one is marked

`-- FAILED:`). The minimizer falls back to replaying this history when the failure depends on how the state was built, not just its contents.

• `schema.json` - table structure at failure time.

4. **Probe the reproduction with `differential_probe`.** It runs a statement-per-line script on Turso and SQLite side by side, prints both outcomes for every statement, marks divergences, and compares the final table contents. Exit code 1 means something diverged.

   cargo run -q -p differential-fuzzer --bin differential_probe -- \
       simulator-output/minimized.sql

Use it instead of piping SQL into the two shells: the tursodb shell cannot `ATTACH ':memory:' AS aux`, so fuzzer reproductions with an `aux` schema only run correctly through the probe. Reading from stdin also works: `echo "SELECT ~X'96';" | cargo run -q -p differential-fuzzer --bin differential_probe`.

5. **Bisect by editing the script.** Copy `minimized.sql`, simplify one thing at a time (replace an expression with a constant, drop a column, drop a state line), and re-run the probe after each edit. The divergence marker tells you immediately whether the edit kept the bug. This loop usually ends at a one-line kernel you can hand to `EXPLAIN` on both engines.

6. **Create a permanent `*.sqltest` regression** from the kernel before the fix is complete. Preserve only the schema, data, statements, and assertions required to expose the bug. The seed and minimized fuzzer artifact are debugging aids, not substitutes for this regression. Use a Rust test only when the behavior cannot be expressed in `.sqltest`, and always load the [Debugging skill for reference](../debugging/).

Understanding Failures

Oracle Failure Types

1. **Row set mismatch** - Turso returned different rows than SQLite 2. **Turso errored but SQLite succeeded** - Turso rejected valid SQL 3. **SQLite errored but Turso succeeded** - Turso accepted invalid SQL 4. **Schema mismatch** - Tables/columns differ after DDL

Warning (non-fatal)

• **Unordered LIMIT mismatch** - LIMIT without ORDER BY may return different valid rows

Key Source Files

| File | Purpose | |------|---------| | `main.rs` | CLI parsing, entry point | | `runner.rs` | Main simulation loop, executes statements on both DBs | | `oracle.rs` | Compares Turso vs SQLite results | | `schema.rs` | Introspects schema from both databases | | `memory/` | In-memory IO for deterministic simulation |

Tracing

Set `RUST_LOG` for more detailed output:

RUST_LOG=debug cargo run --bin differential_f