WalScope
SQLite WAL Observatory
- Python
- SQLite
- WAL
- CLI
Problem
Checkpoint lag and WAL growth are hard to inspect without attaching SQLite to a live database. WalScope is a zero-runtime-dependency CLI that reads the files as bytes.
Safety property
inspect and watch do not open target databases through SQLite. Only lab.py imports sqlite3. Inspection uses open(..., "rb") on the database, -wal, and -shm. No sqlite3.connect on user targets, and no checkpoint or mutation of those files.
System design
How it works
WAL headers are 32 bytes; frame headers 24 bytes. Magic 0x377f0682 / 0x377f0683 selects checksum word order. Stored checksums stay big-endian. Salts and the cumulative checksum are validated; scanning stops on an invalid frame. A commit frame is the non-zero database-size field. 64 KiB pages are supported.
The wal-index is native-endian: duplicate 48-byte headers, header checksum, mxFrame, nBackfill, nBackfillAttempted, and five read marks. Opposite-endian or unsupported state is reported, not guessed. A populated read mark is not treated as proof of an active reader. A single snapshot is not treated as proof of checkpoint starvation.
Key engineering decisions
Parse bytes, don’t connect
Keeps inspection read-only even if SQLite would otherwise take locks or run a checkpoint.
Checksum endianness
Magic selects word order; on-disk checksum fields remain big-endian.
Stop on the first bad frame
Truncation and corruption fail closed instead of inventing a tail of “valid” frames.
Native wal-index
Matches SQLite’s shm layout on the host instead of assuming a wire endianness.
Pinned-reader experiment
Measured checkpoint lag while a reader was pinned, then after release. PASSIVE checkpoint return values matched the file view.
Pinned reader mxFrame 28 nBackfill 3 lag 25 PASSIVE checkpoint (0, 28, 3)
Reader released mxFrame 28 nBackfill 28 lag 0 PASSIVE checkpoint (0, 28, 28)
Local parse performance
Not an industry benchmark — times from this machine on SQLite-generated WAL files.
90,672 B
0.0012s
22 valid frames
1.72 MB
0.0232s
417 valid frames
17.1 MB
0.236s
4,153 valid frames
Verification
- Python 3.12.13 local · SQLite 3.53.1
- Python 3.12 and 3.13 CI passed
- 64 tests, 64 passed, 0 failures, 0 skips
- SQLite-generated WAL, magic 0x377f0682, header checksum passed, 10/10 physical frames valid
- 64 KiB WAL page size verified
Commit c56ae03d71c6f1d209f5505ba45b03e749fdae89 · v0.1.0
What I learned
File-format tools fail by being slightly wrong. Checksum word order, salt continuity, and refusing to over-interpret read marks matter more than a pretty hex dump.