I audited my AI trading fleet and found two zombie trades

The setup

PaperGhost is eight AI trading personas paper-trading in public — four on stocks, four on crypto, each seeded with a virtual $1,000. Every signal, every rejected signal, every fill, every loss gets journaled and published. The deal I made with myself was simple: no cherry-picking, ever. If a ghost blows up, the blowup is the content.

Which is why the bug that kicked this off was existential rather than cosmetic. After a clean wipe-and-reseed — every engine back to exactly $1,000 — the database said one ghost held $997.73. The homepage said $1,068. If I can't trust the numbers on my own transparency project, there is no project.

Two accounting systems, written at different moments

The architecture had grown the way solo projects grow. The trading engine kept its truth in memory and a JSON state file: cash, open positions, realized P&L. A snapshot writer flattened that into a Postgres row every cycle: equity, cash, positions value. And the frontend — burned once by stale snapshots — had started recomputing “live” equity on its own: snapshot.cash + Σ(cost basis) + Σ(unrealized P&L) over the open-trades table, with fresh marks.

Three representations of the same dollar. Each correct in isolation. Written and read at different moments, by different code, from different tables. Every “bug” I'd chased that week was really one of the seams between them leaking.

So instead of patching display formula number four, I stopped and did a forensic audit: pick one ghost, trace one position from signal → fan-out → fill → mark-to-market → snapshot → website, and build a consistency matrix of every surface that claimed to know the number — engine memory, state file on disk, SQLite journal, Postgres, the race API, and two different pages of the site.

The matrix doesn't lie

The backend chain was perfect. State file, SQLite, Postgres, API: all four agreed to the cent — $996.76 at the same timestamp. The lie entered exactly at the display layer, and the matrix pointed straight at the input: the frontend's “live” formula was summing open positions from the trades table, and the ghost showing $1,068 had three open rows there. Its engine held two.

The third row was a short position in a small-cap stock, opened at 16:17 UTC — 43 minutes before the wipe. It existed only in Postgres. Not in SQLite, not in any state file, not in any engine's memory. A zombie. And it had a twin: a second ghost carried the same phantom short.

How a wipe leaks

The wipe had been done carefully — state files deleted, SQLite truncated, Postgres truncated — but not atomically. The agents were still running on their schedulers while the truncation happened. For roughly forty minutes, processes holding pre-wipe state in memory kept writing into freshly truncated tables: a few equity snapshots with pre-wipe balances, and those two open-position rows.

Nothing could ever close them. The close path updates a trade row in place, keyed by the engine's order ID — and no engine held those IDs anymore. Immortal rows, faithfully marked to market by the frontend every thirty seconds, inflating one ghost's displayed worth by exactly the phantom's cost basis plus its imaginary P&L. The site wasn't computing wrong. It was computing correctly on poisoned input, which is so much worse, because every formula you check comes back clean.

What actually fixed it

Deleting two rows took five minutes. The point of the audit was making this class of failure impossible to miss again:

One formula, one place. Equity math had been duplicated across two journal writers and three display paths, all subtly divergent. It now lives in a single compute_equity() on the writer side — pinned by a parity test that ran the old and new math against all eight live engines before cutover (difference: 0.00) — with a single TypeScript counterpart planned for the display side.

Reconciliation as a tripwire, not a hope. Every snapshot write now compares the engine's open symbols against the journal's open rows and logs a warning on mismatch. The zombie hunt took a day; the tripwire turns the next one into a one-cycle alarm. The frontend gets the same guard: if the lots don't reconcile with the snapshot, render the snapshot and a badge — never invent money.

A stop-the-world runbook. Wipes now have a mandatory order: stop the schedulers, prove the processes are dead with pgrep, truncate everything including the write-behind outbox (a queue that would otherwise cheerfully replay pre-wipe rows into your clean database), delete state, restart, verify.

Death to row-cap bugs. The audit surfaced a whole second family: queries with no explicit order silently clamped to 1,000 rows by the API gateway. First the charts froze on the oldest rows; after the obvious fix (descending + limit), the window itself shrank as write volume grew and yesterday quietly lost half its ghosts. The durable fix was moving aggregation into the database — a view that collapses ticks to one row per day per ghost — so the chart reads 8 rows a day instead of fighting a cap.

SQLite-to-Postgres migration honesty. Cutting the journal over to Postgres surfaced every place the code had quietly depended on SQLite's permissiveness: tuple-indexing rows that became dict-like (row[0] → KeyError), tuple-unpacking that silently iterated column names, naive local timestamps stored into timestamptz and reinterpreted as UTC — a five-hour skew on every trade. Each one failed quietly, which is the theme of this entire postmortem: the dangerous failures were the ones wrapped in a well-intentioned try/except.

Lessons, compressed

Parallel representations of money will drift; the only question is whether you find out from a tripwire or from a user. Display code must never be the fourth accounting system. Truncating a database under live writers isn't a wipe, it's a race. And every silent except around a write path is a future forensic investigation with your name on it.

Watch them learn

The fleet now runs autonomously on a $12 VPS — eight ghosts, one accounting formula, every number on the site traceable to a journal row. They are not good traders yet. That's the point: the calibrator refuses to size up any strategy until a hundred logged outcomes prove it deserves capital, and the Morgue is waiting for the ones that don't make it.

The race is public: paperghost.live/ghosts. Every signal, every loss, every death, journaled. Educational only — don't copy these trades. The ghosts are still learning.