Back to open source

Re-reading a crash-safe indexer

An Ethereum indexer that writes each block and its checkpoint in one Postgres transaction, designed so a crash cannot leave half a block. That part held up. Re-reading the rest found three ways it could still lose or duplicate data, and making it testable turned up two more.

Nihal-Pandey-2302/rust-evm-indexer the fix commits Rust, Tokio, SQLx, PostgreSQL 10 tests fail on the old code; all 11 pass after the fixes

The question

An indexer copies blocks, transactions and logs from an Ethereum node into Postgres. The question that decides whether it can be trusted: if the process dies halfway through a block, what is in the database when it comes back?

The first design

The first version, in May 2025, wrote each row straight to the pool, with no transaction around a block and no record of where it had got to. The log insert already carried a note to myself:

// Relies on 'id' BIGSERIAL for conflict. Consider unique constraint on (tx_hash, log_index_in_tx) for content-based idempotency.
From the log insert in the first version (554b6b5). It took eighteen months to act on.

The part that held up

Two days later (327fd40) every block was written in one Postgres transaction, with the sync checkpoint inside it: the block, its transactions, their logs, then the checkpoint, then commit. A crash before the commit leaves nothing behind. A crash after it leaves a complete block and a checkpoint at that block, so the next run starts after it and the block is never processed twice. On the later re-read, this part still held.

Re-reading the rest

A re-read in November 2026, made to check what this site could honestly say about the project, went through the rest. The README claimed reorg handling and idempotent writes. Three things did not hold.

1. A reorg left a hole

When a new block's parent did not match the stored block below it, the code deleted that stored block and everything above it, in three separate statements, and left the checkpoint where it was.

stored
100 101 102 103
block 104 arrives
its parent is not the stored 103
old rollback
deletes 103, checkpoint stays at 103, next cycle fetches 104: the new 103 is never fetched
fixed rollback
walks back to 102, the last height that matches the chain, deletes above it and rewinds the checkpoint to 102 in one transaction: 103 is fetched again
The one-block case. The old code also only ever looked one block back, and a crash between its deletes left logs gone with their transactions still in place.

2. A failed receipt committed an incomplete block

Receipts carry a transaction's status and logs. If fetching one still failed after three tries, the error was turned into "no receipt": the transaction was written with no status and no logs, and the checkpoint moved on. The block was never revisited, which is exactly the state the README said could not happen.

3. Logs were not idempotent

Blocks and transactions ignored duplicates on insert. Logs had nothing to conflict on, the note from the first version still unacted on. Only the checkpoint kept them from being written twice, so any path that re-applied a block would have duplicated every log.

Making it testable first

The repository had no tests. Before changing anything, the ingester moved behind a small trait for the chain, so a stub node could serve blocks, forks and failures on demand, with a real Postgres underneath. Eight tests were written against the old behaviour first, and every one failed. Two more, added while fixing, also fail on the old code.

The fixes

A reorg now walks back from the stored tip until the stored hash matches the chain, up to 64 blocks. Then one transaction deletes everything above that height and rewinds the checkpoint to it. A reorg deeper than 64 blocks stops ingestion and changes nothing, rather than guessing.

pub async fn rollback_to(pool: &PgPool, keep_height: u64) -> Result<(), sqlx::Error> {
    let height = keep_height as i64;
    let mut tx = pool.begin().await?;
    sqlx::query("DELETE FROM logs WHERE block_number > $1")
        .bind(height)
        .execute(&mut *tx)
        .await?;
    sqlx::query("DELETE FROM transactions WHERE block_number > $1")
        .bind(height)
        .execute(&mut *tx)
        .await?;
    sqlx::query("DELETE FROM blocks WHERE block_number > $1")
        .bind(height)
        .execute(&mut *tx)
        .await?;
    sqlx::query("UPDATE indexer_status SET last_processed_block = $2 WHERE indexer_name = $1")
        .bind(INDEXER_NAME)
        .bind(height)
        .execute(&mut *tx)
        .await?;
    tx.commit().await
}
The rollback, from src/db.rs (df82957). The checkpoint rewind is in the same transaction as the deletes.

A receipt that fails, comes back empty, or belongs to a different block now fails the whole block before its transaction starts, so the checkpoint stays put and the block is retried. Logs gained a key, and checking what to key them on turned up a mislabelled column: the stored index was the log's position in the block, not in its transaction. They are now unique on block hash and log index, and a migration for existing databases, tested on a copy of the old schema, adds the key and removes duplicates.

Two more, found while writing the tests

If the node did not return a block yet (a lagging node), the old loop skipped that height, committed the next one and moved the checkpoint past the gap. And a receipt fetched just after the node itself reorged could belong to the other fork, and was stored with its logs. Both now fail the block instead.

What the tests showed

Run against the commit before the fixes, then after:

one_block_reorg_reingests_replaced_block
checkpoint 103, expected 102: the replaced block is never fetched again
two_block_reorg_rolls_back_to_fork_point
checkpoint 103, expected 101: a stale block from the old fork stays
failed_rollback_leaves_data_and_checkpoint_unchanged
6 transactions and 12 logs left, expected 8 and 16
reorg_deeper_than_limit_is_not_rolled_back
no error: one block rolled back regardless of depth
failed_receipt_does_not_commit_block
block committed with a transaction that has no status and no logs
missing_receipt_does_not_commit_block
same, for a receipt the node returns as null
receipt_from_another_block_does_not_commit_block
a receipt from the other fork stored with its logs
reapplying_a_block_keeps_one_copy_of_each_log
16 logs, expected 8
second_block_at_same_height_is_rejected
the second block was inserted
block_missing_from_rpc_does_not_skip_height
checkpoint 102 with no block 101
What each test found on the old code. With the fixes, all 11 pass, including a test of the migration, which is new code with no "before".

Who did what

The first design and the per-block transaction with its checkpoint are mine. The November re-read, the five fixes and the tests were done with Claude, as every fix commit records.

What is still open

  • A reorg is noticed only when the next block arrives. If the new chain is shorter than what is stored, the stale tip stays until the chain grows past it.
  • A reorg deeper than 64 blocks needs a manual rollback; the repository has the statement.
  • The tests run against a stub node, not the real HTTP client.
  • A process crash in the middle of a block write is not tested directly. It rests on Postgres transactions; the injected mid-rollback failure is the closest test.
  • Databases written by the old code may still hold gaps. A check script finds them; the migration does not repair them.

The repository on GitHub