Ask Anvil

A loop over a few thousand records throws on one of them (a timeout, a bad value, a dropped connection) and the whole job dies. You rerun it and either start from record one again, creating duplicate documents, or you have no idea which records finished.

The cause

The job keeps its progress only in memory, and one unhandled error ends the loop. With no durable record of which items succeeded, the only recovery is guessing.

The fix

Record a status for every item under a stable ID, in storage that survives a crash, and skip items already marked done. Catch errors per item so one bad record does not stop the rest. This Node.js version uses a JSON file, but a database table works the same way:

const fs = require('fs');

const STATE_FILE = 'batch-state.json';
const state = fs.existsSync(STATE_FILE)
  ? JSON.parse(fs.readFileSync(STATE_FILE, 'utf8'))
  : {};

function save() {
  fs.writeFileSync(STATE_FILE + '.tmp', JSON.stringify(state));
  fs.renameSync(STATE_FILE + '.tmp', STATE_FILE);
}

async function runBatch(records, generate) {
  for (const record of records) {
    if (state[record.id] && state[record.id].status === 'done') continue;
    try {
      const output = await generate(record);
      state[record.id] = { status: 'done', output };
    } catch (err) {
      state[record.id] = { status: 'failed', error: String(err.message) };
    }
    save();
  }
  return Object.entries(state)
    .filter(([, s]) => s.status === 'failed')
    .map(([id]) => id);
}

module.exports = { runBatch };

Run it again after a failure and only failed or unprocessed records are attempted. In a quick test with four records where record 3 threw on the first run, the second run called the generator for record 3 only.

Caveats

If the process crashes after a document is generated but before save() runs, that record is generated again on resume. Make the generate step safe to repeat, for example by writing to a deterministic file name derived from record.id instead of a random one.

Records that fail because of bad data will fail again on every rerun. Inspect the returned list of failed IDs, fix the data, and retry those deliberately instead of looping on them forever.