How we sync 12,000 banks in under 100ms
A look under the hood at our event-driven ingestion pipeline, lock-free queues, and the boring tricks that made it fast.
When a customer connects a new account to Finley, they expect transactions to appear within seconds, not minutes. Hitting that bar across 12,000 banks, each with its own API quirks, rate limits, and downtime windows, took us two years and a lot of unglamorous engineering.
The three pillars
Our ingestion stack stands on three boring choices: an event-driven core, lock-free queues, and aggressive caching of intermediate state. None of these are novel. The win was in disciplined application.
- Kafka topics per bank cluster, partitioned by customer hash
- Goroutine pool with bounded concurrency per institution
- Redis-backed dedup window with 5-minute TTL
- Postgres logical replication for downstream consumers
What we threw away
We rewrote the original cron-based fetcher three times. Each rewrite was an exercise in deleting code: the v3 ingestion service is 40% the line count of v1 and handles 25x the traffic.
