Engineering
The hard part isn't sending events
Publishing an event is easy. Making sure every service ends up consistent, even when messages arrive late, twice, or out of order, is the real work.
We lean on an event-driven design where each service owns its data and reacts to events from others, rather than reaching into shared tables. That keeps the services decoupled and lets each one scale on its own.
The details that make it reliable are unglamorous: idempotent consumers so a repeated event does no harm, clear ownership so there is one source of truth for each fact, and careful handling of ordering and failure.
The real challenge
Consistency is a promise, not a default
Sending an event is easy; agreeing on the truth is hard.
In a distributed system, the same message can arrive twice, out of order, or after a service has already moved on. The hard engineering isn't the messaging, it's designing so that none of those cases corrupts your data.
We lean on idempotency and clear ownership of each piece of state, so every service can process an event confidently, knowing a replay changes nothing it shouldn't.
How we keep the truth straight
Idempotent handlers
Processing the same event twice yields the same result.
Single ownership
One service owns each fact, so there's no ambiguity to reconcile.
Safe replays
Reprocessing a stream is a routine operation, not a risk.
How we keep it consistent
The principles that matter
- 1
Clear ownership
Each fact has exactly one service that owns it, so there is never a fight over the truth.
- 2
Idempotent consumers
Processing the same event twice produces the same result, so retries and duplicates are safe.
- 3
Handle failure
We design for the message that arrives late, fails, or never comes, not just the happy path.
- 4
Observe everything
When something drifts, we can see it quickly and trace it back to the cause.
Tools serve the principles, not the other way round
The specific queue or stream matters less than the discipline around it. Get ownership, idempotency and failure handling right, and most tools will do the job well.
Deep dive
The boring choices that make realtime reliable
Most of the hard work in realtime sync is not the socket layer, it is deciding what happens when two edits collide, a client replays after a week offline, or a consumer falls behind the event stream. We settle those rules first and write them down.
Events are versioned and idempotent, so a message delivered twice changes nothing the second time. That single property removes a whole class of 3 a.m. incidents, because retries become safe instead of scary.
We also cap fan-out deliberately: presence and typing indicators can be lossy, financial state cannot. Splitting the two onto different delivery guarantees keeps the fast path fast without gambling with the numbers.
We build software for teams who want their tools to fit the way they actually work — web, mobile, AI and the systems that tie them together. We write here about what we learn shipping it.