← home / notes
Thirteen notes on distributed systems, reliability, and LLMs in production - drawn as a graph, because that is how the ideas connect. Start anywhere and follow the edges.
The dependency graph you draw from RPC traffic is real but incomplete. The dependencies that take you down are the ones no edge shows.
When you fuse many noisy event streams into one model, the hard problems are not throughput - they are identity and time.
Most platforms that reason about a system are really stacks of queries against a model of that system. Platform quality is capped by the model underneath.
Reliability effort gets spent on whatever feels scary in the room. Here is how to replace that gut feel with a number.
A live broadcast turns one source into millions of simultaneous viewers in seconds. The hard part is not the video.
When an LLM makes a judgment at scale, do not let it grade itself. Pair a cheap predictor with an expensive ground-truth engine.
Before you wire an LLM into a real workflow, decide how you will know it is good enough - because the eval is the product spec.
A separate model can score outputs you cannot label by hand - but only if you treat it like a measuring instrument.
Most items in a large workload are easy, and a few are genuinely hard. The cheapest reliable pipeline routes compute by criticality.
Two ways to feed a model context: retrieve fuzzy passages by similarity, or call purpose-built tools that return exact answers.
When an LLM is the caller, the interface is the prompt. Typed responses, idempotent writes, granular composable calls.
A correct model or a sharp analysis is necessary but not sufficient. What gets internal AI actually used is the last mile.
In 2018 I helped build an RNN model that turned English questions into SQL. What that project looks like from the LLM era.