tl;dr Over three years, we moved our latency-sensitive services to Rust. LB (load balancer) latency dropped from 600ms to 101ms, publish API latency from ~350µs to ~50µs, and a later Presence API redesign cut peak memory about 6x. We also let an unbounded Tokio workload turn a 100 MiB pod into a 3.7 GiB pod.
Two years ago, I wrote about moving one data-pipeline service from Python to Rust in 120ms to 30ms: Python 🐍 to Rust 🦀🚀. Since then, we have moved the rest of our performance-sensitive stack.
Disclosure: I worked on the original C systems years ago. Now a new team rebuilt these systems in Rust.
Our core services were a mix of Python, Go, JVM, and C. We migrated the critical ones one service and one region at a time, running old and new implementations side by side until we had enough data to move the remaining traffic.
The rewrites cut latency and resource use. Memory behavior became easier to reason about, and the team prefers working in the new codebases.
Once latency became more stable, we could see details the old runtime noise had hidden. A 100µs excursion now sticks out and we can easily investigate changes of 30µs.
Most panels show a baseline and cutover. Panels 8 and 9 compare Rust with Rust. Panel 14 shows a rolling replacement; panels 4 and 13 show steady state only.
Reddit post gets only one image, so I tiled fifteen numbered panels together in the order discussed below.
The nginx replacement
Our load balancer decides where each message goes next. Its average time had sat at 600ms for so long that we had stopped questioning it.
We replaced the nginx-based balancer with Pingora, Cloudflare's Rust proxy framework. We kept the same box, traffic, and routing decisions.
Panel 1: Balancer time, 600ms to 101ms.
The drop starts around 09:40 as the Rust balancer takes over. We shifted traffic in two increments, which produced the brief ledge at ~300ms. Once the cutover finished, latency held at 101ms on the same hardware and traffic.
The right edge of panel 1 includes broker latency. That internal hop stays at ~350µs during the cutover. We had already upgraded the broker to Rust, so the balancer accounted for nearly all of the 600ms.
PubSub: lower publish latency and variance
Publish was already under a millisecond, so I didn't expect much room for improvement.
Panel 2: Publish latency, from a 400-500µs band to roughly 40-60µs.
The three series, average_publish, average_internal_publish, and average_signal, sat in the 400 / 500µs band, with a ~1ms excursion around 14:55.
A smaller bump reaches ~600µs at 15:10 as the cutover starts. Over the next few minutes, the band drops to roughly 40-60µs and stays there for the remaining twenty minutes. We watched the service for weeks before calling the migration complete. Neither the spikes nor the unexplained 1ms tail in customer p99 dashboards returned.
The lower variance mattered more to our latency guarantees than the lower average.
Measuring server hops in microseconds
The migration changed how I measure server-side latency. Microseconds are now a useful unit for these internal hops.
Panel 3 shows the same publish path over a day and a half.
Panel 3: Publish latency over 36 hours, ~350µs to ~50µs.
Green (average_internal_publish) fluctuates between 350µs and 460µs on the left, while orange and yellow (average_signal, average_publish) sit around 300 / 350µs. Around 10-22 08:00, green joins them near 300µs. Just before 10:00, all three fall into a 40 / 60µs band and stay there for the next 24 hours.
The hottest path went from ~350µs to ~50µs, with less variance. The entire chart stays under 500µs, and small regressions are easier to spot.
On the right, green rises to ~100 / 185µs around 10-23 00:00 / 04:00 while orange and yellow stay near 40-60µs. The bump affects internal publish only. A 100µs excursion disappeared inside the old stack's normal variance; here it is clear enough to investigate.
The replication layer runs at 27µs to 31µs.
Panel 4: Replication average latency, 27-31µs.
This steady-state panel shows average replication latency for two nodes, average_21 and average_23, across a vertical range of 27µs to 31µs.
Green moves between 28.5µs and 30.5µs. Yellow sits at 27.5 / 29.5µs and follows the traffic cycle. The ~1.5µs gap between the nodes is visible. Three years ago, GC pauses hid it.
GC did not account for all of the old latency. We owned the code and could have kept improving it. The first migration gave us enough evidence that a Rust rewrite would repay its cost.
We can now investigate whether the NIC, the scheduler, or our code makes one node trail another by 1.5µs. A 5µs improvement in this 30µs hop will show up.
At 3 trillion API calls a month, even single-digit-microsecond changes are measurable.
Presence: latency and memory
Presence tracks who is online and which channel they occupy across regions. It holds state for millions of concurrent occupants while heartbeats arrive, expire, and reconcile across the network. Presence had caused more incidents than any other service, so this was our riskiest migration.
Subscribe join latency, the presence online status event
Panel 5: Presence subscribe join latency (median), az1 from 1-3s to 200-250ms.
Green (az1) is on the old stack, moving between 1s and 3s with a 2.9s median spike at ~13:58. Orange and blue (az2, az4) are already on Rust and hold near ~350ms while az1 spikes.
At 14:03, we cut az1 over. It drops to ~200 / 250ms, below az2 and az4 at around 300ms because az1 received a later build. The other regions have since moved to that version and now match it.
Web heartbeat join latency
The web/HTTP path drops from 700ms to 200ms.
Panel 6: Presence web heartbeat join latency, 700ms to 200ms.
The blue refresh series fluctuates between 600ms and 800ms before the 10:30 cutover, then holds near ~200ms. join_announce (green) stays around 100 / 150ms before moving toward 200ms at the right edge. join_interval (yellow) was near zero and stops reporting around 10:25 because we removed that metric during the cutover.
The internal heartbeat fanout
Panel 7: Subscribe internal heartbeat latency, 1.5-2s to 500ms.
Three regions sit in the 1.4s / 1.8s range because they share an overloaded path. One spikes to 2.1s during the cutover. Green follows the same rhythm at a lower ~1s baseline. After the ~17:35 cutover, all four settle into a ~500 / 600ms band with less variance, about three times faster than before.
Presence memory
Panel 8: Presence pod memory, Rust vs Rust, a 3.4 GiB peak down to a 256-512 MiB band.
Panel 8 shows memory per pod in the presence namespace. Both halves are presence-rust deployments from two ReplicaSets, 6cdfbf75d5 and 6f78d4f459. This is a comparison between Rust designs, separate from the language migration.
In the old ReplicaSet, the top pod peaks at 3.4 GiB while the rest of the fleet ranges from 512 MiB to 2.5 GiB. Memory declines over more than a day as the service trims accumulated occupancy state. That build still carried substantial per-occupant overhead.
After the 04/23 15:00 / 18:00 gap, the new ReplicaSet holds within a ~256 / 512 MiB band. The growth curve disappears, and pods no longer approach an OOMKill during peak traffic.
Removing the GC made the profile easier to reason about, but we still had to fix the data layout. The first Rust build beat the Python service it replaced; the redesign cut peak memory by another 6x.
Push notifications: FCM
Push is a fan-out-and-wait workload. Apple and Google account for much of the latency, so our part needs to stay small and consistent.
Panel 9: FCM Rust average time, Rust vs Rust, from 135-250ms to roughly 135ms.
The single series, fcm rust avg, is the Rust path on both sides of the chart. The left side ranges from 135ms to 250ms and spikes at 08:45. Just after 09:00, it settles at ~135ms for the rest of the window.
At 09:00, we enabled connection pooling and reuse against FCM. The low runtime noise let us isolate the remaining variance and trace it to our code rather than Google's service.
That flat line is my favorite kind of graph.
Event Processing: fewer warnings and retries
Panel 10 counts requests that hit a degraded or retry-worthy path on internal_heartbeat, a high-volume route through event processing.
Panel 10: API warnings on the internal_heartbeat route, 1.5-4.9K down to under 250.
Green (az2) sustains 1.55K to 4.9K warnings throughout the afternoon. Yellow (az4), already migrated, usually stays between 100 and 500 with occasional excursions to ~860.
At 17:03, az2 drops into the same range as az4. Both spike at 17:20, when green reaches 1.02K and yellow ~740, then touch 400 / 600 a few times through 17:45. After ~17:50, both usually stay under ~250 with occasional spikes to ~400.
With roughly an order of magnitude fewer warnings and retries, alerts stand out. We no longer tune thresholds to ignore the service's normal behavior.
Memory and CPU
The migrations reduced memory and CPU use as well.
Memory, per shard
Panel 11: Shard 5 max memory, 95% down to a 20-40% band.
This panel covers shard 5 across all regions for two days. On the left, red sits between 90% and 97%, above the critical threshold, while yellow falls from ~90% to 85%. One traffic spike could have exhausted the remaining memory.
The cutover runs from 10-06 22:30 to after 10-07 00:00, when every series lands in the 30 / 40% band. Peak traffic on 10-07 between 12:00 and 16:00 pushes usage to ~69%. Overnight it falls to 12 / 25%, then returns to 20 / 37% with daytime traffic.
On the same boxes, peak memory moved from the critical range to below the 75% warning threshold.
Memory, per region
Panel 12: Max memory usage by region, with the step down at cutover.
The regional view shows the same step down on 06/22. The tooltip values come from after the cutover. Before it, the lines seesaw as the runtime allocates and collects memory.
After the cutover, the lines become thinner and smoother. iad remains highest at ~72% because this fleet still included unmigrated capacity; the other four regions had completed the rollout. We moved each region only after comparing it with capacity still running the old implementation.
(This iad fleet is now fully migrated. Other services there moved earlier; panels 8 and 10 both show iad clusters. We rolled out each service on its own schedule.)
CPU
Panel 13: Max CPU usage across regions, holding in the 0-40% band.
Across five days, regional CPU stays in the 0 / 40% band with occasional peaks at 60 / 70%. It never reaches the 75% warning or 90% critical lines. The sawtooth pattern follows normal daily traffic.
CPU per pod
Panel 14: CPU usage by pod, 0.5 cores down to 0.22 during a rolling replacement.
This panel shows per-pod CPU during a rolling replacement. The old pods use 0.4 / 0.6 cores before they drain and terminate.
The new pods stabilize at ~0.22 cores, about half the CPU for the same work. We used the savings to run fewer pods with more headroom.
Delivery semantics
We required the new system to match or improve our delivery guarantees. The delivery changes mattered more to us than the latency graphs.
For years, we replicated messages over a hand-rolled TCP protocol. It moved trillions of messages and supported retry and redelivery. It also left us responsible for framing, backpressure, reconnect logic, sequence numbers, and versioning between old and new nodes. Only a few engineers knew all of its failure modes.
We replaced the protocol with gRPC and a store-and-forward model.
Store and forward: persist first, stream second.
We replicate an incoming event to multiple physical nodes while beginning to stream it. We return delivery status only after the durable write. If the stream breaks or the downstream service is rolling, we query the replicas for messages that still need redelivery.
Streaming first leaves an unrecoverable gap if the process crashes between "sent" and "recorded." Persisting first may produce a duplicate delivery attempt. The record remains on the replica nodes, and an idempotency key handles the duplicate. The wire guarantee is at least once; the idempotency key makes delivery effectively once at the application layer. We do not claim exactly-once delivery over an unreliable network. For acknowledged messages, the durable replicas preserve a redelivery path, and the idempotency key prevents a repeated attempt from becoming an observable duplicate.
gRPC's automatic retry policy, configured at the service level.
The gRPC service config defines retryable status codes, attempt limits, and backoff. The channel handles retries below the application call, so many brief network failures no longer reach application code.
Previously, each service implemented its own retry policy. Conservative policies risked dropped messages; aggressive ones could overload a service during a brief failure.
gRPC absorbs transient failures such as momentary TCP resets. Store and forward handles failures that outlast the retry policy.
Rust's compile-time concurrency checks gave us more confidence in the new replication model. We ran both implementations side by side and watched the dashboards shown here.
Why Rust fit this workload
Rust suited this workload: a latency-sensitive message bus that holds millions of concurrent occupants in memory. These results do not mean every CRUD service needs a rewrite.
We spent years profiling and tuning the existing services. Managed runtimes can go far, but the collector still decides when to pause the program. The 1ms publish excursion and 2s heartbeat oscillation show the latency cost; the regional memory chart shows the allocation churn. The Rust services no longer incurred GC pauses.
We use Go for many API services. For this part of the message bus, we could not meet our latency and memory targets without removing GC pauses and reducing per-object overhead.
Our C codebase was 14 years old and needed a major update for current libraries and compilers. Since we faced a substantial rewrite either way, we chose Rust. We could have made some of the same performance improvements in C, and our measurements put Rust within the noise of C. The practical difference was that more engineers felt comfortable changing the concurrent Rust code, while the compiler caught ownership and data-race mistakes before deployment.
The first six months with the borrow checker were rough. Experienced engineers got frustrated, PRs stalled, and people questioned the migration. By month eight, review cycles had shortened and we spent less time debugging ownership mistakes after compilation.
cargo helped too. A shared toolchain, formatter, test runner, and build command save time in infrastructure spread across four languages.
Our Tokio mistake: unbounded in-flight work
Panel 15 shows the costliest mistake in the migration.
Panel 15: Memory usage by pod, from a ~100 MiB baseline to a 3.7 GiB burst.
Bursts arrived faster than we could complete them. We spawned a task per request, Tokio queued the tasks, and each one held buffers and state while waiting on I/O. Leak detection found nothing because the memory was live. Without an explicit cap, in-flight work grew with each burst. Waiting tasks use little CPU, but their state still consumes memory.
We first tuned the HPA with lower scale-up thresholds and faster reaction windows. It reacted sooner, but scaling out gave the unbounded backlog more places to accumulate.
We added backpressure at the ingest edge. A cap on in-flight work sends the burst to a bounded queue instead of letting tasks accumulate on the heap. Excess work waits or gets shed, so memory no longer grows with the arrival rate. We deployed the change in every region, and the same pods now hold their baseline under comparable bursts instead of climbing into GiB. I don't yet have a matching after panel for a comparable burst.
Two years ago, my Python-to-Rust post included mpsc::channel(100) and recommended Tokio MPSC for ingest paths. We failed to follow that advice here. Async can schedule large numbers of in-flight tasks at low CPU cost, but every task still holds state. Without a limit somewhere in the path, bursts consume the available memory.
Advice after three years
- Instrument before you start. Every dashboard in this post existed before the migration, so we can support claims such as "600ms to 101ms."
- Budget for the learning curve. This took three years, and the first six months were the most expensive.
- Optimize for predictable latency. Customers notice the spikes more than the average.
- Treat the first Rust release as a baseline. Panels 8 and 9 compare Rust with Rust: Presence peak memory fell about 6x, and connection pooling removed most FCM variance.
- Work on durability alongside performance. We moved from our TCP protocol to gRPC and store and forward to improve correctness.
This is a follow-up to my 2024 Python-to-Rust post. I could not separate the benefits of Rust from the benefits of rewriting the architecture in that first migration. Panels 8 and 9 provide useful counterexamples: both compare Rust with Rust, and both show that data layout and connection reuse still mattered after the language migration.
Across these services, we now use less memory and about half the CPU per pod, and we see an order of magnitude fewer warnings on the route shown in panel 10. We also retired a wire protocol that only a few engineers understood. Those results justified the three-year migration for this workload.
