For four years the nightly import job was the part of our system that everyone was afraid of. It read about forty gigabytes of partner feeds, normalised them, and wrote the result into the warehouse, and on a good night it finished before the analysts arrived.
On a bad night, which by last autumn meant roughly one night in three, it ran out of memory somewhere in the middle, restarted from scratch, and finished at lunchtime. Nobody could say exactly why, because the memory profile changed with every partner, every feed format, and every library upgrade.
Measuring before cutting
We started with a week of measurement rather than a rewrite. The flame graphs told a consistent story: most of the time, and almost all of the memory, went into building intermediate strings that were thrown away a few microseconds later, usually while trimming, lower-casing, or splitting fields.
The fix for that is not a language, of course. But the new importer had to be written by two people in a quarter, and we wanted the compiler to stop us from reintroducing the same allocations, so we picked a language where borrowing is the default and copying is visible.
The new hot path
The core loop now borrows every field from the input buffer and only allocates when a value actually has to change:
fn normalise<'a>(field: &'a str) -> Cow<'a, str> {
let trimmed = field.trim();
if trimmed.bytes().any(|b| b.is_ascii_uppercase()) {
Cow::Owned(trimmed.to_ascii_lowercase())
} else {
Cow::Borrowed(trimmed)
}
}
That one function, applied consistently, removed about eighty percent of the allocations. The rest came from reading feeds in fixed-size chunks instead of whole files, and from reusing a single output buffer per worker.
Results
- Peak memory went from 14 GB to 900 MB.
- The median run takes 41 minutes instead of three and a half hours.
- We have not had a single out-of-memory restart since January.
The best performance work is the work that makes the next bug impossible, not just the current one fast.
If you are thinking about a rewrite of your own, measure first, write down what you expect to change, and keep the old system running in parallel for longer than feels necessary. Ours ran side by side for six weeks, and the diffs it produced caught three real bugs in the new code.
Great write-up. We had almost exactly the same experience with our billing exports, except that in our case the culprit was a logging library that formatted every record twice, even when the log level was off.
How did you handle partners that send Latin-1 instead of UTF-8? That was always the part of our importer that made the borrowing approach impossible, because every field had to be converted before we could even look at it.
We detect the encoding once per file and transcode the whole chunk up front, so the field-level code only ever sees UTF-8. It costs one copy per chunk instead of one per field, which turned out to be fine.