The Skill Nobody Puts on Their Resume
Ask a senior engineer what separates good engineers from great ones, and you’ll hear the usual answers: systems thinking, debugging instincts, knowing when not to build. Compression almost never comes up. That’s a mistake.
Not compression in the narrow sense of gzip and zstd, though those matter. Compression as a way of thinking: the ability to find the minimum representation of something without losing what makes it meaningful. This applies to data formats, API design, code architecture, written communication, and how you explain a problem to a skeptical executive. Once you start seeing it this way, you notice that the best engineers are doing it constantly, and the mediocre ones almost never are.
What Compression Actually Is
At its technical core, compression exploits redundancy. If your data has patterns, a good compressor finds them and encodes them more efficiently. Shannon’s information theory gives this a rigorous foundation: the theoretical minimum bits needed to represent a message is determined by its entropy, essentially how unpredictable it is. A string of random bytes has high entropy and compresses poorly. A log file full of repeated timestamps and status codes has low entropy and compresses extremely well.
This is why the compression ratio you achieve on a file tells you something real about its structure. A 10:1 ratio on a text file means 90% of that file was redundancy. A 1.1:1 ratio on a JPEG means most of the information is already unique and the encoder has already done the hard work.
But the interesting part isn’t the math. It’s the tradeoffs. Lossless compression preserves everything and accepts a ceiling on how much you can shrink. Lossy compression breaks through that ceiling by deciding some information isn’t worth keeping. JPEG throws away high-frequency spatial detail your eye mostly ignores. MP3 discards audio frequencies that are masked by louder sounds nearby. These aren’t bugs. They’re editorial decisions encoded into algorithms, based on a model of what humans actually perceive.
Every compression scheme contains an implicit theory about what matters.
Where This Plays Out in Real Systems
The practical implications show up everywhere in production systems, often in ways teams don’t frame as compression problems.
Protocol Buffers versus JSON is a compression argument. JSON is human-readable and self-describing: field names travel with every message. Protobuf assigns integer field IDs and relies on a shared schema, so field names don’t need to travel at all. For a message with ten fields, JSON might repeat the same field name thousands of times per second across your services. Protobuf encodes that name exactly once, in the schema file. The bandwidth savings are often 3x to 10x on typical payloads, but more importantly, Protobuf forces you to think explicitly about your schema as a contract rather than letting it drift organically with your JSON keys.
Columnar storage formats like Parquet make a similar argument about access patterns. Row-oriented storage (the default for most databases) is great when you need all the fields for a given record. But analytical queries often touch only a few columns across millions of rows. Columnar storage puts all values for a given column contiguous in memory, which enables much better compression (similar values compress better together) and means you only read the columns you actually need. This is why BigQuery and Snowflake achieve query speeds that would be impossible with row-oriented storage at the same scale.
HTTP/2’s header compression (HPACK) is a solution to a problem most developers don’t think about: the headers on every HTTP request are often larger than the request body. A typical set of cookies, user-agent strings, and authorization headers can run to several kilobytes, sent fresh with every request. HPACK maintains a shared header table between client and server, so repeated headers get referenced by index rather than retransmitted. The savings compound as a session continues.
The Design Skill Hidden Inside the Technical One
Here’s where compression as a discipline gets genuinely interesting, and where it stops being purely about bytes.
When you design an API, you’re making compression decisions constantly. What belongs in the URL versus the body? What should be implicit in the context versus explicit in every call? A well-designed API encodes shared assumptions between client and server, which is exactly what a compression scheme does. A poorly designed API forces the caller to repeat context that the server already knows, which is exactly what an inefficient encoding does.
Consider the difference between an endpoint that returns everything about a user versus one that lets callers specify which fields they need (GraphQL takes this to one extreme, sparse fieldsets in JSON:API take a more conservative approach). The design question is: what does the caller usually need, and how much do you penalize callers who need something different? This is a compression tradeoff dressed in API design clothing.
Code itself follows the same logic. A codebase that repeats business logic in multiple places hasn’t found the right abstraction yet, which is another way of saying it hasn’t been compressed. The DRY principle (Don’t Repeat Yourself) is essentially a call for better compression. But over-abstraction is lossy compression gone wrong: you’ve discarded information (the specific context of each use case) that you actually needed.
The Communication Angle
Engineers who understand compression are almost always better written communicators, and the connection is not coincidental.
Good technical writing compresses without losing fidelity. A precise sentence is one where removing any word loses meaning. A bloated design document is one with high redundancy: the same point restated three ways, caveats on top of caveats, context the reader already has explained again. The editing process is decompression followed by recompression: unpack what you meant, then find the minimum representation that communicates it accurately.
This is why engineers who truly understand a system can explain it concisely, and engineers who only partially understand it tend toward verbose explanations. Verbosity is often a symptom of incomplete mental compression. You haven’t yet found the underlying structure that makes the pieces cohere, so you have to enumerate them instead.
The latency numbers engineers quote are interesting through this lens: part of their value is compression. Instead of explaining cache hierarchy and memory bandwidth from first principles, you share a table of numbers and an engineer can reconstruct the reasoning. The numbers are a compressed representation of a much larger body of physical and empirical knowledge.
The Tradeoff You Can’t Avoid
Every compression decision involves a bet about the future. Lossless compression bets that you’ll need all the original information. Lossy compression bets that you won’t. Shared schemas bet that both sides will stay in sync. Aggressive caching bets that the underlying data won’t change faster than the cache expires.
These bets fail in predictable ways. Video codecs designed for natural footage produce terrible artifacts on screen-recorded content (sharp edges, text, solid colors) because the perceptual model underlying the lossy compression doesn’t match the signal. Protobuf schema drift between services produces silent data corruption when field IDs get reused. Cache invalidation bugs are wrong bets about how often data changes.
The engineers who handle this best aren’t the ones who avoid lossy tradeoffs. They’re the ones who are explicit about what assumptions their compression scheme makes, and who build in ways to detect when those assumptions stop holding. This is why good systems have checksums, schema versioning, and cache-busting mechanisms: not because compression is dangerous, but because unexamined compression is.
What This Means
Compression thinking shows up across almost everything a software engineer does, from choosing a wire format to designing an abstraction to writing a postmortem. The underlying skill is the same: find the minimum representation that preserves what matters, be explicit about what you’re discarding, and understand the assumptions your encoding bakes in.
The engineers who do this well tend to write cleaner APIs, build systems that stay maintainable longer, and communicate more clearly about technical decisions. They’ve internalized that redundancy is a cost, that every abstraction is a bet, and that the right level of compression depends entirely on what you’re optimizing for and who’s doing the decoding.
If you’re not thinking about what your system is implicitly assuming and discarding, someone else will discover it for you, usually in production.
