People usually reach for gRPC because they’ve heard it’s faster. It can be, but the size of the bytes is the least interesting thing it changes about your system. What changes more is who owns the contract, how failures show up on your dashboards, and how your load balancer behaves. Teams find those out after they’ve migrated, which is the expensive time to find out.
So let’s do the bytes first, get them out of the way, and then talk about the parts that’ll actually show up in your incident reviews.
The bytes, honestly
Protobuf’s wire format is simple enough to write by hand. Each field is a tag (field number and wire type packed into a varint) followed by either a varint or a length and some bytes. Here’s an encoder for an order with line items, next to the JSON for the same data:
"""Protobuf's wire format, by hand, next to the JSON for the same order."""
import gzip, json
def varint(n: int) -> bytes:
out = bytearray()
while True:
byte, n = n & 0x7F, n >> 7
out.append(byte | (0x80 if n else 0))
if not n:
return bytes(out)
def field(num: int, value) -> bytes:
if isinstance(value, int): # wire type 0: varint
return varint(num << 3 | 0) + varint(value)
data = value.encode() if isinstance(value, str) else value
return varint(num << 3 | 2) + varint(len(data)) + data # type 2: length-delimited
def line_item(sku, qty, price):
return field(1, sku) + field(2, qty) + field(3, price)
def order(o):
msg = field(1, o["id"]) + field(2, o["customer_id"])
for it in o["items"]:
msg += field(3, line_item(it["sku"], it["quantity"], it["price_cents"]))
return msg + field(4, o["total_cents"]) + field(5, o["status"])
o = {"id": "ord_8f3a2c", "customer_id": 1048576,
"items": [{"sku": "KB-104", "quantity": 1, "price_cents": 12900},
{"sku": "USB-C-2M", "quantity": 3, "price_cents": 1499}],
"total_cents": 17397, "status": 2}
pb, js = order(o), json.dumps(o, separators=(",", ":")).encode()
print(f"JSON {len(js):4d} bytes gzip {len(gzip.compress(js)):4d}")
print(f"protobuf {len(pb):4d} bytes gzip {len(gzip.compress(pb)):4d}")
print("first bytes:", pb[:14].hex(" "))
# 1,000 different orders in one response: JSON repeats every field name
batch = [dict(o, id=f"ord_{i:06x}", customer_id=1_048_576 + i * 7919,
total_cents=17397 + i * 13) for i in range(1000)]
many_js = json.dumps(batch, separators=(",", ":")).encode()
many_pb = b"".join(field(1, order(x)) for x in batch)
print(f"x1000 JSON {len(many_js):6d} B, gzip {len(gzip.compress(many_js)):5d} B | "
f"protobuf {len(many_pb):6d} B, gzip {len(gzip.compress(many_pb)):5d} B")JSON 183 bytes gzip 151
protobuf 54 bytes gzip 74
first bytes: 0a 0a 6f 72 64 5f 38 66 33 61 32 63 10 80
x1000 JSON 184001 B, gzip 11493 B | protobuf 56867 B, gzip 7763 BI checked the hand-rolled bytes against Google’s protobuf library with a matching schema, and they’re identical.
Read the first bytes and you can see where the savings come from. 0a is field 1, wire type 2. The next 0a is a length of 10. Then ord_8f3a2c. Then 10 is field 2, a varint. The name "customer_id" never goes over the wire, because both sides already agreed that field 2 is the customer.
Now look at the gzip column. A single small message gets bigger under gzip, since the compression header costs more than it saves. For a thousand orders, the 3.2× raw advantage shrinks to about 1.5× once both sides are compressed, because gzip is very good at removing repeated field names. If your REST API already serves gzip or brotli, the bandwidth argument for gRPC is much weaker than the raw numbers suggest.
The bigger performance win is usually CPU. Parsing JSON means scanning text, handling escapes and building strings for every key, while protobuf decoding mostly reads varints and copies byte ranges. At a few hundred requests a second nobody notices. At a few hundred thousand, across a fleet, it shows up on the bill.
The contract is the real product
With REST and JSON, the contract is whatever the server happens to return, plus an OpenAPI document if you’re disciplined, which may or may not match. With gRPC the .proto file is the contract. Clients and servers in every language are generated from it, and a breaking change is something you can see in a diff of one file.
That brings a rule you need to internalize: field numbers are forever. The number is the only thing on the wire. Rename a field and nothing breaks. Reuse a deleted field’s number for something new and old clients will happily decode your new data as the old meaning. Mark retired numbers reserved, and add the check to CI. I went into this in schema evolution; protobuf makes the rule simple to follow and catastrophic to break.
The format also makes forward compatibility nearly free. Every field carries its own length or terminator, so a reader can skip fields it doesn’t know:
# A reader built against v1 (fields 1-5) still parses a v2 message, because
# every field carries its own length or terminator and can be skipped.
def read(buf):
i, known = 0, {}
while i < len(buf):
key, shift = 0, 0
while True:
b = buf[i]; i += 1; key |= (b & 0x7F) << shift; shift += 7
if b < 0x80: break
num, wtype = key >> 3, key & 7
val, shift = 0, 0
while True:
b = buf[i]; i += 1; val |= (b & 0x7F) << shift; shift += 7
if b < 0x80: break
if wtype == 2:
val, i = buf[i:i + val], i + val
if num <= 5:
known[num] = val
else:
print(f" skipped unknown field {num} (wire type {wtype})")
return known
newer = pb + field(9, "gift-wrap") # v2 adds field 9
print("v1 reader kept fields:", sorted(read(newer))) skipped unknown field 9 (wire type 2)
v1 reader kept fields: [1, 2, 3, 4, 5]That’s why you can deploy a new server before upgrading every client. The skip logic is in the format, not in somebody’s careful handling of optional JSON keys.
Failures look different, and your dashboards may not notice
This is the one I’d tell every team before they migrate.
gRPC calls normally come back with HTTP status 200. The actual result is in a grpc-status trailer, using gRPC’s own codes: UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED, PERMISSION_DENIED and so on. If your SLO dashboards and alerts count 5xx responses at the load balancer, a fully broken gRPC service can look perfectly healthy. Every call fails, and the graph says 100% 200s.
Before cutover, make sure every layer that computes an error rate (load balancer, mesh, APM, SLO tooling) reads grpc-status. Then deliberately fail a call in staging and watch it show up as an error.
The codes themselves are an improvement once you’re reading them. UNAVAILABLE means “try again” and INVALID_ARGUMENT means “don’t,” which is a cleaner retry signal than guessing from a 500. Retry policy can key on them directly; the general rules from retries and backoff still apply.
Deadlines travel with the call
A gRPC client sets a deadline, and it travels with the request as a grpc-timeout header. A server that calls other services can pass the remaining budget on, so a request that has already used 800 ms of a 1-second budget doesn’t start a 5-second downstream call. That’s the deadline propagation pattern, built into the framework rather than bolted on by convention. It only helps if you actually set deadlines. A gRPC call without one can wait forever, and plenty of client libraries don’t set one by default.
The Kubernetes load-balancing trap
gRPC uses HTTP/2, which keeps a connection open and multiplexes many requests over it. A normal Kubernetes ClusterIP Service is balanced by kube-proxy at the connection level. Put the two together and each client sticks to whichever pod it connected to first.
Then the autoscaler adds five pods, and they sit idle while the original three stay overloaded, because the existing clients never open new connections. It looks like the HPA isn’t working. It’s working fine. The traffic just can’t move.
Three fixes, roughly in order of how often I’d reach for them:
- Balance per request at L7. A service mesh or an Envoy-based gateway balances individual RPCs rather than connections. If you’re on Gateway API,
GRPCRouteexists for this. - Client-side balancing. Point clients at a headless Service so DNS returns every pod IP, and use the gRPC client’s round-robin policy. It works, but now every client library’s configuration is part of your load-balancing story.
- Cap connection age. Have servers close connections after a maximum age so clients reconnect and spread out over time. Crude, but it needs no new infrastructure.
Browsers, curl, and caches
Browsers can’t speak native gRPC, because JavaScript doesn’t get the low-level HTTP/2 control and trailer access it needs. You need gRPC-Web with a translating proxy, or a protocol like Connect that runs over plain HTTP.
curl stops being your debugging tool, too. You need grpcurl plus either server reflection or the .proto files, which matters at 3am when the person on call has never touched the service.
And unary gRPC calls are POSTs, so the HTTP caching you get for free with GET (CDNs, browser caches, ETag revalidation) is gone. For read-heavy public data that’s a real loss.
How I’d choose
gRPC for internal service-to-service traffic where you control both ends, have several languages, care about streaming, or run enough volume that serialization CPU shows up. Budget for the load-balancing and observability work up front.
REST with JSON for public APIs, browser clients, partner integrations, webhooks, and anything where cacheability and curl-ability matter more than raw efficiency. My older REST vs. GraphQL comparison covers that side.
Both is a perfectly good answer. Keep gRPC internally and expose a JSON façade at the edge, generated from the same .proto through gRPC-JSON transcoding, so there’s still only one contract.
Whichever you pick, run the two checks that catch most migration pain before it reaches production. Fail a call on purpose and confirm your error-rate graph moves. Then scale the backend while a long-lived client is connected and confirm the new pods get traffic.
Related: REST vs. GraphQL APIs – how to choose? · Schema Evolution: The Field You Removed Is Still in Someone’s Queue · Timeouts and Deadline Propagation: The Budget Nobody Sets
Comments