The case of the square stream
For weeks the 24/7 stream played inside a small box of black bars — bars above, bars beside, on every viewer, on every phone that refused to rotate. The encoder was provably sending perfect widescreen the entire time. This is the story of how we measured our way through five wrong theories to the one component nobody could see.
The symptom that survived every fix
The report was simple: the stream sits in a little rectangle with black
all around it. The first fixes were the sensible ones — a capture
that really had been square once (a stale 1080×1080
size in the deployment), a device-scale bug that had genuinely blurred
and boxed the picture before. Both were found, both were fixed, both
were verified fixed on the box. The bars didn't move.
The number that made the feeling concrete came from asking YouTube itself what it was serving. Every live stream gets a ladder of renditions — the qualities the player switches between — and a healthy 1080p ladder reads 1920×1080, 1280×720, and so on down. Ours read:
144×144 · 240×240 · 360×360 · 480×480 · 720×720 · 1080×1080
Square at every rung — width equal to height all the way down, as if the transcoder knew how tall the video was but had never learned how wide. The 16:9 game sat letterboxed inside YouTube's square canvas, and the square canvas sat pillarboxed inside everyone's 16:9 player. Bars on all four sides, none of them ours.
Proving the encoder innocent
The stream box records its own output to disk — the same encode it sends to YouTube, cut into ten-second files for the highlight reel. That gave us something rare in a bug hunt: the exact bytes on trial, available for forensics. We put them under every instrument we had. The frames measured full-bleed — the game touched all four edges, no baked bars. The H.264 stream headers were textbook: 1920 wide, 1080 tall, square pixels, standard cropping. There was no way to write a cleaner widescreen bitstream. And still, live on the channel at that very minute: 1080×1080.
The wrong theories, in order
What followed was several hours of murder-mystery. Maybe the deploy never actually reached production — it had. Maybe a forgotten second encoder somewhere was feeding the channel with an old stream key — a real hazard, since minting a new key on YouTube does not invalidate the old ones, and we chased that ghost hard enough to stop the real encoder and watch whether the channel died. (It did — eventually. Live thumbnails lag a few minutes behind the pixels, and a "fresh" frame that had actually been rendered before the shutdown sent us hunting zombies for another hour.) Maybe YouTube pins a broadcast's ladder forever once minted — so we rotated keys, ended broadcasts, let the ingest session go stone cold. Maybe creating the broadcast before the video connects made YouTube derive the ladder too early — so we taught the auto-live supervisor to wait until the ingest measured active before creating one. Each theory explained everything right up until the next experiment killed it. Fresh key, fresh broadcast, cold session, guarded creation order: square, square, square, square.
The experiment that ended it
The breakthrough was an A/B test with no moving parts left: take one of
the recorded ten-second files — the convicted-innocent bytes
— and replay it straight at the same stream key with a plain
ffmpeg -c copy push. No re-encode, no capture, no Chrome,
nothing but a different path to the wire. The ladder came back
1920×1080. The same bytes, byte for byte, transcoded square from
the streaming container and widescreen from the replay.
That left exactly one structural difference. The container fanned its encode out through ffmpeg's tee muxer — one encoder feeding YouTube and the disk recorder from a single session. The replay pushed a plain, direct FLV output. Something about the tee-wrapped RTMP stream makes YouTube's transcoder mis-derive the video's width and build its ladder square; the direct output, carrying the identical bitstream, transcodes correctly. The recorder never showed it because the recorder's tee branch writes a different container format entirely. The one component we couldn't inspect — the wire between our muxer and their ingest — was the one that was lying.
The fix deleted the tee: every destination now gets its own encoder session and its own direct connection. That costs a second NVENC session for the recorder, which the GPU shrugs at — and it bought the channel this, first try, on the very next broadcast:
256×144 · 426×240 · 640×360 · 854×480 · 1280×720 · 1920×1080
Full screen, every rung the right shape, and phones finally rotate to landscape — the square ingest had been quietly convincing YouTube's mobile player the video wasn't landscape at all.
What the hunt left behind
The scariest part of this bug wasn't the bars — it was how politely everything failed. The tee muxer's per-output "ignore failures" flag meant a dead destination vanished silently. A crashed encoder looked identical to a healthy restart. So the stream box now audits itself out loud: it test-fires the GPU encoder at startup and says plainly when the stream cannot go out; it verifies its color pipeline before trusting it; it counts rapid crash loops and names which process died; and it watches its own recording folder, because ten-second files that stop appearing are proof the encoder has stopped producing for everyone. Silence is no longer an acceptable failure mode.
One confession, in the tradition of this dev log: the very watchdog we
added to end silent failures shipped with a bug that made it cry wolf
— a misspelled find flag meant it reported "no fresh
recordings" forever, even while fresh recordings sat right there on the
disk. The tripwire tripped itself. It got caught within the hour by the
same habit that eventually caught the tee: don't trust the report,
measure the artifact. The files were forty seconds old; the watchdog
said they didn't exist; exactly one of those could be true.