The failure mode is familiar. A wall of green tiles in the control room. Encoder CPU at 40 percent. Uplink utilization nominal. Bridge port count well under capacity. Then a moderator says into the program feed, “I think we lost the remote speaker,” and the desk operator looks up to find that the attendee-visible stream has been dead for ninety seconds. The dashboard was fine. The show was not.
This is not a tooling problem. It is a routing problem. The dashboard was answering the wrong question. It was reporting on the health of components the operator could see, not on the experience the attendee was having. The fix is not more tiles. The fix is separating the signal that pages a human from the signal that helps that human diagnose the problem once they are awake.
Two monitoring modes, two jobs
Google’s SRE book draws a distinction that maps cleanly onto a show network. Black-box monitoring is symptom-oriented and represents active, not predicted, problems: “The system isn’t working correctly, right now.” White-box monitoring depends on inspecting the internals of the system and allows detection of imminent problems and failures masked by retries.
For a live event, the black box is the attendee’s experience. Can they join? Is there audio? Is there video? Is the video moving? The white box is everything behind that: RTP sequence numbers, RTCP receiver reports, encoder buffer depth, ingest bitrate, egress segment availability. Both matter. They do not matter equally at 2 a.m. when the pager goes off.
The SRE guidance is explicit that for paging, black-box monitoring forces discipline to only nag a human when a problem is both already ongoing and contributing to real symptoms. That discipline is exactly what a show-network desk needs. A dashboard full of green internals while the stream is dead is the predictable result of inverting the two layers: paging on white-box signals that look healthy, and diagnosing with black-box signals that nobody is watching.
The four golden signals on a show network
The SRE book names four golden signals: latency, traffic, errors, and saturation. If you can only measure four things about your user-facing system, measure those. The mapping onto a live event is an editorial framing choice, not a claim from any source, but it holds up under pressure.
Latency is glass-to-glass delay and join time. For a remote speaker, it is the time between the speaker’s mouth moving and the attendee’s speaker emitting sound. For an attendee, it is the time between clicking the join link and seeing the first frame. Latency increases are often a leading indicator of saturation, which is why this signal belongs on the paging surface even when nothing has failed yet.
Traffic is concurrent sessions and bitrate. For a distribution path, it is the number of active viewers and the aggregate egress rate. For a conference bridge, it is the number of active RTP flows. Traffic alone rarely pages. Traffic dropping to zero does.
Errors are failed joins, dropped frames, and rejected authentication attempts. The SRE book notes that errors can be explicit, implicit, or by policy, and that catching failures at the load balancer does a decent job of catching completely failed requests while only end-to-end tests can detect that you are serving the wrong content. A show network has the same problem: a 200 response from the join endpoint does not mean the attendee heard anything.
Saturation is encoder CPU, uplink headroom, and bridge port capacity. This is the signal most likely to be green right up until it is not. Measuring the 99th percentile response time over a small window such as one minute can give a very early signal of saturation, and the same logic applies to encoder frame times and uplink queue depth.
What RTCP actually gives you
RTP itself does not provide any mechanism to ensure timely delivery or provide other quality-of-service guarantees; it relies on lower-layer services to do so. That is the first thing to internalize. The protocol is not going to tell you the stream is fine. It is going to give you the raw material to figure that out yourself.
RTCP exists to monitor the quality of service and to convey information about the participants in an ongoing session. The sequence number in the RTP header can be used by the receiver to estimate how many packets are being lost. The reception report indicates how well the current speaker is being received and may be used to control adaptive encodings. These are the standard tools. They are coarse.
RFC 8888 specifies a finer-grained feedback packet for congestion control, motivated by the fact that an effective RTP congestion control algorithm requires more fine-grained feedback on packet loss, timing, and ECN marks than is provided by the standard RTCP Sender Report and Receiver Report packets. Each 16-bit packet metric block in that feedback packet carries a received bit, a two-bit ECN mark, and a thirteen-bit arrival time offset. That is per-packet arrival timing, not a summary statistic. For a monitoring desk, this is the difference between knowing that jitter is elevated and knowing which packets arrived late and by how much.
Two operational consequences follow. First, the feedback packet is a diagnostic instrument, not a paging instrument. It tells you why the stream is degraded. It does not tell you whether the attendee noticed. Second, the feedback packet can be lost. RTCP packets do not contain a sequence number, so loss of feedback packets has to be inferred based on the time since the last feedback packet. If multiple consecutive congestion control feedback packets are lost, the media sender should rapidly reduce its sending rate as this likely indicates a path failure.
The alert class nobody configures
Most monitoring desks alert on “feedback says bad.” Packet loss above a threshold. Jitter above a threshold. That is one alert class. There is a second, and it is the one that catches the failure mode in the title.
“No feedback received” is a distinct condition from “feedback received and it says the path is bad.” The RFC 8888 guidance treats loss of feedback as its own signal requiring its own response, precisely because the absence of data is not the same as good data. A desk that only alerts on the content of feedback will sit quietly while the feedback channel itself has gone dark.
The practical implementation is a timer. For each monitored RTP flow, track the time since the last RTCP packet. If that time exceeds the expected reporting interval by a margin, raise a white-box alert. This is a ticket or a dashboard signal in most cases. It becomes a page when the flow in question is on the critical path for a live session and the timer has expired by enough that the desk cannot distinguish “feedback lost” from “path dead.”
Routing, not thresholds
The instinct when building an alerting system is to pick numbers. Page at this packet loss percentage. Ticket at that jitter value. The SRE book warns against this in a specific way: rules that generate alerts for humans should be simple to understand and represent a clear failure. It also notes that when pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes ignoring a real page that is masked by the noise.
The more useful structure is routing. Every signal gets one of three destinations: page, ticket, or dashboard. The routing decision is the threshold. A signal pages when it represents an ongoing, attendee-visible failure. A signal tickets when it represents a condition that will become attendee-visible if left alone. A signal lives on a dashboard when it is diagnostic context for a human who is already investigating something else.
This structure has a property that single-number thresholds do not: it is testable before doors open. You can walk the routing table with the show team and ask, for each row, “if this fires during the keynote, who wakes up?” If the answer is “nobody” for a signal that represents a dead stream, the routing is wrong. If the answer is “everybody” for a signal that represents a transient jitter spike, the routing is also wrong.
The SRE book also notes that Google has trended toward simpler and faster monitoring systems with better tools for post hoc analysis, and avoids magic systems that try to learn thresholds or automatically detect causality. That is a useful constraint for a show network. Learned thresholds are hard to explain to a show caller at 9:58 a.m. A static routing table is not.
Pre-show, live, and post-event windows
The same signal can have different routing in different windows. During load-in, a failed join test is a ticket. During a live session, a failed join rate above the baseline is a page. After the event, a failed join is a data-integrity question for the content pipeline, not an operational emergency.
The cleanest way to handle this is to make the window an explicit input to the routing table rather than an implicit assumption. A signal row that says “page” without saying “during live session” is a signal that will page during load-in, which is exactly the kind of noise the SRE guidance warns degrades response to real pages.
There is a related handoff problem worth naming here. The transition from the room to the chat and back is where hybrid events tend to fail, and the monitoring desk needs to know which side of that handoff it is watching at any given moment. The room-to-chat handoff is a useful reference for the specific failure patterns at that boundary.
A worksheet for the monitoring desk operator
The artifact below is designed to be copied into a runbook and filled in before load-in. It deliberately avoids learned thresholds and magic numbers. The threshold basis column is where the show team writes down why a given routing decision was made, so that the decision can be revisited after the event without re-litigating it from memory.
SHOW-NETWORK MONITORING DESK — ALERT ROUTING WORKSHEET
Role: Show-Network Monitoring Desk Operator
Event: ______________________ Window: [ ] Load-in [ ] Live [ ] Post-event
Columns:
Signal | Mode | Source | Threshold basis | Routing | Owner
Rows to fill in (one per monitored signal):
1. Attendee join success rate
Mode: black-box
Source: join endpoint telemetry
Threshold basis: baseline from load-in test
Routing: page (live) / ticket (load-in)
Owner: ______________
2. First-frame render time (p50, p99)
Mode: black-box
Source: client-side beacon
Threshold basis: p99 over 1-minute window
Routing: page (live) / dashboard (load-in)
Owner: ______________
3. Audio continuity (dropout count per minute)
Mode: black-box
Source: client-side beacon
Threshold basis: any dropout during live session
Routing: page
Owner: ______________
4. Encoder frame time (p99)
Mode: white-box
Source: encoder stats endpoint
Threshold basis: leading indicator of saturation
Routing: ticket / dashboard
Owner: ______________
5. Ingest bitrate deviation from target
Mode: white-box
Source: ingest receiver
Threshold basis: sustained deviation, not transient
Routing: ticket
Owner: ______________
6. RTCP receiver report loss estimate
Mode: white-box
Source: RTCP RR
Threshold basis: trend, not single sample
Routing: dashboard
Owner: ______________
7. RTCP congestion control feedback arrival
Mode: white-box
Source: RTCP feedback packet (RFC 8888)
Threshold basis: time since last packet vs. expected interval
Routing: ticket / page (critical path only)
Owner: ______________
8. Egress segment availability
Mode: black-box
Source: CDN origin fetch log
Threshold basis: any gap during live session
Routing: page
Owner: ______________
9. Concurrent session count
Mode: black-box
Source: distribution telemetry
Threshold basis: drop to zero during live session
Routing: page
Owner: ______________
10. Bridge port utilization
Mode: white-box
Source: bridge stats
Threshold basis: headroom, not absolute
Routing: ticket
Owner: ______________
Post-incident artifact (fill in after any page):
- Timeline: first symptom, first alert, first human action, resolution
- Metric snapshot: the four golden signals at T-5min, T-0, T+5min
- Reversible fix applied: ______________________
- Rollback procedure: ______________________
- Routing table change proposed: ______________________
The worksheet is not a monitoring system. It is a forcing function for the conversation that has to happen before the monitoring system is configured. The SRE book notes that SRE teams carefully avoid any situation that requires someone to stare at a screen to watch for problems. A routing table that has been argued through in advance is how a show-network desk avoids becoming that situation.
What to do with this
Pick one live session from the last event. Walk the routing table with the person who was on the desk. For each row, ask whether the routing matched what actually happened. The rows where the answer is no are the rows that need to change before the next load-in. The rows where the answer is yes are the rows that can be left alone.
The dashboard will still be there. It will still be full of green tiles. The difference is that the tiles will be answering the diagnostic question, and the pager will be answering the attendee question. Those are not the same question, and a desk that treats them as one will keep discovering the stream is dead the same way it did last time.