Real-Time Events: How We Replaced 12 Polls per Minute with a Single Persistent Connection

A colleague resolves an alert. You are looking at the same alert on your screen. Nothing changes. You hit F5. Still there. You wait. You hit F5 again. Now it shows resolved.

In a monitoring platform, every second of stale data is a second where an analyst might make a wrong call. Start investigating something already closed. Miss a new critical alert because the list hasn’t refreshed. Duplicate work because two people see the same open alert and both reach for it.

We shipped SilentPulse with polling. It was the right call at the time, because polling is simple, debuggable, and works everywhere. But the tradeoffs caught up with us.

This post is a technical walkthrough of how we replaced polling with Server-Sent Events. It covers the database layer, the Go backend, the React frontend, and the infrastructure configuration. If you run SilentPulse behind nginx (most deployments do), the infrastructure section at the end matters for you.

The polling problem, quantified

The SilentPulse frontend had a useFlowStatus hook that called GET /api/v1/flows/status every 5 seconds. The dashboard overview, alert stats, and pipeline coverage widgets each had their own polling cycles at 10-15 second intervals.

Add it up for a single browser tab:

Endpoint	Interval	Requests/min
/api/v1/flows/status	5s	12
/api/v1/dashboard/overview	15s	4
/api/v1/alerts/stats	10s	6
/api/v1/dashboard/pipeline-coverage	15s	4
Total per tab		26

A SOC with 8 analysts, each with one or two dashboard tabs open, generates 200-400 GET requests per minute to the API. The vast majority return identical JSON because nothing changed in the last 5 seconds. The responses are small (1-4 KB), but the connection overhead, auth token validation, database queries, and JSON serialization happen every time.

The worse problem is latency. With a 5-second poll on the flow status endpoint, the average delay between an event and the analyst seeing it is 2.5 seconds. The worst case is 4.99 seconds. For alert stats at 10-second intervals, worst case is 9.99 seconds. Almost 10 seconds of stale state on a security monitoring dashboard.

We can’t fix this by polling faster. At 1-second intervals we’d generate 1,560 requests per minute from the same 8 analysts and still have up to 999ms of delay. The polling model hits a wall where reduced latency directly increases server load, and there’s no configuration that satisfies both.

Why SSE and not WebSockets

Three options were on the table: WebSockets, Server-Sent Events (SSE), and HTTP/2 Server Push.

WebSockets open a full-duplex TCP connection upgraded from HTTP. They’re the right choice when the client needs to send frequent messages to the server, like a chat application or collaborative editor. Our case is strictly unidirectional - the server pushes events, the browser never sends events back. WebSockets also require specific proxy configuration (Upgrade header pass-through, connection timeout tuning), and some corporate proxies and firewalls still interfere with the upgrade handshake.

HTTP/2 Server Push was designed for preemptively sending resources (CSS, JS) before the browser requests them. It was never intended for application-level event streaming and has been removed from Chrome since version 106.

SSE runs over a plain HTTP response with Content-Type: text/event-stream. The server writes events as text lines, the connection stays open, and the browser reads them as they arrive. It uses standard HTTP semantics, works through every proxy that handles HTTP, and degrades to a normal response if anything goes wrong. The protocol is defined in the WHATWG HTML Living Standard, section 9.2.

The SSE wire format is human-readable:

event: alert:created
data: {"id":"7a9c...","tenant_id":"a000...","status":"open","alert_type":"missing"}

event: alert:resolved
data: {"id":"7a9c...","status":"resolved"}

: heartbeat

Each event is a pair of event: and data: lines, followed by a blank line. Lines starting with : are comments, used for keepalive heartbeats. That’s the entire protocol.

One limitation: the browser’s native EventSource API does not support setting custom HTTP headers. It sends cookies but not Authorization: Bearer ... headers. Since SilentPulse authenticates API requests with JWT Bearer tokens (not cookies), we can’t use EventSource directly.

We solved this by implementing SSE over fetch() with the Streams API, the same approach we already use for the AI chat streaming feature. The pattern works in every modern browser and gives us full control over headers, error handling, and reconnection logic.

The backend: three new components

1. PostgreSQL NOTIFY (existing)

SilentPulse already had a PG trigger on the alerts table, added in migration 000003_alert_notify_trigger. When a new row is inserted into alerts, the trigger fires pg_notify('alerts_channel', ...) with the alert’s ID, tenant_id, flow_pulse_id, asset_id, status, alert_type, and started_at serialized as JSON.

CREATE OR REPLACE FUNCTION notify_alert_created()
RETURNS TRIGGER AS $$
BEGIN
    PERFORM pg_notify('alerts_channel', json_build_object(
        'id', NEW.id,
        'tenant_id', NEW.tenant_id,
        'flow_pulse_id', NEW.flow_pulse_id,
        'asset_id', NEW.asset_id,
        'status', NEW.status,
        'alert_type', NEW.alert_type,
        'started_at', NEW.started_at
    )::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

This trigger was built for the notification service (email, Slack, webhook dispatch). The notification service runs its own LISTEN alerts_channel connection and processes incoming alerts into outbound notifications. Nothing changes about that path. PostgreSQL NOTIFY delivers to all listeners on the channel, so adding a second listener for SSE is transparent.

One implementation note: pg_notify has an 8,000-byte payload limit. Our alert JSON payloads are around 300 bytes, well within the limit. If we ever need to push larger payloads, the approach would be to send only the alert ID via NOTIFY and have the listener fetch the full record.

2. SSE Hub (internal/sse/hub.go)

The hub is the central coordinator for SSE connections. It manages a registry of connected clients, receives events from the PG listener, and fans them out to the right browsers.

The core data structures:

type Event struct {
    Type     string      // "alert:created", "alert:resolved", etc.
    TenantID uuid.UUID   // target tenant; uuid.Nil = broadcast to all
    Data     []byte      // JSON payload
}

type Client struct {
    id       string
    tenantID uuid.UUID
    events   chan Event   // buffered, capacity 64
}

type Hub struct {
    register   chan *Client
    unregister chan *Client
    broadcast  chan Event  // buffered, capacity 256
    mu         sync.RWMutex
    clients    map[string]*Client
}

The hub runs as a single goroutine with a select loop over four channels: register, unregister, broadcast, and a 30-second heartbeat ticker. This is the same concurrency pattern used in Gorilla WebSocket’s chat example and in countless production Go services. One goroutine owns the map, no concurrent map access, no races.

The broadcast path is tenant-scoped. When an event arrives with a specific TenantID, the hub iterates over registered clients and sends only to clients whose tenantID matches. This is a hard security boundary: tenant A’s browser never receives tenant B’s alert events, even if they share the same API pod.

case e := <-h.broadcast:
    h.mu.RLock()
    for _, c := range h.clients {
        if e.TenantID != uuid.Nil && c.tenantID != e.TenantID {
            continue
        }
        select {
        case c.events <- e:
        default:
            log.Printf("[sse] client %s buffer full, dropping event", c.id)
        }
    }
    h.mu.RUnlock()

The select with default is critical. If a client’s 64-event buffer is full (the browser stopped reading, network stall, slow consumer), the event is dropped for that client rather than blocking the broadcast to all other clients. A slow reader can’t hold up the entire fan-out. The dropped event is logged, and the client will resync via its next poll interval (30 seconds) or by React Query’s staleTime logic.

The 30-second heartbeat sends an SSE comment (: heartbeat\n\n) to every client. This serves two purposes: it prevents proxies and load balancers from closing idle connections (nginx’s default proxy_read_timeout would kill a quiet connection after 60 seconds), and it lets the frontend detect a dead connection faster than TCP keepalive would.

3. PG Listener (internal/sse/listener.go)

The listener holds a dedicated PostgreSQL connection (acquired from the pool) and runs LISTEN alerts_channel. When a notification arrives, it deserializes the JSON payload and pushes an Event to the hub’s broadcast channel.

func (l *Listener) listen(ctx context.Context) error {
    conn, err := l.pool.Acquire(ctx)
    if err != nil {
        return err
    }
    defer conn.Release()

    if _, err := conn.Exec(ctx, "LISTEN alerts_channel"); err != nil {
        return err
    }

    for {
        notification, err := conn.Conn().WaitForNotification(ctx)
        if err != nil {
            return err
        }

        var payload AlertPayload
        if err := json.Unmarshal([]byte(notification.Payload), &payload); err != nil {
            log.Printf("[sse-listener] invalid payload: %v", err)
            continue
        }

        tenantID, err := uuid.Parse(payload.TenantID)
        if err != nil {
            continue
        }

        l.hub.Broadcast(Event{
            Type:     "alert:created",
            TenantID: tenantID,
            Data:     []byte(notification.Payload),
        })
    }
}

The outer Run() method wraps listen() in a reconnect loop with a 5-second backoff. If the PG connection drops (network partition, PostgreSQL restart, pod migration), the listener reconnects and re-issues LISTEN. The LISTEN subscription is per-connection in PostgreSQL, so a new connection always starts with zero subscriptions and must re-subscribe.

This is a separate PG connection from the one the notification service uses. Both receive all notifications on the channel. This is intentional and matches PostgreSQL’s documented behavior: pg_notify delivers to every session that has issued LISTEN on that channel.

4. HTTP Handler (internal/handler/events.go)

The SSE endpoint at GET /api/v1/events/stream follows the same auth middleware chain as every other protected route. The handler extracts the tenant ID from JWT claims, subscribes to the hub, sets the SSE response headers, and enters a streaming loop.

func (h *EventsHandler) Stream(w http.ResponseWriter, r *http.Request) {
    claims := middleware.GetClaims(r.Context())
    if claims == nil {
        http.Error(w, "unauthorized", http.StatusUnauthorized)
        return
    }

    flusher, ok := w.(http.Flusher)
    if !ok {
        http.Error(w, "streaming not supported", http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")
    w.Header().Set("X-Accel-Buffering", "no")

    client := h.hub.Subscribe(claims.TenantID)
    defer h.hub.Unsubscribe(client)

    fmt.Fprintf(w, "event: connected\ndata: {\"status\":\"ok\"}\n\n")
    flusher.Flush()

    for {
        select {
        case <-r.Context().Done():
            return
        case event, ok := <-client.Events():
            if !ok {
                return
            }
            if event.Type == "" {
                fmt.Fprintf(w, ": heartbeat\n\n")
            } else {
                fmt.Fprintf(w, "event: %s\ndata: %s\n\n", event.Type, event.Data)
            }
            flusher.Flush()
        }
    }
}

Four response headers matter:

• Content-Type: text/event-stream tells the browser (and any intermediary) that this is an SSE stream.
• Cache-Control: no-cache prevents caching proxies from buffering the response.
• Connection: keep-alive is technically redundant in HTTP/1.1 (persistent connections are the default) but some proxies need the explicit signal.
• X-Accel-Buffering: no tells nginx to disable response buffering for this connection. Without it, nginx holds the response in a buffer and delivers it in chunks, defeating the purpose of streaming.

The http.Flusher assertion is the same pattern used in the AI chat handler (ai_chat.go). Go’s net/http ResponseWriter implements Flusher when the underlying connection supports it, which is always the case for direct HTTP connections. The Flush() call after each event pushes the bytes to the TCP socket immediately.

The r.Context().Done() case fires when the client disconnects (browser tab closed, navigation, network loss). Go’s HTTP server cancels the request context when the client goes away, which cleanly exits the loop and triggers the deferred Unsubscribe.

Alert handler broadcast

When an analyst acknowledges or resolves an alert through the API (or through the MCP server), the alert handler broadcasts the status change to all connected browsers in the same tenant:

func (h *AlertHandler) Acknowledge(w http.ResponseWriter, r *http.Request) {
    // ... validation, DB update, escalation stop ...

    if h.hub != nil {
        data, _ := json.Marshal(map[string]string{
            "id": id.String(), "status": "acknowledged",
        })
        BroadcastAlertEvent(h.hub, "alert:acknowledged", claims.TenantID, data)
    }

    WriteData(w, http.StatusOK, map[string]string{"status": "acknowledged"})
}

The hub field is nil-safe. If SSE is somehow disabled (hub not initialized), the broadcast is a no-op and the API behaves exactly as before.

Wiring in main.go

The hub and listener start after the Redis connection, before the HTTP server:

sseHub := sse.NewHub()
hubCtx, hubCancel := context.WithCancel(ctx)
defer hubCancel()
go sseHub.Run(hubCtx)

sseListener := sse.NewListener(pool, sseHub)
go func() {
    if err := sseListener.Run(hubCtx); err != nil && err != context.Canceled {
        log.Printf("[api] SSE listener stopped: %v", err)
    }
}()

Two goroutines added to the API process. The hub goroutine handles the fan-out event loop. The listener goroutine holds a PG connection and feeds events into the hub. Both are tied to hubCtx and shut down cleanly when the API process exits.

The hub is passed to the router via InfraDeps.SSEHub, which wires it into the alert handler and registers the events stream route. The entire SSE subsystem is behind a nil check, so removing the hub from InfraDeps disables SSE completely without any other code changes.

The frontend: useEventSource hook

The React hook lives in src/frontend/src/hooks/use-event-source.ts and is called once in the dashboard layout component. It connects when the user has a valid access token and disconnects on unmount (logout, tab close).

The connection uses fetch() instead of EventSource:

const resp = await fetch("/api/v1/events/stream", {
    headers: { Authorization: `Bearer ${accessToken}` },
    signal: abortController.signal,
});

const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n");
    buffer = lines.pop() || "";

    for (const line of lines) {
        if (line.startsWith("event: ")) {
            eventType = line.slice(7).trim();
        } else if (line.startsWith("data: ")) {
            handleEvent(eventType, line.slice(6));
            eventType = "";
        }
    }
}

The { stream: true } option on TextDecoder.decode handles the case where a multi-byte UTF-8 character is split across two chunks. Without it, a chunk ending mid-character would produce a replacement character instead of waiting for the next chunk.

The SSE line parsing handles incomplete messages naturally. When a chunk arrives that doesn’t end with \n, the partial line stays in buffer and gets prepended to the next chunk. This covers the case where a single SSE event is split across TCP packets.

Reconnection strategy

When the connection drops (server restart, network hiccup, proxy timeout), the hook reconnects with exponential backoff:

const delay = Math.min(1000 * Math.pow(2, retriesRef.current), MAX_BACKOFF);
retriesRef.current++;
reconnectTimer = setTimeout(connect, delay);

The sequence: 1s, 2s, 4s, 8s, 16s, 30s, 30s, 30s… The counter resets to zero when the server sends the connected event, confirming a successful reconnection.

One exception: a 401 response (expired or revoked token) does not trigger a retry. Reconnecting with the same expired token would just produce another 401. The auth store handles token refresh through its own mechanism, and the hook will reconnect when a new accessToken value appears in the auth store.

Cache invalidation

Each event type maps to a set of React Query cache keys to invalidate:

function handleEvent(type: string, dataStr: string) {
    if (type === "alert:created") {
        queryClient.invalidateQueries({ queryKey: queryKeys.alerts.all });
        queryClient.invalidateQueries({ queryKey: queryKeys.alerts.stats() });
        queryClient.invalidateQueries({ queryKey: queryKeys.dashboard.all });
        toast.warning(`New alert: ${data.alert_type || "visibility gap detected"}`);
    }

    if (type === "alert:acknowledged" || type === "alert:resolved") {
        queryClient.invalidateQueries({ queryKey: queryKeys.alerts.all });
        queryClient.invalidateQueries({ queryKey: queryKeys.alerts.stats() });
        queryClient.invalidateQueries({ queryKey: queryKeys.dashboard.all });
    }

    if (type === "flow:status" || type === "worker:status") {
        queryClient.invalidateQueries({ queryKey: queryKeys.flows.status() });
        queryClient.invalidateQueries({ queryKey: queryKeys.dashboard.all });
    }
}

invalidateQueries doesn’t refetch immediately. It marks the cached data as stale. If the query is currently mounted (a component is rendering that data), React Query refetches it. If not, the stale data waits until the next time a component mounts with that query key.

This is the right behavior. An alert event invalidates queryKeys.dashboard.all, which covers the overview widget, the pipeline coverage canvas, and the operations flow diagram. If the analyst is on the Alerts page, only the alerts-related queries refetch. The dashboard queries stay stale until the analyst navigates back to the dashboard, at which point they refetch with fresh data.

The flow:status and worker:status handlers are wired but dormant. The backend doesn’t emit these event types yet because there are no PG triggers on the flow or worker tables. When those triggers ship in a future release, the frontend will handle them without any additional changes.

Polling as fallback

The useFlowStatus hook’s polling interval changed from 5 seconds to 30 seconds:

return useQuery({
    queryKey: queryKeys.flows.status(),
    queryFn: () => api.get<FlowStatusResponse>("/api/v1/flows/status"),
    refetchInterval: 30_000,
    refetchIntervalInBackground: false,
});

SSE handles the fast path. Polling at 30 seconds is a safety net: if the SSE connection is down, the dashboard still converges to the correct state within 30 seconds. This is a 6x reduction in background API traffic per tab (from 12 requests/min to 2 requests/min on the status endpoint alone).

Infrastructure: making long-lived connections work

SSE connections are long-lived by design. A browser opens GET /api/v1/events/stream when the dashboard loads and keeps it open for the entire session, which could be hours. Most reverse proxy defaults are tuned for short-lived request-response cycles and will kill SSE connections.

nginx ingress

Two annotations changed in the Helm values.yaml:

ingress:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"

proxy-read-timeout went from 180 seconds (3 minutes, which was already extended for the AI chat feature) to 3600 seconds (1 hour). This is the maximum time nginx will wait for the upstream to send data before closing the connection. The 30-second heartbeat ensures nginx always sees activity well within this window.

proxy-buffering: off disables nginx’s response buffering globally for this ingress. By default, nginx buffers the upstream response in memory (or on disk for large responses) and delivers it to the client in optimized chunks. This is good for normal HTTP responses but breaks SSE, because events sit in the buffer until the buffer fills or the connection closes. Disabling buffering makes nginx forward each Flush() from the Go server directly to the client’s TCP socket.

A more targeted approach would be to use a server-snippet annotation to disable buffering only for the /api/v1/events/stream path. But OrbStack’s nginx ingress controller (and some other distributions) block server-snippet for security reasons. The global proxy-buffering: off annotation is the simplest configuration that works everywhere. The performance impact on non-streaming responses is negligible; nginx still uses sendfile for static assets.

Other reverse proxies

If you run SilentPulse behind a different reverse proxy:

HAProxy. Set timeout server 3600s and no option httpclose on the backend. HAProxy does not buffer responses by default, so no additional configuration is needed for streaming.

Traefik. Traefik handles SSE out of the box. The only consideration is respondingTimeouts.readTimeout in the entry point configuration, which defaults to 0 (no timeout). If you’ve set a custom value, make sure it’s at least 3600 seconds.

Cloudflare. Cloudflare proxies SSE connections correctly but enforces a 100-second idle timeout on free and pro plans. The 30-second heartbeat keeps the connection active well within this limit.

AWS ALB. Set the idle timeout to 3600 seconds in the target group configuration. The default is 60 seconds, which would kill SSE connections between heartbeats.

What changed, file by file

File	Change
internal/sse/hub.go	New. Hub with tenant-scoped fan-out, heartbeat ticker, non-blocking sends.
internal/sse/listener.go	New. PG LISTEN on alerts_channel with reconnect loop.
internal/handler/events.go	New. GET /api/v1/events/stream SSE handler.
internal/handler/alert.go	Added hub field. Broadcasts alert:acknowledged and alert:resolved after successful DB update.
internal/handler/router.go	Added SSEHub to InfraDeps. Registers events stream route. Wires hub into alert handler.
cmd/api/main.go	Starts hub goroutine and PG listener after Redis connect. Passes hub to router via InfraDeps.
hooks/use-event-source.ts	New. SSE hook with fetch/ReadableStream, exponential backoff, query invalidation, toast notifications.
app/(dashboard)/layout.tsx	Calls useEventSource() in the dashboard layout.
hooks/queries/use-flow-status.ts	Polling interval 5s to 30s.
values.yaml	proxy-read-timeout: 3600, proxy-buffering: off.

Total: 4 new files, 6 modified files. Zero database migrations. Zero breaking changes. The SSE endpoint is additive, and polling still works if the browser doesn’t connect to the stream.

What fires now, and what’s coming

Three event types are live:

Event	Source	Trigger
alert:created	PG NOTIFY trigger	INSERT into alerts table (scheduler creates an alert)
alert:acknowledged	Alert handler	POST /api/v1/alerts/{id}/acknowledge succeeds
alert:resolved	Alert handler	POST /api/v1/alerts/{id}/resolve succeeds

The frontend hook also handles flow:status and worker:status events, but the backend doesn’t emit them yet. Adding PG triggers on the flow_pulses and worker_heartbeats tables is straightforward and will ship in a follow-up. When those triggers land, the frontend picks up the events without any code changes, because the invalidation logic is already there.

Longer term, the SSE hub is generic. Any part of the backend can broadcast an Event with a type, tenant ID, and JSON payload. Candidates on the roadmap:

• Configuration drift detection. The Dead Man’s Switch detects config hash changes every 5 minutes. Pushing that event to the browser would surface tampering detection in real time instead of on the next poll cycle.
• Compliance SLA breaches. When an asset group’s availability drops below its KPI target, the breach is currently written to the database and waits for the next dashboard refresh. An SSE event would surface it immediately.
• Worker health transitions. When a worker’s circuit breaker opens (integration point unreachable), the analyst sees it on the next 30-second poll. An SSE event would make that instant.

Deployment

Update your Helm chart:

helm upgrade silentpulse silentpulse/silentpulse

Or pull the latest Docker Compose:

docker compose pull && docker compose up -d

SSE is enabled by default. No environment variables to set, no feature flags to flip. The browser connects to the stream automatically after login.

If you see SSE connections dropping after 60 seconds, check your reverse proxy’s read timeout. The heartbeat fires every 30 seconds, so any timeout above 30 seconds should work, but we recommend 3600 seconds to handle cases where heartbeats are delayed by GC pauses or high load.

Verify the connection is working by opening your browser’s DevTools, going to the Network tab, and filtering by events/stream. You should see a single long-lived request with Content-Type: text/event-stream and periodic : heartbeat comments in the response.

---

The SSE implementation covers issues #360, #361, and #362 in the Real-Time Events milestone.