SilentPulse monitors whether your security telemetry is flowing. To do that, it needs credentials to your security stack, including Splunk tokens, Kafka certificates, Elasticsearch API keys, and SIEM service accounts. If someone compromises SilentPulse, they don’t just blind your monitoring. They get the keys to the castle.
We knew this from day one. But knowing it and proving it are different things. So we ran a formal white-box penetration test against our own code, and then ran it three more times until every finding was closed.
This post walks through the full process: what we tested, what we found, how we fixed it, and what broke along the way. No redactions, no marketing spin.
Why pentest a monitoring tool?
Most security products treat their own security as a checkbox. Ship a SOC 2 report, add “encrypted at rest” to the features page, move on.
We can’t afford that. SilentPulse sits at a unique intersection:
- It stores credentials to every security system it monitors (integration points)
- It’s multi-tenant, so a compromise in one tenant shouldn’t expose another
- It’s a security product, and if the tool that watches the watchers is itself vulnerable, the irony writes itself
- It has a Dead Man’s Switch, and if an attacker can silently disable monitoring, the organization doesn’t even know it’s blind
So we didn’t treat security as a feature. We treated it as a threat model.
The setup: white-box, 4 attacker profiles
We scoped the review around the “Security Hardening: Self-Protection” milestone, covering three features that handle the most sensitive parts of the system:
- Slow-drip degradation detection - Redis-backed rotation analysis that catches sophisticated evasion patterns where an attacker gradually reduces telemetry instead of cutting it off
- Dead Man’s Switch - SHA-256 config integrity monitoring that detects unauthorized changes to monitoring configurations
- Per-tenant credential protection - HKDF-derived AES-256-GCM encryption with per-tenant key isolation and credential access auditing
We tested against four attacker profiles:
| Attacker | Access | Goal |
|---|---|---|
| Insider with DB access | PostgreSQL credentials | Exfiltrate other tenants’ data, suppress monitoring |
| Compromised tenant | Valid JWT for one tenant | Pivot to other tenants, escalate privileges |
| Monitoring disabler | Network access to Redis | Silently disable detection without triggering alerts |
| Supply-chain attacker | Modify dependencies/config | Inject backdoor or weaken cryptographic protections |
Standards: OWASP ASVS 4.0, CWE/SANS Top 25, NIST SP 800-57 for key management.
Round 1: the initial review
The first pass found 14 findings: 0 Critical, 3 High, 5 Medium, 3 Low, 3 Informational.
The most significant ones are detailed below.
Timing oracle on API key comparison (High)
The health metrics API key was compared with Go’s != operator, a byte-by-byte comparison that returns early on first mismatch. Classic timing side-channel.
// Before: early-return comparison leaks key byte-by-byte
if providedKey == "" || providedKey != h.apiKey {
http.Error(w, "unauthorized", http.StatusUnauthorized)
}An attacker on the same network segment could recover the full API key in ~8,000 requests by measuring response time differences for each character position.
Fix:
// After: constant-time comparison — no timing leak regardless of mismatch position
providedKey := r.Header.Get("X-Health-API-Key")
if providedKey == "" || subtle.ConstantTimeCompare([]byte(providedKey), []byte(h.apiKey)) != 1 {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}One import, one function call. The entire fix is replacing != with subtle.ConstantTimeCompare(). The response time is now identical whether the attacker guesses zero bytes correctly or all of them.
Cross-tenant data exposure in health metrics (High)
The health endpoint queried alerts, worker heartbeats, and evaluation stats across all tenants without scoping. A single shared API key gave full visibility into every tenant’s operational state.
// Before: no tenant filter — returns counts across ALL tenants
h.pool.QueryRow(ctx,
`SELECT COUNT(*) FROM alerts WHERE status = 'open'`,
).Scan(&resp.AlertsOpen)Fix: Documented as intentional admin-only aggregate endpoint, replaced KEYS with SCAN (cursor-based iteration to avoid blocking Redis), added result caps, and added X-Data-Scope: system-wide header for transparency.
Per-tenant encryption was dead code in production (High)
The most painful finding. We built per-tenant HKDF key derivation and then never wired it into the worker binary. SetPerTenantEncryption() existed, was tested, and was never called.
What this means in practice: an admin follows the hardening guide, enables per-tenant encryption, runs the migration tool, restarts workers, and all monitoring stops. Workers can’t decrypt credentials encrypted with per-tenant keys because they only know the master key. SilentPulse goes blind by following its own documentation.
Fix: Wired SetPerTenantEncryption in the worker and notifications binaries:
// cmd/worker/main.go — this block was entirely absent before
if cfg.PerTenantEncryption && cfg.EncryptionKey != "" {
auditLogger := crypto.NewCredentialAuditLogger(pool)
collector.SetPerTenantEncryption(cfg.EncryptionKey, auditLogger)
log.Printf("[worker] per-tenant encryption enabled")
}Four lines. That’s the difference between “encrypted” and “not encrypted” for every credential the worker touches. This one came back to haunt us, though: the API entry point was still missing the same wiring, caught only in the Round 2 re-test.
Config hash only covered row IDs, not values (Medium)
The Dead Man’s Switch computed a SHA-256 hash of monitoring configurations but only hashed row IDs, not column values. An attacker with database access could modify any setting (widen time windows, create permanent exclusion windows, disable validation) without changing the hash.
// Before: only hashes IDs — content changes are invisible
rows, err := h.pool.Query(ctx,
fmt.Sprintf("SELECT id::text FROM %s WHERE ...", table),
)Fix: Two separate changes. First, the hash now includes all security-relevant columns, not just IDs but every field an attacker could modify to weaken monitoring:
// After: every security-relevant column is part of the hash
"flow_pulses": `SELECT id::text || '|' || flow_id::text || '|' ||
integration_point_id::text || '|' ||
integration_config::text || '|' ||
COALESCE(parser_type, '') || '|' || COALESCE(parser_config::text, '') || '|' ||
validation_type || '|' || validation_config::text || '|' ||
time_window::text || '|' || scheduler_interval::text || '|' ||
hostname_field || '|' || collector_mode || '|' ||
query_config::text || '|' || enabled::text
FROM flow_pulses WHERE deleted_at IS NULL ORDER BY id`,Second, the fmt.Sprintf pattern for table names was itself a SQL injection vector. We replaced it with pre-built queries in a map, validated at startup:
var validTableName = regexp.MustCompile(`^[a-z_]+$`)
func init() {
for table := range criticalTableQueries {
if !validTableName.MatchString(table) {
panic(fmt.Sprintf("integrity: invalid table name %q", table))
}
}
}Migration tool wasn’t transactional (Medium)
The credential re-encryption tool updated rows one at a time without a transaction. If the pod was killed mid-migration (OOM, preemption), half the credentials would be at encryption version 2 (per-tenant key) and half at version 1 (master key). No advisory lock meant concurrent runs could double-encrypt.
Fix:
// After: entire migration is atomic — concurrent runs block, failures roll back
tx, err := conn.Begin(ctx)
if err != nil {
return 0, 0, 0, fmt.Errorf("begin tx: %w", err)
}
defer tx.Rollback(ctx)
// Advisory lock prevents concurrent migration runs from double-encrypting
if _, err := tx.Exec(ctx,
`SELECT pg_advisory_xact_lock(hashtext('migrate-credentials'))`); err != nil {
return 0, 0, 0, fmt.Errorf("advisory lock: %w", err)
}Now if a pod is killed mid-migration, the transaction rolls back and no credentials are left in an inconsistent state. The advisory lock releases automatically when the transaction ends.
Round 2: the re-test that found more bugs
After fixing all 14 findings across three prioritized commits, we ran a full re-test. 13 of 14 passed clean. One was partial, and we found 3 new issues.
The most significant: the flow_pulses query in the config hasher was missing 6 critical columns. The fix for SP-SEC-006 included content hashing for most tables, but flow_pulses was missing flow_id, integration_point_id, integration_config, parser_type, parser_config, and query_config. An attacker could modify what a pulse monitors, how it parses data, and where it connects, all without triggering the integrity checker.
We also found a potential deadlock: emitSystemEvent() (which writes to the database) was called under mu.Lock() in the integrity verifier. Under load, this would block all goroutines waiting for the mutex while the DB write completed.
// Fixed: capture data under lock, release, THEN perform I/O
baselineForEvent = h.baselineHash
h.mu.Unlock()
if firstDrift {
h.emitSystemEvent(ctx, "config_drift", "critical",
"Configuration hash drift detected",
map[string]string{"baseline": baselineForEvent, "current": hash})
}The pattern is textbook: copy what you need from shared state, release the lock, then do the slow work. But under the pressure of “fix the hash coverage finding,” we introduced a new concurrency issue in the fix itself.
Round 3: the schema drift problem
Round 3 expanded scope beyond code to deployment parity. We discovered that 6 SQL migration files were missing from the Helm deployment target. They existed in docker-compose but were never copied to Helm.
This meant Helm-deployed environments were missing:
- The alerts entity FK re-pointing (pointing at wrong table)
- Alert type/details columns for EPS degradation
- MITRE dashboard indexes
- The credential_access_log table (no audit trail)
- The system_events table (no drift detection)
Multiple features would fail silently in Helm environments. The schema parity check was sobering: docker-compose had 31 files, Helm had 25.
Fix: Byte-identical sync of all migration files between both deployment targets. Verified with diff.
Round 4: the broadest scope
The fourth round was the most comprehensive: full schema audit, per-tenant encryption lifecycle traced across all services, JWT middleware review, route authentication audit.
Three more medium-severity findings:
-
notification_channelslackedencryption_version. Unlike integration points, channels had no way to record which encryption scheme was used. The dispatcher relied on trial-decrypt as its primary mechanism (try per-tenant, fall back to master), adding latency and log noise for every legacy channel. Adding the version column demoted trial-decrypt to a last-resort fallback and made the system auditable. You can now query exactly which channels use per-tenant keys. -
The migration tool only migrated integration points. Notification channels encrypted with the master key were never migrated. If the master key were rotated, channels would become undecryptable.
-
Helm schema used
DROP INDEXinstead ofALTER TABLE DROP CONSTRAINT. PostgreSQL doesn’t allow dropping an index that backs a UNIQUE constraint. The old constraint would survive, breaking soft-delete email reuse.
The final count
After 4 rounds:
| Round | Findings | Fixed |
|---|---|---|
| Initial review | 14 (3H / 5M / 3L / 3I) | All 14 |
| Re-test #1 | 3 new (1H / 1M / 1L) | All 3 |
| Re-test #2 | 2 new (1H / 1L) | All 2 |
| Re-test #3 | 3 new (1H / 1M / 1L) | All 3 |
| Re-test #4 | 5 new (0H / 3M / 2L) + 1 info | All 5 |
| Total | 27 unique findings | 27/27 fixed |
Final schema parity: 32/32 SQL files identical between Helm and docker-compose. Zero compilation errors. All 7 call sites for crypto.NewTenantAES() verified as using identical parameters across API, worker, notifications, and migration tool.
By the numbers
The hardening effort spanned 22 days (February 1-22, 2026) and produced 32 commits across the codebase.
| Metric | Value |
|---|---|
| Total commits | 32 |
| Files changed | 99 unique files |
| Lines added | +4,904 |
| Lines deleted | -752 |
| Net new code | +4,152 lines |
| Timeline | 22 days |
The breakdown by file type:
| Type | Files |
|---|---|
| Go production code | 65 |
| SQL migrations | 12 |
| TypeScript/TSX (frontend) | 5 |
| Helm/K8s/Docker Compose YAML | 5 |
| Documentation | 4 |
| Go test files | 4 |
| Other (Makefile, scripts, config) | 4 |
Go files account for 70% of all changes. This was overwhelmingly a backend security effort.
Three new Go packages were created from scratch: internal/integrity/ (Dead Man’s Switch and config hash verification), internal/crypto/ (per-tenant credential encryption with access auditing), and internal/safeclient/ (centralized SSRF protection). A standalone CLI tool (cmd/migrate-credentials/) was built for encryption key rotation with full transactional safety.
The most telling stat: router.go and cmd/api/main.go were each modified in 7 of the 32 commits. Every round of findings touched the same critical entry points, the router where authentication is enforced and the main binary where security features are wired. The config integrity hasher (internal/integrity/hasher.go) was modified 5 times across rounds, each time because re-testing revealed columns or tables that the previous fix had missed.
The entire initial pentest, all 14 findings fixed, re-tested, and verified, was completed in a single day. The subsequent 11 days of post-pentest hardening added SSRF protection, JWT token revocation, API rate limiting, and encryption key rotation support. These weren’t findings from the pentest itself; they were attack surfaces we identified because the pentest forced us to think like an attacker.
What we learned
1. Dead code is a security risk
The per-tenant encryption finding (SP-SEC-003) wasn’t a logic bug. The code was correct, tested, and reviewed. It just wasn’t called. In a monolith, a single integration test would catch this. In a multi-binary system (API, worker, scheduler, notifications), each binary’s main.go is its own integration surface. A feature wired in one binary but not another is worse than no feature at all, because it creates a false sense of protection.
Our takeaway: every security feature now has an integration test that exercises the full path from configuration to decryption across all binaries.
2. Schema drift is a silent killer
Six missing SQL files between deployment targets. Not six different files, six absent files. Both targets pass go build, both start successfully, both serve API requests. The missing tables only surface when specific features are used: credential audit fails silently (no table), drift detection fails silently (no table), degradation alerts have wrong types (missing migration).
Our takeaway: CI now runs a byte-level diff between all deployment targets. Any schema divergence fails the build.
3. Fixing a finding can introduce a finding
The config hash fix (SP-SEC-006) was correct in intent but incomplete in execution. Adding content hashing to 3 of 4 tables left one table with the original vulnerability. The re-test caught it, but it illustrates why re-testing matters. A fix that’s 75% complete is still 100% vulnerable for the remaining 25%.
4. Trial-decrypt is a code smell
The notification dispatcher’s trial-decrypt pattern (try per-tenant key, catch error, fall back to master key) worked but masked a design gap: the absence of encryption_version on the channel table. Adding the version column demoted trial-decrypt from the primary path to a last-resort fallback and made the system auditable. You can now query exactly which channels are protected by per-tenant keys instead of guessing at runtime.
5. Timing oracles are everywhere
if key != expectedKey is one of the most common patterns in Go. It’s also a timing side-channel in every authentication context. We found it in one place; we then audited every comparison in the codebase. crypto/subtle.ConstantTimeCompare should be the default, not the exception.
This wasn’t a one-off
The 4-round pentest described in this post wasn’t a special event. It’s how we work.
Every commit to our main branch triggers a security review pass. Not a checkbox scan, but an actual review against the same attacker profiles, the same standards, the same “would this survive an insider with DB access?” mindset. Every release goes through the full cycle: threat model, code audit, re-test until clean.
We do this because security is what we do. SilentPulse exists to tell you when your defenses have gaps. If we can’t hold ourselves to that standard, we have no business holding anyone else to it.
This process keeps teaching us the same uncomfortable lesson: everyone makes mistakes. We built per-tenant encryption, wrote tests for it, reviewed the code, and still shipped it without wiring it into production. We fixed a config hash vulnerability in 3 out of 4 tables and called it done. We maintained two deployment targets and let them drift apart for weeks without noticing.
These aren’t rookie mistakes. They’re the kind of mistakes that experienced engineers make when they’re moving fast, when the code compiles, when the tests pass, when everything looks right. The only reliable defense isn’t being smarter or more careful. It’s assuming you missed something and going back to check.
That’s why we re-test. That’s why we don’t stop at round 1.
Why we’re publishing this
We could have written a blog post that says “SilentPulse is secure” and linked to a SOC 2 badge. Instead, we’re showing you the 27 things we got wrong.
Most vendors treat security findings as liabilities, something to minimize, bury in a PDF, disclose only when forced. We think that’s backwards. If you’re trusting a product with your Splunk tokens and Kafka certificates, you deserve to know how that product handles its own vulnerabilities. Not “we take security seriously,” but these are the 27 specific things we found, this is exactly how we fixed them, and this is the process that caught them.
The right question when evaluating a security product isn’t “do they have vulnerabilities?” Every software does. The right question is “do they find them before you do, and what happens next?”
We found 27. We fixed 27. We re-tested 4 times until nothing was left. And now you know exactly what they were.
The full technical report, with CWE classifications, severity ratings, code evidence, and commit references for every fix, is available to customers and prospects upon request. We’ll keep publishing what we find, because the moment we stop looking is the moment we stop deserving your trust.
SilentPulse: See when security goes silent. Even when it’s our own security under the microscope.