EvidenceOps — Agentic Incident Command

ORGANIZATIONAL CONTEXT

Blast radius

CockroachDB · live

SERVERcampaigns.lumenpost.examplePlesk · UK

→

SITEcampaigns.lumenpost.exampleSite · campaign operations

→

SERVICEMailWizz Redis queue and mutex store

CAMPAIGNAffected campaign sends

OwnerLumen Post (synthetic demo)Policystandard SLAConsumers3 active

GROUNDED INVESTIGATION

Competing hypotheses

AI output · evidence validated

Adversarial review

Selected hypothesis: Redis memory pressure. Evidence strongly supports Redis memory pressure as the primary cause; verify live memory usage before remediation.

Evidence: EVD-003, EVD-004

STRUCTURED AGENT FLOW

Evidence-gated investigation

4 bounded steps

I
Map current blast radius
Three synthetic campaign sends have stopped advancing while other sends continue via SES; Redis is near its container memory ceiling and PCNTL mutex reports locks without verified ownership.
Evidence: EVD-001, EVD-003, EVD-002
grounded
H
Retrieve comparable resolution
Historical memory (EVD-004) shows a 89% similarity to a prior incident where Redis memory pressure caused campaign stalls; however, the current site differs and the incident remains open.
Evidence: EVD-004
grounded
P
Draft bounded remediation
First verify current Redis memory headroom and lock ownership; if pressure persists, increase container memory limit and adjust maxmemory settings before considering PCNTL mutex changes.
Evidence: EVD-003, EVD-004
grounded
R
Challenge action preconditions
Precondition for approval: independently confirm Redis memory usage and lock state; hypothesis "redis-memory-pressure" remains leading, others are secondary.
Evidence: EVD-003, EVD-004
challenged

89%match

PERSISTED HISTORICAL MATCH

40000000-0000-4000-8000-000000000001 · MailWizz campaigns stalled under Redis container memory pressure

4 stored resolutions searched

Retrieved by vector similarity from 4 stored resolutions. MailWizz campaigns stalled under Redis container memory pressure has an operator-confirmed resolution with intentionally date-only timing.

Root cause: Redis memory pressure near the 256 MB container limit caused intermittent instability. It appeared first as Predis connections dropping mid-request and later as PCNTL mutex and lock confusion, but both symptoms had the same underlying cause rather than representing separate bugs.
Winning action: Raised the Redis container memory limit from 256 MB to 512 MB and applied maxmemory 400mb, maxmemory-policy allkeys-lru, and save "" over SSH to remove RDB snapshot pressure. No manual lock clearing or PCNTL disabling was required.
Outcome: Campaigns 5006, 5002, and 5001 resumed and completed after the Redis memory increase on 2026-07-25.

Evidence: EVD-004

MATCH DELTA · 40000000-0000-4000-8000-000000000001

Shared context

Same redis service kind.
Both services run under Plesk.

Changed context

Different site: current campaigns.lumenpost.example; historical campaigns.riddimstream.com.
The historical record has no precise duration; the current incident remains open.

Do not transfer

Do not describe this as a full MailWizz outage; other campaigns kept succeeding.
Do not split the connection drops and mutex behavior into two root causes.
Do not claim FLUSHDB, manual lock clearing, or disabling PCNTL resolved the incident.
Do not claim a confirmed container OOM kill; the observed fact was severe memory pressure and intermittent Redis instability.

Reuse the diagnostic sequence from 40000000-0000-4000-8000-000000000001, not its resolution. Verify current Redis pressure and live lock ownership before proposing any capacity or lock change.

Investigation log

live

15:15
Incident evidence receivedOperator
18:30
Infrastructure context loadedCockroachDB catalog
18:30
AI investigation completed and evidence-validatedRecallOps AI agent
00:59
AI investigation completed and evidence-validatedEvidenceOps AI agent
01:16
AI investigation completed and evidence-validatedEvidenceOps AI agent

READ-ONLY COCKROACHDB CATALOG

Verified catalog context

connected

campaigns.lumenpost.example

campaigns.lumenpost.example · Plesk · UK

MailWizz Redis queue and mutex store · degraded_under_investigation

Via CockroachDB SQL · Observed 1:55:49 AM

✓

HUMAN DECISION GATE · SIMULATE

Verify Redis pressure and simulate a bounded memory-headroom change without clearing live locks.

Incident command

MailWizz campaigns intermittently stalled while Redis approached its container ceiling

Blast radius

Competing hypotheses

Evidence-gated investigation