DB/HTTP pools, queues, DNS/TLS và downstream latency
P99 cao với CPU thấp thường là waiting: pool acquisition, locks, queues, I/O, throttling hoặc downstream.
Queueing map
Load balancer/backlog → Tomcat threads → app executor → DB/HTTP client pool → downstream queue/locks. Find first queue whose wait/depth grows before latency. Increasing all pools moves bottleneck and increases in-flight.
Database
Pool active/idle/pending/acquire timeout; query latency/locks/connections; transaction duration; DB CPU/IO/cache. Pool size bounded by DB capacity; long external call inside transaction holds connection. Trace spans separate acquire vs execute.
HTTP/network
DNS resolution/cache, connect, TLS handshake, pool acquire, time-to-first-byte, body download. Connection reuse/keepalive, stale connections, ephemeral ports/NAT, proxy timeouts. Trace/client metrics must expose phases.
Queues/cache
Arrival/service rate, oldest age, retries/DLQ, partition skew; cache hit/miss/eviction/load latency, hot key and stampede. Queue/cache hides dependency until burst/expiry then cliff.
Hypothesis workflow
Define impact/time/scope; compare healthy dimensions; correlate release/config/traffic; RED at entry then dependency spans, USE resources, logs. Change one mitigation, verify SLI and watch secondary saturation.