Hypothesis-driven CPU, memory, hang và resource debugging
Bắt đầu từ user impact/time/scope và resource hierarchy; restart là mitigation có thể xóa evidence, không phải root cause.
Debugging
CPU incident
Check workload/release, host vs cgroup utilization/quota, run queue/PSI and per-process/thread CPU. Sample stacks/JFR/perf; distinguish useful compute, GC/JIT, spin, lock contention, syscall/IRQ. Reduce load/rollback/bound concurrency then verify SLI and secondary queues.
Latency/hang
CPU low may mean blocked. Multiple thread dumps, futex/syscall latency, pool/queue depth, D-state I/O, network/downstream traces. Deadlock has stable cycle; saturation has queues/progress; livelock consumes CPU without progress.
Memory incident
Compare cgroup current/events, RSS/PSS composition, heap-after-GC, NMT, thread count/stacks, direct buffers/mmap and allocator behavior. Decide dump/profile safety. OOMKilled vs runtime OOME vs host pressure have different evidence/recovery.
FD/PID incident
Counts by type and rate, limits, allocation/open/connect failures; identify ownership/error path. PIDs can exhaust from threads/processes/zombies. Raise temporary limit only with monitoring while fixing leak/reaping/concurrency.
Signals/shutdown
Verify supervisor sends correct signal/tree, app stops intake/drains/closes before deadline, child processes forward/reap. Core dumps/diagnostic signals require storage/security. Kill -9 only last containment.
Postmortem evidence
Timeline metrics/stacks/counters/config/deploy, hypotheses and decisions; contributing capacity/guardrail/visibility gaps. Actions need owner/deadline/test such as load under quota, OOM drill, FD leak regression and SIGTERM game day.