procfs, counters, sampling profilers và eBPF
Tool output là sampled/cumulative evidence với scope/overhead; luôn ghi timestamp, namespace, cgroup, PID/TID mapping và workload.
Observability
procfs and basic tools
/proc/PID stat/status/maps/smaps/fd/task/io, /proc/stat/meminfo/vmstat/pressure/net expose kernel views. ps/top/pidstat read/derive snapshots; units/averaging differ. ss/nstat/ip inspect sockets/network; vmstat/iostat distinguish CPU, paging and device queues.
Tracing system calls
strace shows syscall arguments/results/timing and attach/follow threads; useful for blocked/open/connect/permissions but overhead and sensitive data risk. Trace focused syscalls/PIDs/duration; ptrace permissions/Yama/container capabilities apply.
perf
perf stat counters, record sampling stacks, sched events and lock tools reveal CPU/cache/scheduling. Symbols/frame pointers/DWARF/JIT maps affect attribution. Sampling frequency/stack unwinding overhead and skid mean approximate, not exact accounting.
eBPF
Attach programs to tracepoints/kprobes/uprobes/perf/network with verifier and maps; tools like bpftrace/BCC/libbpf. Powerful fleet observability but kernel/version/BTF/privilege compatibility, overhead/cardinality and data governance matter. Prefer stable tracepoints when available.
Pressure and saturation
PSI reports time tasks stalled on CPU/memory/I/O; cgroup PSI localizes workload. Combine utilization, run queue, throttling, faults/reclaim, lock/futex, syscall latency and app SLI. One high metric is correlation, not root cause.
Evidence safety
Stacks, cmdline, environment, file paths, packets and syscall buffers can reveal secrets/PII. Least privilege, filters, short duration, encrypted access/retention and redaction. Production attach/profiling requires rollback/overhead threshold.