Supplement · Tomcat

8 failure labs external và embedded Tomcat

Mỗi lab lưu version, config, workload, raw evidence, baseline → failure → mitigation → after và cleanup. “Thiết kế xong” không đồng nghĩa “đã thực thi”.

8 labs

Lab 01 · Request path và effective configuration

Mục tiêu: deploy servlet nhỏ vào external Tomcat và embedded Boot; correlate access log, Valve, Filter, Servlet, thread và native TID. Failure: sai Host, Context hoặc filter dispatch. Pass: sequence diagram khớp timestamps và effective config.

Lab 02 · Connector saturation

Giữ workload constant, thay max connections, threads, queue và simulated DB pool. Đo p50, p99, reject, busy threads, queue, CPU và DB acquire. Pass: tìm đúng queue đầu tiên tăng và overload policy recover sau spike.

Lab 03 · Async lifecycle và disconnect

Tạo sync và async endpoints có timeout, client cancel và double-complete race. Lấy traces cùng thread dumps. Pass: không leak request/context, timeout hoặc cancel dừng downstream và graceful drain thấy async work.

Lab 04 · Classloader redeploy leak

WAR cố ý tạo thread, ThreadLocal và registered driver; redeploy lặp. Đo old loader, loaded classes, heap dominator rồi sửa cleanup. Pass: loader cũ collectible và regression thất bại khi reintroduce leak.

Lab 05 · Parallel deployment và sessions

Deploy hai version cùng context, tạo session payload và rolling traffic; kill node hoặc version. Failure: incompatible session và stale route. Pass: documented migration/invalidation policy cùng failover evidence.

Lab 06 · TLS và forwarded-header spoofing

Đặt reverse proxy trước Tomcat; test direct backend access, forged Forwarded/XFF, redirect, secure cookie, SNI, protocol và rotation. Pass: client không spoof scheme/IP; trusted chain và certificate gates tự động.

Lab 07 · Security và management audit

Inventory ports, deployed apps, Manager, JMX, AJP, file permissions, error pages và access logs. Chạy negative requests oversized/malformed trong safe bounds. Pass: unused surface removed, limits verified runtime, secrets và PII không lộ.

Lab 08 · SIGTERM game day

Long request, async work, keep-alive và background consumer đang chạy khi remove readiness rồi gửi TERM. Pass: stop intake, committed-work policy đúng, pools đóng, exit trước deadline và SLI recovery verified.

Execution evidence · 2026-08-23

Environment and provenance: Windows host, Java/Javac 21.0.9; Apache Tomcat 10.1.44 downloaded from the Apache archive into the OS temporary directory. No Tomcat binary, dependency cache, generated class, or runtime instance is part of the canonical lesson.

Lab	Status	Evidence completed	Still pending
01 Request path/config	PARTIAL EXECUTED	External deployment, NIO connector, request thread, effective fixture config, access/startup logs.	Embedded comparison, Valve/Filter ordering, native TID correlation.
02 Connector saturation	PENDING	Fixture connector bounded with `maxThreads=4`, `acceptCount=2`.	Controlled workload, p50/p99/reject/queue/CPU evidence, recovery after spike.
03 Async lifecycle	PARTIAL EXECUTED	Async timeout callback ran; active count returned to zero.	Client/downstream cancellation, redispatch, double-complete race, thread dumps.
04 Classloader leak	PENDING	Re-deployable external webapp fixture exists.	Intentional leak, repeated redeploy, old-loader retention/collection and heap evidence.
05 Deployment/session	PARTIAL EXECUTED	Cookie-backed session persisted across two requests; HttpOnly and context path observed.	Parallel versions, incompatible payload, routing/node failure and migration policy.
06 TLS/proxy trust	PARTIAL EXECUTED	Direct forged XFF did not change `remoteAddr` or scheme without a trust Valve.	Reverse proxy, trusted-hop configuration, TLS/SNI/protocol/rotation, secure-cookie gates.
07 Security audit	PARTIAL EXECUTED	Minimal instance exposed one HTTP connector and only the lab webapp; runtime logs/config retained.	Management/JMX/AJP/file audit and bounded malformed/oversized negative requests.
08 Shutdown game day	PARTIAL EXECUTED	Authenticated shutdown paused connector and allowed one in-flight sync request to complete.	OS SIGTERM, readiness removal, keep-alive, async/background work, deadline and recovery SLI.

Interpretation limit: 0/8 labs are complete against their full specification; 6/8 have reproducible partial execution and 2/8 remain pending. Partial evidence is not counted as completion. These observations do not prove connector saturation, classloader collectability, clustered failover, TLS rotation, full management hardening, production proxy trust, or complete orchestrated graceful shutdown.

Executed observations retained from the evidence log

External deployment: the fixture ran under context /lab on the configured NIO connector; logs captured startup, pause, stop and destroy.
Async timeout: timeout callback executed and fixture active accounting returned to zero; cancellation/race paths remain pending.
Session: one cookie jar preserved the same session ID while visits advanced 1 → 2; this does not prove Secure-cookie or cluster compatibility.
Forwarded-header baseline: direct client XFF remained a header; remoteAddr stayed loopback and scheme stayed HTTP without a trust Valve.
Controlled shutdown: one 1500 ms sync request completed after shutdown initiation; this is not equivalent to a full SIGTERM/readiness/orchestrator game day.