# E. Trace / log / dump audit — CompressSpec `feature/delta-compaction-v3` (dcd1ce8)

- 문제: 엔진 안에 88개 log/debug env + 12개 output-dir env 가 흩어져 있고, 대부분 hot path 안에서 `os.environ.get` 으로 매번 읽힘. 무엇이 default 로 켜져 있고 무엇이 비용을 내는지 코드 근거 없이 판단 불가.
- 현상: default(디버그 env 미설정) 에서도 CPU verify 쪽 compaction 경로가 request×round 마다 3~4 줄 `logger.info` 를 무조건 emit 하고, GPU scheduler 는 recompaction 마다 1줄 emit. 나머지는 µs 급 env read 뿐. GPU sync 를 유발하는 default-ON 은 없음.
- 원인: 파일·줄 단위 근거는 §1. (1) `vllm_cpu/.../gpu_model_runner.py:3983,4107,5233,5387,4758/4766/4898` 무조건 info, (2) `scheduler_recompact.py:297,416` 무조건 info, (3) `_diag()`/`emit()`/`fsm_log()` 가 호출마다 env 를 읽음 (6개 사본).
- 결과: §2 KEEP/MERGE/REMOVE 표, §3 단일 output root 설계, §4 최소 스위치 세트(`CS_TRACE=0/1/2/3` + `BREAKDOWN` + `CS_DUMP`) 제안. 측정하지 않은 수치는 "추정" 으로 표시하고 확인할 줄을 적음.

Evidence tree: `/home2/qwg724/001_CompressSpec` (branch `feature/delta-compaction-v3`, HEAD `dcd1ce8`). Co-author tree `/home2/ysshin/CompressSpec` checked only where noted. Login node = read/grep only; **no run-time measurement was possible**, so every "cost" below is a code-derived estimate with the line to check.

Default assumed throughout: no debug env set, `BREAKDOWN_DIR` unset, `COMPRESS_SPEC_DEBUG` unset, `print_log=False` (`config/connector_config.py:150` → `cpu_verification_print_log_opt=False`, so every `if self.log_print_opt:` line in `cpu_verify_engine.py` is OFF), `upload_pend_print_opt=False` (`connector_config.py:98`).

---

## 1. (a) Default-ON cost, ranked by estimated per-step impact

Freq units: **step** = one GPU engine step; **round** = one verify cycle of one request; **B** = batch size.

| # | What runs with NO debug env | file:line | Freq | GPU sync | Est. cost / note |
|---|---|---|---|---|---|
| 1 | `logger.info(f"max_selected_token_cnts …")` — **f-string built before any level check**, also `os.environ.get('COMPACT_TOKEN_HEAD_AGG')` inside the f-string | `vllm_cpu/vllm/v1/worker/gpu_model_runner.py:3983` (token selector), `:4107` (block selector, default), `:3805` (legacy) | 1 per request per compaction round | no (CPU engine) | ~16 lines/round at B=16 into `cpu_server.log`, inside the `/verify` critical path. Unmeasured; µs-scale format + one `write()` each. |
| 2 | `logger.info("[COMPACT_THRESHOLD_CHECK] …", os.getpid())` | same file `:5233-5237` | 1 per request per compaction round | no | config echo; should be once per process. |
| 3 | `_should_compact` verdict: exactly one of `[gate-skip-reason] code=1` (`:4758`), `[gate] reject → compact (forced)` (`:4766`), `[gate] sufficiency=` (`:4898`) fires per call | `:4758`, `:4766`, `:4898` | 1 per request per round | no | With `COMPACT_GATE_SCORE=1` default and CPU LSE present `:4898` fires every round. |
| 4 | `logger.info("[PARTITION][CPU] alloc …")` | `:5387-5391` | 1 per request per copy-path compaction | no | block method (default) mostly takes the zero-copy path `:5359`; still unconditional. |
| 5 | `[COMPSPEC RECOMPACT] … (landing)` and `[COMPSPEC RECOMPACT] …` | `vllm/vllm/cpu_verify/compaction_upload/scheduler_recompact.py:297`, `:416` | 1 per recompaction plan | no | GPU-side scheduler process, unconditional. |
| 6 | Verify server `Verify done | req=… accepted=…` | `verify/cpu_verify_server_score.py:462-469` | 1 per `/verify` | no | The per-round accept stat scripts grep; KEEP but 4th copy of the same info (§3.1). `Mismatch at draft_idx` `:266` per reject. |
| 7 | `[COMPSPEC TRACKER_INIT]` | `vllm/vllm/cpu_verify/channels/temp_buffer.py:283` | once per request | no | fine |
| 8 | Per-call `os.environ.get` in the CPU compaction path: `:5226,5228,5232,5243,5244,5295,5351,5363,5422,5447`; selector `:3725-3737,3828-3895,3929-3930,4024,4075-4081`; gate `:4743,4751,4752,4756,4884,4892,4708(×2)`; `_diag()` ×12 `vllm_cpu/vllm/compaction/runner_diff_upload.py:52,88,119,122,181,197,227,230,266,269,328,358`, ×2 `index_map.py:251,351`, ×4 `cpu_verify_engine.py:2265,4343,4768,4779`, ×3 `keep_map_reader.py:171,205,220` | ~35–45 reads (+`int()` in `_diag`) per request per round | no | ≈0.5–1 µs each → 20–40 µs/req/round. Negligible vs verify latency but is what makes the code read "heavy". **Six identical `_diag()` copies**: `keep_map_reader.py:215`, `cpu_verify_engine.py:4871`, `gpu_recompact.py:88`, `scheduler_recompact.py:425`, `index_map.py:79(_dlog)+409`, `runner_diff_upload.py:402`. |
| 9 | GPU-side per-step env reads: `LOG_DELTA_SELECTION` (`vllm/vllm/v1/worker/gpu_model_runner.py:3885`), `KV_DUMP_NCT` (`:3934`), `SCORE_DUMP_DIR` (`:3483` via `:4077`; `offloading_attn_bp_connector.py:326,401`), `ATTN_BP_DEBUG_FILL` (`:336`), `LS_PEAK_PROBE` (`scheduler.py:1914`), `COMPSPEC_PREEMPT_OFFLOAD` (`:1319`), `LOG_DELTA_SELECTION` per request (`:1782`), `CS_STOP_LOG` per request (`sched/utils.py:93`), `ATTN_KV_DUMP_DIR` ×2 per step (`debug_utils.py:98` via `scheduler.py:1760,1819`) | ~8 + 2·B per step | no | ≈10–30 µs/step. |
| 10 | `fsm_log()` reads `FSM_RUN_DIR` per call; 10 sites (`_tracker_registry.py` ×7, `_verify_dispatcher.py:130,133`, `cpu_verify_engine.py` ×1) | ~6 per request per round | no | `_fsm_log.py:16`; negligible. |
| 11 | `compaction_trace.enabled()/emit()` env read per call: `cpu_tracker.py:212` (per reload), `cpu_verify_engine.py:1857` (`HELD_TRACE` per request per after-verify) | per event | no | `compaction_trace.py:10-18`; negligible. |
| 12 | Breakdown disabled path: `BreakdownLogger.get()` returns `None` via `_disabled` (`breakdown_logger.py:137-138`); wrappers return `_NULL` (`:579-590`). 21 sites GPU runner, 7 sched, 7 mixin, 3 dispatcher, 5 tracker | ~40 no-op calls/step | no | ≈10 µs/step. OK. |
| 13 | `_ktime()` returns shared `nullcontext` when disabled (`_cpu_kernel_timer.py:75-77`; `ENABLED` at import `:17-19`); 10 sites `llama.py`, 7 `qwen3.py`, 10 `vllm_cpu gpu_model_runner.py` | ~10×L per CPU forward | no | ≈30 µs/forward at L=32. OK. |
| 14 | `record_function_or_nullcontext`, 26 sites in `vllm_cpu gpu_model_runner.py`; fast path `nullcontext()` (`vllm_cpu/vllm/v1/utils.py:387-396`) | per step | no | becomes torch `record_function` only if `VLLM_CUSTOM_SCOPES_FOR_PROFILING=1`, which `cpu_breakdown_setting()` sets **only** when `BREAKDOWN_KERNEL_PROFILE_MODE=1 && INLINE=0 && BREAKDOWN_DIR` (`breakdown_debugger/breakdown_util.py:147-158`). Not default-ON. |
| 15 | `KEEPMAP_SLOT_LOG` default `"1"` | `vllm/vllm/cpu_verify/keep_map_reader.py:157` | only inside `if not _same:` (`:154`), warn once per request | no | **Zero cost in the normal path** — a warn-once drift gate, not an emitter. |
| 16 | `_trace.trace("adv", …, f"backpressure …")` f-string built before the gate | `cpu_verify_engine.py:2366` | per backpressure event (rare) | no | the other 7 `_trace.trace` sites are guarded by `if _trace.on():` (`:1382,1427,1606,1885,3630`). |
| — | GPU-sync-for-logging in the default path | — | — | **none found** | every logging `.cpu()/.item()/.tolist()/synchronize()` on the GPU side is env-gated: `KV_DUMP_NCT` (`gpu_model_runner.py:867-911`), `LOG_DELTA_SELECTION=2` (`:915-960`), `SCORE_DUMP_DIR` (`attn_bp_connector.py:327`), `COMPACT_DIFF_STAT` (`gpu_recompact.py:22-23,79`), `RECOMPACT_DIAG` (`:53-54`), `RECOMPACT_FP_LOG` (`:32`), `COMPRESSION_SPEC_DEBUG=true` (`gpu_model_runner.py:2035-2038`, whole-tensor repr → implicit sync). |

**Bottom line (a):** the only default-ON emitters that scale with batch × rounds are #1–#6 (≈4 lines per request per verify round on the CPU side, +1 per recompaction on the GPU scheduler, +1 on the verify server). Everything else is µs-scale env noise. Whether #1–#5 measurably slow the verify RPC is **unmeasured**; instrument `vllm_cpu/.../gpu_model_runner.py:3983`, `:4107`, `:5233`, `:4898`.

Not in the inventory: `CPU_FWD_READ_HASH` (`vllm_cpu/vllm/v1/attention/backends/cpu_attn.py:442`, per-forward env read) and `COMPRESS_SPEC_CYCLE_ANNOTATE` (`cpu_verify_engine.py:3705-3719`, writes to **hard-coded** `.realmode-run/cycle_map.jsonl`, comment "uncommitted").

---

## 2. (b) Enabled cost

| Switch | Emitter | Per-event work | GPU sync? | Files |
|---|---|---|---|---|
| `BREAKDOWN_DIR`/`BREAKDOWN_RUN_DIR` | `vllm/vllm/breakdown_logger.py` | **per row**: `json.dumps` + `_fh.write` on a **line-buffered** handle (`:102` `buffering=1` → 1 `write()` per row) + explicit `flush()` every `BREAKDOWN_FLUSH_EVERY=1` rows (`:254-255`, redundant). `log_per_req` **broadcasts one row per request** (`:311-317`). Rows/step: worker PREPROCESS, PP_BUILD_ATTN, FORWARD_TOTAL, SOFTMAX_AFTER_FW, COMPUTE_LOGITS, SAMPLE, DRAFT_PROPOSE, BOOKKEEP ×B (`gpu_model_runner.py:3879,3994,4045,4062,4081,4205,4235,4288`) + 10 single PI_*/BA_* rows; sched SCHED_PLAN ×B, SCHED_UPDATE ×B, CPU_VERIFY_POSTPROC ×jobs (`scheduler.py:1265,2261,1848`), 7 KV_* (`kv_connector_model_runner_mixin.py:137-160`); + CPU_VERIFY_RPC/QUEUE_WAIT per RPC (`_verify_dispatcher.py:146,172`), KV_RELOAD_WAIT per reload (`cpu_tracker.py:208`). **Estimate B=16: ~180 `write()`+`json.dumps` per step**; at 10–20 µs each ≈ 2–4 ms/step (~10% of a 20–30 ms decode step). Unmeasured; check `:250-255`. | CUDA events only (`:395-407`); anchor sync once (`:420`); forced sync only >1024 pending (`:406-407`) or `close()`. | `step_<role>.jsonl`, `sched.jsonl` ~200 B/row |
| `BREAKDOWN_SKIP_WARMUP=1` (set by `breakdown_util.py:93`) + `.warmup_gating` (`test/test_multi_bp.py:81`) | `tick()` | while not recording and no `.measure_ready`: **2 `os.path.exists` per step per process** (`breakdown_logger.py:184-189`; called `scheduler.py:464`, `gpu_model_runner.py:3861`) | no | none |
| `BREAKDOWN_LIGHT_LAT=1` | aggregate mode (`:208-239`) | in-memory; `lat_<role>.json` every 2000 rows (`:237`), `reload_ts_` every 500 (`:231-234`) | no | tiny. **No in-tree runner sets it** (0 hits in `jh_script scripts breakdown_debugger test verify compressspec .env`); appears only in old `gpu.log` env dumps (accept_sweep Aug 21–Sep 4), i.e. exported from a shell. |
| `RECOMPACT_TRACE=1` (18 setters) | `compaction_trace.emit()` → `<BREAKDOWN_DIR>/<name>.jsonl` (`compaction_trace.py:24`, **root dir not RUN_DIR**, mode `"a"` → appends across runs) | reload_trace (`cpu_tracker.py:224`), verify_trace (`_verify_dispatcher.py:163`, needs breakdown on), recompact_trace (`scheduler_recompact.py:384`), draft_trace (`breakdown_logger.py:294-306`, **only under `_light`** → never written by current runners) | no | small |
| `HELD_TRACE=1` | `cpu_verify_engine.py:1857-1874`, `runner_diff_upload.py:251` | per request per after-verify: SHM `read_keep_latest` + emit; forces held-map path in full mode | no | small |
| `COMPACT_DIFF_STAT=1` (**19 setters**) | `gpu_recompact.py:21-24,78-83` | **`torch.cuda.synchronize()` ×2 per step with recompact plans** + `RECOMPACT_TIME` line | **yes** | lines: `runner_diff_upload.py:52-228`, `index_map.py:251,351` + `_diff_stat` set-diffs per selection (`:377-405`), `DIFF_META_SCHED` per request per step (`vllm_cpu/.../scheduler.py:1321`), `DIFF_META_SRV` (`cpu_verify_server_score.py:524`, `cpu_verify_server.py:667`), `FIN_REC/FIN_DRAIN` (`cpu_verify_engine.py:4343,4779`) |
| `DELTA_COMPACTION_LOG=N` (3 setters incl. ysshin `sweep_engine.sh:57`) | all `_diag(name, level)` fall back to it | 1 = `COMPACT_DIFF_STAT`+`KEEPMAP_DIAG`+`UPLOAD_TRACE`+`PRUNE_HOLD_LOG` (+ the 2 syncs); 2 adds `RECOMPACT_TRACE`, `COMPACT_ACK_TRACE`, **`RECOMPACT_DIAG` → `int(src.max().item())` per layer per plan (`gpu_recompact.py:53-54`)**; 3 adds `POOL_SET_DIAG`, `KEEPMAP_SIZE_DIAG` (`runner_diff_upload.py:269-343`) and `_HOLD_DUMP` → `$OUT_ROOT/held.jsonl` per round (`index_map.py:91-93,125`) | ≥2 yes | level 3 held.jsonl ≈ S×L ints/round |
| `LOG_DELTA_SELECTION=1` | `_delta_stamp` (`vllm_cpu gpu_model_runner.py:5445-5453`) | 1 line per compaction | no | small |
| `LOG_DELTA_SELECTION=2` | GPU `_log_draft_pool` per step (`vllm/.../gpu_model_runner.py:915-943`) + **`_log_pool_kv_fp` reads the whole selected pool for every layer and `.cpu()` per changed request (`:945-960`)**; sched `[EMIT]` crc32 over all emitted tokens per request per step (`scheduler.py:1782-1790`); `[TRACK_GAP]` per request per step (`temp_buffer.py:325-335`); `SEL_FP`/`ZC_ROWS` (`vllm_cpu gpu_model_runner.py:5295-5304,5351-5355`); `KM_READ_FP` (`keep_map_reader.py:198-202`) | **yes** | log only |
| `COMPACT_ACK_LOG=1` | `cpu_verify_engine.py:2068,2092,2801,3748` | 1–3 lines per ack | no | small |
| `KEEPMAP_DIAG=1` | `keep_map_reader.py:171,205`, `runner_diff_upload.py:266,358`, `scheduler_recompact.py:284,345,362,387` | per keep-map read/publish | no | small |
| `FSM_RUN_DIR` (from `FSM_DEBUG_DIR`, 3 setters; `breakdown_util.py:62-69`) | `_fsm_log.py:15-26` | **`open("a")`+write+close per transition** (no cached handle), ~6/request/round → ~100 opens/round at B=16 | no | one file per request |
| `COMPRESS_SPEC_TRACE` / `COMPRESS_SPEC_DEBUG≥2` (0 setters) | `_trace.py` | per-cycle in-memory buffer, one multi-line `logger.info` per verify cycle (`:141`) | no | log |
| `COMPRESS_SPEC_DEBUG≥1` | `cspec_debug.SUMMARY` | **apparently not wired**: `cpu_verify_engine.py:392-394` sets `compression_spec_debug = self.log_print_opt`, which comes from the harness `print_log`, not from `cspec_debug` (unverified: grep `SUMMARY` consumers) | — | — |
| `COMPRESSION_SPEC_DEBUG=true` (legacy, 14 reads; test sets 3) | `vllm/.../gpu_model_runner.py:1266,2034-2038,3887`, `single_type_kv_cache_manager.py:81` | per-step `logger.info` of **whole block-table tensors** | **yes** | huge log |
| `SCORE_DUMP_DIR` (`.env` commented only) | `attn_bp_connector.py:327` **sync per step**, `:401-409` `.pt` per finished job; engine `_maybe_dump_score_job` per submit (`cpu_verify_engine.py:2502-2668`: json + SHM `.pt` slices + 2 jsonl); GPU `_score_dump_pool` per step (`:3478`); server `verify/utils.py:1500-1562` | **yes** | `.pt` per job → GBs |
| `DEBUG_METADATA_DUMP_DIR` (18 setters — accept-meta pipeline) | `verify/utils.py:1592-1675` `accept_rate.jsonl`+`compact_ratio.jsonl` (open/append/close per `/verify`); `REJECT_DISTRIB_DUMP` → `reject_distrib.jsonl` (+`.pt`); `COMPACTION_IDX_DUMP=1` → `compaction_idx.jsonl` with **full kept-token list per layer per compaction** (`vllm_cpu gpu_model_runner.py:4189-4230`) | no | idx dump S×L ints/round |
| `ATTN_KV_DUMP_DIR` | `debug_utils.py:90-178` pretty-JSON per after_job per step incl. `all_token_ids` (called unconditionally `scheduler.py:1760,1819`, early-return when unset); `cpu_verify/utils.py:580` tracker json | no | O(context) per job |
| `KV_DUMP_NCT`/`KV_DUMP_SEQ` (+`KV_DUMP_DIR=/tmp/qwg_kvdump`, `ARM`) | GPU `_dump_draft_inputs` per step: stacks `kv[idx]` all layers, `.cpu()`, `torch.save` (`vllm/.../gpu_model_runner.py:866-911`); `temp_buffer.py:566-580`; `gpu_recompact.py:41-77`; CPU `_dump_cpu_upload` (`vllm_cpu gpu_model_runner.py:5456-5487`) | **yes** per step | `.pt` per step |
| `CPU_KV_HASH_DBG=1` (1 setter) | `vllm_cpu gpu_model_runner.py:5106-5130` | per step, per request, **per block** `.float().abs().sum().item()` → O(pool)/step | no (CPU) | log, very slow |
| `BREAKDOWN_KERNEL_PROFILE_MODE=1`+`INLINE=1` (default) | `_cpu_kernel_timer.py` | 2×`time.monotonic()` + append per op (`:60-68`); flush every 4096 ops or per forward (`:110`): `json.dumps` per row + 1 write (`:90-98`); ~550 rows/forward | no | `_cpu_kernels/kern_<host>_<pid>.jsonl` |
| `…PROFILE_MODE=1`+`INLINE=0` | torch.profiler with shapes (`breakdown_util.py:158-167`) | ysshin `scripts/amx_ablation/DESIGN.md:123`: **never use** (stalls async batching → width-1 forwards) | n/a | traces per job |
| `RECOMPACT_FP_LOG=1` | `gpu_recompact.py:26-37` | `_slot_sums().numpy()` per plan = full pool read + D2H | **yes** | log |
| `VERIFY_BATCH_LOG=1` | `cpu_attn.py:231` | 1 line per CPU forward | no | ysshin ablation cross-check (`DESIGN.md:182`) |
| `DRAFT_DISPATCH_LOG`, `LAST_VERIFY_LOG`, `PRESERVE_LOG`, `CS_STOP_LOG`, `DRAFT_TOKEN_LOG`, `LOGPROB_READ_DEBUG`, `PAYLOAD_BUILD_DBG`, `TEMPBUF_DBG`, `DUMP_TRACKER_DISPATCH`, PR4 `*_DBG` family | one `logger.info` per event, most capped by `_pr4_*` first-N counters | no | small |

---

## 3. (c) Duplicates and disposition

### 3.1 Same information in several places

| Information | Copies |
|---|---|
| per-round verify result | `CPU_VERIFY_RPC` row (`_verify_dispatcher.py:146`) · `verify_trace` (`:163`) · server `Verify done` (`cpu_verify_server_score.py:462`) · `accept_rate.jsonl` (`verify/utils.py:1592`) · engine finish summary (`cpu_verify_engine.py:4394`) · `cspec_accept_stats` (`scheduler.py:1842`) · `cycle_map.jsonl` (`cpu_verify_engine.py:3705`) |
| KV reload timing | `KV_RELOAD_WAIT` (`cpu_tracker.py:208`) · `reload_trace` (`:224`) · `reload_ts_` light (`breakdown_logger.py:231`) · `RELOAD_LIST` (`PRESERVE_LOG`) |
| recompaction event | `[COMPSPEC RECOMPACT]` ×2 unconditional (`scheduler_recompact.py:297,416`) · `recompact_trace` (`:384`) · `RC_DBG` ×3 (`:284,345,387`) · `RC_RANGE` (`:362`) · `RECOMPACT_TIME` (`gpu_recompact.py:82`) · `RECOMPACT_APPLIED` (`runner_diff_upload.py:92`) |
| upload mode/gen | `DIFF_META_RUNNER` (`runner_diff_upload.py:228`) · `DIFF_META_SCHED` (`vllm_cpu scheduler.py:1322`) · `DIFF_META_SRV` (`cpu_verify_server_score.py:525`, `cpu_verify_server.py:667`) · `DIFF_ACK` (`index_map.py:252`) · `ACK_ENTER/PROMOTE/STAGE` (`COMPACT_ACK_LOG`) · `ACK_CONSUME` (`COMPACT_ACK_TRACE`) |
| selection / keep-map fingerprints | `SEL_FP` (`vllm_cpu gpu_model_runner.py:5299`) · `KM_PUB_FP`/`KM_READ_FP` (`keep_map_reader.py:200`) · `POOL_KV_FP` (`vllm gpu_model_runner.py:959`) · `RECOMPACT_FP` (`gpu_recompact.py:34`) · `_diff_stat` (`index_map.py:377-405`) · `KEEPMAP_SIZE`/`POOL_SET` |
| score dumps | GPU `_score_dump_pool` (`:3478`) · connector `debug_dump_pt` (`attn_bp_connector.py:405`) · engine `_maybe_dump_score_job` (`cpu_verify_engine.py:2502`) · server `verify/utils.py:1500` |
| level knobs | `COMPRESS_SPEC_DEBUG` 0–3 · legacy `COMPRESSION_SPEC_DEBUG=true` · `DELTA_COMPACTION_LOG` 1–3 · `VERIFY_DEBUG` · harness `print_log` — five mechanisms |
| `_diag()` | 6 copies (§1 #8) |
| output roots | `BREAKDOWN_DIR` vs `BREAKDOWN_RUN_DIR` (logger → RUN_DIR `breakdown_logger.py:147`; `compaction_trace.py:24`, `_cpu_kernel_timer.py:39`, `tick()` `:185` → DIR root, cross-run append) · `FSM_DEBUG_DIR→FSM_RUN_DIR` · `COMPRESS_SPEC_DUMP_DIR` (`.realmode-run/debug`, `cspec_debug.py:37`) · `SCORE_DUMP_DIR` · `DEBUG_METADATA_DUMP_DIR` · `ATTN_KV_DUMP_DIR` · `LOGPROB_DUMP_DIR` · `CPU_VERIFY_DUMP_DIR`+`INIT_FILE_PATH` (`cpu_verify/utils.py:649-668`) · `KV_DUMP_DIR=/tmp/qwg_kvdump` · `OUT_ROOT=/tmp` (`index_map.py:92`) · `COMPACT_HOLD_DUMP` · `LOG_DIR`/`CPU_LOG`/`GPU_LOG` (`compressspec/cli.py:447-450`) · hard-coded `.realmode-run/cycle_map.jsonl` · `.env`-only `DUMP_DIR`/`BREAKDOWN_DUMP_DIR` (dead) |

### 3.2 Disposition

**KEEP**

| Switch | Where | Why |
|---|---|---|
| `BREAKDOWN_DIR` (+`BREAKDOWN_RUN_DIR`, `BREAKDOWN_SKIP_WARMUP`) | `breakdown_logger.py:147-160`, `breakdown_util.py:53-94` | 9 setters; pipeline (`breakdown_debugger.py`, `collect_cells.py`, `run_compressspec.sh`) reads `sched.jsonl`/`step_*.jsonl`. Fix line-buffer+flush duplication; route trace files into RUN_DIR. |
| `BREAKDOWN_KERNEL_PROFILE_MODE` (+`_INLINE`) | `_cpu_kernel_timer.py:17-18` | 29 setters; `cpu_inline_kernels.py` reads `_cpu_kernels/`. Keep INLINE=1; document INLINE=0 distortion. |
| `DEBUG_METADATA_DUMP_DIR` | `verify/utils.py:1288`, `vllm_cpu gpu_model_runner.py:4195` | 18 setters; `dump_debugger/run10/*` reads `accept_meta`. |
| `RECOMPACT_TRACE` | `compaction_trace.py` | 18 setters; `analysis/recompact_*.py`, `save_overhead.sh`, `refactor_trace.py` read the jsonl. Fix output dir. |
| `COMPACT_DIFF_STAT` | §2 | 19 setters; `patch_diagcost.py`, `save_ab.sh`, `run_nd.sh` grep `diff_stat`. Keep the lines; **move the syncs `gpu_recompact.py:22-23,79` to level 2**. |
| `DELTA_COMPACTION_LOG` | 6× `_diag()` | 3 setters incl. ysshin; becomes the single level knob. |
| `HELD_TRACE` | `cpu_verify_engine.py:1857` | 1 setter; `refactor_trace.py` reads it. Fold into level 2. |
| `FSM_DEBUG_DIR`/`FSM_RUN_DIR` | `_fsm_log.py` | 3 setters; stuck-request diagnosis. Level 2; cache handle. |
| `SCORE_DUMP_DIR` | 4 writers | `dump_debugger/score_debugger.py` reads `score_job_*`. Dump category (syncs every step). |
| `VERIFY_BATCH_LOG` | `cpu_attn.py:231` | ysshin AMX ablation cross-check. Level 1. |
| `LOG_DELTA_SELECTION=1` | `:5445` | cheap timing. Level 1. |
| `COMPRESS_SPEC_DEBUG` 0–3 + `COMPRESS_SPEC_TRACE[_REQ]` | `cspec_debug.py`, `_trace.py` | already the intended unified level; rename to `CS_TRACE`. |
| `KEEPMAP_SLOT_LOG` | `keep_map_reader.py:157` | zero cost; keep behaviour, drop the env. |
| `LOG_DIR`/`CPU_LOG`/`GPU_LOG` | `cli.py:447-450` | keep, derive from new root. |

**MERGE**

| Switch | Into | Evidence |
|---|---|---|
| `KEEPMAP_DIAG`, `UPLOAD_TRACE`, `PRUNE_HOLD_LOG`, `COMPACT_ACK_LOG`, `COMPACT_ACK_TRACE`, `RECOMPACT_DIAG`, `POOL_SET_DIAG`, `KEEPMAP_SIZE_DIAG` | `CS_TRACE` levels (already fall back to `DELTA_COMPACTION_LOG`; named overrides have 0 setters except `COMPACT_ACK_LOG` 2 passthroughs) | `runner_diff_upload.py:52-358`, `scheduler_recompact.py:126-388`, `gpu_recompact.py:53` |
| `LOG_DELTA_SELECTION=2`, `PRESERVE_LOG`, `LAST_VERIFY_LOG`, `DRAFT_DISPATCH_LOG`, `CS_STOP_LOG`, `DRAFT_TOKEN_LOG` | `CS_TRACE=3` (all from the 09-19 parity hunt `2d7f91b2a`; runners pass `${X:-0}`) | `vllm/.../gpu_model_runner.py:3885`, `temp_buffer.py:516,701`, `_tracker_registry.py:141,170`, `sched/utils.py:93`, `cpu_verify_server_score.py:455` |
| `DIFF_META_SCHED/SRV/RUNNER` | keep `DIFF_META_RUNNER` | 3 lines per event |
| `[COMPSPEC RECOMPACT]` ×2 + `RC_DBG` ×3 + `recompact_trace` | one jsonl record + one level-1 line | `scheduler_recompact.py:297,416,284,345,387,384` |
| `COMPRESSION_SPEC_DEBUG=true` | `CS_TRACE=3` or delete | `vllm/.../gpu_model_runner.py:329,1266,2034,3887`, `single_type_kv_cache_manager.py:81`; `c0a496c72` 2026-01-27 |
| `VERIFY_DEBUG`, `VERIFY_DEBUG_PAYLOAD` | `CS_TRACE≥2` on the server | `cpu_verify_server_score.py:114`, `cpu_verify_server.py:131,409` |
| all `*_DIR`/`OUT_ROOT`/`COMPACT_HOLD_DUMP`/`LOG_DIR`/`BREAKDOWN_RUN_DIR` | one root (§4.1) | |
| `BREAKDOWN_LIGHT_LAT`, `_PER_REQ`, `_FLUSH_EVERY`, `_GPU_EVENT_DEPTH` | `BREAKDOWN=rows|light`; drop the rest (FLUSH_EVERY redundant with `buffering=1`; PER_REQ 0 setters) | `breakdown_logger.py:88-91,105` |
| `_diag()` ×6, `_dlog()`, `compaction_trace.enabled()` | one `cs_trace.level()` resolved once at import | §1 #8 |

**REMOVE** (one-off instrumentation, `_pr4_*` first-N counters, no runner sets it)

| Switch | file:line | Added in | Setters |
|---|---|---|---|
| `PR4_8_DBG_EXTENDED` | `vllm/vllm/cpu_verify/channels/naive_copy.py:327` | vllm `f369fd175` 2026-06-07 fix(hybrid-sw) | 0 |
| `PR4_SW_GUARD_DEBUG`, `PR4_DISABLE_COMPRESS_CALL` | `flashinfer.py:1841`, `:2874` | `f369fd175` | 0 |
| `DUMP_FORWARD_OUT`, `DUMP_LAST_HIDDEN`, `DUMP_LOGITS_OUT` | `gemma3.py:373,532,550` | `f369fd175` | 0 |
| `DUMP_SAMPLED_IDS` | `vllm/.../gpu_model_runner.py:4213` | `f369fd175` | 0 |
| `TEMPBUF_DBG`, `DUMP_TRACKER_DISPATCH` | `temp_buffer.py:666,907` | `f369fd175` | 0 |
| `PAYLOAD_BUILD_DBG` | `cpu_verify_engine.py:2899` | `f369fd175` | 0 |
| `KV_DST_PTR_DBG` | `cpu_gpu.py:262,779` | `f369fd175` | 0 |
| `KV_TRANSFER_DBG`, `NAIVE_STORE_DBG`, `SAVE_KV_LAYER_DBG` | `offloading_naivecopy_connector.py:2,335,471` | `f369fd175` | 0 |
| `DUMP_EXTEND_BEFORE_VERIFY` | `vllm/.../scheduler.py:2303` | `f369fd175` | 1 |
| `ALLOC_NEW_COMPUTED_DBG`, `CPU_ATTN_BUILD_DBG`, `DUMMY_BATCH_DBG`, `FREE_BLOCKS_DBG`, `FWD_TENSOR_STATS`, `INPUT_PROCESSOR_DBG`, `RESERVE_DBG`, `CPU_FWD_READ_HASH` | `vllm_cpu/.../single_type_kv_cache_manager.py:244`, `cpu_attn.py:195,442,484,531`, `engine/core.py:1373`, `block_pool.py:514,386`, `input_processor.py:612` | vllm_cpu `2ecde1a2b` 2026-06-11 pr4-item8 | 0 |
| `CPU_KV_HASH_DBG` | `vllm_cpu gpu_model_runner.py:5106-5130` | vllm_cpu `730f89e59` 2026-06-11 (issue #34) | 1 |
| `COMPACT_DUMP_INDICES`, `COMPACT_DUMP_KV` | `:4298`, `:4464` | vllm_cpu `68cbf974a` 2026-05-06 | 0 |
| `FAKE_COMPACTION_VERIFY` | `:5525` | `88187ac4c` 2026-03-30 | 0 |
| `FAKE_NEEDS_KV_RELOAD` | `_fake_verify.py:247` | vllm `bebdb88ca` 2026-06-02 | 0 |
| `KV_DUMP_NCT`, `KV_DUMP_SEQ`, `KV_DUMP_DIR`, `RECOMPACT_FP_LOG` | `vllm gpu_model_runner.py:866-911,3934`, `temp_buffer.py:566-580`, `gpu_recompact.py:26-77`, `vllm_cpu gpu_model_runner.py:5363,5456-5487` | vllm `2d7f91b2a` / vllm_cpu `deaf471f3` 2026-09-19 (full-vs-diff parity hunt, since resolved) | 0 |
| `COMPRESS_SPEC_CYCLE_ANNOTATE` | `cpu_verify_engine.py:3705-3719` | not checked | 0 |
| `LOGPROB_READ_DEBUG`, `LOGPROB_READ_LEGACY_ANCHOR`, `LOGPROB_READ_POS_OFFSET` | `cpu_verify_engine.py:3478,3481,3517` | vllm `e39c49308` 2026-08-21 | 0 |
| `NUMA_VERIFY_DEBUG` | `temp_cpu.py:80` | vllm `0ef90d5dc` 2026-07-29 | 0 |
| `ATTN_BP_DEBUG_FILL`, `ATTN_BP_DUMP_SCORE_CHECK` | `attn_bp_connector.py:336`, `vllm gpu_model_runner.py:6836` | `97f6af266` / `bdf92c114` | `.env`=0 only |
| `COMPACTION_IDX_DUMP` (+`_EVERY`, `_LAYERS`) | `vllm_cpu gpu_model_runner.py:4189-4230` | vllm_cpu `915f8700c` 2026-07-19 | 0 (only `dump_debugger/saturate_logscripts/diff_sim.py` reads it) |
| `ATTN_KV_DUMP_DIR`, `CPU_VERIFY_DUMP_DIR`+`INIT_FILE_PATH`, `LOGPROB_DUMP_DIR` | `debug_utils.py:98`, `cpu_verify/utils.py:586,649`, `verify/utils.py:85,227` | `a1313a187` / `19f58c9` May 2026 | `.env` commented only |
| `.env`-only `DUMP_DIR`, `BREAKDOWN_DUMP_DIR`, legacy `COMPRESSSPEC_*` | `env_dead.csv` | | dead |

---

## 4. (c) Single output root and (d) minimal switch set

### 4.1 One root: `CS_OUT` (fallback `./.realmode-run`)

Resolve once per process in one small module (e.g. `vllm/vllm/cpu_verify/cs_out.py`) imported by both engines and the verify server. The launcher (`compressspec` CLI / `gpu_breakdown_setting`) creates `CS_OUT_RUN=$CS_OUT/<YYYYmmdd_HHMMSS[_tag]>` and exports it; children never mint their own timestamp (today three do: `breakdown_logger.py:154`, `debug_utils.py:103`, `verify/utils.py:1281,1289`, `attn_bp_connector._run_timestamp`).

| new subpath under `$CS_OUT_RUN/` | replaces | writers |
|---|---|---|
| `logs/gpu_client.log`, `logs/cpu_server.log` | `LOG_DIR`, `GPU_LOG`, `CPU_LOG` | `compressspec/cli.py:447-450` |
| `breakdown/` (`sched.jsonl`, `step_<role>.jsonl`, `cpu_verify.jsonl`, `lat_<role>.json`, `meta.json`, `model_config.json`) | `BREAKDOWN_DIR`+`BREAKDOWN_RUN_DIR` | `breakdown_logger.py:101,147-154,493,522` |
| `breakdown/cpu_kernels/` | `$BREAKDOWN_DIR/_cpu_kernels` | `_cpu_kernel_timer.py:39-46` |
| `breakdown/torch_profile/`, `breakdown/cpu_profile/` | `VLLM_TORCH_PROFILER_DIR` defaults | `breakdown_util.py:98-100,158-161` |
| `trace/<name>.jsonl` (reload/verify/recompact/held/draft_trace) | `$BREAKDOWN_DIR/<name>.jsonl` | `compaction_trace.py:24` |
| `trace/fsm/<req>.jsonl` | `FSM_DEBUG_DIR`/`FSM_RUN_DIR` | `_fsm_log.py:16-23`, `breakdown_util.py:62-69` |
| `trace/held.jsonl` | `$OUT_ROOT/held.jsonl`, `COMPACT_HOLD_DUMP` | `index_map.py:91-93` |
| `trace/cycle_map.jsonl` | hard-coded `.realmode-run/cycle_map.jsonl` | `cpu_verify_engine.py:3708` (or delete) |
| `dump/accept/` (`accept_rate.jsonl`, `compact_ratio.jsonl`, `reject_distrib.jsonl`, `reject_pt/`, `compaction_idx.jsonl`) | `DEBUG_METADATA_DUMP_DIR` | `verify/utils.py:1288,1379,1398,1616,1665`, `vllm_cpu gpu_model_runner.py:4195` |
| `dump/score/` | `SCORE_DUMP_DIR`, `COMPRESS_SPEC_DUMP_DIR/score` | `vllm gpu_model_runner.py:3483`, `attn_bp_connector.py:326,401`, `cpu_verify_engine.py:2480,2506`, `verify/utils.py:1280` |
| `dump/kv/` | `ATTN_KV_DUMP_DIR`, `KV_DUMP_DIR`, `CPU_VERIFY_DUMP_DIR`+`INIT_FILE_PATH` | `debug_utils.py:98`, `cpu_verify/utils.py:586,650`, KV_DUMP writers |
| `dump/logprob/` | `LOGPROB_DUMP_DIR` | `verify/utils.py:85` |
| `dump/debug/` | `COMPRESS_SPEC_DUMP_DIR` (`cspec_debug.py:37`) | mem-check / transport dumps |

Rules: no writer reads a path env directly (all call `cs_out.path("trace/…")`); `.env`-only `DUMP_DIR`/`BREAKDOWN_DUMP_DIR` deleted.

### 4.2 Minimal switch set

| Switch | Values | Turns on |
|---|---|---|
| `CS_OUT` | path | output root. Absent → `./.realmode-run`. |
| `CS_TRACE` | `0` (default) | **nothing** — also silences today's default-ON lines (§1 #1–#5 move to `≥1`; `THRESHOLD_CHECK` becomes once per process). Stays always-on: rare warnings (`cpu_verify_engine.py:2126,3142,3210`), `KEEPMAP_SLOT_MISMATCH` warn-once, `TRACKER_INIT`, server `Verify done` (or move to `≥1` if `accept_rate.jsonl` is canonical). |
| | `1` counters | today's `DELTA_COMPACTION_LOG=1` names (`COMPACT_DIFF_STAT` lines, `KEEPMAP_DIAG`, `UPLOAD_TRACE`, `PRUNE_HOLD_LOG`), `DELTA_STAMP`, `VERIFY_BATCH_LOG`, gate verdicts, selection summary, one `[COMPSPEC RECOMPACT]` line, and the jsonl traces reload/verify/recompact (today's `RECOMPACT_TRACE`). **No GPU sync at this level** (move `gpu_recompact.py:22-23,79` to 2). |
| | `2` cycle | + CSpecTrace blocks (`_trace.py`), `held_trace`, FSM jsonl, `COMPACT_ACK_*`, `RECOMPACT_TIME` with sync, `RECOMPACT_DIAG`, server `VERIFY_DEBUG`. |
| | `3` parity | + `LOG_DELTA_SELECTION=2` fingerprints, `PRESERVE_LOG`, `LAST_VERIFY_LOG`, `DRAFT_DISPATCH_LOG`, `CS_STOP_LOG`, `DRAFT_TOKEN_LOG`, `POOL_SET/KEEPMAP_SIZE`, `held.jsonl` (if retained). GPU syncs allowed. |
| `CS_DUMP` | comma list of `accept,score,kv,logprob,debug` (default empty) | file dumps into `dump/<name>/`; replaces the per-dir vars and `COMPRESS_SPEC_DEBUG=3`. `accept` is what runners use (18 setters). |
| `BREAKDOWN` | `0` (default) / `rows` / `light` | breakdown logger; `rows` = today's per-request jsonl, `light` = `BREAKDOWN_LIGHT_LAT=1`. Drop `_FLUSH_EVERY`, `_PER_REQ`, `_GPU_EVENT_DEPTH`, `_RUN_DIR`. |
| `BREAKDOWN_KERNEL` | `0` / `inline` / `profiler` | today's `_KERNEL_PROFILE_MODE`+`_KERNEL_INLINE`; `profiler` prints the batching-distortion warning. |
| `CS_TRACE_REQ` | request id | today's `COMPRESS_SPEC_TRACE_REQ`; cheap filter, keep. |

Implementation notes from the evidence: resolve `CS_TRACE`/`CS_DUMP`/`BREAKDOWN` **once at import** into module constants (as `cspec_debug.py` and `_cpu_kernel_timer.py` already do) and delete the six `_diag()` copies; guard every f-string emitter with the constant (today `vllm_cpu gpu_model_runner.py:3983,4107,3805` format unconditionally); give `_fsm_log` and `compaction_trace` cached handles under `CS_OUT_RUN`; drop either the explicit `flush()` in `breakdown_logger.log()` (`:254-255`) or `buffering=1` (`:102`), not both.

### 4.3 Not verified (login node, no run)

- Actual ms cost of §1 #1–#5 and of the breakdown row path at B=16 — estimates are syscall-count based. Instrument `vllm_cpu gpu_model_runner.py:3983`, `:4898`, `breakdown_logger.py:250-255`.
- Whether `COMPRESS_SPEC_DEBUG≥1` (`cspec_debug.SUMMARY`) reaches `log_print_opt` anywhere (`cpu_verify_engine.py:392-394` suggests not).
- Who exported `BREAKDOWN_LIGHT_LAT=1` in the Aug 21–Sep 4 accept-sweep runs; no in-tree setter.
- ysshin tree checked only for `sweep_engine.sh:57` (`DELTA_COMPACTION_LOG` from `ST_DELTA_LOG`), `run_compressspec.sh:68` (`BREAKDOWN_DIR` passthrough), both verify servers (`cpu_breakdown_setting()` same early-return) and `amx_ablation/DESIGN.md`.