# Stage 5 report: global Otsu reflectivity candidates Date: 14 July 2026 Project: `classical-lidar-lane-detection` Architecture plan: ## 1. Outcome Stage 5 is complete. The package now computes one deterministic global Otsu threshold from each frame's raw Stage-4 road-point reflectivity values and returns a Boolean candidate mask aligned to the Stage-3 road-point rows. The real-data command produces a four-panel PNG per labeled frame. It shows the retained and rejected road points, the raw reflectivity histogram and Otsu threshold, the K-Lane ground-truth bird's-eye-view label, and a display-only 144 x 144 candidate/label overlay. The label is deliberately downstream of the decision and is used only to make the result easy to inspect. This stage decides which road points have high-reflectivity evidence. It does not clean the BEV evidence or fit lane geometry. ## 2. Fixed Stage-5 contract | Contract | Stage-5 behavior | |---|---| | Input | Raw Stage-4 reflectivity values aligned to Stage-3 road points | | Threshold | One global Otsu threshold per frame | | Otsu domain | Exact observed values; no 8-bit quantization | | Candidate rule | `reflectivity > threshold` | | Output | Boolean mask aligned to the input road-point rows | | Label use | Diagnostic PNG only; never threshold computation | The implementation evaluates every split between consecutive observed values and selects the split with maximum between-class variance. The reported threshold is the largest value in the lower class. NumPy's first-maximum rule makes ties deterministic. A constant frame has no valid two-class split, so its single value becomes the threshold and the strict comparison returns no candidates. No parameter was tuned against labels. There is no Stage-5 configuration object because global Otsu and the strict decision rule are the fixed plan. ## 3. Code and interfaces added - `src/lidar_lane_detection/threshold.py` contains exact global Otsu, the aligned candidate result, and its machine-readable report. - `src/lidar_lane_detection/threshold_diagnostics.py` creates the four-panel visual artifact with the ground-truth BEV comparison. - `src/lidar_lane_detection/threshold_cli.py` repeats Stages 1-4, thresholds labeled frames, and writes per-frame and run-level artifacts. - `tests/test_threshold.py` covers a known bimodal split, alignment and input preservation, constant input, non-8-bit float values, and invalid inputs. - `pyproject.toml` exposes the workflow as `klane-stage5`. `ReflectivityThresholdResult` contains only the Otsu threshold, candidate mask, winning between-class variance, and observed-value count. Candidate point rows are obtained by applying the mask to the existing Stage-3 rows; they are not copied into a second result structure. ## 4. How to read the PNG The artifact landing page is [`../artifacts/stage5/README.md`](../artifacts/stage5/README.md). One representative panel is [`../artifacts/stage5/seq_01_001270427447090_otsu.png`](../artifacts/stage5/seq_01_001270427447090_otsu.png). The four views are: 1. **Otsu road-point candidates:** green points pass Otsu; dim gray points are rejected Stage-3 road points. 2. **Raw reflectivity histogram:** the orange line is the exact threshold; green bins lie on the candidate side. Log bar height is display-only. 3. **K-Lane ground-truth BEV:** magenta cells are the supplied binary lane confidence label, with forward at the top and positive lateral `y` at left. 4. **Display-only overlay:** green is candidate occupancy, magenta is ground truth, and white means both occupy that 144 x 144 cell. The overlay rasterizes candidate points only to make orientation and rough agreement visible. It is not Stage-6 cleanup, is not exposed as the detector's final mask, and is not scored. ## 5. Five-frame real-data run The command selected five evenly spaced **labeled** sequence-1 frames. This is why the middle timestamps differ slightly from the unlabeled-capable Stage-4 smoke selection. | Timestamp | Road points | Otsu threshold | Candidates | Candidate share | Otsu time | |---|---:|---:|---:|---:|---:| | `001270427447090` | 13,714 | 10,581 | 897 | 6.54% | 0.87 ms | | `001334120928180` | 14,675 | 11,551 | 1,231 | 8.39% | 0.86 ms | | `001413813614490` | 13,303 | 12,951 | 697 | 5.24% | 0.80 ms | | `001463808561830` | 12,202 | 7,253 | 1,055 | 8.65% | 0.71 ms | | `001501204924330` | 8,530 | 16,529 | 159 | 1.86% | 0.61 ms | The Otsu-only median was 0.80 ms on this smoke run. It excludes PCD parsing, ROI cropping, road-plane fitting, label loading, reporting, and PNG rendering, so it is a component timing rather than an end-to-end benchmark. The panels make the intended limitation visible. Otsu isolates a small bright tail, and some retained evidence follows the ground-truth lane direction, but many retained cells still trace cross-road scan structure or compact bright objects. The last frame is especially sparse: it inherits limited Stage-3 road support and Otsu retains only 159 points. Stage 6 is responsible for testing whether simple spatial cleanup can reject these artifacts; Stage 5 does not hide them. ## 6. Automated behavior verified The complete project suite result is: ```text Ran 18 tests in 0.258s OK ``` The complete source and test trees also pass Python bytecode compilation. The five-frame real-data command completed successfully, and representative dense and sparse panels were visually inspected at original resolution. ## 7. Explicitly not done To preserve the YAGNI boundary, Stage 5 did not add: - a high-resolution detector raster, morphology, gap closing, connected components, elongation checks, or component-size thresholds; - line fitting, lane identity, polynomial geometry, or temporal support; - strict or tolerant F1, precision, recall, threshold tuning, or label-driven selection; - 8-bit conversion, range correction, ring-wise thresholds, normalization, intensity fusion, a second threshold method, or fallback heuristics; - OpenCV, SciPy, scikit-image, a plotting dependency, ML, GPU, or ROS. The only label-related addition is the requested visible ground-truth BEV comparison in the diagnostic artifact. ## 8. Reproduction Run from the project root: ```bash env UV_CACHE_DIR=/tmp/uv-cache uv sync env UV_CACHE_DIR=/tmp/uv-cache uv run klane-stage5 \ /mnt/arnavs_hdd/data/02-lane-detection/K-Lane \ --sequence 1 \ --sample-count 5 \ --output-dir artifacts/stage5 env UV_CACHE_DIR=/tmp/uv-cache uv run python -m unittest discover -s tests -v env UV_CACHE_DIR=/tmp/uv-cache uv run python -m compileall -q src tests ``` ## 9. Next boundary Stage 5 is implemented, tested, exercised on labeled K-Lane data, and visually inspectable. The next architecture stage may rasterize and spatially clean these candidates with the plan's connected components and light morphology. None of that work is folded into this stage.