Refactor radar system

README.md

@@ -44,8 +44,6 @@ controls plus a pixel-identical live plot through an embedded web UI.
 
 - **librevna** — single LibreVNA.
 - **librevna_multi** — synchronized multi-device LibreVNA.
-- **compact_m_k209** — Compact-M K209 / S2VNA (including Raspberry Pi remote mode).
-- **sn9000** — SNVNA SN9000 over VISA (pure-Python `@py` backend).
 - **kamil_adc** — external Kamil ADC acquisition process, with optional laser control.
 - **mock / simulator** — driver-less mode for development without hardware.
 
@@ -56,13 +54,28 @@ controls plus a pixel-identical live plot through an embedded web UI.
 - **gpr** — coherent back-projection GPR imaging with object detection.
 - **legacy_gpr** — ellipse-intersection MIMO localization (point and extended/region modes).
 - **sar3d** — 3D SAR of the reference notebook `Multi_antenna_3D_clean_v1.ipynb`
-  on an NVIDIA GPU: front (X-Y), side (Z-Y), top (X-Z) and 3D views of the imaged
+  on an NVIDIA GPU through CUDA: front (X-Y), side (Z-Y), top (X-Z)
+  and oblique maximum projections of the imaged
   volume. Its settings carry the notebook's parameter names; the scene window is
   fixed (X −2…2 m, height −0.5…2 m above the ground), its range extent follows
   `MIN_DEPTH`/`MAX_DEPTH`, and the voxels are cubes with edge `SAR_VOXEL_M`
   (default 2 cm). The radar motion is corrected from each trace's capture
   timestamps, so the sweep timing needs no parameters.
 
+For matrix-radar B-scans, **Next combination** advances output first, then input,
+and wraps to the first pair. The desktop fields and browser channel selector
+redraw the selected cached B-scan immediately without restarting acquisition.
+**Save Last 1000** and **Save S21 JSON** use a fixed limit of 1000 recent
+measurements from each available stage; older GUI profiles may still contain
+`save_count`, which is ignored. **Start + Record** retains its separate count for
+recording future measurements to disk.
+
+Tx/Rx antenna positions are edited in tables with switch position and X/Y/Z in
+metres, on both the desktop and web interface. Coordinate and legacy-GPR medium
+changes apply to the running processor and recompute its retained result. Profiles
+preserve all three coordinates. GPR/SAR views wait for a result marked with the
+current settings revision, so an older in-flight result cannot overwrite new geometry.
+
 ## Repository layout
 
 | Path | Contents |
@@ -72,22 +85,26 @@ controls plus a pixel-identical live plot through an embedded web UI.
 | `run_config_examples/` | Ready-made configuration profiles, one per hardware setup. |
 | `deploy/` | systemd daemon installer, and the USB update kit for the Pi and Jetson (`pack-usb.sh`, `update.sh`). |
 | `build/bin/` | Compiled C++ binaries. |
-| `device_firmware/`, `docs/` | Device firmware and vendor manuals. |
+| `device_firmware/` | Device firmware and protocol documentation. |
 
 ## Requirements
 
 - Linux (POSIX shared memory; GPIO on Raspberry Pi). The deployment target is a
   **Raspberry Pi 5**; hardware PWM gating for the generator sweep is Pi 5
   specific (GPIO12/13/18/19, `dtoverlay=pwm-2chan`).
-- A C++20 compiler and `make` — Eigen and nlohmann/json are vendored, so no extra C++ dependencies are needed.
+- A C++20 compiler and `make`; Eigen and nlohmann/json are vendored.
+- `libusb-1.0` development headers and `libatomic.so.1` (`libatomic1` on Debian/Ubuntu).
+  Python ring readers/writers use lock-free atomic loads, stores and cursor CAS
+  through libatomic, including memory barriers for ARM. Slot sizes must be multiples
+  of 8 bytes, and the complete ring mapping must not exceed 1 TiB.
 - **Python 3.10 or newer**, in a virtual environment.
-- For the `sar3d` processor: an NVIDIA GPU with its driver (`libcuda`). The
+- For `sar3d`: an NVIDIA GPU with its driver (`libcuda`). The
   kernels are compiled at run time by NVRTC, so there is no nvcc step; `make`
   needs only `nvrtc.h`, `cuda.h` and `libnvrtc`, taken from the pip
-  `nvidia-cuda-*` packages in `.venv` (`requirements.txt`, x86_64; CUDA 13 covers
-  Turing and newer GPUs) or from a system toolkit in `/usr/local/cuda` (Jetson).
-  Without them everything else still builds, and sar3d reports that CUDA is
-  missing — a Raspberry Pi needs none of it.
+  `nvidia-cuda-*` packages in `.venv` (`requirements-cuda.txt`, x86_64) or from
+  a system toolkit in `/usr/local/cuda` (Jetson).
+  Without these dependencies or a usable GPU, SAR3D shows a warning above the
+  plots and does not calculate an image. Other processing modes remain available.
 
 ## Build and run
 
@@ -103,6 +120,8 @@ make                     # build the C++ binaries into build/bin/
 `make` builds the sar3d GPU code when it finds CUDA (see Requirements) and
 prints a note when it does not. After installing the CUDA requirements into an
 existing venv, run `make` again: it notices the change and rebuilds that part.
+For SAR3D on x86_64, install the optional packages with
+`.venv/bin/python -m pip install -r requirements-cuda.txt`, then run `make`.
 
 Run the desktop GUI:
 
@@ -153,17 +172,27 @@ Two ways to run the full pipeline with no radar attached:
 
 **Mock producer (synthetic data).** The default `run_config.json` already sets
 `radar.driver_mode: "mock"`, so the producer generates synthetic S-parameters.
-The preprocessor still requires S21 calibration/reference sets; create them once:
+Mock acquisition supports the LibreVNA models; Kamil ADC uses native acquisition
+or recorded-dataset replay without devices.
+Both LibreVNA modes share one simulator: a point reflector moves back and forth
+along the antenna array and between depths 3.25–4.25 m over an eight-second cycle.
+Each S21 uses the actual transmitter–target–receiver distance from the configured
+XYZ coordinates; coincident antennas produce identical signals. All combinations
+share the same target position and measured capture window, with collections paced
+at 50 ms intervals. B-scan's default range is 0–8 m; stored profiles retain their
+chosen range. The simulator models a stationary radar, so radar speed should be zero.
+The scene uses the antenna positions from its startup profile; live geometry edits
+change the reconstruction of captured sweeps, whose samples remain unchanged.
+The preprocessor still requires S21 calibration/reference sets. Start the GUI:
 
 ```bash
-.venv/bin/python -m python_app.scripts.manual_smoke_run --duration 1.0  # creates smoke_cal/smoke_ref
-./start.sh   # then in the GUI: Preprocessing Panel -> pick the sets -> Start
+./start.sh
 ```
 
-(Alternatively use the GUI's "Create neutral sets" button, which makes
-identity calibration + zero reference so preprocessing is a pass-through.)
+In **Preprocessing Panel**, use **Create neutral sets**, then **Start**. This makes
+identity calibration + zero reference, preserving the simulated S21 samples.
 
-**Dataset replay (recorded scene, meaningful sar3d picture).** When
+**Dataset replay (recorded scene).** When
 `radar.replay.dataset_dir` is set in the profile, the supervisor launches
 `python_app.scripts.replay_raw_producer` instead of the normal producer: it
 replays recorded snapshots into the raw ring at the recorded cadence, looping
@@ -179,15 +208,32 @@ A ready profile for the VNA_AlGo tank005 dataset is provided:
 ./start.sh --profile run_config_examples/run_config_replay_vna.example.json
 ```
 
-The sar3d processor keeps about 75 MB of buffers on the GPU at the default
-2 cm voxel and depth 2…8 m (plus the CUDA context). The volume grows as
-1/`SAR_VOXEL_M`³: 1 cm takes about 600 MB and ran about 6 times slower in
-testing. If the GPU has no room, it logs the CUDA error once and publishes empty
-results until memory frees.
+Finer voxels increase SAR3D work as 1/`SAR_VOXEL_M`³. Increase the voxel size when
+a faster preview matters more than spatial resolution. Processing failures appear
+above the plots and a failed single capture
+finishes with its explanation while retaining the acquired data.
 
 The web interface is served in-process at `http://<host>:8080`. It mirrors the live
 plot and offers Start / Stop / Capture controls and the active processor settings.
 
+Manual dataset and JSON export keep the last 1000 available collections from each
+stage independently, including raw or preprocessed sweeps without a result.
+Streaming recording keeps every collection received after arming until the requested
+result count or Stop; Stop and closing the window flush accepted data. Acquisition
+rings use latest-wins delivery, so disk recording cannot recover sweeps overwritten
+before the GUI receives them. NumPy datasets preserve both S-parameters, capture
+windows, every supported result payload, and processing time. Failed exports remove
+their staging files, leaving existing datasets intact.
+
+Calibration and reference sets keep their trace arrays and metadata together in one
+atomically replaced NPZ archive, with a readable JSON sidecar. Earlier NPZ/JSON pairs
+remain supported. Invalid trace shapes, duplicate combinations and non-finite
+calibration/reference values are rejected before a saved set is replaced.
+Median captures also require identical frequency axes and channel coverage.
+Dataset replay restores both recorded S-parameters and orders long recordings by
+their numeric index; older S21-only datasets still replay with zero S11. Offline
+JSON conversion and the GUI share the same sweep-matching code.
+
 ## Configuration
 
 The active profile is `run_config.json`. It selects the radar model and sweep, the
@@ -217,13 +263,34 @@ the current version is renamed to `radar_system<N>`, the next free number. Kept
 it: `run_config.json`, profiles in `run_config_examples/` changed or added on the
 device (the release version of a changed one lands next to it as `.new`),
 `python_app/runtime/` without logs, `python_app/data*` and any other top-level
-folder that is not part of the release, as hard links. Then it carries the venv over
-(or builds a new one with `--python`), builds the C++ part and restarts
+file or folder that is not part of the release, as hard links. The venv is copied
+independently so package updates cannot modify the rollback environment (or a new
+one is built with `--python`). It builds the C++ pipeline and ADC collector and restarts
 `radar.service` if it was running. To go back:
 `rm -rf radar_system && mv radar_system<N> radar_system`.
 
 ## Tests
 
 ```bash
-.venv/bin/python -m unittest discover -s python_app/tests
+QT_QPA_PLATFORM=offscreen .venv/bin/python -m unittest discover -s python_app/tests
+make check-native
+make check-sar3d PYTHON=.venv/bin/python
+ASAN_OPTIONS=detect_leaks=0 make check-native NATIVE_SANITIZERS=address,undefined
 ```
+
+The tests cover failed writes and settings commits, recordings during manual Save,
+repeated Start/Stop/Close, web-action cancellation, malformed IPC frames, hardware
+resource cleanup with mock failures, and an isolated deployment update. Native tests
+also exercise mutated frames, ring concurrency, locator input and ADC spool recovery.
+Firmware host tests check cart timer rollover and SBUS packing; they do not validate
+physical timing. Real devices, GPIO and the CUDA backend need their own hardware run.
+
+Invalid device/switch settings and undersized raw/preprocessed slots fail before
+acquisition starts. Oversized processed results report the required slot size in
+the log; switching to a mode that fits resumes processing without a restart.
+
+GUI profiles, run configs, session state and live processing settings use shared
+atomic writers with unique temporary files. Profiles and run/session settings sync
+their file and directory; frequent live settings avoid extra disk syncs. A failed
+write before replacement leaves the previous file intact. The launcher and supervisor
+share acquisition producer selection, with replay taking precedence over the model.