awe/radar_system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rework radar system

Browse Source
This commit is contained in:
Ayzen
2026-10-06 12:32:26 +03:00
parent 552cdc688a
commit 5bb86b7165
299 changed files with 13537 additions and 27763 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+317 -116
View File
@@ -2,25 +2,25 @@
A real-time radar acquisition and imaging system for vector-network-analyzer (VNA)
and ADC front-ends. It drives the radar through a frequency sweep, calibrates and
preprocesses the measured S-parameters, runs a selectable imaging/detection
preprocesses the measured S21 traces, runs a selectable imaging/detection
processor, and presents the live result in a desktop GUI and an in-process web
interface. The same application runs interactively on a workstation or unattended
as a headless Raspberry Pi appliance.
## Overview
The system is split into a **C++ real-time pipeline** and a **Python application**
that supervises it and handles visualization and control.
The system uses **Python acquisition producers**, **C++ preprocessing and processing**,
and a **Python application** for supervision, visualization and control.
The pipeline runs as three independent processes connected by lock-free POSIX
shared-memory ring buffers. Each stage advances on its own, and a slow consumer
never blocks a producer — the rings are latest-wins, so a reader that falls behind
simply skips to the freshest frame:
skips overwritten frames and continues from the retained tail:
1. **Acquisition** — `sweep_orchestrator` (or a hardware-specific producer)
configures the radar, performs the frequency sweep, and publishes raw S11/S21
traces.
2. **Preprocessing** — `data_preprocessor` applies through/OSL calibration,
1. **Acquisition** — Python producers configure the radar, acquire frequency sweeps,
and publish raw S21 traces. Both single and multi-device LibreVNA use
`matrix_raw_producer`; Kamil ADC uses `kamil_adc_raw_producer` with its C++ collector.
2. **Preprocessing** — `data_preprocessor` applies through calibration,
reference subtraction, optional frequency-domain notch filtering, and an
optional object-based MIMO correction of S21.
3. **Processing** — `data_processor` runs the selected processing mode and emits
@@ -28,9 +28,9 @@ simply skips to the freshest frame:
or 3D SAR views). Detected objects are additionally broadcast to subscribers
over a TCP locator server.
The **Python application** (`python_app`) starts and monitors the C++ binaries,
reads the result rings, renders the live view with pyqtgraph, and exposes the same
controls plus a pixel-identical live plot through an embedded web UI.
The **Python application** (`python_app`) starts and monitors the producer and C++ processes,
reads the result rings, renders the live view with pyqtgraph, and exposes acquisition
controls, processing settings and a copy of the live plot through an embedded web UI.
```
radar hardware
@@ -45,12 +45,13 @@ controls plus a pixel-identical live plot through an embedded web UI.
- **librevna** — single LibreVNA.
- **librevna_multi** — synchronized multi-device LibreVNA.
- **kamil_adc** — external Kamil ADC acquisition process, with optional laser control.
- **kamil_adc** — [Kamil ADC collector](data_acq_and_processing/kamil_adc_collector/README.md),
with optional laser control and [temperature monitoring](python_app/hardware_full/laser_control/monitoring/README.md).
- **mock / simulator** — driver-less mode for development without hardware.
## Processing modes
- **pass_through** — magnitude and phase of the selected channel for each combo.
- **pass_through** — magnitude and phase of S21 for each combo.
- **bscan** — depth-versus-sweep B-scan heatmap.
- **gpr** — coherent back-projection GPR imaging with object detection.
- **legacy_gpr** — ellipse-intersection MIMO localization (point and extended/region modes).
@@ -58,30 +59,29 @@ controls plus a pixel-identical live plot through an embedded web UI.
on an NVIDIA GPU through CUDA: front (X-Y), side (Z-Y), top (X-Z)
and oblique maximum projections of the imaged
volume. Its settings carry the notebook's parameter names; the scene window is
fixed (X −2…2 m, height −0.5…2 m above the ground), its range extent follows
set by the selected antennas with a 2 m X margin and height −0.5…2 m above the ground; its range extent follows
`MIN_DEPTH`/`MAX_DEPTH`, and the voxels are cubes with edge `SAR_VOXEL_M`
(default 2 cm). The radar motion is corrected from each trace's capture
timestamps, so the sweep timing needs no parameters.
For matrix-radar B-scans, **Next combination** advances output first, then input,
and wraps to the first pair. The desktop fields and browser channel selector
redraw the selected cached B-scan immediately without restarting acquisition.
**Save Last 1000** and **Save S21 JSON** use a fixed limit of 1000 recent
measurements from each available stage; older GUI profiles may still contain
`save_count`, which is ignored. **Start + Record** retains its separate count for
recording future measurements to disk.
For B-scans on every radar model, **B-scan pair** selects a pair from the displayed
history independently of the run combinations. **Next combination** advances output
first, then input, and wraps to the first recorded pair; it is enabled with at least
two pairs. Desktop and browser selectors redraw the cached B-scan immediately.
Tx/Rx antenna positions are edited in tables with switch position and X/Y/Z in
metres, on both the desktop and web interface. Coordinate and legacy-GPR medium
changes apply to the running processor and recompute its retained result. Profiles
preserve all three coordinates. GPR/SAR views wait for a result marked with the
current settings revision, so an older in-flight result cannot overwrite new geometry.
preserve all three coordinates. A new table row becomes active after X is entered;
desktop, web and profile geometry use one Python validator; C++ validates its own
input. GPR/SAR views wait for a result marked with the current settings revision,
so an older in-flight result cannot overwrite new geometry.
## Repository layout
| Path | Contents |
| --- | --- |
| `data_acq_and_processing/` | C++ pipeline: `sweep_orchestrator/`, `preprocessing/`, `processing/` (including `locator/`), shared `common_cpp/` (config, IPC rings), and vendored `third_party/` (Eigen, nlohmann/json). |
| `data_acq_and_processing/` | C++ pipeline: `preprocessing/`, `processing/` (including `locator/`), `kamil_adc_collector/`, shared `common_cpp/` (config, IPC rings, H7992 driver), and vendored `third_party/` (Eigen, nlohmann/json). |
| `python_app/` | Python application: `gui/`, `orchestration/` (process supervisor, shared-memory readers, config), `hardware_full/` (device drivers and services), `webui/`, `storage/`, `models/`, `workflows/`, `scripts/` (producers and tools), and `tests/`. |
| `run_config_examples/` | Ready-made configuration profiles, one per hardware setup. |
| `deploy/` | systemd daemon installer, and the USB update kit for the Pi and Jetson (`pack-usb.sh`, `update.sh`). |
@@ -94,7 +94,7 @@ current settings revision, so an older in-flight result cannot overwrite new geo
**Raspberry Pi 5**; hardware PWM gating for the generator sweep is Pi 5
specific (GPIO12/13/18/19, `dtoverlay=pwm-2chan`).
- A C++20 compiler and `make`; Eigen and nlohmann/json are vendored.
- `libusb-1.0` development headers and `libatomic.so.1` (`libatomic1` on Debian/Ubuntu).
- `libusb-1.0` runtime and `libatomic.so.1` (`libatomic1` on Debian/Ubuntu).
Python ring readers/writers use lock-free atomic loads, stores and cursor CAS
through libatomic, including memory barriers for ARM. Slot sizes must be multiples
of 8 bytes, and the complete ring mapping must not exceed 1 TiB.
@@ -102,7 +102,7 @@ current settings revision, so an older in-flight result cannot overwrite new geo
- For `sar3d`: an NVIDIA GPU with its driver (`libcuda`). The
kernels are compiled at run time by NVRTC, so there is no nvcc step; `make`
needs only `nvrtc.h`, `cuda.h` and `libnvrtc`, taken from the pip
`nvidia-cuda-*` packages in `.venv` (`requirements-cuda.txt`, x86_64) or from
`nvidia-cuda-*` packages in `.venv` (`requirements.txt`, x86_64) or from
a system toolkit in `/usr/local/cuda` (Jetson).
Without these dependencies or a usable GPU, SAR3D shows a warning above the
plots and does not calculate an image. Other processing modes remain available.
@@ -115,57 +115,50 @@ From the repository root:
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -r requirements.txt
make # build the C++ binaries into build/bin/
make # data_preprocessor and data_processor in build/bin/
make kamil_adc_collector # also required for Kamil ADC acquisition
```
`make` builds the sar3d GPU code when it finds CUDA (see Requirements) and
prints a note when it does not. After installing the CUDA requirements into an
existing venv, run `make` again: it notices the change and rebuilds that part.
For SAR3D on x86_64, install the optional packages with
`.venv/bin/python -m pip install -r requirements-cuda.txt`, then run `make`.
CUDA 13 is installed from `requirements.txt` on x86_64. For GPUs older than
sm_75, install NVRTC 12 outside `.venv` and build with
`make BUILD_DIR=/tmp/build-nvrtc12 CUDA_ROOTS="<dir>/nvidia/cuda_nvrtc <dir>/nvidia/cuda_runtime"`.
A separate build directory keeps the two CUDA builds independent.
Run the desktop GUI:
Run the desktop GUI through the launcher for build and single-instance handling:
```bash
.venv/bin/python -m python_app.gui.main
# or, with the build step and single-instance handling:
./start.sh
./start.sh --profile run_config.json # explicit local mock profile
# or choose a hardware profile from run_config_examples/
```
`start.sh` supports `--headless` (offscreen, auto-start — for unattended Pi use),
`--profile PATH`, `--producer-only`, `--skip-build`, and `--clean-shm`; see
`./start.sh --help`. The acquisition device (LibreVNA, Kamil ADC, …) is detected
automatically from the active config's `radar.model` — there is no per-device flag.
`start.sh` also supports `--headless` (offscreen, auto-start), `--auto-start`,
`--producer-only`, `--skip-build`, `--build-only`, and `--clean-shm`; see
`./start.sh --help`. Pass `--profile PATH` so launcher provisioning and the GUI
use the same profile. The producer is selected from `radar.model`, with dataset
replay taking precedence. There is no per-device flag. Provision dependencies
before unattended use: headless startup does not install missing packages.
For development, `.venv/bin/python -m python_app.gui.main` starts the GUI directly,
without the launcher's build step or single-instance lock.
## Jetson (aarch64) bring-up
Ubuntu 22.04 on Jetson has glibc 2.35, and PyQt6 6.8+ ships only
`manylinux_2_39` aarch64 wheels — `requirements.txt` therefore pins PyQt6 to
`<6.8` on aarch64 (6.7.1 is the last release with a compatible wheel). With
that pin the stock `./start.sh` provisioning works; the extra steps are only
for the CUDA-backed `sar3d` processor:
`requirements.txt` pins PyQt6 below 6.8 on aarch64 for the older glibc used by
Jetson images. CUDA comes from JetPack in `/usr/local/cuda`; pip CUDA packages
are installed only on x86_64. Use the same build commands and launcher as above.
For dataset replay, first copy the dataset locally and update
`radar.replay.dataset_dir` in the profile; example paths are machine-specific:
```bash
# 1. copy the replay dataset (for dataset-replay profiles) and point
# radar.replay.dataset_dir in the profile at the local copy — the path in
# run_config_examples/*.json is machine-specific.
# 2. normal launch; make takes the JetPack CUDA from /usr/local/cuda
# (requirements.txt installs no pip nvidia-* packages on aarch64) and the
# driver's /usr/lib/aarch64-linux-gnu/nvidia/libcuda.so.1:
./start.sh --profile run_config_examples/run_config_replay_vna.example.json
```
Ubuntu 22.04 ships Python 3.10, and the host parses the radar stream in Python:
the same parsing code ran 1.6 times slower on 3.10 than on 3.13 (the Pi's version)
in a benchmark. Use Python 3.13 for the venv, installed with uv so the system
`python3` stays untouched (apt depends on it):
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh # then open a new terminal
uv python install 3.13
bash radar_update/update.sh --python 3.13 # see Deployment
```
The USB updater can create a fresh venv with an already installed interpreter,
for example `bash radar_update/update.sh --python 3.13`. The system Python does
not need to be replaced.
## Running without hardware (emulation)
@@ -180,31 +173,38 @@ along the antenna array and between depths 3.25–4.25 m over an eight-second cy
Each S21 uses the actual transmitter–target–receiver distance from the configured
XYZ coordinates; coincident antennas produce identical signals. All combinations
share the same target position and measured capture window, with collections paced
at 50 ms intervals. B-scan's default range is 0–8 m; stored profiles retain their
chosen range. The simulator models a stationary radar, so radar speed should be zero.
at 50 ms intervals. The root mock profile explicitly selects a 0–8 m B-scan
range. Both shipped mock profiles use a 0.445 m vertical antenna baseline
(Rx1/Rx3 and, for the four-output matrix, Tx2/Tx3), avoiding coincident pairs.
The general defaults are 0.824–1.0 m; stored profiles retain their range. The
simulator models a stationary radar, so radar speed should be zero.
The scene uses the antenna positions from its startup profile; live geometry edits
change the reconstruction of captured sweeps, whose samples remain unchanged.
The preprocessor still requires S21 calibration/reference sets. Start the GUI:
```bash
./start.sh
./start.sh --profile run_config.json
```
In **Preprocessing Panel**, use **Create neutral sets**, then **Start**. This makes
Open **Preprocessing**, use **Create Neutral S21 Sets**, then **Start**. This makes
identity calibration + zero reference, preserving the simulated S21 samples.
**Dataset replay (recorded scene).** When
`radar.replay.dataset_dir` is set in the profile, the supervisor launches
`python_app.scripts.replay_raw_producer` instead of the normal producer: it
replays recorded snapshots into the raw ring at the recorded cadence, looping
forever with a monotonic timeline. Pair it with NEUTRAL preprocess sets so the
already-preprocessed recording passes through the preprocessor bit-exactly.
A ready profile for the VNA_AlGo tank005 dataset is provided:
replays the collection directories in that folder into the raw ring. Point the
path at a saved stage such as `dataset/preprocessed`, containing
`NNNN_idK_ns<timestamp>/` directories with `meta.json` and `i*_o*_{freq,s21}.npy`.
Replay follows the recorded cadence scaled by `radar.replay.rate_scale`;
`radar.replay.loop` defaults to true. Trace windows preserve the recorded spacing
for SAR, even with accelerated replay. Each collection starts at its actual publication
time for acquisition identity and retains the original duration for legacy GPR.
For an already-preprocessed recording, select neutral through/reference sets
and disable notch and object correction to preserve the recorded S21 values.
A ready profile for the VNA_AlGo tank005 dataset is provided. Its neutral sets are
created once with the **Create Neutral S21 Sets** button of the preprocessing panel:
```bash
# one-time: neutral sets for this profile (or use the GUI button)
.venv/bin/python -m python_app.scripts.create_neutral_preprocess_sets \
--config run_config_examples/run_config_replay_vna.example.json
# interactive GUI (mode sar3d is pre-selected by the profile's gui section):
./start.sh --profile run_config_examples/run_config_replay_vna.example.json
```
@@ -214,77 +214,195 @@ a faster preview matters more than spatial resolution. Processing failures appea
above the plots and a failed single capture
finishes with its explanation while retaining the acquired data.
The web interface is served in-process at `http://<host>:8080`. It mirrors the live
plot and offers Start / Stop / Capture controls and the active processor settings.
## Web interface
Manual dataset and JSON export keep the last 1000 available collections from each
stage independently, including raw or preprocessed sweeps without a result.
Streaming recording keeps every collection received after arming until the requested
result count or Stop; Stop and closing the window flush accepted data. Acquisition
rings use latest-wins delivery, so disk recording cannot recover sweeps overwritten
before the GUI receives them. NumPy datasets preserve both S-parameters, capture
The embedded server listens at `http://<host>:8080`; set
`RADAR_SYSTEM_WEBUI_PORT` to use another port. It is available in desktop and
headless mode. Browser actions are:
- **Start**, **Single Capture**, **Stop**, **Tmp Reference** and **Remove Last Measurement**;
- load a profile from `run_configs/*.json` using **Load**;
- **Save Last 1000** and **Start + Record** with a path, optional name and recording count;
- edit processor settings and antenna tables with **Apply**, or clear history with **Reset history**;
- select a **B-scan pair** or advance with **Next combination**.
Copy profiles from `run_config_examples/` into `run_configs/` to offer them in the
browser. JSON export and calibration dialogs are desktop actions. PNG capture
runs only while a browser is connected and the plot has changed, including zoom
and resize, at most five times per second. Each client retains the latest frame,
status and settings independently; reconnecting refreshes settings, and **Apply**
sends only edited fields.
## Data and preprocessing
### Snapshots and recording
**Save Last 1000** and **Save S21 JSON** keep up to 1000 available collections from
each stage independently, including raw or preprocessed sweeps without a result.
The limit is fixed; legacy profile `save_count` values are ignored. Saving
keeps an active disk recording running.
Without raw/preprocessed taps, result snapshots remain available, but S21 JSON
requires retained traces.
**Start + Record to Disk** (**Start + Record** in the browser) has a separate count
for future results. It keeps collections received after arming until that count,
Stop, or exhaustion of the bounded writer queue. If storage falls behind,
recording stops accepting data, drains the queue and final chunk, and reports the limit.
Stop and closing the window also flush accepted data without blocking the GUI.
Pausing acquisition for a capture or history action ends an active recording:
the log reports the accepted count, requested count and stop reason before draining
the writer. Latest-wins rings cannot recover sweeps overwritten before the GUI
receives them. NumPy datasets preserve S21, capture
windows, every supported result payload, and processing time. Failed exports remove
their staging files, leaving existing datasets intact. Companion profiles retain
the radar, antenna layout and preprocessing used for acquisition, including when
another profile is loaded after Stop.
another profile is loaded after Stop. Dataset name prefixes also describe the acquisition;
a manual save is cancelled if acquisition is restarted during its final history drain.
Each NumPy dataset includes `processing_settings.json`, mapping result revisions to
the live settings used for processing. Results retain their revision, capture window,
and actual `speed_m_s` (including speed supplied by the locator). Settings
changed while acquisition is stopped are recorded for the reprocessed results as well.
**Remove Last Measurement** uses capture order and removes matching collections
from every stage; the processor suppresses that sweep if it is still pending.
**Clear Runtime History** (**Reset history** in the browser) clears all retained
stages and rejects late results from before the cutoff. These actions leave saved
files intact. A sweep is identified by `(collection_id, capture_start_ns)`; all
stages preserve the producer's capture window. `monotonic_ns` is diagnostic only.
**Save S21 JSON** can add exports to an existing NumPy dataset. Existing JSON files
are never overwritten; missing raw/preprocessed counterparts produce empty arrays,
and stages pair only by the exact acquisition key. Its `vna_config_profile.json` companion
leaves the NumPy dataset's `config_profile.json` intact. A streaming write failure
keeps all collections already published; only the incomplete collection is discarded.
The offline snapshot converter uses this same exporter. Legacy snapshots without
capture windows retain repeated IDs through one-to-one time/index matching;
processing completion timestamps are not compared with acquisition timestamps.
### Calibration and reference sets
Calibration and reference sets keep their trace arrays and metadata together in one
atomically replaced NPZ archive, with a readable JSON sidecar. Earlier NPZ/JSON pairs
atomically replaced NPZ archive, with a readable JSON sidecar. Both files and their
directory entries are synced before successful completion. Existing file permissions
are retained; new files and directories respect the process umask. Earlier NPZ/JSON pairs
remain supported. Invalid trace shapes, duplicate combinations and non-finite
calibration/reference values are rejected before a saved set is replaced.
Median captures also require identical frequency axes and channel coverage.
In **Preprocessing Panel**, create or select the ordinary S21 calibration and
empty-scene reference first. **Create…** in the **Object Calibration** row opens a separate step:
take fresh sweeps, then enter the measured reflector centre X/Y/Z in metres.
Height above ground, radar pitch and sphere radius follow `mimo_calibration.py`;
several reflector positions may contribute to one calibration. The calculation
shows each pair's phase at the target before/after correction and its coherence.
The selected reference is already subtracted, and the previous object correction
is disabled while these sweeps are acquired. Saved object corrections apply on the
next **Start**, after notch filtering and before any processor; **Off** disables
them. Changing antenna geometry disables an incompatible correction and reloads
acquisition without restarting the application. **Import…** in that row
accepts the example's `cal_*.npy` / `cal_meta.json` format; the example and GUI share
one numerical implementation.
Open **Preprocessing** to create or select the ordinary S21 calibration and
empty-scene reference. The processing order is through calibration → calibrated
reference subtraction → notch → object correction → the selected processor.
Dataset replay restores both recorded S-parameters and orders long recordings by
their numeric index; older S21-only datasets still replay with zero S11. Offline
JSON conversion and the GUI share the same sweep-matching code.
### Object calibration
**Create…** in the **Object Calibration** row opens a separate dialog. Use
**Capture data** to take fresh preprocessed sweeps, then enter the measured
reflector centre X/Y/Z in metres and use **Calculate and save**.
Coordinates follow the notebook: X across the array, Y above the ground, Z along
the ground.
Height above ground, radar pitch and sphere radius follow `MIMO_calibration.ipynb`;
several reflector positions may contribute to one calibration. The only diagnostic
plot shows the phase of each antenna pair's contribution at the target point,
before and after correction, as in cell 5 of `MIMO_calibration.ipynb`: Tx rows,
Rx columns, phase relative to the complex sum of each map, with cell values and
standard deviation in degrees on a cyclic ±180° scale.
With partial antenna geometry, only the specified Tx × Rx pairs contribute to
the calculation and phase maps. Acquisition still retains every physical stage;
pairs outside that geometry receive unit correction and appear in `unchanged_combos`.
The selected reference is already subtracted, and the previous object correction
is disabled while these sweeps are acquired. A capture owns an exclusive pipeline
pause until it finishes or is cancelled, including during automatic-restart backoff.
An explicit **Stop** cancels automatic resumption.
Object corrections use the same NPZ collection format as other preprocessing sets:
S21 holds each pair's multiplier on the captured frequency grid. Metadata retain
the capture radar key, antenna geometry, through-set name and content hash, target
positions, estimation settings and phase diagnostics. Saving uses the capture's
configuration even if the current profile changes during calculation. The next
**Start** checks the selected through set's S21 and frequency arrays, geometry and
coverage of the requested combinations before applying the correction after notch
filtering and before any processor. **Off** disables it. Changing antenna geometry
disables an incompatible correction and reloads acquisition without restarting
the application. Changing relative permittivity only updates processing settings.
Legacy `cal_*.npy` / `cal_meta.json` folders are not application calibration sets.
Capture a set through **Create…** to use object correction; the legacy files on
disk are left intact. Ordinary through/reference NPZ sets need no conversion.
Older NPZ sets and snapshots remain readable: extra S11 arrays are ignored;
new trace files contain only frequency and S21.
## Runtime interfaces
Tap rings are optional: an empty `rings.raw_tap.name` or
`rings.preprocessed_tap.name` disables that copy and its GUI history. Core processing
and single capture still work without taps; object calibration needs the preprocessed
tap. All producers share raw publication and hardware retry handling.
Raw and preprocessed IPC use **RAW3** and **PRP3** respectively: each trace contains
float32 frequencies followed by complex64 S21 (12 bytes per point). The complete
frame requires collection and per-trace capture windows; its size is `44 + combos * (28 + 12 * points)` bytes.
Older RAW2/PPR2 bundles are incompatible and are rebuilt from NPZ sets at Start;
update Python and C++ together. Results use RSL3; older RSL2 frames are rejected.
RF switches use H7992 with 1–4 positions and two GPIO control lines. Profiles use
`driver_mode`, GPIO settings and the position count. An old `driver: "h7992"` is accepted;
other explicit driver values or enabled inversion are rejected before GPIO opens.
Two positions use the existing H7992 mapping `(A,B) = (0,0), (0,1)`; physical
wiring and channel numbers follow the profile.
Single-device LibreVNA acquires the configured `run.combos` in order. Multi-device
acquisition keeps the full matrix: SAR3D derives intra-sweep stage timing from
the complete list of outputs sharing a capture window.
## Configuration
The active profile is `run_config.json`. It selects the radar model and sweep, the
switch matrix and acquisition combos, the preprocessing calibration/reference
assets, the GPR geometry, and the shared-memory ring sizes. Profiles for each
hardware setup are provided in `run_config_examples/` — copy one to
`run_config.json`, or pass it with `--profile`, to use it.
A profile selects the radar model and sweep, switches and acquisition combos,
preprocessing sets, antenna geometry, ring sizes and GUI settings. The GUI loads
`RADAR_SYSTEM_PROFILE` (set by `start.sh --profile`), otherwise the last profile in
`python_app/runtime/gui_session_state.json`, otherwise root `run_config.json`.
If a selected external profile cannot be loaded, it reports the error and falls
back to the root profile. **Load Config** and **Save Config** manage desktop profiles.
Hardware examples are in `run_config_examples/`. The root `run_config.json` is
intentionally mock; select a hardware profile explicitly on a real device. A
change of model, serials or sweep parameters changes the `radar_key` used to find
calibration/reference sets; select or capture matching sets before **Start**.
## Deployment
On a Raspberry Pi, `deploy/install-daemon.sh` installs a systemd service that runs
the application headless and restarts it on failure. A single-instance lock ensures
the interactive GUI and the daemon never run at the same time, since they share the
radar, the shared-memory rings, and the locator port.
the application through `start.sh --headless --skip-build` and restarts it on
failure. The launcher holds a single-instance lock for the radar, rings and locator
port; an interactive launch first asks systemd to stop an active daemon.
**Updating a device from a USB flash drive.** On the development machine
`deploy/pack-usb.sh` writes the host part of the last commit (no venv, build, data
or local configs) and the update script into `radar_update/` on the flash drive.
`deploy/pack-usb.sh /path/to/drive` writes the host part of the **last commit**
and its update script into `radar_update/` on the flash drive. Uncommitted changes,
venv, build outputs, recorded data and local configs are excluded.
On the Pi or Jetson, close the program and run, from the drive's folder:
```bash
bash radar_update/update.sh # or: --python 3.13 for a fresh venv
```
It installs the newest archive into `~/Documents/radar_system` (or `RADAR_DIR`);
the current version is renamed to `radar_system<N>`, the next free number. Kept from
it: `run_config.json`, profiles in `run_config_examples/` changed or added on the
device (the release version of a changed one lands next to it as `.new`),
`python_app/runtime/` without logs, `python_app/data*` and any other top-level
file or folder that is not part of the release, as hard links. The venv is copied
independently so package updates cannot modify the rollback environment (or a new
one is built with `--python`). It builds the C++ pipeline and ADC collector and restarts
`radar.service` if it was running. To go back:
`rm -rf radar_system && mv radar_system<N> radar_system`.
Run the updater as the normal device user. It installs the newest archive into
`~/Documents/radar_system` (or `RADAR_DIR`), renaming the current version to the
next free `radar_system<N>`. It preserves:
- `run_config.json` and changed/added `run_config_examples/*.json` by independent
copy; a conflicting release profile is placed beside the device profile as `.new`;
- `python_app/runtime/` by copy, excluding logs and `*.pids`;
- `python_app/data*` and extra non-hidden top-level entries absent from the release
by hard links, excluding `build`, `graphify-out`, `device_firmware`, `docs` and `RELEASE`;
- the venv by independent copy, or a fresh one when `--python` is supplied.
Saved data is written through replacement or new files, preserving the backup's
hard links. `RELEASE` gets a new inode. The updater builds the C++ pipeline and ADC
collector, then restarts `radar.service` only if it was running before the update.
Keep the device's real hardware profile: the release's root profile is mock.
For rollback, stop `radar.service` and close the GUI, rename the new installation
aside to preserve measurements taken since the update, then rename the chosen
backup to `radar_system`. Restart the service if it is normally used.
## Tests
@@ -302,10 +420,93 @@ also exercise mutated frames, ring concurrency, locator input and ADC spool reco
Firmware host tests check cart timer rollover and SBUS packing; they do not validate
physical timing. Real devices, GPIO and the CUDA backend need their own hardware run.
`check-sar3d` runs the real CUDA processor (`data_acq_and_processing/tests/test_sar3d.py`).
It compares it with `sar3d_reference.npz`: compact float32/complex64 inputs, the three
maximum projections of the notebook's volume, and its joint peak/level for four
compensation/normalization and timing cases. The expected values come from unchanged
functions in cells 0 and 4 of the original notebook. The same test checks the published
status for invalid input, a result too large for the ring slot, an empty and a very weak
signal. Without a usable GPU it checks only the CUDA warning and skips the rest; the
notebook itself is not needed to run it. To regenerate the reference, pass the
original notebook (it is kept outside the repository):
```bash
.venv/bin/python data_acq_and_processing/tests/generate_sar3d_reference.py \
--notebook /path/to/Multi_antenna_3D_clean_v1.ipynb
```
The archive records cell-source SHA256 hashes and NumPy/SciPy versions. Another build
can be checked with `make check-sar3d BUILD_DIR=/path/to/build` or by running
`test_sar3d.py` with `SAR3D_FIXTURE=/path/to/build/native_tests/sar3d_fixture`.
For older GPUs, pass the same `BUILD_DIR` and NVRTC 12 `CUDA_ROOTS` used to build.
For development, install `pyright` into the venv, then run
`.venv/bin/pyright python_app` to check all Python modules and tests.
## Application architecture
Invalid device/switch settings and undersized raw/preprocessed slots fail before
acquisition starts. Oversized processed results report the required slot size in
the log; switching to a mode that fits resumes processing without a restart.
`AppWindow` assembles controllers and shuts them down. `WindowView` owns layout
and typed sections; controllers receive concrete dependencies, without GUI mixins
or duplicate window state. `ProfileController` owns startup selection, session
state and profile persistence. `RadarSettingsController` applies device limits,
while `LiveSettingsController` publishes revisions and updates plots.
GUI scalar settings use dataclass metadata in `models/gui_profile_schema.py`:
profile validation, desktop values, live processing fields, processing log summaries
and web forms share those bindings. Hardware fields and geometry have their own
typed mappings and validators. Section builders return typed widget groups; their
controllers connect signals directly, with no string table of action callbacks.
`gui/controllers/config_controller.py` reads and applies scalar metadata without owning
the window or acquisition processes. Web batches roll back together when validation
or live-file publication fails; desktop settings also revert to the last publication.
Profile loading blocks editor signals until all settings are populated.
`HistoryController` owns retained acquisitions; `PlotController` selects the view
and supplies independent B-scan, GPR, trace and SAR renderers with typed settings.
Renderers own their plots and display caches, without access to the acquisition
processes. Stopped B-scan replay uses the last run's ring, matches exact acquisition
keys and the requested settings revision, and cancels when that run changes. With
no retained preprocessed sweeps (including disabled taps), the processor recomputes
its own retained history. Capture previews preserve the selected processing view.
`PipelineController` owns processes, ring readers, polling/recovery, Single Capture
and replay. Its pause prevents recovery from starting a producer over a hardware
capture; Stop or Close cancels any pending resumption, including during a drain.
Validation rejects Start before stopping existing processes. Pauses resume only
continuous acquisition, retry failed resumption with the usual backoff, and release
ownership before presenting capture errors. GPIO errors are logged without dialogs.
Rendering failures leave acquisition and the locator running and log each cause once;
repeated ring-read failures retain their separate recovery and stop policy.
`PreprocessController` owns calibration/reference sessions and asset selection.
One `SequentialCaptureSession` handles both current settings and a list of sweep
variants; a single variant adds no extra acquisition or reconfiguration. Multi-device
collection always captures one complete synchronized cycle. The native
`DataPreprocessor` owns all three bundles in `preprocessing/bundles`: through,
through-calibrated reference and object correction, applied in that order with notch
filtering before object correction.
Object capture checks both acquisition processes and releases its pause only after
completion or cancellation; losing the producer reports a failure immediately.
`SnapshotController` coordinates manual exports, paused Clear/Remove operations
and Start + Record.
History deletion is retained before publishing it to the processor, so a failed
settings write can be retried without restoring deleted sweeps. `RecordingController`
owns a bounded writer session; its disk thread receives data and storage only.
A processor restart during Start + Record does not finish the newly armed session.
Temporary references and neutral sets clear retained processing history without
restarting the locator while the pipeline is paused.
`WebBridge` queues browser actions onto Qt and exposes immutable status/frame
snapshots. Clear and Remove use separate action endpoints; live settings only edit
settings. Trace display options (including its Y range) do not publish processing
revisions. A timeout cancels queued actions; an action already running reports that
state without inviting a duplicate request. The GPIO controller invokes the same
reference-capture action as desktop and web, and ignores queued presses after Close.
`LogController` batches worker logs and returns web-action errors to the browser
without opening a desktop modal. Stop/Close also cancels delayed launcher startup.
GUI profiles, run configs, session state and live processing settings use shared
atomic writers with unique temporary files. Profiles and run/session settings sync
their file and directory; frequent live settings avoid extra disk syncs. A failed