added docs
This commit is contained in:
Ayzen
@@ -1,599 +1,84 @@
|
||||
# Radar System
|
||||
|
||||
A real-time radar acquisition and imaging system for vector-network-analyzer (VNA)
|
||||
and ADC front-ends. It drives the radar through a frequency sweep, calibrates and
|
||||
preprocesses the measured S21 traces, runs a selectable imaging/detection
|
||||
processor, and presents the live result in a desktop GUI and an in-process web
|
||||
interface. The same application runs interactively on a workstation or unattended
|
||||
as a headless Raspberry Pi appliance.
|
||||
Программа радиолокационной системы. Она управляет радаром, снимает развёртки
|
||||
коэффициента передачи S21 по всем парам антенн, калибрует их, строит изображения сцены
|
||||
и показывает их в окне программы и в браузере. Найденные объекты она отдаёт внешним
|
||||
программам по TCP. Программа работает на рабочей станции с экраном и на Raspberry Pi 5
|
||||
без оператора.
|
||||
|
||||
## Overview
|
||||
Радары, с которыми она работает:
|
||||
|
||||
The system uses **Python acquisition producers**, **C++ preprocessing and processing**,
|
||||
and a **Python application** for supervision, visualization and control.
|
||||
|
||||
The pipeline runs as three independent processes connected by lock-free POSIX
|
||||
shared-memory ring buffers. Each stage advances on its own, and a slow consumer
|
||||
never blocks a producer — the rings are latest-wins: a full ring overwrites its oldest
|
||||
unread frame, a reader that falls behind continues with the frames still retained, and
|
||||
one overtaken during a read resumes at the newest frame:
|
||||
|
||||
1. **Acquisition** — Python producers configure the radar, acquire frequency sweeps,
|
||||
and publish raw S21 traces. Both single and multi-device LibreVNA use
|
||||
`matrix_raw_producer`; RadioPhotonic radar (RPH) uses `rph_radar_raw_producer` with its C++
|
||||
collector (E-502) or the RFG board's UDP stream.
|
||||
2. **Preprocessing** — `data_preprocessor` applies through calibration,
|
||||
reference subtraction, optional frequency-domain notch filtering, and an
|
||||
optional object-based MIMO correction of S21.
|
||||
3. **Processing** — `data_processor` runs the selected processing mode and emits
|
||||
results (pass-through traces, B-scan images, GPR images with detected objects,
|
||||
or 3D SAR views). Detected objects are additionally broadcast to subscribers
|
||||
over a TCP locator server.
|
||||
|
||||
The **Python application** (`python_app`) starts and monitors the producer and C++ processes,
|
||||
reads the result rings, renders the live view with pyqtgraph, and exposes acquisition
|
||||
controls, processing settings and a copy of the live plot through an embedded web UI.
|
||||
|
||||
```
|
||||
radar hardware
|
||||
│
|
||||
▼
|
||||
Acquisition ──[raw ring]──▶ Preprocessing ──[preprocessed ring]──▶ Processing ──[result ring]──▶ GUI / Web
|
||||
│
|
||||
└──▶ TCP locator (detected objects)
|
||||
```
|
||||
|
||||
## Supported hardware
|
||||
|
||||
- **librevna** — single LibreVNA.
|
||||
- **librevna_multi** — synchronized multi-device LibreVNA.
|
||||
- **rph_radar** — [RadioPhotonic radar (RPH) collector](data_acq_and_processing/rph_radar_collector/README.md),
|
||||
with manual laser setpoints or LD1 current variation via **Apply Radar**.
|
||||
`radar.rph_radar.source` selects the ADC:
|
||||
- `e502` (default) — the application owns the collector and exchanges complete
|
||||
sweeps over a PTY. Switching to a different antenna pair starts a fresh ADC session
|
||||
after settling, discarding queued samples and an initial partial sweep.
|
||||
- `rfg_udp` — the RFG board sends averaged sweep points to `udp_bind`
|
||||
(`run_config_rph_radar_rfg.example.json`); main = ch0 − ch1, reference = ch2. It
|
||||
needs `laser_control.mode: variation` (even when the laser is started elsewhere):
|
||||
a sweep must have exactly the task's step count, `ceil((max − min) / step)` in DAC
|
||||
codes, with no lost points; others are dropped (the board loses a few percent).
|
||||
Capture windows come from kernel receive timestamps. Only one antenna pair (both
|
||||
switches with one position): without a source-side marker a sweep after a switch
|
||||
cannot be proven fresh.
|
||||
- **mock / simulator** — driver-less mode for development without hardware.
|
||||
|
||||
## Processing modes
|
||||
|
||||
- **pass_through** — magnitude and phase of S21 for each combo.
|
||||
- **bscan** — depth-versus-sweep B-scan heatmap. A band selecting fewer than two
|
||||
sweep points gives no range profile, and such sweeps are left out of the image.
|
||||
- **gpr** — coherent back-projection on the ground plane, as in `Horns_motion_4x4_ground.py`,
|
||||
with object detection. Radar height and look angle define the slice in radar X/Z axes;
|
||||
the wave speed is 3×10⁸ m/s (air).
|
||||
- **sar3d** — 3D SAR of the reference notebook `Multi_depthnorm_clean.ipynb`
|
||||
on an NVIDIA GPU through CUDA: front (X-Y), side (Z-Y), top (X-Z)
|
||||
and oblique maximum projections of the imaged
|
||||
volume. Its settings carry the notebook's parameter names; the scene window is
|
||||
set by the selected antennas with a 2 m X margin and height −0.5…2 m above the ground; its range extent follows
|
||||
`MIN_DEPTH`/`MAX_DEPTH`, and the voxel steps `SAR_VOXEL_M` X, Y and Z default to the
|
||||
notebook's 2.5, 2 and 2 cm. The radar motion is corrected from each trace's capture
|
||||
timestamps, so the sweep timing needs no parameters.
|
||||
|
||||
Both coherent GPR and SAR apply `PAIR_DEPTH_EQUALIZE` (on by default) before the
|
||||
per-pair median normalization. The notebook's 1.5 m sliding median estimates each
|
||||
pair's depth-dependent level; positive gains bring the pairs to their geometric
|
||||
mean while preserving phase and the common depth envelope. **Subtract mean of previous
|
||||
collections** subtracts up to **Mean count** preceding spectra for each pair before
|
||||
motion correction and the IFFT; zero count disables it. SAR uses a causal aperture:
|
||||
`SAR_FRAME_RADIUS` previous frames plus the current frame, with the current scene origin.
|
||||
The processor retains enough history for the aperture and each frame's preceding
|
||||
mean; transport ring capacity does not shorten this background window.
|
||||
Scene width `SAR_X_SIZE_M` and length `SAR_Z_SIZE_M` use 0 for the automatic notebook
|
||||
bounds; positive values resize the axes about their automatic centres. `SAR_Y_RANGE_M`
|
||||
sets the height limits. The depth gate remains independent of these scene bounds.
|
||||
|
||||
SAR **Zoom (%)** shows the full scene at 100%; increasing it shrinks all four
|
||||
views uniformly towards the brightest response. **SAR_CLOUD_THRESHOLD** hides
|
||||
values below the selected fraction of the normalization maximum. Both controls
|
||||
redraw the existing result without recalculating it. Tx/Rx markers appear on the
|
||||
front, side and top projections in ground coordinates. A hollow square marks the
|
||||
peak; its size follows the contiguous half-maximum width along the volume axes
|
||||
(FWHM, at least one voxel). This measures the imaged response rather than the
|
||||
reflector's physical dimensions. The square is left out when CLEAN objects are shown
|
||||
or when the background gate empties the frame.
|
||||
|
||||
**BG_GATE_ENABLE** shows a frame relative to its background, the median of the volume
|
||||
over its working voxels: below 6 backgrounds the frame is empty, at 6 its maximum
|
||||
appears at the cloud threshold, and from 8.5 the scale is full. **SAR_CLEAN_ENABLE**
|
||||
runs the notebook's CLEAN on a one-frame aperture: always with `SAR_FRAME_RADIUS` 0,
|
||||
otherwise while no previous frame is available, as after Start. It subtracts the strongest point response
|
||||
from the A-scans of every pair, step by step on a 4 × 10 × 4 cm grid over the whole scene
|
||||
region, until the residual is background. The search ignores `SAR_X_SIZE_M` and
|
||||
`SAR_Z_SIZE_M`, so reflectors just outside a narrowed scene are subtracted too. Each object is then imaged apart and drawn in its own 0–1
|
||||
scale over the residual, so a weak object is not hidden by the sidelobes of a strong
|
||||
one. **SAR_MARK_ENABLE** draws boxes around groups of CLEAN objects: red for very low
|
||||
objects (like a mine), orange on the ground, blue above it, dashed grey at the scene
|
||||
boundary, where the response may lie partly outside. The legend is in the title of the
|
||||
3D view. Without boxes the objects are numbered with their level in backgrounds. All
|
||||
three switches are on by default, like in the notebook, and the status line reports the
|
||||
background ratio and the CLEAN result. The gate and CLEAN are processing settings; boxes
|
||||
only redraw. Their thresholds and rules are the notebook's fixed values. Both cost time:
|
||||
on a GTX 1060 with the default scene of 10 million voxels, the gate adds about 0.1 s per
|
||||
frame and CLEAN about 0.4 s more.
|
||||
|
||||
The desktop uses a light theme with white plot canvases. **Mode** stays below the
|
||||
**Actions** buttons while the settings sections scroll; the settings panel and
|
||||
runtime log can be collapsed with their arrow buttons. SAR3D status, peak
|
||||
coordinates and level appear above the projections.
|
||||
|
||||
For B-scans on every radar model, **B-scan pair** selects a pair from the displayed
|
||||
history independently of the run combinations. **Next Combination** advances output
|
||||
first, then input, and wraps to the first recorded pair; it is enabled with at least
|
||||
two pairs. Desktop and browser selectors redraw the cached B-scan immediately.
|
||||
|
||||
Tx/Rx antenna positions are edited in tables with switch position and X/Y/Z in
|
||||
metres, on both the desktop and web interface. Coordinate changes apply to the
|
||||
running processor and recompute its retained result. Profiles
|
||||
preserve all three coordinates. A new table row becomes active after X is entered;
|
||||
desktop, web and profile geometry use one Python validator; C++ validates its own
|
||||
input. GPR/SAR views wait for a result marked with the current settings revision,
|
||||
so an older in-flight result cannot overwrite new geometry.
|
||||
|
||||
## Repository layout
|
||||
|
||||
| Path | Contents |
|
||||
| Радар | Что это |
|
||||
| --- | --- |
|
||||
| `data_acq_and_processing/` | C++ pipeline: `preprocessing/`, `processing/` (including `locator/`), `rph_radar_collector/`, shared `common_cpp/` (config, IPC rings, H7992 driver), and vendored `third_party/` (Eigen, nlohmann/json). |
|
||||
| `python_app/` | Python application: `gui/`, `orchestration/` (process supervisor, shared-memory readers, config), `hardware_full/` (device drivers and services), `webui/`, `storage/`, `models/`, `workflows/`, `scripts/` (producers and tools), and `tests/`. |
|
||||
| `run_config_examples/` | Ready-made configuration profiles, one per hardware setup. |
|
||||
| `deploy/` | systemd daemon installer, and the USB update kit for the Pi and Jetson (`pack-usb.sh`, `update.sh`). |
|
||||
| `build/bin/` | Compiled C++ binaries. |
|
||||
| `device_firmware/` | Device firmware and protocol documentation. |
|
||||
| одиночный LibreVNA | один векторный анализатор цепей и ВЧ-ключи на антеннах |
|
||||
| многоприборная схема | три синхронных LibreVNA; с контроллером ключей — матрица 4 × 4 пар за одну развёртку |
|
||||
| радиофотонный радар | СВЧ-сигнал формируют лазеры, данные оцифровывает АЦП L-Card E-502 или плата RFG |
|
||||
| симулятор | работа без железа: один движущийся точечный отражатель |
|
||||
|
||||
## Requirements
|
||||
Режимы обработки: pass-through (S21 по частоте), B-scan (дальность во времени), GPR
|
||||
(карта земли перед радаром с поиском объектов), 3D SAR (трёхмерное изображение на
|
||||
видеокарте NVIDIA).
|
||||
|
||||
- Linux (POSIX shared memory; GPIO on Raspberry Pi). The deployment target is a
|
||||
**Raspberry Pi 5**.
|
||||
- A C++20 compiler and `make`; Eigen and nlohmann/json are vendored.
|
||||
- `libusb-1.0` runtime and `libatomic.so.1` (`libatomic1` on Debian/Ubuntu).
|
||||
Python ring readers/writers use lock-free atomic loads, stores and cursor CAS
|
||||
through libatomic, including memory barriers for ARM. Slot sizes must be multiples
|
||||
of 8 bytes, and the complete ring mapping must not exceed 1 TiB.
|
||||
- **Python 3.10 or newer**, in a virtual environment.
|
||||
- For `sar3d`: an NVIDIA GPU with its driver (`libcuda`). The
|
||||
kernels are compiled at run time by NVRTC, so there is no nvcc step; `make`
|
||||
needs only `nvrtc.h`, `cuda.h` and `libnvrtc`, taken from the pip
|
||||
`nvidia-cuda-*` packages in `.venv` (`requirements.txt`, x86_64) or from
|
||||
a system toolkit in `/usr/local/cuda` (Jetson).
|
||||
Without these dependencies or a usable GPU, SAR3D shows a warning above the
|
||||
plots and does not calculate an image. Other processing modes remain available.
|
||||
## Быстрый старт без железа
|
||||
|
||||
## Build and run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install --upgrade pip
|
||||
.venv/bin/python -m pip install -r requirements.txt
|
||||
make # data_preprocessor and data_processor in build/bin/
|
||||
make rph_radar_collector # RadioPhotonic radar (RPH) with radar.rph_radar.source: e502
|
||||
```sh
|
||||
./start.sh
|
||||
```
|
||||
|
||||
`make` builds the sar3d GPU code when it finds CUDA (see Requirements) and
|
||||
prints a note when it does not. After installing the CUDA requirements into an
|
||||
existing venv, run `make` again: it notices the change and rebuilds that part.
|
||||
CUDA 13 is installed from `requirements.txt` on x86_64. For GPUs older than
|
||||
sm_75, install NVRTC 12 outside `.venv` and build with
|
||||
`make BUILD_DIR=/tmp/build-nvrtc12 CUDA_ROOTS="<dir>/nvidia/cuda_nvrtc <dir>/nvidia/cuda_runtime"`.
|
||||
A separate build directory keeps the two CUDA builds independent.
|
||||
`start.sh` создаёт окружение Python, ставит пакеты и собирает C++. Корневой профиль
|
||||
`run_config.json` — симулятор. В окне нажмите **Preprocessing…** →
|
||||
**Create Neutral S21 Sets**, затем **Start** и выберите режим в списке **Mode**.
|
||||
|
||||
Run the desktop GUI through the launcher for build and single-instance handling:
|
||||
Установка, запуск с железом и служба для работы без экрана — в главе
|
||||
[Установка и запуск](docs/installation.md).
|
||||
|
||||
```bash
|
||||
./start.sh --profile run_config.json # explicit local mock profile
|
||||
# or choose a hardware profile from run_config_examples/
|
||||
```
|
||||
## Документация
|
||||
|
||||
`start.sh` also supports `--headless` (offscreen, auto-start), `--auto-start`,
|
||||
`--producer-only`, `--skip-build`, `--build-only`, and `--clean-shm`; see
|
||||
`./start.sh --help`. Pass `--profile PATH` so launcher provisioning and the GUI
|
||||
use the same profile. The producer is selected from `radar.model`; there is no
|
||||
per-device flag. Provision dependencies
|
||||
before unattended use: headless startup does not install missing packages.
|
||||
Для работы с радаром:
|
||||
|
||||
For development, `.venv/bin/python -m python_app.gui.main` starts the GUI directly,
|
||||
without the launcher's build step or single-instance lock.
|
||||
1. [Как работает радар](docs/principles.md) — физика измерения и обработки для тех,
|
||||
кто не работал с векторными анализаторами и радарами.
|
||||
2. [Оборудование](docs/hardware.md) — приборы, подключение, прошивки.
|
||||
3. [Установка и запуск](docs/installation.md) — установка, `start.sh`, служба,
|
||||
обновление с флешки.
|
||||
4. [Типовые сценарии](docs/workflow.md) — порядок действий от подготовки до
|
||||
сохранения данных.
|
||||
5. [Окно программы](docs/desktop.md) — каждая кнопка, поле и сообщение окна.
|
||||
6. [Режимы обработки](docs/processing.md) — что показывает каждый режим и что делает
|
||||
каждая его настройка.
|
||||
7. [Калибровка и предобработка](docs/preprocessing.md) — наборы, съём, калибровка по
|
||||
объекту.
|
||||
8. [Веб-интерфейс, кнопка и локатор](docs/web.md) — управление по сети, GPIO-кнопка,
|
||||
протокол локатора, работа без экрана.
|
||||
9. [Файл конфигурации](docs/configuration.md) — каждый ключ профиля.
|
||||
10. [Данные на диске](docs/data.md) — форматы наборов, снимков, журналов.
|
||||
11. [Неполадки](docs/troubleshooting.md) — признак, причина, что делать.
|
||||
12. [Словарь](docs/glossary.md).
|
||||
|
||||
## Jetson (aarch64) bring-up
|
||||
Для разработки:
|
||||
|
||||
`requirements.txt` installs PyQt6 from wheels only, so pip picks the newest release
|
||||
the system glibc can run: 6.7 on Jetson's glibc 2.35, current releases on Debian 13. CUDA comes from JetPack in `/usr/local/cuda`; pip CUDA packages
|
||||
are installed only on x86_64. Use the same build commands and launcher as above.
|
||||
- [Устройство программы](docs/architecture.md) — процессы, кольца, кадры, состав кода,
|
||||
где что править.
|
||||
- [Разработка](docs/development.md) — сборка, тесты, эталоны, правила.
|
||||
- [Протокол LibreVNA и кадры колец](device_firmware/librevna_protocol.md),
|
||||
[переключение стимула](device_firmware/per_point_switching.md),
|
||||
[контроллер ключей](device_firmware/switch_controller/README.md),
|
||||
[сборка прошивок](device_firmware/multi_libreVNA_firmware/reproduce/README.md),
|
||||
[протокол пульта тележки](device_firmware/cart_firmware/docs/protocol.md),
|
||||
[коллектор E-502](data_acq_and_processing/rph_radar_collector/README.md).
|
||||
|
||||
The USB updater can create a fresh venv with an already installed interpreter,
|
||||
for example `bash radar_update/update.sh --python 3.13`. The system Python does
|
||||
not need to be replaced.
|
||||
## Состав репозитория
|
||||
|
||||
## Running without hardware (emulation)
|
||||
|
||||
The default `run_config.json` already sets `radar.driver_mode: "mock"`, so the
|
||||
producer generates synthetic S-parameters and no radar is needed. Mock acquisition
|
||||
supports the LibreVNA models; RadioPhotonic radar (RPH) needs native acquisition.
|
||||
Both LibreVNA modes share one simulator: a point reflector moves back and forth
|
||||
along the antenna array and between depths 3.25–4.25 m over an eight-second cycle.
|
||||
Each S21 uses the actual transmitter–target–receiver distance from the configured
|
||||
XYZ coordinates; coincident antennas produce identical signals. All combinations
|
||||
share the same target position and measured capture window, with collections paced
|
||||
at 50 ms intervals. The root mock profile explicitly selects a 0–8 m B-scan
|
||||
range. Both shipped mock profiles use a 0.445 m vertical antenna baseline
|
||||
(Rx1/Rx3 and, for the four-output matrix, Tx2/Tx3), avoiding coincident pairs.
|
||||
The general defaults are 0.824–1.0 m; stored profiles retain their range. The
|
||||
simulator models a stationary radar, so radar speed should be zero.
|
||||
The scene uses the antenna positions from its startup profile; live geometry edits
|
||||
change the reconstruction of captured sweeps, whose samples remain unchanged.
|
||||
The preprocessor still requires S21 calibration/reference sets. Start the GUI:
|
||||
|
||||
```bash
|
||||
./start.sh --profile run_config.json
|
||||
```
|
||||
|
||||
Open **Preprocessing…**, use **Create Neutral S21 Sets**, then **Start**. This makes
|
||||
identity calibration + zero reference, preserving the simulated S21 samples.
|
||||
|
||||
SAR3D work grows inversely with the product of the three `SAR_VOXEL_M` steps. Increase them when
|
||||
a faster preview matters more than spatial resolution. Processing failures appear
|
||||
above the plots and a failed single capture
|
||||
finishes with its explanation while retaining the acquired data.
|
||||
|
||||
## Web interface
|
||||
|
||||
The embedded server listens at `http://<host>:8080`; set
|
||||
`RADAR_SYSTEM_WEBUI_PORT` to use another port. It is available in desktop and
|
||||
headless mode. Browser actions are:
|
||||
|
||||
- **Start**, **Single Capture**, **Stop**, **Capture Temporary Reference** and **Remove Last Measurement**;
|
||||
- load a profile from `run_configs/*.json` using **Load**;
|
||||
- **Save Last 1000** and **Start + Record to Disk** with a path, optional name and recording count;
|
||||
- edit processor settings and antenna tables with **Apply**, or clear history with **Clear Runtime History**;
|
||||
- select a **B-scan pair** or advance with **Next Combination**.
|
||||
|
||||
Copy profiles from `run_config_examples/` into `run_configs/` to offer them in the
|
||||
browser. JSON export and calibration dialogs are desktop actions. PNG capture
|
||||
runs only while a browser is connected and the plot has changed, including zoom
|
||||
and resize, at most five times per second. Each client retains the latest frame,
|
||||
status and settings independently; reconnecting refreshes settings, and **Apply**
|
||||
sends only edited fields. Field labels, units and checkboxes match the desktop form.
|
||||
|
||||
## Data and preprocessing
|
||||
|
||||
### Snapshots and recording
|
||||
|
||||
**Save Last 1000** and **Save S21 JSON** keep up to 1000 available collections from
|
||||
each stage independently, including raw or preprocessed sweeps without a result.
|
||||
The limit is fixed; legacy profile `save_count` values are ignored. Saving
|
||||
keeps an active disk recording running.
|
||||
Without raw/preprocessed taps, result snapshots remain available, but S21 JSON
|
||||
requires retained traces.
|
||||
**Start + Record to Disk** has a separate count
|
||||
for future results. It keeps collections received after arming until that count,
|
||||
Stop, or exhaustion of the bounded writer queue. If storage falls behind,
|
||||
recording stops accepting data, drains the queue and final chunk, and reports the limit.
|
||||
Stop and closing the window also flush accepted data without blocking the GUI.
|
||||
Pausing acquisition for a capture or history action ends an active recording:
|
||||
the log reports the accepted count, requested count and stop reason before draining
|
||||
the writer. Latest-wins rings cannot recover sweeps overwritten before the GUI
|
||||
receives them. NumPy datasets preserve S21, capture
|
||||
windows, every supported result payload, and processing time. Failed exports remove
|
||||
their staging files, leaving existing datasets intact. Companion profiles retain
|
||||
the radar, antenna layout and preprocessing used for acquisition, including when
|
||||
another profile is loaded after Stop. Dataset name prefixes also describe the acquisition;
|
||||
a manual save is cancelled if acquisition is restarted during its final history drain.
|
||||
|
||||
Each NumPy dataset includes `processing_settings.json`, mapping result revisions to
|
||||
the live settings used for processing. Results retain their revision, capture window,
|
||||
and actual `speed_m_s` (including speed supplied by the locator). Settings
|
||||
changed while acquisition is stopped are recorded for the reprocessed results as well.
|
||||
**Remove Last Measurement** uses capture order and removes matching collections
|
||||
from every stage; the processor suppresses that sweep if it is still pending.
|
||||
**Clear Runtime History** clears all retained
|
||||
stages and rejects late results from before the cutoff. These actions leave saved
|
||||
files intact. A sweep is identified by `(collection_id, capture_start_ns)`; all
|
||||
stages preserve the producer's capture window. `monotonic_ns` is diagnostic only.
|
||||
|
||||
**Save S21 JSON** can add exports to an existing NumPy dataset. Existing JSON files
|
||||
are never overwritten; missing raw/preprocessed counterparts produce empty arrays,
|
||||
and stages pair only by the exact acquisition key. Its `vna_config_profile.json` companion
|
||||
leaves the NumPy dataset's `config_profile.json` intact. A streaming write failure
|
||||
keeps all collections already published; only the incomplete collection is discarded.
|
||||
Legacy snapshots without capture windows retain repeated IDs through one-to-one
|
||||
time/index matching; processing completion timestamps are not compared with
|
||||
acquisition timestamps.
|
||||
|
||||
### Calibration and reference sets
|
||||
|
||||
Calibration and reference sets keep their trace arrays and metadata together in one
|
||||
atomically replaced NPZ archive, with a readable JSON sidecar. Both files and their
|
||||
directory entries are synced before successful completion. Existing file permissions
|
||||
are retained; new files and directories respect the process umask. Earlier NPZ/JSON pairs
|
||||
remain supported. Invalid trace shapes, duplicate combinations and non-finite
|
||||
calibration/reference values are rejected before a saved set is replaced.
|
||||
Median captures also require identical frequency axes and channel coverage.
|
||||
Open **Preprocessing…** to create or select the ordinary S21 calibration and
|
||||
empty-scene reference. The processing order is through calibration → calibrated
|
||||
reference subtraction → notch → object correction → the selected processor.
|
||||
|
||||
### Object calibration
|
||||
|
||||
**Create…** in the **Object calibration** row opens a separate dialog. Use
|
||||
**Capture Data** to take fresh preprocessed sweeps, then enter the measured
|
||||
reflector centre X/Y/Z in metres and use **Calculate and Save**. The optional
|
||||
**Calculation Settings** section expands inside the scrollable dialog body;
|
||||
the calculation, cancellation and close buttons remain visible below it.
|
||||
Coordinates follow the notebook: X across the array, Y above the ground, Z along
|
||||
the ground.
|
||||
Height above ground, radar pitch and sphere radius follow `MIMO_calibration.ipynb`;
|
||||
several reflector positions may contribute to one calibration. The only diagnostic
|
||||
plot shows the phase of each antenna pair's contribution at the target point,
|
||||
before and after correction, as in cell 5 of `MIMO_calibration.ipynb`: Tx rows,
|
||||
Rx columns, phase relative to the complex sum of each map, with cell values and
|
||||
standard deviation in degrees on a cyclic ±180° scale.
|
||||
With partial antenna geometry, only the specified Tx × Rx pairs contribute to
|
||||
the calculation and phase maps. Acquisition still retains every physical stage;
|
||||
pairs outside that geometry receive unit correction and appear in `unchanged_combos`.
|
||||
The selected reference is already subtracted, and the previous object correction
|
||||
is disabled while these sweeps are acquired. A capture owns an exclusive pipeline
|
||||
pause until it finishes or is cancelled, including during automatic-restart backoff.
|
||||
An explicit **Stop** cancels automatic resumption.
|
||||
|
||||
Object corrections use the same NPZ collection format as other preprocessing sets:
|
||||
S21 holds each pair's multiplier on the captured frequency grid. Metadata retain
|
||||
the capture radar key, antenna geometry, through-set name and content hash, target
|
||||
positions, estimation settings and phase diagnostics. Saving uses the capture's
|
||||
configuration even if the current profile changes during calculation. The next
|
||||
**Start** checks the selected through set's S21 and frequency arrays, geometry and
|
||||
coverage of the requested combinations before applying the correction after notch
|
||||
filtering and before any processor. **Off** disables it. Changing antenna geometry
|
||||
disables an incompatible correction and reloads acquisition without restarting
|
||||
the application. Changing relative permittivity only updates processing settings.
|
||||
|
||||
Legacy `cal_*.npy` / `cal_meta.json` folders are not application calibration sets.
|
||||
Capture a set through **Create…** to use object correction; the legacy files on
|
||||
disk are left intact. Ordinary through/reference NPZ sets need no conversion.
|
||||
|
||||
Older NPZ sets and snapshots remain readable: extra S11 arrays are ignored;
|
||||
new trace files contain only frequency and S21.
|
||||
|
||||
## Runtime interfaces
|
||||
|
||||
Tap rings are optional: an empty `rings.raw_tap.name` or
|
||||
`rings.preprocessed_tap.name` disables that copy and its GUI history. Core processing
|
||||
and single capture still work without taps; object calibration needs the preprocessed
|
||||
tap. All producers share raw publication and hardware retry handling.
|
||||
|
||||
Raw and preprocessed IPC use **RAW3** and **PRP3** respectively: each trace contains
|
||||
float32 frequencies followed by complex64 S21 (12 bytes per point). The complete
|
||||
frame requires collection and per-trace capture windows; its size is `44 + combos * (28 + 12 * points)` bytes.
|
||||
Older RAW2/PPR2 bundles are incompatible and are rebuilt from NPZ sets at Start;
|
||||
update Python and C++ together. Results use RSL3; older RSL2 frames are rejected.
|
||||
|
||||
RF switches use H7992 with 1–4 positions and two GPIO control lines. Profiles use
|
||||
`driver_mode`, GPIO settings and the position count. An old `driver: "h7992"` is accepted;
|
||||
other explicit driver values or enabled inversion are rejected before GPIO opens.
|
||||
Two positions use the existing H7992 mapping `(A,B) = (0,0), (0,1)`; physical
|
||||
wiring and channel numbers follow the profile.
|
||||
|
||||
Single-device LibreVNA acquires the configured `run.combos` in order. Multi-device
|
||||
acquisition keeps the full matrix: SAR3D derives intra-sweep stage timing from
|
||||
the complete list of outputs sharing a capture window.
|
||||
|
||||
## Configuration
|
||||
|
||||
A profile selects the radar model and sweep, switches and acquisition combos,
|
||||
preprocessing sets, antenna geometry, ring sizes and GUI settings. The GUI loads
|
||||
`RADAR_SYSTEM_PROFILE` (set by `start.sh --profile`), otherwise the last profile in
|
||||
`python_app/runtime/gui_session_state.json`, otherwise root `run_config.json`.
|
||||
If a selected external profile cannot be loaded, it reports the error and falls
|
||||
back to the root profile. **Load Config** and **Save Config** manage desktop profiles.
|
||||
|
||||
Hardware examples are in `run_config_examples/`. The root `run_config.json` is
|
||||
intentionally mock; select a hardware profile explicitly on a real device. A
|
||||
change of model, serials, sweep parameters or multi-device switch positions changes
|
||||
the `radar_key` used to find calibration/reference sets; select or capture matching
|
||||
sets before **Start**. Changing the multi-device reference-clock source also resets
|
||||
display history. For RadioPhotonic radar (RPH), identity follows the phase anchors, output
|
||||
band and acquisition/laser settings; executable/TTY paths and LibreVNA-only sweep
|
||||
fields do not affect it. RPH processing frequency controls use its output band.
|
||||
Phase, band and timeout errors are rejected before a running pipeline is stopped.
|
||||
|
||||
Profiles with model `kamil_adc` are rejected with an explicit error: use
|
||||
`rph_radar` and the `radar.rph_radar` section. An inactive `radar.kamil_adc`
|
||||
section left in older saved profiles is ignored; there are no aliases or automatic
|
||||
data migrations. Old RPH calibration/reference sets use a different
|
||||
measurement key and are not selected by the renamed mode.
|
||||
|
||||
### RPH phase calibration
|
||||
|
||||
Use `run_config_examples/run_config_rph_radar.example.json` (E-502) or
|
||||
`run_config_rph_radar_rfg.example.json` (RFG board) for **RadioPhotonic radar (RPH)**.
|
||||
Apply the desired laser settings with **Apply Radar**, then stop acquisition before
|
||||
running the calibration tool:
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m python_app.scripts.rph_radar_calibrate --config run_config.json --sweeps 200
|
||||
.venv/bin/python -m python_app.scripts.rph_radar_calibrate --config run_config.json --sweeps 200 --apply
|
||||
```
|
||||
|
||||
Pass the path to your active RPH profile. The first command reports measured phase
|
||||
anchors; `--apply` captures again and writes only `phase0_rad` and `phase1_rad` in
|
||||
`radar.rph_radar.phase_calibration`. Frequency anchors must already describe the
|
||||
known sweep endpoints. The tool uses current acquisition settings, sends no laser
|
||||
commands, preserves every other JSON byte, and refuses to overwrite a profile
|
||||
edited during capture. Band and point count stay unchanged. `--warmup` controls
|
||||
how many initial sweeps to discard (default 10).
|
||||
|
||||
## Deployment
|
||||
|
||||
On a Raspberry Pi, `deploy/install-daemon.sh` installs a systemd service that runs
|
||||
the application through `start.sh --headless --skip-build` and restarts it on
|
||||
failure. The launcher holds a single-instance lock for the radar, rings and locator
|
||||
port; an interactive launch first asks systemd to stop an active daemon.
|
||||
|
||||
**Updating a device from a USB flash drive.** On the development machine
|
||||
`deploy/pack-usb.sh /path/to/drive` writes the host part of the **last commit**
|
||||
and its update script into `radar_update/` on the flash drive. Uncommitted changes,
|
||||
venv, build outputs, recorded data and local configs are excluded.
|
||||
On the Pi or Jetson, close the program and run, from the drive's folder:
|
||||
|
||||
```bash
|
||||
bash radar_update/update.sh # or: --python 3.13 for a fresh venv
|
||||
```
|
||||
|
||||
Run the updater as the normal device user. It installs the newest archive into
|
||||
`~/Documents/radar_system` (or `RADAR_DIR`), renaming the current version to the
|
||||
next free `radar_system<N>`. It preserves:
|
||||
|
||||
- `run_config.json` and changed/added `run_config_examples/*.json` by independent
|
||||
copy; a conflicting release profile is placed beside the device profile as `.new`;
|
||||
- `python_app/runtime/` by copy, excluding logs and `*.pids`;
|
||||
- `python_app/data*` and extra non-hidden top-level entries absent from the release
|
||||
by hard links, excluding `build`, `graphify-out`, `device_firmware`, `docs` and `RELEASE`;
|
||||
- the venv by independent copy, or a fresh one when `--python` is supplied.
|
||||
|
||||
Saved data is written through replacement or new files, preserving the backup's
|
||||
hard links. `RELEASE` gets a new inode. The updater builds the C++ pipeline and ADC
|
||||
collector, then restarts `radar.service` only if it was running before the update.
|
||||
Keep the device's real hardware profile: the release's root profile is mock.
|
||||
|
||||
For rollback, stop `radar.service` and close the GUI, rename the new installation
|
||||
aside to preserve measurements taken since the update, then rename the chosen
|
||||
backup to `radar_system`. Restart the service if it is normally used.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
QT_QPA_PLATFORM=offscreen .venv/bin/python -m unittest discover -s python_app/tests
|
||||
make check-native
|
||||
make check-sar3d PYTHON=.venv/bin/python
|
||||
make check-gpr
|
||||
ASAN_OPTIONS=detect_leaks=0 make check-native NATIVE_SANITIZERS=address,undefined
|
||||
```
|
||||
|
||||
The tests cover failed writes and settings commits, recordings during manual Save,
|
||||
repeated Start/Stop/Close, web-action cancellation, malformed IPC frames, hardware
|
||||
resource cleanup with mock failures, and an isolated deployment update. Native tests
|
||||
also exercise mutated frames, ring concurrency, locator input and ADC whole-sweep PTY overflow recovery.
|
||||
Firmware host tests check cart timer rollover and SBUS packing; they do not validate
|
||||
physical timing. Real devices, GPIO and the CUDA backend need their own hardware run.
|
||||
|
||||
`check-sar3d` runs the real CUDA processor (`data_acq_and_processing/tests/test_backprojection.py`).
|
||||
It compares it with `sar3d_reference.npz`: compact float32/complex64 inputs, the three
|
||||
maximum projections of the notebook's displayed volume, its peak/level, response
|
||||
widths and background ratio for ten cases. Six cover compensation/normalization,
|
||||
equalization, background, scene and timing. Four are frames of point reflectors in noise
|
||||
for CLEAN, which also compare the objects and their boxes: with and without the background
|
||||
gate, with a scene where the stop at each range's background adds steps, and with a
|
||||
narrowed scene. The box rules are also checked
|
||||
one by one on synthetic objects. The expected values come from unchanged functions in cells 1 and 5
|
||||
of the original notebook, with each frame's previous-collection mean subtracted
|
||||
from its input spectra by NumPy. The same test checks the published
|
||||
status for invalid input, a volume beyond float32, a result too large for the ring slot,
|
||||
an empty and a very weak signal, and that the views are normalized at their peak on
|
||||
finer grids. Without a usable GPU it checks only the CUDA warning and the box rules and
|
||||
skips the rest; the notebook itself is not needed to run it. To regenerate the references,
|
||||
pass the original notebook and the Horns script (both are kept outside the repository):
|
||||
|
||||
```bash
|
||||
.venv/bin/python data_acq_and_processing/tests/generate_backprojection_reference.py \
|
||||
--notebook /path/to/Multi_depthnorm_clean.ipynb \
|
||||
--gpr-script /path/to/Horns_motion_4x4_ground.py
|
||||
```
|
||||
|
||||
The generator also writes `gpr_reference.npz` from the unchanged Horns coherent BP
|
||||
functions, with the notebook equalizer inserted before scalar normalization.
|
||||
`make check-gpr` compares the CPU processor with those images, including motion and
|
||||
previous-collection subtraction. Neither source file is needed to run the tests.
|
||||
|
||||
The archives record source SHA256 hashes and NumPy/SciPy versions. Another build
|
||||
can be checked with `make check-sar3d BUILD_DIR=/path/to/build` or by running
|
||||
`test_backprojection.py Sar3dTest` with
|
||||
`BACKPROJECTION_FIXTURE=/path/to/build/native_tests/backprojection_fixture`.
|
||||
For older GPUs, pass the same `BUILD_DIR` and NVRTC 12 `CUDA_ROOTS` used to build.
|
||||
|
||||
For development, install `pyright` into the venv, then run
|
||||
`.venv/bin/pyright python_app` to check all Python modules and tests.
|
||||
|
||||
## Application architecture
|
||||
|
||||
Invalid device/switch settings and undersized raw/preprocessed slots fail before
|
||||
acquisition starts. Oversized processed results report the required slot size in
|
||||
the log; switching to a mode that fits resumes processing without a restart.
|
||||
|
||||
`AppWindow` assembles controllers and shuts them down. `WindowView` owns layout
|
||||
and typed sections; controllers receive concrete dependencies, without GUI mixins
|
||||
or duplicate window state. `ProfileController` owns startup selection, session
|
||||
state and profile persistence. `RadarSettingsController` applies device limits,
|
||||
while `LiveSettingsController` publishes revisions and updates plots.
|
||||
|
||||
GUI scalar settings use dataclass metadata in `models/gui_profile_schema.py`:
|
||||
profile validation, desktop values, live processing fields, processing log summaries
|
||||
and web forms share those bindings. Hardware fields and geometry have their own
|
||||
typed mappings and validators. Section builders return typed widget groups; their
|
||||
controllers connect signals directly, with no string table of action callbacks.
|
||||
`gui/controllers/config_controller.py` reads and applies scalar metadata without owning
|
||||
the window or acquisition processes. Web batches roll back together when validation
|
||||
or live-file publication fails; desktop settings also revert to the last publication.
|
||||
Profile loading blocks editor signals until all settings are populated.
|
||||
|
||||
`HistoryController` owns retained acquisitions; `PlotController` selects the view
|
||||
and supplies independent B-scan, GPR, trace and SAR renderers with typed settings.
|
||||
Renderers own their plots and display caches, without access to the acquisition
|
||||
processes. Every plot change passes through `PlotController`, which counts changes
|
||||
of the shown image for the browser stream. The display shows the newest acquisition,
|
||||
including results read while stopping. Stopped B-scan replay uses the last run's
|
||||
ring, matches exact acquisition keys and the requested settings revision, and
|
||||
cancels when that run changes; sweeps the processor does not return are left out
|
||||
rather than mixing settings in one image. With no retained preprocessed sweeps
|
||||
(including disabled taps), the processor recomputes its own retained history.
|
||||
Capture previews preserve the selected processing view.
|
||||
`PipelineController` owns processes, ring readers, polling/recovery, Single Capture
|
||||
and replay. Its pause prevents recovery from starting a producer over a hardware
|
||||
capture; Stop or Close cancels any pending resumption, including during a drain.
|
||||
Validation rejects Start before stopping existing processes. Pauses resume only
|
||||
continuous acquisition, retry failed resumption with the usual backoff, and release
|
||||
ownership before presenting capture errors. GPIO errors are logged without dialogs.
|
||||
Rendering failures leave acquisition and the locator running and log each cause once;
|
||||
repeated ring-read failures retain their separate recovery and stop policy.
|
||||
`PreprocessController` owns calibration/reference sessions and asset selection.
|
||||
One `SequentialCaptureSession` handles both current settings and a list of sweep
|
||||
variants; a single variant adds no extra acquisition or reconfiguration. Multi-device
|
||||
collection always captures one complete synchronized cycle. The native
|
||||
`DataPreprocessor` owns all three bundles in `preprocessing/bundles`: through,
|
||||
through-calibrated reference and object correction, applied in that order with notch
|
||||
filtering before object correction.
|
||||
Object capture checks both acquisition processes and releases its pause only after
|
||||
completion or cancellation; losing the producer reports a failure immediately.
|
||||
`SnapshotController` coordinates manual exports, paused Clear/Remove operations
|
||||
and Start + Record to Disk.
|
||||
History deletion is retained before publishing it to the processor, so a failed
|
||||
settings write can be retried without restoring deleted sweeps. `RecordingController`
|
||||
owns a bounded writer session; its disk thread receives data and storage only.
|
||||
A processor restart during Start + Record to Disk does not finish the newly armed session.
|
||||
Temporary references and neutral sets clear retained processing history without
|
||||
restarting the locator while the pipeline is paused.
|
||||
|
||||
`WebBridge` queues browser actions onto Qt and exposes immutable status/frame
|
||||
snapshots. Clear and Remove use separate action endpoints; live settings only edit
|
||||
settings. Trace display options (including its Y range) do not publish processing
|
||||
revisions. A timeout cancels queued actions; an action already running reports that
|
||||
state without inviting a duplicate request. The GPIO controller invokes the same
|
||||
reference-capture action as desktop and web, and ignores queued presses after Close.
|
||||
`LogController` batches worker logs and returns web-action errors to the browser
|
||||
without opening a desktop modal. Stop/Close also cancels delayed launcher startup.
|
||||
|
||||
GUI profiles, run configs, session state and live processing settings use shared
|
||||
atomic writers with unique temporary files. Profiles and run/session settings sync
|
||||
their file and directory; frequent live settings avoid extra disk syncs. A failed
|
||||
write before replacement leaves the previous file intact. The launcher and supervisor
|
||||
share acquisition producer selection.
|
||||
| Каталог | Что |
|
||||
| --- | --- |
|
||||
| `python_app/` | окно программы, драйверы приборов, веб-интерфейс, хранение данных, тесты |
|
||||
| `data_acq_and_processing/` | C++: препроцессор, процессор с режимами обработки и локатором, коллектор E-502, тесты |
|
||||
| `run_config_examples/` | профили под каждое железо |
|
||||
| `run_configs/` | профили для загрузки из веб-интерфейса |
|
||||
| `deploy/` | служба systemd и обновление с флешки |
|
||||
| `device_firmware/` | прошивки LibreVNA, контроллера ключей и пульта тележки |
|
||||
| `docs/` | документация |
|
||||
| `start.sh` | запуск: окружение Python, сборка, программа |
|
||||
| `run_config.json` | профиль по умолчанию — симулятор |
|
||||
| `Makefile` | сборка и тесты C++ |
|
||||
| `requirements.txt` | Python-пакеты |
|
||||
|
||||
Reference in New Issue
Block a user