deploy: USB update kit for the Pi and Jetson

pack-usb.sh writes the host part of the last commit and update.sh to a flash drive. update.sh installs it over ~/Documents/radar_system, keeps configs, calibrations and data, and leaves the previous version next to it as radar_system<N>.
2026-10-02 16:21:16 +03:00
parent d283ecfdf1
commit 515049d2ed
3 changed files with 262 additions and 1 deletions
@@ -70,7 +70,7 @@ controls plus a pixel-identical live plot through an embedded web UI.
 | `data_acq_and_processing/` | C++ pipeline: `sweep_orchestrator/`, `preprocessing/`, `processing/` (including `locator/`), shared `common_cpp/` (config, IPC rings), and vendored `third_party/` (Eigen, nlohmann/json). |
 | `python_app/` | Python application: `gui/`, `orchestration/` (process supervisor, shared-memory readers, config), `hardware_full/` (device drivers and services), `webui/`, `storage/`, `models/`, `workflows/`, `scripts/` (producers and tools), and `tests/`. |
 | `run_config_examples/` | Ready-made configuration profiles, one per hardware setup. |
-| `deploy/` | systemd daemon installer for Raspberry Pi appliances. |
+| `deploy/` | systemd daemon installer, and the USB update kit for the Pi and Jetson (`pack-usb.sh`, `update.sh`). |
 | `build/bin/` | Compiled C++ binaries. |
 | `device_firmware/`, `docs/` | Device firmware and vendor manuals. |

@@ -136,6 +136,17 @@ for the CUDA-backed `sar3d` processor:
 ./start.sh --profile run_config_examples/run_config_replay_vna.example.json
 ```

+Ubuntu 22.04 ships Python 3.10, and the host parses the radar stream in Python:
+the same parsing code ran 1.6 times slower on 3.10 than on 3.13 (the Pi's version)
+in a benchmark. Use Python 3.13 for the venv, installed with uv so the system
+`python3` stays untouched (apt depends on it):
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh   # then open a new terminal
+uv python install 3.13
+bash radar_update/update.sh --python 3.13         # see Deployment
+```
+
 ## Running without hardware (emulation)

 Two ways to run the full pipeline with no radar attached:
@@ -192,6 +203,25 @@ the application headless and restarts it on failure. A single-instance lock ensu
 the interactive GUI and the daemon never run at the same time, since they share the
 radar, the shared-memory rings, and the locator port.

+**Updating a device from a USB flash drive.** On the development machine
+`deploy/pack-usb.sh` writes the host part of the last commit (no venv, build, data
+or local configs) and the update script into `radar_update/` on the flash drive.
+On the Pi or Jetson, close the program and run, from the drive's folder:
+
+```bash
+bash radar_update/update.sh                  # or: --python 3.13 for a fresh venv
+```
+
+It installs the newest archive into `~/Documents/radar_system` (or `RADAR_DIR`);
+the current version is renamed to `radar_system<N>`, the next free number. Kept from
+it: `run_config.json`, profiles in `run_config_examples/` changed or added on the
+device (the release version of a changed one lands next to it as `.new`),
+`python_app/runtime/` without logs, `python_app/data*` and any other top-level
+folder that is not part of the release, as hard links. Then it carries the venv over
+(or builds a new one with `--python`), builds the C++ part and restarts
+`radar.service` if it was running. To go back:
+`rm -rf radar_system && mv radar_system<N> radar_system`.
+
 ## Tests

 ```bash
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# Put the project on a USB flash drive for deploy/update.sh: the host part of the
+# last commit as an archive (no venv, build, data or local configs) and the update
+# script, both in <drive>/radar_update/.
+#
+#     deploy/pack-usb.sh [DRIVE]
+#
+# DRIVE defaults to the only mounted removable drive.
+set -euo pipefail
+
+PROJECT_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
+# What runs on the Pi or Jetson; device_firmware/ and docs/ stay on this machine.
+HOST_PATHS=(data_acq_and_processing deploy python_app run_config_examples Makefile
+            requirements.txt run_config.json start.sh locator_test_client.py)
+
+drive="${1:-}"
+if [[ -z "${drive}" ]]; then
+    mapfile -t drives < <(lsblk -nlo HOTPLUG,MOUNTPOINT | sed -n 's|^ *1 \+\(/.*\)$|\1|p')
+    if (( ${#drives[@]} != 1 )); then
+        echo "Found ${#drives[@]} mounted removable drives; name one: $0 /path/to/drive" >&2
+        printf '  %s\n' "${drives[@]}" >&2
+        exit 1
+    fi
+    drive="${drives[0]}"
+fi
+if [[ ! -d "${drive}" || ! -w "${drive}" ]]; then
+    echo "Not a writable directory: ${drive}" >&2
+    exit 1
+fi
+
+cd "${PROJECT_ROOT}"
+if [[ -n "$(git status --porcelain -- "${HOST_PATHS[@]}")" ]]; then
+    echo "Note: uncommitted changes are left out, the archive holds the last commit." >&2
+fi
+release="radar_system-$(git log -1 --format=%cd --date=format:%Y%m%d-%H%M)-$(git rev-parse --short HEAD)"
+out="${drive}/radar_update"
+mkdir -p "${out}"
+git archive --format=tar.gz --prefix=radar_system/ -o "${out}/${release}.tar.gz" HEAD "${HOST_PATHS[@]}"
+git show HEAD:deploy/update.sh > "${out}/update.sh"
+sync
+
+echo "Written to ${out}:"
+echo "  ${release}.tar.gz"
+echo "  update.sh"
+echo "On the device, from the drive's folder:  bash radar_update/update.sh"
@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+# Install a release archive made by deploy/pack-usb.sh over the project on this
+# device. The code is replaced; configs, calibrations and data are kept; the current
+# version stays next to it as radar_system<N> (the next free number) for a rollback.
+#
+#     bash update.sh [--python VERSION|PATH] [ARCHIVE]
+#
+# ARCHIVE   the newest radar_system-*.tar.gz next to this script by default
+# --python  build a fresh venv with this interpreter, e.g. 3.13 (found with uv or as
+#           python3.13); without it the current venv is carried over
+# The project is radar_system in the Documents folder unless RADAR_DIR names it.
+set -Eeuo pipefail
+shopt -s nullglob
+
+say() { echo "[update] $*"; }
+fail() { echo "[update] $*" >&2; exit 1; }
+
+(( EUID != 0 )) || fail "Run it as your usual user, not with sudo: the project files must stay yours."
+if [[ -n "${RADAR_DIR:-}" ]]; then
+    target="${RADAR_DIR}"
+else
+    target="${HOME}/Documents/radar_system"
+    localized="$(xdg-user-dir DOCUMENTS 2>/dev/null || true)/radar_system"   # e.g. ~/Документы
+    if [[ ! -e "${target}" && -d "${localized}" ]]; then target="${localized}"; fi
+fi
+[[ ! -L "${target}" ]] || fail "${target} is a symlink; set RADAR_DIR to the real directory."
+target="$(realpath -m -- "${target}")"
+script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+python_request=""
+archive=""
+while (( $# > 0 )); do
+    case "$1" in
+        --python) python_request="${2:?--python needs a version or a path}"; shift 2 ;;
+        -h|--help) sed -n '2,12p' "${BASH_SOURCE[0]}"; exit 0 ;;
+        *) archive="$1"; shift ;;
+    esac
+done
+cd "${HOME}"   # never stand in the directory that is about to be renamed
+
+# --- check everything before touching the project ------------------------------
+if [[ -z "${archive}" ]]; then
+    releases=("${script_dir}"/radar_system-*.tar.gz)
+    (( ${#releases[@]} > 0 )) || fail "No radar_system-*.tar.gz next to ${BASH_SOURCE[0]}."
+    archive="${releases[-1]}"   # the names sort by commit date
+fi
+listing="$(tar -tzf "${archive}")" || fail "Damaged archive: ${archive}"
+grep -qx 'radar_system/start.sh' <<< "${listing}" || fail "Not a radar_system release: ${archive}"
+if [[ ! -e "${target}" ]]; then
+    read -r -p "[update] No project at ${target}. Install a fresh one there? [y/N] " answer
+    [[ "${answer}" == [yY]* ]] || exit 1
+fi
+
+python=""
+if [[ -n "${python_request}" ]]; then
+    if [[ "${python_request}" =~ ^3\.[0-9]+$ ]]; then
+        uv="$(command -v uv || echo "${HOME}/.local/bin/uv")"
+        if [[ -x "${uv}" ]]; then
+            python="$("${uv}" python find "${python_request}" 2>/dev/null || true)"
+        fi
+        [[ -n "${python}" ]] || python="$(command -v "python${python_request}" || true)"
+        [[ -n "${python}" ]] || fail "Python ${python_request} not found; install it: uv python install ${python_request}"
+    else
+        python="$(command -v -- "${python_request}" || true)"
+        [[ -n "${python}" ]] || fail "Interpreter not found: ${python_request}"
+    fi
+    say "venv interpreter: ${python} ($("${python}" --version 2>&1))"
+fi
+
+parent="$(dirname -- "${target}")"
+mkdir -p -- "${parent}"
+staging="$(mktemp -d "${parent}/.radar_update.XXXXXX")"
+trap 'rm -rf -- "${staging}"' EXIT
+tar -xzf "${archive}" -C "${staging}"
+
+# --- stop everything that runs from the project ----------------------------------
+daemon_stopped=0
+if command -v systemctl >/dev/null && systemctl is-active --quiet radar.service; then
+    say "stopping radar.service"
+    sudo systemctl stop radar.service
+    daemon_stopped=1
+fi
+start_daemon() {
+    if (( daemon_stopped )); then
+        say "starting radar.service"
+        sudo systemctl start radar.service
+    fi
+}
+busy=()
+for proc in /proc/[0-9]*; do
+    pid="${proc#/proc/}"
+    [[ "${pid}" == "$$" || "${pid}" == "${PPID}" ]] && continue
+    cwd="$(readlink "${proc}/cwd" 2>/dev/null)" || continue
+    [[ "${cwd}" == "${target}" || "${cwd}" == "${target}/"* ]] && busy+=("${pid}")
+done
+if (( ${#busy[@]} > 0 )); then
+    echo "[update] The program still runs from ${target}; close it and run the update again:" >&2
+    ps -o pid=,args= -p "$(IFS=,; echo "${busy[*]}")" >&2 || true
+    start_daemon
+    exit 1
+fi
+
+# --- swap the versions --------------------------------------------------------------
+backup=""
+if [[ -e "${target}" ]]; then
+    n=1
+    while [[ -e "${target}${n}" ]]; do n=$((n + 1)); done
+    backup="${target}${n}"
+    mv -- "${target}" "${backup}"
+    on_failure() {
+        echo "[update] FAILED. The previous version is ${backup}; to go back to it:" >&2
+        echo "    rm -rf '${target}' && mv '${backup}' '${target}'" >&2
+        if (( daemon_stopped )); then echo "    sudo systemctl start radar.service" >&2; fi
+    }
+    trap on_failure ERR
+fi
+mv -- "${staging}/radar_system" "${target}"
+say "installed ${archive##*/} into ${target}"
+
+# --- keep this device's state ----------------------------------------------------------
+if [[ -n "${backup}" ]]; then
+    say "previous version: ${backup}"
+    kept=()
+    # The active profile; the release carries the development one (mock driver).
+    if [[ -f "${backup}/run_config.json" ]]; then
+        cp -a -- "${backup}/run_config.json" "${target}/run_config.json"
+        kept+=(run_config.json)
+    fi
+    # Profiles changed or added on this device win; a changed one gets the release
+    # version next to it as .new.
+    renamed=()
+    for profile in "${backup}"/run_config_examples/*.json; do
+        shipped="${target}/run_config_examples/${profile##*/}"
+        if [[ -f "${shipped}" ]]; then
+            cmp -s -- "${profile}" "${shipped}" && continue
+            mv -- "${shipped}" "${shipped}.new"
+            renamed+=("${profile##*/}")
+        fi
+        cp -a -- "${profile}" "${shipped}"
+        kept+=("run_config_examples/${profile##*/}")
+    done
+    # Runtime state (last profile, live settings), without logs and pid files.
+    if [[ -d "${backup}/python_app/runtime" ]]; then
+        cp -a -- "${backup}/python_app/runtime" "${target}/python_app/runtime"
+        rm -rf -- "${target}/python_app/runtime/logs" "${target}/python_app/runtime/"*.pids
+        kept+=(python_app/runtime)
+    fi
+    # Data: calibrations, references, snapshots, and every top-level folder that is
+    # not part of the release. Hard links are instant and take no space; the program
+    # writes new files instead of rewriting old ones.
+    data=()
+    for entry in "${backup}"/python_app/data*; do data+=("python_app/${entry##*/}"); done
+    for dir in "${backup}"/*/; do
+        name="$(basename -- "${dir}")"
+        case "${name}" in build|graphify-out|device_firmware|docs) continue ;; esac
+        [[ -e "${target}/${name}" ]] || data+=("${name}")
+    done
+    for entry in "${data[@]}"; do
+        cp -al -- "${backup}/${entry}" "${target}/${entry}"
+        kept+=("${entry}")
+    done
+    say "kept from ${backup##*/}: ${kept[*]:-nothing}"
+    if (( ${#renamed[@]} > 0 )); then
+        say "profiles changed on this device are kept, the release ones are next to them as .new: ${renamed[*]}"
+    fi
+fi
+
+# --- Python environment and C++ build ------------------------------------------------
+venv="${target}/.venv"
+if [[ -z "${python}" && -n "${backup}" && -x "${backup}/.venv/bin/python" ]]; then
+    say "carrying the venv over"
+    cp -al -- "${backup}/.venv" "${venv}"
+    "${venv}/bin/python" -m pip install --quiet -r "${target}/requirements.txt"
+else
+    python="${python:-$(command -v python3)}"
+    say "creating a venv with ${python} (downloads the packages)"
+    "${python}" -m venv "${venv}"
+    "${venv}/bin/python" -m pip install --upgrade pip
+    "${venv}/bin/python" -m pip install -r "${target}/requirements.txt"
+fi
+say "building the C++ part"
+make -C "${target}" -j"$(nproc)" all
+
+echo "${archive##*/}" > "${target}/RELEASE"
+trap - ERR
+start_daemon
+say "done ($("${venv}/bin/python" --version 2>&1)). Start it: cd '${target}' && ./start.sh"