Commit graph

2176 commits

Author SHA1 Message Date
Roland
ef6d470f52 Antenna calculator: harden wavelength/MHz input handlers.
Clear stale results when wavelength is emptied or invalid, and reject
non-finite overflow values so NaN is never displayed or persisted.
2026-07-15 19:57:48 -06:00
Roland
2fbf0a9b0f Antenna calculator: extract pure math, fix bugs, add tests.
Verify ARRL wire-antenna formulas (936/468/234/585), fix feet/inches
rollover ("N ft 12 in"), make wavelength edits update lengths, and cover
the math with 50 unit tests.
2026-07-15 19:57:48 -06:00
Roland
bad16684a4
install_map_tools: fix pmtiles x86_64 asset name (was 404 on amd64).
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
go-pmtiles release assets are named ..._Linux_x86_64.tar.gz, not _amd64;
the #538 consolidation used $arch=amd64 in the URL, so every amd64 build
(Debian Live ISO, docker) 404'd installing pmtiles and the wrolpi chroot
hook failed.  Use x86_64 for the URL arch on x86_64 hosts; the SHA var and
the aarch64 (arm64) path are unchanged and already correct.

Verified: both v1.30.1 tarballs download and match the pinned SHAs
(x86_64 23a2a222..., arm64 a1f9f42d...).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 16:20:53 -06:00
lrnselfreliance
899ab13964
Consolidate map-tool install into wrolpi/scripts/install_map_tools.sh (#538)
The pmtiles CLI + tippecanoe install was duplicated across three places
(scripts/install_protomaps.sh, the Debian Live hook, and pi-gen), so a version
bump or logic change had to be made in all of them in lockstep. Move it into a
single shared, version-guarded, SHA-pinned installer and have all three callers
invoke it.

- New wrolpi/scripts/install_map_tools.sh: installs pmtiles (per-arch binary,
  SHA-pinned for amd64 and arm64) and builds tippecanoe (source tarball,
  arch-independent SHA). Version-guarded, so an image that already baked in the
  pinned versions makes the on-device run a no-op until a version bumps.
- pi-gen 03-run-chroot.sh, the Debian hook, and install_protomaps.sh now just
  call the shared script. This also adds SHA pinning to the on-device pmtiles
  path (previously unpinned).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 15:55:06 -06:00
lrnselfreliance
f621e0bd5c
Bake tippecanoe into the Raspberry Pi image (#537)
The Debian Live image builds tippecanoe at image time, but pi-gen did not,
so a fresh Pi compiled it on first boot/upgrade via install_protomaps.sh --
a multi-minute, CPU-heavy C++ build on the weakest hardware WROLPi targets.
This is the same "Debian bakes it, pi-gen doesn't" gap recently fixed for
the React app build.

Build tippecanoe 2.79.0 in the stage2 chroot, mirroring the Debian hook
(SHA-pinned source tarball, which is arch-independent). Add the two missing
build deps (libsqlite3-dev, zlib1g-dev) to the package list; build-essential
was already present.

With tippecanoe present at the pinned version, install_protomaps.sh's version
guard skips the build on first boot and on upgrades, so the on-device compile
only recurs when the pinned version actually bumps.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 15:48:25 -06:00
lrnselfreliance
944d95c717
Bake the app build into images; rebuild on-device only when stale (#536)
* Bake the app build into images; rebuild on-device only when stale

Previously the Raspberry Pi image shipped without app/build/ and compiled
the React bundle on first boot via repair.sh -- a multi-minute, memory-hungry
step that can OOM a 2GB Pi.  The Debian Live ISO already baked the build in,
so the two paths had drifted.

Add scripts/build_app.sh as the single source of truth: it hashes the build
inputs (src/, public/, package.json, package-lock.json, tsconfig.json) and
runs `npm run build` (with GENERATE_SOURCEMAP=false) only when build/ is
missing or those inputs changed, then writes build/.build-stamp.

Wire all three call sites to it:
- pi-gen 03-run-chroot.sh now builds after npm install, baking build/ into
  the Pi image like the Debian hook already does.
- debian-live 9999-wrolpi.hook.chroot uses it so the baked build is stamped.
- repair.sh uses it, so first-boot/manual-repair/install/upgrade all become
  no-ops when the shipped build is current.

Result: fresh images ship a stamped build/, so a weak Pi never compiles on
boot or repair; an on-device rebuild happens only when an upgrade actually
changes app/.  build_app.sh never runs npm install, so repair.sh keeps its
offline guarantee.

* build_app.sh: rebuild when built output is incomplete, not just missing

Addresses review feedback: a matching stamp only proves the inputs are
unchanged, not that the output survived.  repair.sh exists to fix corruption,
so a build/ with index.html + stamp but a missing JS/CSS asset must still
rebuild.  Verify every file referenced by build/asset-manifest.json exists
(and that the manifest itself parses); rebuild otherwise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 15:19:25 -06:00
lrnselfreliance
6266052998
Frontend: npm audit fix for shell-quote, form-data, and ws. (#535)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Applies non-breaking transitive lockfile bumps for three still-present
advisories on the app dependency tree:

* shell-quote 1.8.0 -> 1.10.0 (critical: unescaped newlines in quote())
* form-data 3.0.4 -> 3.0.5 / 4.0.4 -> 4.0.6 (high: CRLF injection via
  unescaped multipart field names/filenames)
* ws 7.5.10 -> 7.5.12 / 8.18.3 -> 8.21.1 (high: memory-exhaustion DoS)

package-lock.json only; no package.json / direct-dependency changes. Runs
`npm audit fix` without --force, so no breaking dev-chain (react-scripts)
upgrades are pulled in. Verified on trixie/Node 20: jest 760/760 and the
production build both green.

Supersedes the stale dependabot PRs #487 (ws), #486 (form-data), and
equal-or-newer patched versions.

Co-authored-by: Roland <roland.henrie@o2x.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 20:33:04 -06:00
Roland
9e92732a0c
kiwix: drop --address 0.0.0.0 so kiwix-serve works on trixie.
Debian 13 ships kiwix-tools 3.7.0, whose kiwix-serve rejects the literal
--address 0.0.0.0 ("IP address is not available on this system: 0.0.0.0")
and exits 1, so wrolpi-kiwix fails to start on a fresh trixie image.
kiwix-serve binds all interfaces by default when --address is omitted, on
both the old (bookworm) and new (trixie) versions, so dropping the flag is
portable.

Verified on a trixie Pi (10.0.0.8): wrolpi-kiwix active, serving 200 on
9085 and via Caddy on 8085.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 20:32:09 -06:00
dependabot[bot]
6b7bd07976
Bump the npm-dependencies group across 1 directory with 6 updates (#534)
Bumps the npm-dependencies group with 6 updates in the /app directory:

| Package | From | To |
| --- | --- | --- |
| [query-string](https://github.com/sindresorhus/query-string) | `9.4.0` | `9.4.1` |
| [react-colorful](https://github.com/omgovich/react-colorful) | `5.7.0` | `5.8.0` |
| [react-hotkeys-hook](https://github.com/JohannesKlauss/react-keymap-hook) | `5.3.2` | `5.3.3` |
| [react-qr-code](https://github.com/rosskhanas/react-qr-code) | `2.0.21` | `2.2.0` |
| [semantic-ui-offline](https://github.com/arthurlacoste/semantic-ui-offline) | `2.5.0` | `2.5.1` |
| [three](https://github.com/mrdoob/three.js) | `0.184.0` | `0.185.1` |



Updates `query-string` from 9.4.0 to 9.4.1
- [Release notes](https://github.com/sindresorhus/query-string/releases)
- [Commits](https://github.com/sindresorhus/query-string/compare/v9.4.0...v9.4.1)

Updates `react-colorful` from 5.7.0 to 5.8.0
- [Release notes](https://github.com/omgovich/react-colorful/releases)
- [Changelog](https://github.com/omgovich/react-colorful/blob/master/CHANGELOG.md)
- [Commits](https://github.com/omgovich/react-colorful/compare/5.7.0...v5.8.0)

Updates `react-hotkeys-hook` from 5.3.2 to 5.3.3
- [Release notes](https://github.com/JohannesKlauss/react-keymap-hook/releases)
- [Changelog](https://github.com/JohannesKlauss/react-hotkeys-hook/blob/main/CHANGELOG.md)
- [Commits](https://github.com/JohannesKlauss/react-keymap-hook/compare/v.5.3.2...v5.3.3)

Updates `react-qr-code` from 2.0.21 to 2.2.0
- [Release notes](https://github.com/rosskhanas/react-qr-code/releases)
- [Changelog](https://github.com/rosskhanas/react-qr-code/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rosskhanas/react-qr-code/compare/2.0.21...2.2.0)

Updates `semantic-ui-offline` from 2.5.0 to 2.5.1
- [Commits](https://github.com/arthurlacoste/semantic-ui-offline/commits/v2.5.1)

Updates `three` from 0.184.0 to 0.185.1
- [Release notes](https://github.com/mrdoob/three.js/releases)
- [Commits](https://github.com/mrdoob/three.js/commits)

---
updated-dependencies:
- dependency-name: query-string
  dependency-version: 9.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: react-colorful
  dependency-version: 5.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: react-hotkeys-hook
  dependency-version: 5.3.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: react-qr-code
  dependency-version: 2.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
- dependency-name: semantic-ui-offline
  dependency-version: 2.5.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-dependencies
- dependency-name: three
  dependency-version: 0.185.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-14 20:25:34 -06:00
dependabot[bot]
cb265b99ed Bump react-router from 7.16.0 to 7.18.1 in /app
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Bumps [react-router](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router) from 7.16.0 to 7.18.1.
- [Release notes](https://github.com/remix-run/react-router/releases)
- [Changelog](https://github.com/remix-run/react-router/blob/react-router@7.18.1/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/remix-run/react-router/commits/react-router@7.18.1/packages/react-router)

---
updated-dependencies:
- dependency-name: react-router
  dependency-version: 7.18.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-14 16:48:48 -06:00
dependabot[bot]
b0ae532a99 Bump @artsy/fresnel from 6.1.0 to 8.5.0 in /app
Bumps [@artsy/fresnel](https://github.com/artsy/fresnel) from 6.1.0 to 8.5.0.
- [Release notes](https://github.com/artsy/fresnel/releases)
- [Changelog](https://github.com/artsy/fresnel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/artsy/fresnel/compare/v6.1.0...v8.5.0)

---
updated-dependencies:
- dependency-name: "@artsy/fresnel"
  dependency-version: 8.5.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-14 16:48:26 -06:00
Roland
3967a49989
SQLite: make the browse pages index-only.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.

- Browse archives with file_group as the outer table (CROSS JOIN) so
  SQLite walks the date index and probes archive's covering index;
  driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
  rowid is present in every index, the path expression forced full-row
  reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
  mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
  sqlite_stat1 at all after the rebuild.

Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 07:46:29 -06:00
Roland
b810ab2fc0
Archives: an unpaired-surrogate title must not kill the modeler.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Scraped pages can carry half an emoji in their titles; sqlite3 refuses
to store lone surrogates, so one poisoned title failed archive_modeler's
batch commit and ended the whole modeling run on 10.0.0.9.  Every
archive after it stayed unmodeled, and cleanup then deleted their
'empty' domain collections (1,670 -> 578).

Strip surrogates at every point Archive ingests scraped strings (titles,
author, description, url, article text), and make the modeler skip a
batch whose commit fails instead of dying.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 19:25:30 -06:00
Roland
9481ad2d2b
SQLite: keep the indexing batch out of a long transaction.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
apply_indexers held one ORM session across a whole batch of indexers --
minutes of readability/subprocess work -- so its final commit had to
upgrade a stale read snapshot to a write.  SQLite answers that with
"database is locked" (SQLITE_BUSY_SNAPSHOT, which busy_timeout does not
apply to) whenever any other connection commits a write in the meantime,
which killed the refresh on 10.0.0.9 at 75%.

Read the batch as plain columns, index on transient FileGroups with no
open transaction, then write the results in one short BEGIN IMMEDIATE
transaction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 16:37:47 -06:00
Roland
e01795ad7e
Debian 13 (trixie) rebase: docker, debian-live-config, and pi-gen builds.
All three build targets move from Debian 12/bookworm to Debian 13/trixie
for fresh installs; existing Debian 12 systems remain supported in place.

Docker: api/help on python:3.13-trixie, controller on 3.13-slim-trixie,
app on node:20-trixie (matches trixie's nodejs 20), samba/zim on
debian:13.  Full backend suite passes on Python 3.13 in the api container.

debian-live-config: --distribution trixie; postgres teardown for the
SQLite cutover (pg_dropcluster hook block removed, wrolpi-bootstrap.sh no
longer creates a cluster or calls the deleted initialize_api_db.sh — the
API creates wrolpi.db itself on first start).  README documents the
SQLite layout and drops Etcher recommendations.

pi-gen: upstream has no trixie branch — the arm64 branch now defaults to
RELEASE=trixie; KEYBOARD_KEYMAP=en_US is rejected by trixie's
console-setup (ckbcomp validates), use 'us'; stop stamping a Postgres
.pgpass into /etc/skel.

Package lists (both targets): postgresql-15/postgresql-doc/libpq-dev
deleted; libtiff5-dev renamed to libtiff-dev; software-properties-common
and cpufrequtils removed from trixie (Controller CPU throttle falls back
to sysfs, dead cpufreq-set sudoers rule removed); locales and gnupg added
(no longer arrive transitively under --apt-recommends false); sqlite3 CLI
added for reset_tags.sh.

help.sh: replace the Postgres service/port/socket/cluster/psql checks
(all false FAILEDs after the SQLite cutover) with SQLite checks:
wrolpi.db exists, PRAGMA quick_check, file_group table + row count, run
as the wrolpi user so root never creates the -shm sidecar.

Both build scripts take a WROLPI_BUILD_DIR override so development builds
can run beside the release pipeline's build trees; the sudo re-exec
passes it through explicitly (a plain sudo would drop it).

test_common.py: the truncate tests hard-coded sys.getsizeof results from
the 3.11 object layout (str/list shrank in 3.12); assert the contract
(fits budget, prefix, monotonic in budget) so they pass on 3.11 and 3.13.

Verified: trixie ISO (3.7G) and pi-gen desktop image (4.3G) both built
from scratch and passed their sanity checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 15:43:08 -06:00
Roland
943b3a0b60
SQLite: keep large collection/file queries under the expression-tree depth limit.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Collection.batch_from_config OR'd a (name AND kind) pair per collection, so
importing more than ~1000 collections (e.g. a 1,670-domain domains.yaml)
failed with "Expression tree is too large (maximum depth 1000)".  Query a
flat name IN (...) and match (name, kind) in Python instead.

get_bulk_tag_preview had the same unbounded pattern (an OR'd LIKE per stem);
chunk the query at 500 stems.

Found upgrading 10.0.0.9 (decay).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 15:45:27 -06:00
Roland
d9303a0ea8
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.

Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
  foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
  sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
  adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
  Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
  A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
  Stemming contract pinned by dedicated tests (watch/watches/watched/
  watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
  baseline and test DBs; fixes the channel video_count DELETE/UPDATE
  bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
  written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
  network-filesystem refusal; never creates/migrates/trusts a database
  beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
  revision ids are dates (2026_07_10_1350) so filenames self-sort and
  alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
  HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
  generate_series histograms->strftime GROUP BY, DISTINCT ON->
  row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
  NULLS FIRST/LAST, LIMIT-before-OFFSET).  Simple statements converted
  to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
  channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
  after a lossy import so partial imports can never overwrite YAML
  configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
  treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
  (BIGINT is not a rowid alias; FTS5 content_rowid requires it).

Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed.  upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.

Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index.  QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.

Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.

Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:09:37 -06:00
Roland
7da89d883e
Bump version to 0.27.2-beta
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-07-09 16:03:43 -06:00
Roland
715035e5dc
help.sh: port 80 is the Controller's, not Caddy's; fix IPv4 port regex.
The Caddyfile only binds 443/8084/8085/8086 and reverse-proxies to the
Controller on port 80, which the Controller health check already covers.
The IPv4 alternative in the port grep was missing its colon, so only the
IPv6 ":::" listeners ever matched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 16:00:53 -06:00
Roland
0e67cd6c74 help.sh: only print OK/FAILED/Note lines; robust hostname SAN check.
Connect to Postgres directly instead of via `sudo -i -u wrolpi` so
sudo/PAM stop spamming the journal when help.sh runs as a systemd
service.  Drop the raw pg_lsclusters table and the per-file PMTiles
listing.  Use `openssl x509 -checkhost` for the hostname SAN check so a
short hostname (e.g. "pi") cannot substring-match another SAN and
wrapped SAN lines cannot cause a false failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 16:00:43 -06:00
Roland
e42e4ed06e Controller: add help.sh to the Maintenance Scripts.
A new wrolpi-help-script.service oneshot runs /opt/wrolpi/help.sh so the
Controller UI can run system diagnostics and show the output from the
journal.  The confirmation modal hides the warnings block for scripts
with no warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 16:00:43 -06:00
Roland
d6b019c24e Certificates: help.sh validates the CA and leaf certs; fix renew exec bit.
help.sh now detects invalid Caddy certificates: expired or unparseable
CA/leaf certs, a leaf expiring within 30 days, a leaf that does not match
its private key or the current hostname, and Caddy serving a stale cert
after regeneration.  It also checks that the renewal script is executable
and its timer is enabled.

scripts/renew_certificates.sh was committed without the executable bit,
so wrolpi-cert-renew.service failed with "Permission denied".  Set the
bit and make repair.sh restore executable bits on all scripts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 16:00:43 -06:00
Roland
b172a4c9d6 Calculators: never store NaN as the Kelvin temperature.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Clearing the temperature input stored parseFloat('') = NaN, rendering
every Kelvin cell as NaN and persisting the invalid value in localStorage
across reloads. Clearing now falls back to room temperature, non-numeric
input is ignored, and a bad stored value decodes to the default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 15:27:34 -06:00
Roland
53f9386c85 Theme: fix white corner cell on inverted definition tables.
Semantic UI paints a definition table's empty corner cell white to fake a
"cut-out" corner against a white page. Our dark theme has a dark page, so
the corner rendered as a white square. Override the background and border
box-shadows to transparent for inverted tables only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 15:27:34 -06:00
Roland
ee0e85a69c Calculators: add 4-Wire Kelvin segment to the Electrical Calculator.
Shows the expected millivolt drop across a single wire at 1A, 5A, and 10A
so a wire can be verified with a known current and sense leads. Shares the
length, units, and wire type with the Power Loss calculator (state lifted
to the parent, same localStorage keys). Length is not doubled because the
sense leads measure across one conductor.

Includes a wire temperature input (default room temperature, °F/°C follows
the SAE toggle, stored canonically in Celsius) which corrects resistance
using the material's temperature coefficient.

Adds Solid/Stranded Aluminum wire types with published resistance tables
for both unit systems, and aluminum's 0.403%/°C coefficient.

Fixes erroneous copper resistance values: solid 6 AWG was 0.3134 instead
of the published 0.3951 Ohm/kft, and stranded 0/2/4 AWG were decimal typos
(0.022/0.019/0.038 instead of 0.122/0.194/0.308 - the table is solid x
1.24 throughout). New tests enforce monotonicity and ratio consistency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 15:27:34 -06:00
Roland
b98ebf2a7f
Hotspot: guard on all three binaries so iw/rfkill install independently.
On systems that already have dnsmasq (e.g. Ubuntu-style installs), the
previous guard skipped the whole block, silently leaving iw and rfkill
uninstalled. apt-get is a no-op for packages already present, so one
combined install stays idempotent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 07:51:23 -06:00
Roland
93d9af1480 Hotspot: ship dnsmasq-base; NetworkManager requires it for hotspot DHCP.
The amd64 ISO is built with --apt-recommends false, so network-manager's
Recommends (including dnsmasq-base) were never installed. NetworkManager's
shared/hotspot mode spawns dnsmasq for DHCP, so the hotspot failed to
activate on every ISO install. Add it to the package list (which
install_debian_12.sh also installs from) and as an idempotent upgrade.sh
step so existing installs are fixed on their next upgrade.

Also add iw and rfkill, the standard WiFi diagnostic tools.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 07:51:13 -06:00
Roland
cdd28d0a39
Videos: a Channel's directory beats source_id when assigning a Video.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
A user reported videos placed in a Channel's directory did not appear on
that Channel's page. Their info json named another existing Channel, and
get_channel preferred the source_id match over the directory, so the
videos were assigned to the other Channel.

Reorder get_channel so the directory match (with its parent-directory
walk) runs before the source_id/url matches. A Video's files must live
in their Channel's directory, so the directory reflects the user's
intent and beats yt-dlp metadata. This also aligns validation with
VideoDownloader._get_channel, which already prefers the destination
directory, and stops validation from clobbering that decision.

channel_id still wins over all other params; source_id/url still apply
when the file is not in any Channel's directory.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 18:33:15 -06:00
Roland
f65f0aca6b
Restore "Most Views" download order
yt-dlp 2026.7.4 (via upstream PR #16965) again extracts view_count from
YouTube's lockupViewModel channel/playlist listings, so the 'views'
download order deprecated in fcb64fed can come back — verified live
against a real channel listing.

Unlike the old implementation, the sort degrades gracefully:

- Entries missing view_count sort last instead of raising KeyError.
- If NO entry has a view_count (yt-dlp older than ~2026.07), the
  download fails cleanly with UnrecoverableDownloadError telling the
  user to update yt-dlp or switch to Newest/Oldest — no retry loop.

Listing counts are approximate ("3K" -> 3000), which is fine for
ordering. No requirements.txt change needed; the >=2026.03.17 range
picks up 2026.7.4 automatically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 16:34:50 -06:00
Roland
6730235b7d
Bump version to 0.27.1-beta
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-07-07 07:55:29 -06:00
Roland
ce05a073dd
Upgrade: mark the pinned GPG key trusted in the throwaway keyring.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
verify-commit judges signature validity regardless of ownertrust, so the
upgrade's security model (possession of the pinned key is the trust
anchor) is unchanged — but gpg printed "WARNING: This key is not
certified with a trusted signature!" on every healthy upgrade, making
valid-signature logs look like something continued past a failure.
Import ownertrust for the sole key in the keyring to silence it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 20:57:01 -06:00
Roland
fdf65e218e
repair.sh: heal Postgres cluster ownership non-destructively.
The only self-service fix for a permission-broken cluster was
reset_api_db.sh, which drops the database.  Repair now restores
postgres ownership of /var/lib/postgresql and /etc/postgresql and
restarts Postgres before configuring the database, same as the reset
script but keeping the data.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 20:30:30 -06:00
Roland
3180a29e82
help.sh: real Postgres cluster diagnostics.
The umbrella postgresql.service reports active with every cluster down,
which made two cluster failures look green.  Show pg_lsclusters, check
the first cluster is online, verify the data directory is under
/var/lib/postgresql, and reveal a mount over /var/lib/postgresql (the
Portable bind layout — wrong on an installed system).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 20:30:30 -06:00
Roland
51a12b199e
Controller: keep every wrolpi-* service visible in the UI.
show_only_when_running hid wrolpi-upgrade the moment it exited — exactly
when its logs matter after a failed upgrade — and hid wrolpi-api-dev
whenever it was stopped.  Reinterpret the flag: hide a service only when
its unit is not installed on the box (LoadState=not-found).  Discovery
now unions loaded units (--all, catching transient and exited units)
with installed unit files, and discovered services are listed regardless
of running state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 20:30:30 -06:00
Roland
910503b641
Reconciler: uid/gid and ntfsfix for NTFS secondary mounts.
QA on 10.0.0.8 (mount-issues.md findings 2 and 3) showed the reconciler
path missed both NTFS accommodations that mount_drive() and
add_fstab_entry() already have: secondary NTFS drives mounted owned by
root (writable only via ntfs-3g's permissive default), and a dirty NTFS
volume would fail to mount at boot with no cleanup attempted.

Inject uid/gid for ntfs/ntfs3 in the reconciler, and run ntfsfix -d in
SubprocessMountExecutor before NTFS mounts (translating UUID=/LABEL=
specs to /dev/disk/by-* paths, which ntfsfix needs).  Also return an
error MountResult instead of raising when a binary is missing, per the
executor protocol.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 20:30:29 -06:00
Roland
9f2d9ed82b
repair.sh: key the Portable postgres ownership restore off the active bind.
Portable is defined by the live boot medium (wrolpi-bootstrap MODE), and
its visible effect at repair time is the bind of the cluster over
/var/lib/postgresql.  Restore postgres ownership only when that bind is
active, and restore the bind targets themselves; on installed systems a
config/postgresql directory is inert (dead weight from the old layout,
or a user's backup on the drive) and must not be touched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 15:39:03 -06:00
Roland
4dfb997bfc
Focusing on lone input when opening modals. 2026-07-06 15:35:23 -06:00
Roland
756c063d15
Upgrade/repair: a down database must never block the pipeline.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
scripts/upgrade.sh ran `main.py db upgrade` under set -e before
repair.sh, so a broken Postgres bricked the very upgrade carrying the
fix for it (seen live: the bootstrap bind-mount fix could not be
delivered because the binds had already killed the cluster).  repair.sh
had the same problem with initialize_api_db.sh and the superuser psql.

Warn and continue instead: the migration is retried by
initialize_api_db.sh during repair, and if the database is still down
afterwards the API reports it loudly while the system-level work
(configs, services, bootstrap refresh) still completes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 15:40:47 -06:00
Roland
9d778d25a5
repair.sh: refresh the Portable bootstrap script on installed boxes.
Boxes installed from the Live USB run /usr/local/sbin/wrolpi-bootstrap.sh
at every boot, but nothing ever updated it after image build — fixes to
the bootstrap (e.g. Postgres bind-mounts on live boots only) could never
reach installed systems.  Copy the script (and its unit, if installed)
from the repo during repair, which both upgrade.sh and install runs
execute.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 15:34:40 -06:00
Roland
f500600025
Bootstrap: Postgres bind-mounts on live boots only.
wrolpi-bootstrap.service applied the /media/wrolpi/config/postgresql
bind-mounts over /var/lib/postgresql and /etc/postgresql at every boot,
including on installed systems.  The binds exist for live boots, where
the root filesystem is an ephemeral overlay; on an installed system they
put the cluster in the path of repair.sh's config chown
(pg_filenode.map permission errors) and, once the user mounts an
NTFS/exFAT primary drive over /media/wrolpi, point Postgres at a
filesystem that cannot host a data directory at all — no clusters,
connection refused, while the umbrella postgresql.service stays green.

Installed systems now keep Postgres at the normal Debian paths on the
root filesystem.  A box that still has a stale bind active from an old
boot gets a log note to reboot; on the first clean boot the existing
pg_createcluster/initialize_api_db logic builds the root-disk cluster
automatically and the database rebuilds from configs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 15:33:54 -06:00
Roland
2e660cdbb9
Disks: mount the primary drive directly; clean phantom fstab.yaml entries.
POST /disks/mount wrote every mount into fstab.yaml and waited for the
reconciler — but the reconciler skips /media/wrolpi as a reserved mount
point, so mounting the primary drive from the Disks page could never
succeed and left a phantom fstab.yaml entry behind (visible as
"1 desired, 1 skipped: reserved mount point" in the journal).  Mount the
primary drive directly via mount_drive() and persist it to /etc/fstab,
as onboarding does.

Phantom entries are now also invisible to the fstab list endpoint (they
were reported as active persistence) and are removed whenever the
primary mount is mounted, persisted, or un-persisted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 15:05:20 -06:00
Roland
c25902b92a
Disks: allow persisting the primary mount from the UI; clean NTFS before mounting.
The Disks page treated /media/wrolpi as fully protected, so a
manually-mounted primary drive could not be persisted from the UI — and
the Persist toggle would have been wrong anyway: it wrote fstab.yaml,
which the wrolpi-mounts reconciler refuses to apply for the primary
mount (RESERVED_MOUNT_POINTS).  Route Persist for the primary mount to
the host's /etc/fstab (as onboarding does), report its entry in the
fstab list endpoint, and enable the toggle in both UIs while keeping
Unmount blocked.  Mounts outside /media (e.g. /boot/efi, which showed
an Unmount button) are now entirely untouchable System mounts.

Also:
- Run `ntfsfix -d` before mounting ntfs/ntfs3 volumes so the Windows
  dirty flag (Fast Startup, unclean unplug) cannot block mounting;
  non-fatal and a no-op on clean volumes.
- Add uid/gid fstab options for ntfs/ntfs3, matching mount_drive().
- Log reconcile diagnostics (desired/already-mounted/skipped with
  reasons) so an idle wrolpi-mounts run explains itself in the journal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 14:50:56 -06:00
Roland
85bd5619fa
Open in Slicer: launch slicer programmatically from dropdown items.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Semantic UI may activate a Dropdown.Item from the keyboard without
following its href, which would save the slicer preference without
launching the slicer. Prevent the default anchor navigation and call
window.location.assign() so every activation path behaves the same.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 13:31:40 -06:00
Roland
a580317911
File preview: Open in Slicer button for STL/3MF files.
Adds a split button to the STL and 3MF preview toolbars that opens the
model directly in a desktop slicer via its URL scheme
(scheme://open?file=<absolute media URL>). The slicer downloads the
file itself over the network.

Offers Orca Slicer, Bambu Studio, and Cura; the last-used slicer is
remembered in localStorage and becomes the split button's default,
since browsers cannot detect which protocol handlers are installed.

PrusaSlicer is deliberately omitted: it only downloads from domains
whitelisted by Prusa (printables.com, thingiverse.com, cults3d.com),
so it cannot fetch from a WROLPi.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 13:31:40 -06:00
Roland
da35d39d53
repair.sh: keep Postgres ownership of config/postgresql on bootstrapped systems.
On bootstrapped (portable) systems the cluster lives under
/media/wrolpi/config/postgresql, bind-mounted over /var/lib/postgresql;
the recursive chown of the config directory swept the cluster to
wrolpi:wrolpi, breaking new backends.  No-op on normal installs where
that directory does not exist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 13:31:38 -06:00
Roland
7bf7d9aa66
reset_api_db.sh: heal Postgres cluster ownership before resetting.
If cluster files lose postgres ownership (stray recursive chown,
interrupted install), the postmaster stays up but every new backend
dies with FATAL: could not open file "global/pg_filenode.map":
Permission denied — and dropdb/psql cannot work either.  Restore
ownership of /var/lib/postgresql and /etc/postgresql and restart
postgres before recreating the database, and start the API when done.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 13:31:38 -06:00
Roland
eefebe1788
Upgrade: make apt installs non-interactive and heal interrupted dpkg.
The caddy package install hit a dpkg conffile prompt because repair.sh
already wrote /etc/caddy/Caddyfile; under the tty-less
wrolpi-upgrade.service the prompt got EOF on stdin and apt exited 100.
Run the caddy/samba/gnome-disk-utility installs with
DEBIAN_FRONTEND=noninteractive and --force-confdef/--force-confold
(keep the current conffile; repair.sh manages the Caddyfile anyway),
and run `dpkg --configure -a` first so a box left half-configured by a
previously interrupted run is healed automatically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 13:09:16 -06:00
Roland
02eb8d70e4
Caddy install: run gpg --dearmor with --batch --yes.
If the caddy keyring file already exists (left by a prior partial
install), gpg prompts to overwrite via /dev/tty, which fails under the
tty-less wrolpi-upgrade.service ("cannot open '/dev/tty'") and aborts
the upgrade. --batch --yes overwrites silently, making the caddy
install idempotent in all four scripts that share this pattern.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 13:03:11 -06:00
Roland
e6ace5942b
Disambiguate 'git checkout release' in upgrade.sh.
Same fix install.sh received in bf88ac9f: on a box with no local
release branch (installed before that patch), the tracked release/
directory makes a bare `git checkout -f release` fail with
"could be both a local file and a tracking branch", aborting upgrades.
`checkout -f -B <branch> origin/<branch>` is unambiguous and already
resets the branch to origin, replacing the separate reset --hard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 13:03:11 -06:00
Roland
e8c6d8415b
Certificates: only include SANs permitted by the Root CA name constraints.
Global IPs from `hostname -I` (e.g. routable IPv6 assigned via SLAAC)
violated the Root CA's critical nameConstraints, failing leaf verification
with "permitted subtree violation" (error 47) and aborting
install/repair/upgrade. Filter SAN candidates, including EXTRA_SANS, to the
permitted private ranges and log skipped entries; normalize the IP:/DNS:
prefix case in EXTRA_SANS since openssl rejects lowercase prefixes.

Also guard the service stops in upgrade.sh so upgrading a box whose
install died before the units were installed can still recover.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 12:38:08 -06:00