During onboarding the media drive is not mounted and ca.crt does not exist
yet (or still holds only a throwaway pre-drive state). The cert banner still
offered Download WROLPi Root CA, which 404'd. Gate the button and HTTPS
cert-setup banner on drive mounted, ca.crt present, and repair not running
so an existing CA on the drive can be served after mount/repair.
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A new wrolpi-help-script.service oneshot runs /opt/wrolpi/help.sh so the
Controller UI can run system diagnostics and show the output from the
journal. The confirmation modal hides the warnings block for scripts
with no warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
show_only_when_running hid wrolpi-upgrade the moment it exited — exactly
when its logs matter after a failed upgrade — and hid wrolpi-api-dev
whenever it was stopped. Reinterpret the flag: hide a service only when
its unit is not installed on the box (LoadState=not-found). Discovery
now unions loaded units (--all, catching transient and exited units)
with installed unit files, and discovered services are listed regardless
of running state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
QA on 10.0.0.8 (mount-issues.md findings 2 and 3) showed the reconciler
path missed both NTFS accommodations that mount_drive() and
add_fstab_entry() already have: secondary NTFS drives mounted owned by
root (writable only via ntfs-3g's permissive default), and a dirty NTFS
volume would fail to mount at boot with no cleanup attempted.
Inject uid/gid for ntfs/ntfs3 in the reconciler, and run ntfsfix -d in
SubprocessMountExecutor before NTFS mounts (translating UUID=/LABEL=
specs to /dev/disk/by-* paths, which ntfsfix needs). Also return an
error MountResult instead of raising when a binary is missing, per the
executor protocol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /disks/mount wrote every mount into fstab.yaml and waited for the
reconciler — but the reconciler skips /media/wrolpi as a reserved mount
point, so mounting the primary drive from the Disks page could never
succeed and left a phantom fstab.yaml entry behind (visible as
"1 desired, 1 skipped: reserved mount point" in the journal). Mount the
primary drive directly via mount_drive() and persist it to /etc/fstab,
as onboarding does.
Phantom entries are now also invisible to the fstab list endpoint (they
were reported as active persistence) and are removed whenever the
primary mount is mounted, persisted, or un-persisted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Disks page treated /media/wrolpi as fully protected, so a
manually-mounted primary drive could not be persisted from the UI — and
the Persist toggle would have been wrong anyway: it wrote fstab.yaml,
which the wrolpi-mounts reconciler refuses to apply for the primary
mount (RESERVED_MOUNT_POINTS). Route Persist for the primary mount to
the host's /etc/fstab (as onboarding does), report its entry in the
fstab list endpoint, and enable the toggle in both UIs while keeping
Unmount blocked. Mounts outside /media (e.g. /boot/efi, which showed
an Unmount button) are now entirely untouchable System mounts.
Also:
- Run `ntfsfix -d` before mounting ntfs/ntfs3 volumes so the Windows
dirty flag (Fast Startup, unclean unplug) cannot block mounting;
non-fatal and a no-op on clean volumes.
- Add uid/gid fstab options for ntfs/ntfs3, matching mount_drive().
- Log reconcile diagnostics (desired/already-mounted/skipped with
reasons) so an idle wrolpi-mounts run explains itself in the journal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* POST /api/hotspot/settings checks is_docker_mode() directly instead of
string-matching the error message (still 500, matching the sibling
hotspot/bluetooth/throttle endpoints).
* Escape \ ; , " in the WiFi QR code per the WPA QR format.
* Skip the controller.yaml write when a settings update changes nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hotspot was reported as "not supported" on x86 machines (Ubuntu/Debian PCs)
because the Controller only watched the hardcoded default wlan0 device, while
predictable interface naming produces names like wlp2s0. The Settings page
saved hotspot_device to wrolpi.yaml, which the Controller never read.
The Controller now owns hotspot settings in controller.yaml:
* New endpoints: GET/POST /api/hotspot/settings, GET /api/hotspot/devices
(WiFi interfaces discovered via nmcli).
* Legacy hotspot_* values in wrolpi.yaml are migrated into controller.yaml
once on startup, unless controller.yaml already sets them.
* The Hotspot SSID/password/device fields moved from the Settings page to the
Control page next to the Hotspot toggle; the device is now a dropdown of
the WiFi devices found on the machine.
* The Controller fallback UI gets the same hotspot settings form for parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Surface coarse SMART health for USB-bridge drives
USB enclosures that reject ATA pass-through (e.g. Seagate Expansion) are
invisible to pySMART/DeviceList, so the drive shows no SMART at all on the
Controller page. Fall back to smartctl over the SCSI transport, which many
such bridges still answer, to report a coarse PASS/FAIL health plus drive
identity (model/serial/capacity).
The health (-d scsi -H) and identity (-d scsi -i) reads are issued as
separate calls: combining them makes smartctl suppress the health line when
the identity read reports "SMART support is: Unavailable" (which these
bridges do). The entry is marked smart_limited (labelled "(limited)" in both
UIs) so a green PASS is not mistaken for full monitoring — these drives
expose no attributes or temperature, and some bridges report "OK" statically.
* Harden SCSI health fallback (Greptile review)
- _scsi_health: treat any unrecognised status (e.g. "UNKNOWN", "N/A") as
unreadable (None) instead of a false FAIL. Only "OK" -> PASS and "FAIL*"
-> FAIL; anything else omits the drive rather than showing a red FAIL that
contradicts the smart_limited disclaimer.
- get_all_smart_status: isolate the SCSI supplement loop in its own
try/except so an error there cannot discard the pySMART results already
collected.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Warn in the navbar when a hard drive runs hot.
The navbar already warns about high CPU temperature; add the equivalent
for drive temperature, since an overheating disk risks data loss and a
user running an off-grid library won't be watching for it.
- controller/lib/smart.py: build_smart_stats() returns the per-drive list
plus warning thresholds (55C warn / 65C critical) so the frontend stays
dumb. Reading SMART wakes a spun-down drive, so drives in standby are
detected with `smartctl -n standby` (which works through the USB-SATA
bridges where `hdparm -C` only reports "unknown") and NOT read; their
last-known reading is reused instead.
- controller/lib/status_worker.py: collect SMART on a slow 60s cadence
(cached on app.state) rather than the 5s status loop, so the poll does
not keep USB drives awake. The result flows into the same cached payload
via a new collect_all_status(smart_stats=...) kwarg.
- Frontend: useDriveTemperature() reports the hottest drive; Nav.js shows
a yellow/red `hdd` icon with a popup, mirroring the CPU warning.
Verified on hardware: smart_stats reports real temps and SMART runs once
a minute, not every 5s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Greptile review on HDD temperature warning.
- _drive_is_asleep also treats SLEEP mode as asleep, not just STANDBY.
smartctl -n standby exits "Device is in SLEEP mode, exit(2)" for the
deeper power mode; matching only "STANDBY" meant a SLEEP drive was read
and woken — the exact failure the check prevents. (TDD: added a failing
SLEEP test, then fixed.)
- useDriveTemperature uses ?? instead of || for the temperature fallback
to express "fall back only on null/undefined".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Applied directly since the dependabot PR could not auto-rebase after the
grouped merges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Real drives report attribute 194 raw values like "52 (Min/Max 16/57)"
which int() cannot parse, so every drive's temperature was null.
Prefer pySMART's `temperature` property, which parses these formats
and also works for NVMe devices that have no ATA attribute table.
Verified on real hardware (WD100EMAZ over USB-SAT): 51-53C now
reported where null was returned before.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The /api/disks/unmount endpoint only removed entries from fstab.yaml and
asked the reconciler to unmount, but the reconciler deliberately never
unmounts paths it has not claimed. Foreign mounts (udisks2 desktop
automounts, manual mounts) therefore could not be unmounted from the UI,
failing with a confusing "check journalctl" error.
- Fall back to a direct umount(8) when a reconcile leaves the requested
mount point live.
- Reject unmounting the primary WROLPi drive (/media/wrolpi) with a
clear 400, normalized to catch path traversal.
- Require the normalized mount point to be strictly under /media/
before any direct umount, since the fallback runs as root.
- Show "System" instead of an Unmount button for /media/wrolpi in both
the React UI and the Controller's simplified UI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A WROLPi drive can move between hosts and across deployment styles
(Portable, RPi, Debian). /etc/fstab is host-specific and on the Portable
image it's part of a read-only squashfs, so persisting secondary mounts
there was both wrong-host and lost-on-reboot. This change makes
/media/wrolpi/config/fstab.yaml the single source of truth for WROLPi-
managed mounts and introduces a new wrolpi-mounts.service that reconciles
live state to match.
Architecture:
- The Controller writes intent (fstab.yaml) but never calls mount(8) or
umount(8) itself. Every mutation triggers `systemctl restart
wrolpi-mounts.service`; the Controller then verifies the outcome via
the same MountExecutor abstraction the service uses.
- wrolpi-mounts.service runs a thin entry point that constructs a real
Reconciler with a SubprocessMountExecutor and calls apply(). The
reconciler mounts what's in fstab.yaml but missing from the live state,
and unmounts paths it previously claimed that are no longer in
fstab.yaml. A managed-set file in /run/wrolpi-mounts/managed prevents
the reconciler from ever touching user-mounted paths it didn't own.
- One-time migration (marker-gated) lifts both /etc/fstab WROLPi entries
and any controller.yaml drives.mounts entries into fstab.yaml.
/etc/fstab is backed up to /etc/fstab.wrolpi-migration.backup;
migrated lines are commented in place with a "migrated to fstab.yaml:"
prefix so a host admin can hand-recover.
Test approach: dependency injection in place of patching. The
MountExecutor protocol has SubprocessMountExecutor (production) and
FakeMountExecutor (in-memory, records calls). The Reconciler accepts an
executor in its constructor, so the 20 reconciler tests + 13 migration
tests + 18 fstab_yaml tests use real files in tmp_path with zero
mock.patch calls. test_disks_api.py drops to mocks only for the
host-touching read helpers; the mount/unmount paths now run the real
Reconciler against the fake executor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Map viewer has simple search.
Map search databases can be downloaded from CDN, or built locally. Map searches can be built offline, but will not have enriched things like state names downloaded from Wikidata.
Introduce an onboarding system in the Controller that guides users
through selecting and mounting their primary media drive. This is
the fallback UI flow used when the main React app is unavailable.
Key changes:
- Add onboarding API endpoints and schemas (controller/api/onboarding.py)
- Add onboarding logic to detect drives, handle already-mounted drives
(needed for RPi which auto-mounts), and save primary drive selection
to the controller config when no mounts are configured
- Extend the Controller's index.html template with the onboarding UI
- Skip root cert creation until onboarding is complete
- Update first_startup.sh to support the onboarding/repair flow
- Add comprehensive tests for onboarding logic and API
