A refresh on a large library aborted the whole doc_modeler run on the
first lock error, so the ~20k stranded docs never got modeled.
doc_modeler held one write transaction across a whole batch of 10 docs
while doing slow per-doc extraction (a PDF/epub can be tens of MB). When
another connection committed mid-batch, the deferred transaction failed
to upgrade to a write with SQLITE_BUSY_SNAPSHOT ("database is locked",
which busy_timeout does not absorb -- the same trap that motivated the
short-transaction rewrite of apply_indexers). The per-item error handler
then did `logger.error(f'... {file_group}')`, which lazy-loaded an
attribute on the now-rolled-back session and raised a *second* exception,
escaping the handler and aborting the entire modeler.
Restructure:
- Pick each batch of candidate ids in a short read-only transaction, then
model each doc in its own short write transaction. A lock/error on one
doc rolls back only that doc and cannot poison the rest of the run.
- Retry a doc a few times on "database is locked" -- a fresh transaction
gets a fresh snapshot and succeeds.
- Log the plain file_group id on failure; never touch the ORM object on a
possibly rolled-back session.
- Track only *failed* ids for the termination guard (successes are
excluded by the Doc.id IS NULL gate), keeping the notin_ set small and
well under SQLite's bound-variable limit.
Adds test_doc_modeler_survives_database_locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The zim Docker image is FROM debian:13, whose kiwix-tools 3.7.0 rejects the
literal "--address 0.0.0.0" ("IP address is not available on this system"),
so the container failed to start after loading the library. Omit --address;
kiwix-serve then defaults to binding all interfaces.
This mirrors 9e92732a, which applied the same fix to the systemd
start_kiwix_serve.sh path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: the new `Doc.id IS NULL` gate keeps matching a
doc whose _model_doc fails without persisting a Doc row. With
>=DOC_PROCESSING_LIMIT such docs, every batch re-selected the same failing
files, `processed` never dropped below the limit, and the loop hung --
starving later docs. Reproduced with a patched _model_doc that always
raises.
Track attempted FileGroup ids per run and exclude them from the query, so
the candidate pool strictly shrinks and the loop always terminates. Failed
docs are retried on the next refresh (unchanged behavior).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The doc_modeler gated its query on `FileGroup.indexed == False`, but
`indexed` is a full-text-search flag that `apply_indexers` sets on every
FileGroup during a refresh. Because the refresh pipeline runs
apply_modelers -> apply_indexers, any doc file that reached apply_indexers
without first receiving a Doc row (modeler errored, was interrupted, or
files imported already-indexed) got `indexed=1` and was then excluded by
the `indexed == False` gate forever -- leaving it with model=NULL, no Doc
row, and invisible in /api/docs.
On one library this stranded 20,795 PDFs/epubs/mobis (only the 10
individually-tagged docs, modeled via the per-file do_model path, showed
up). No doc-mimetype files were left with indexed=0, so nothing would
ever recover them.
Gate the modeler on "no Doc row yet OR explicitly flagged for
re-indexing" instead, so already-indexed docs are picked up on the next
refresh while preserving re-index behavior and skipping already-modeled
docs. Self-healing: a global refresh now models the backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A corrupt or partial *.zim (e.g. an interrupted download) is rejected by
kiwix-manage ("Cannot add zim ..."). start_kiwix_serve.sh could then leave
Kiwix down and reported a misleading "Could not find any Zim files" — and
if the bad zim was the only one, exit 2 killed the service with no restart.
- Import each zim independently; log and skip a failed add so the remaining
valid zims are still served.
- Always ensure a library.xml exists (empty if needed) and exec kiwix-serve
unconditionally, so a corrupt/absent zim never stops the server starting.
- Fix the mount-wait loop, which never incremented COUNT (infinite wait
instead of a 30s timeout).
- Add MEDIA_DIRECTORY override so the script is testable.
- Add Restart=on-failure to wrolpi-kiwix.service (matches wrolpi-api.service)
so transient failures self-heal.
- New test harness runs the script with fake kiwix-manage/kiwix-serve shims.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The themed Button forwards its ref straight to SButton, a class
component, so the ref resolves to the class instance instead of the DOM
node. Semantic UI's Modal/Popup Portal calls node.contains(...) on the
trigger's ref during handleDocumentClick, which throws on a non-DOM
value. This crashed the Control page whenever a Service Logs (or QR /
download-error) modal was open and the document was clicked.
Wrap SButton in Ref (findDOMNode) so a forwarded ref always resolves to
the underlying <button> element, fixing every Modal/Popup that uses a
themed Button as its trigger.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refreshing all files failed on a large media directory with
RuntimeError: values_clause: too many parameters (44454 rows x 3).
After the Postgres->SQLite cutover, values_clause guards against
exceeding SQLite's ~32k bound-variable limit, but upsert_directories
still built a single multi-row VALUES clause over every discovered
directory.
Insert in BULK_SQL_CHUNK_SIZE batches, matching the existing bulk
tag-file callers. Adds a regression test that inserts more directories
than a single statement can bind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Debian major upgrade (e.g. 12 -> 13) replaces /usr/bin/python3
(3.11 -> 3.13). Existing venvs still resolve `python3` to the system
interpreter, but their packages live in lib/python3.11/site-packages,
which the new interpreter never searches -- every import fails and
`pip install --upgrade` cannot repair them, taking down the API,
Controller, yt-dlp, and Help.
upgrade.sh now compares each venv's recorded Python version against the
current one and does a full rebuild via reset_virtual_environments.sh
when they differ. Detecting a mismatch (rather than literally "3.11")
also self-heals future bumps and is a no-op once versions match or on a
fresh install with no pyvenv.cfg yet.
reset_virtual_environments.sh gains a -y/--yes flag to skip its
confirmation prompt; the interactive yes_or_no read loops forever on EOF
under the tty-less wrolpi-upgrade.service.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When importing hundreds of tag_files for files that exist on disk but
have no FileGroup yet (a fresh import before a file refresh), the import
crawled at ~a dozen rows/second. Every tagged file hit a fallback that
globbed its parent directory (O(files²) for large Channel/Domain dirs)
and flushed the DB once per row.
Batch the work instead:
- Cache each directory's listing so it is scanned once, not per file.
- Create new FileGroups under no_autoflush and flush them in a single
batch, then create TagFiles in a second pass (one batched INSERT).
- Cache every member path of a newly-created FileGroup so a file tagged
by multiple tags, or another config entry for a different member of
the same shared-stem group, reuses one FileGroup instead of inserting
a duplicate primary_path that would abort the import.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_special_directories runs during every refresh ignore filter. Calling
get_videos_directory/get_no_channel_directory created videos/ and
NO CHANNEL under an empty media tree, which broke empty-refresh and
directory listing tests. Resolve those paths without creating them.
Zim Manage and search depend on FileGroup modeling of files under
zims/. Ignoring that directory left Kiwix serving files on disk while
/api/zim stayed empty. Refuse to ignore zims (alongside other special
dirs), strip it from existing configs on settings load, and share
normalized ignore handling between refresh and upsert.
`git rev-parse --abbrev-ref HEAD` prints the literal "HEAD" on a detached
HEAD, which get_current_branch() returned as if it were a branch. check_for_update
then compared origin/HEAD (the remote's default branch, usually master) and
reported a bogus "N commits behind on HEAD" — a false "update available".
Return None for a detached HEAD so no phantom upgrade is offered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Debian 12 guard fetched the cutoff tag with a non-forcing refspec, so a
box that had already fetched an earlier `debian12-final` kept its stale local
tag ("would clobber existing tag" is rejected). When the tag later moved
(e.g. a version-bumped final release), the guard pinned to the OLD target
instead of the new one. Force the fetch (+ / --force) so a moved tag wins.
Safe: the commit's signature is still verified before checkout.
Caught on real Debian 12 hardware (10.0.0.8) after moving the tag for a patch
bump; added a sandbox scenario covering the moved-tag/stale-local-tag case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Debian 12 (bookworm) cannot run the Postgres->SQLite / Debian 13 rebase,
and there is no in-place 12->13 upgrade. Guard the on-device upgrade so a
Debian 12 machine pins to the signed `debian12-final` tag and never advances
past it, while Debian 13+ upgrades normally.
./upgrade.sh already fetched+checked out the latest release by the time
scripts/upgrade.sh runs, so the guard runs first: on Debian 12 it verifies
the guard commit's signature, rolls back to debian12-final, re-execs that
version's script (which warns to reinstall on Debian 13 and then runs its
normal upgrade so services restart). Aborts rather than run newer code if the
tag is missing or unsigned. Inert on Debian 13+.
Verified with a sandbox harness covering: D13 pass-through, D12-at-tag
warn+proceed, D12-past-tag verify+rollback+re-exec (no loop), and
missing-tag refusal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the four test_list_tar_*_contents and four
test_download_single_member_tar* functions into two parametrized tests
covering gz/xz/plain/bz2. All eight format cases still run; no coverage
loss. Identified via pytest-deduplicate (identical line coverage).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dashboard nudge is useless on Safari and other browsers that cannot
run the WROLPi extension. Share detectBrowser and gate the banner with
isExtensionSupportedBrowser.
During onboarding the media drive is not mounted and ca.crt does not exist
yet (or still holds only a throwaway pre-drive state). The cert banner still
offered Download WROLPi Root CA, which 404'd. Gate the button and HTTPS
cert-setup banner on drive mounted, ca.crt present, and repair not running
so an existing CA on the drive can be served after mount/repair.
Add `git config --global --add safe.directory /opt/wrolpi` before the
`git reset --hard`, guarding against git's "dubious ownership" refusal.
Set HOME=/root first so the --global write cannot fail (and abort the ISO
build under `set -e`) if the live-build chroot leaves HOME unset -- the
same pattern the pi-gen hook already uses.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pi-gen and Debian Live chroot hooks run `npm install`, which rewrites
the tracked app/package-lock.json, then build_app.sh stamps
build/.build-stamp from that dirty working tree. On first boot repair.sh
runs `git reset --hard` (reverting the lock) *before* it recomputes the
stamp, so the baked stamp can never match -- and the device rebuilds the
React app every time, the exact OOM-prone step baking the build in (#536)
was meant to avoid.
Verified against WROLPi-v0.27.2: the shipped image had `M app/package-lock.json`,
baked stamp a9c6ef61..., post-reset stamp 2b908c6c... (guaranteed mismatch).
Mirror repair.sh in both image hooks: `git reset --hard HEAD` right after
npm install and before build_app.sh, so the baked stamp represents committed
source and matches what the device recomputes. Only the lock is reverted;
build/, node_modules, and the venvs are untracked and survive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop .pgpass handling, Postgres ownership repairs, and the docker db
container viewability rules. Update comments and pi-gen map docs for
SQLite/PMTiles. Help submodule documents the new database layout.
Catch useForm fetcher errors and surface them so DomainEditPage can show a not-found message instead of hanging on the loader (and so the webpack overlay no longer flakes Cypress).
- strip_youtube_caption_positioning now writes to a temp file in the same
directory and os.replace()s the original, so a failed/partial write
(e.g. disk full) can never truncate the existing caption file.
- Replace the substring `'youtube.com' in url` gate with is_youtube_url(),
which parses the hostname and matches youtube.com / youtu.be (and
subdomains), rejecting lookalikes like youtube.com.example.org and
notyoutube.com. Add a parametrized test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
YouTube auto-generated captions (via yt-dlp) stamp every cue timing line
with `align:start position:0%`, pinning the on-screen text to the
bottom-left of the player. This strips that exact cue setting from the
downloaded .vtt files so browsers render the captions centered.
- add strip_youtube_caption_positioning() to wrolpi/captions.py — a
targeted text replacement (not a webvtt round-trip) so the `<c>`
word-timing tags in the cue text are preserved; matches only the exact
auto-caption signature, so deliberately positioned cues (e.g. with a
`line` setting) are left untouched; idempotent.
- call it from VideoDownloader after normalize_video_file_names, gated on
a youtube.com / youtu.be source URL. It runs before the FileGroup is
created, so the first index reads the corrected file — no re-index
plumbing needed.
Complements the client-side fix in #543 (which centers captions in the
player for existing/any-source files); this makes new YouTube downloads
correct at the source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Require `line === 'auto'` in the auto-caption signature so a cue that
is deliberately placed (e.g. `line:0` for top-edge) is left alone even
if it is also `align:start position:0%`. YouTube auto-captions never
set `line`.
- For lazily-loaded language tracks, only detach the cuechange listener
after the cue list is actually populated, in case a cuechange fires
before cues load.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
YouTube auto-generated captions (downloaded via yt-dlp) stamp every cue
with `align:start position:0%`, which pins the caption text to the
bottom-left of the player and makes it hard to read. Properly-authored
VTT files have no cue settings and already render centered.
Detect that auto-caption signature and reset only those cues to the
browser's centered default (align:center / position:auto / line:auto).
This mutates the in-memory VTTCue objects the browser parsed; the .vtt
file on disk is never modified, and deliberately-positioned cues are
left untouched.
Wired via each <track>'s onLoad. Non-default language tracks are
`disabled` at load (no cues yet), so they are normalized lazily on the
first cuechange when the user selects them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The React /map page rendered blank (only pins) on fresh builds while the
standalone :8084 viewer worked. maplibre-gl's Web Worker (inlined into the
production main.js) threw `ReferenceError: a is not defined` in its setLayers
handler, so the worker's style-layer index never built and every vector
source's tiles failed to parse ("Cannot read properties of undefined
(reading '<source>')"). The unminified dev build was unaffected — it is a
production minification bug: Terser mangles the inlined worker into broken code.
Root cause: the "npm audit fix" in 62660529 (intended for shell-quote/
form-data/ws) regenerated the lockfile and dragged @babel/core 7.20.12 ->
7.29.7 (whole @babel core pipeline). Babel 7.29 transpiles the maplibre worker
differently; CRA's (unchanged) Terser mangle pass then breaks it. Proven on
10.0.0.8: rebuilding the minified prod bundle with @babel 7.20 renders the map
(0 worker errors); rebuilding back to 7.29 breaks it again with the identical
bundle hash. The transform plugins were never bumped, so pinning @babel/core
alone is insufficient — the core AST pipeline (traverse/generator/types/
helpers) must be pinned too.
Fix: pin the 16 @babel packages that differ from the last-good tree
(6b7bd079) via package.json overrides, keeping the shell-quote/form-data/ws
security fixes. Frontend jest suite passes (810/810).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verify ARRL wire-antenna formulas (936/468/234/585), fix feet/inches
rollover ("N ft 12 in"), make wavelength edits update lengths, and cover
the math with 50 unit tests.
go-pmtiles release assets are named ..._Linux_x86_64.tar.gz, not _amd64;
the #538 consolidation used $arch=amd64 in the URL, so every amd64 build
(Debian Live ISO, docker) 404'd installing pmtiles and the wrolpi chroot
hook failed. Use x86_64 for the URL arch on x86_64 hosts; the SHA var and
the aarch64 (arm64) path are unchanged and already correct.
Verified: both v1.30.1 tarballs download and match the pinned SHAs
(x86_64 23a2a222..., arm64 a1f9f42d...).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pmtiles CLI + tippecanoe install was duplicated across three places
(scripts/install_protomaps.sh, the Debian Live hook, and pi-gen), so a version
bump or logic change had to be made in all of them in lockstep. Move it into a
single shared, version-guarded, SHA-pinned installer and have all three callers
invoke it.
- New wrolpi/scripts/install_map_tools.sh: installs pmtiles (per-arch binary,
SHA-pinned for amd64 and arm64) and builds tippecanoe (source tarball,
arch-independent SHA). Version-guarded, so an image that already baked in the
pinned versions makes the on-device run a no-op until a version bumps.
- pi-gen 03-run-chroot.sh, the Debian hook, and install_protomaps.sh now just
call the shared script. This also adds SHA pinning to the on-device pmtiles
path (previously unpinned).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The Debian Live image builds tippecanoe at image time, but pi-gen did not,
so a fresh Pi compiled it on first boot/upgrade via install_protomaps.sh --
a multi-minute, CPU-heavy C++ build on the weakest hardware WROLPi targets.
This is the same "Debian bakes it, pi-gen doesn't" gap recently fixed for
the React app build.
Build tippecanoe 2.79.0 in the stage2 chroot, mirroring the Debian hook
(SHA-pinned source tarball, which is arch-independent). Add the two missing
build deps (libsqlite3-dev, zlib1g-dev) to the package list; build-essential
was already present.
With tippecanoe present at the pinned version, install_protomaps.sh's version
guard skips the build on first boot and on upgrades, so the on-device compile
only recurs when the pinned version actually bumps.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bake the app build into images; rebuild on-device only when stale
Previously the Raspberry Pi image shipped without app/build/ and compiled
the React bundle on first boot via repair.sh -- a multi-minute, memory-hungry
step that can OOM a 2GB Pi. The Debian Live ISO already baked the build in,
so the two paths had drifted.
Add scripts/build_app.sh as the single source of truth: it hashes the build
inputs (src/, public/, package.json, package-lock.json, tsconfig.json) and
runs `npm run build` (with GENERATE_SOURCEMAP=false) only when build/ is
missing or those inputs changed, then writes build/.build-stamp.
Wire all three call sites to it:
- pi-gen 03-run-chroot.sh now builds after npm install, baking build/ into
the Pi image like the Debian hook already does.
- debian-live 9999-wrolpi.hook.chroot uses it so the baked build is stamped.
- repair.sh uses it, so first-boot/manual-repair/install/upgrade all become
no-ops when the shipped build is current.
Result: fresh images ship a stamped build/, so a weak Pi never compiles on
boot or repair; an on-device rebuild happens only when an upgrade actually
changes app/. build_app.sh never runs npm install, so repair.sh keeps its
offline guarantee.
* build_app.sh: rebuild when built output is incomplete, not just missing
Addresses review feedback: a matching stamp only proves the inputs are
unchanged, not that the output survived. repair.sh exists to fix corruption,
so a build/ with index.html + stamp but a missing JS/CSS asset must still
rebuild. Verify every file referenced by build/asset-manifest.json exists
(and that the manifest itself parses); rebuild otherwise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Applies non-breaking transitive lockfile bumps for three still-present
advisories on the app dependency tree:
* shell-quote 1.8.0 -> 1.10.0 (critical: unescaped newlines in quote())
* form-data 3.0.4 -> 3.0.5 / 4.0.4 -> 4.0.6 (high: CRLF injection via
unescaped multipart field names/filenames)
* ws 7.5.10 -> 7.5.12 / 8.18.3 -> 8.21.1 (high: memory-exhaustion DoS)
package-lock.json only; no package.json / direct-dependency changes. Runs
`npm audit fix` without --force, so no breaking dev-chain (react-scripts)
upgrades are pulled in. Verified on trixie/Node 20: jest 760/760 and the
production build both green.
Supersedes the stale dependabot PRs #487 (ws), #486 (form-data), and
equal-or-newer patched versions.
Co-authored-by: Roland <roland.henrie@o2x.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Debian 13 ships kiwix-tools 3.7.0, whose kiwix-serve rejects the literal
--address 0.0.0.0 ("IP address is not available on this system: 0.0.0.0")
and exits 1, so wrolpi-kiwix fails to start on a fresh trixie image.
kiwix-serve binds all interfaces by default when --address is omitted, on
both the old (bookworm) and new (trixie) versions, so dropping the flag is
portable.
Verified on a trixie Pi (10.0.0.8): wrolpi-kiwix active, serving 200 on
9085 and via Caddy on 8085.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cold archive/video browsing took 18-90 seconds on a Pi: file_group rows
carry their indexed texts inline (SQLite has no out-of-line TOAST like
Postgres), so any plan that reads file_group rows to filter or sort
drags gigabytes of text pages through USB.
- Browse archives with file_group as the outer table (CROSS JOIN) so
SQLite walks the date index and probes archive's covering index;
driving from archive probed every file_group row for the sort key.
- Tie-break orders with fg.id instead of LOWER(fg.primary_path); the
rowid is present in every index, the path expression forced full-row
reads (and a full scan on the videos browse).
- Add a (mimetype, effective_datetime) covering index so the videos
mimetype filter + date order resolve index-only.
- PRAGMA optimize after a global refresh; the planner had no
sqlite_stat1 at all after the rebuild.
Warm timings on 10.0.0.9 (124k archives / 24k videos): archive browse
0.4s -> 0.22s index-only, videos browse 0.38s -> 0.07s; cold I/O drops
from ~124k random fat-page probes to a few MB of contiguous index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scraped pages can carry half an emoji in their titles; sqlite3 refuses
to store lone surrogates, so one poisoned title failed archive_modeler's
batch commit and ended the whole modeling run on 10.0.0.9. Every
archive after it stayed unmodeled, and cleanup then deleted their
'empty' domain collections (1,670 -> 578).
Strip surrogates at every point Archive ingests scraped strings (titles,
author, description, url, article text), and make the modeler skip a
batch whose commit fails instead of dying.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
apply_indexers held one ORM session across a whole batch of indexers --
minutes of readability/subprocess work -- so its final commit had to
upgrade a stale read snapshot to a write. SQLite answers that with
"database is locked" (SQLITE_BUSY_SNAPSHOT, which busy_timeout does not
apply to) whenever any other connection commits a write in the meantime,
which killed the refresh on 10.0.0.9 at 75%.
Read the batch as plain columns, index on transient FileGroups with no
open transaction, then write the results in one short BEGIN IMMEDIATE
transaction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three build targets move from Debian 12/bookworm to Debian 13/trixie
for fresh installs; existing Debian 12 systems remain supported in place.
Docker: api/help on python:3.13-trixie, controller on 3.13-slim-trixie,
app on node:20-trixie (matches trixie's nodejs 20), samba/zim on
debian:13. Full backend suite passes on Python 3.13 in the api container.
debian-live-config: --distribution trixie; postgres teardown for the
SQLite cutover (pg_dropcluster hook block removed, wrolpi-bootstrap.sh no
longer creates a cluster or calls the deleted initialize_api_db.sh — the
API creates wrolpi.db itself on first start). README documents the
SQLite layout and drops Etcher recommendations.
pi-gen: upstream has no trixie branch — the arm64 branch now defaults to
RELEASE=trixie; KEYBOARD_KEYMAP=en_US is rejected by trixie's
console-setup (ckbcomp validates), use 'us'; stop stamping a Postgres
.pgpass into /etc/skel.
Package lists (both targets): postgresql-15/postgresql-doc/libpq-dev
deleted; libtiff5-dev renamed to libtiff-dev; software-properties-common
and cpufrequtils removed from trixie (Controller CPU throttle falls back
to sysfs, dead cpufreq-set sudoers rule removed); locales and gnupg added
(no longer arrive transitively under --apt-recommends false); sqlite3 CLI
added for reset_tags.sh.
help.sh: replace the Postgres service/port/socket/cluster/psql checks
(all false FAILEDs after the SQLite cutover) with SQLite checks:
wrolpi.db exists, PRAGMA quick_check, file_group table + row count, run
as the wrolpi user so root never creates the -shm sidecar.
Both build scripts take a WROLPI_BUILD_DIR override so development builds
can run beside the release pipeline's build trees; the sudo re-exec
passes it through explicitly (a plain sudo would drop it).
test_common.py: the truncate tests hard-coded sys.getsizeof results from
the 3.11 object layout (str/list shrank in 3.12); assert the contract
(fits budget, prefix, monotonic in budget) so they pass on 3.11 and 3.13.
Verified: trixie ISO (3.7G) and pi-gen desktop image (4.3G) both built
from scratch and passed their sanity checks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collection.batch_from_config OR'd a (name AND kind) pair per collection, so
importing more than ~1000 collections (e.g. a 1,670-domain domains.yaml)
failed with "Expression tree is too large (maximum depth 1000)". Query a
flat name IN (...) and match (name, kind) in Python instead.
get_bulk_tag_preview had the same unbounded pattern (an OR'd LIKE per stem);
chunk the query at 500 stems.
Found upgrading 10.0.0.9 (decay).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Caddyfile only binds 443/8084/8085/8086 and reverse-proxies to the
Controller on port 80, which the Controller health check already covers.
The IPv4 alternative in the port grep was missing its colon, so only the
IPv6 ":::" listeners ever matched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
