2021-11-01 16:42:52 -06:00
|
|
|
#! /usr/bin/env python3
|
|
|
|
|
import argparse
|
2023-01-31 17:11:31 -07:00
|
|
|
import asyncio
|
2021-11-01 16:42:52 -06:00
|
|
|
import logging
|
2024-08-15 21:45:06 -06:00
|
|
|
import os
|
2021-11-01 16:42:52 -06:00
|
|
|
import sys
|
|
|
|
|
|
2021-12-20 16:08:13 -07:00
|
|
|
from sanic import Sanic
|
2022-07-09 18:51:45 -06:00
|
|
|
from sanic.signals import Event
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2026-02-14 17:21:39 -07:00
|
|
|
from modules.videos.lib import get_videos_downloader_config
|
2025-12-19 17:40:43 -07:00
|
|
|
from wrolpi import flags
|
2024-05-01 18:59:25 -06:00
|
|
|
from wrolpi import root_api # noqa
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
from wrolpi.api_utils import api_app, perpetual_signal, per_worker_task
|
2024-04-17 11:25:26 -06:00
|
|
|
from wrolpi.common import logger, check_media_directory, set_log_level, limit_concurrent, \
|
The Internet check is configurable and no longer stalls background work (#899)
* The Internet check is configurable and no longer stalls background work
The check connected to one host (one.one.one.one, port 80) with no timeout, in
the perpetual process's event loop. On a network that drops traffic to that
host, each attempt hung for about two minutes and froze every other perpetual
loop (config saves, downloads, the file worker, the media mount check), so
WROLPi reported "No Internet" and "No drive mounted" though both were fine.
- wrolpi.yaml's internet_check_hostname (default one.one.one.one, needs DNS) and
internet_check_ip (default 1.1.1.1) choose what to connect to; the Internet is
up when either answers. Either may be empty. The INTERNET_SERVER environment
variable is gone.
- A connection gives up after 5 seconds.
- The check runs in a thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Internet check tries its IP address first
socket.create_connection resolves a hostname before its timeout applies, so a
name lookup that hangs (a firewall dropping DNS) postponed the IP address, which
needs no lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:05:45 -06:00
|
|
|
cancel_refresh_tasks, cancel_background_tasks, get_wrolpi_config, have_internet_connection, wrol_mode_enabled, \
|
Sync and backup: a secondary WROLPi and a backup drive copy the library (#957)
* A secondary WROLPi copies its primary's library over HTTP
A WROLPi with role "secondary" pulls from its primary: a manifest of every
FileGroup's files (/api/sync/manifest), then each missing or changed file
from the primary's Caddy /media/ (resumable, written to a hidden partial and
renamed with the primary's mtime), a refresh, then the library configs,
imported as the primary wrote them. wrolpi.yaml contributes only its library
settings. Files deleted on the primary are kept.
A secondary never saves the synced configs and never downloads. Its API
answers 503 SYNCING while a sync runs, and every page warns that it is a
copy. It syncs in a daily window or on "Sync now"; WROL Mode stops it.
On a Pi 4 (no AES instructions) the client uses TLS 1.2 ChaCha20, which
pulls ~110 MB/s from Caddy versus ~41 MB/s for rsync over ssh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync has its own /admin tab; a sync's reads from the primary are bounded
The sync settings and progress move from Settings (and the component
gallery, which is for library components) to /admin/sync.
A download never writes past the size in the primary's manifest, and configs,
listings, and manifest pages are capped. A primary address may not carry a
user name or password (it is logged and shown). While a sync runs, the API
answers only the exact requests that show its progress or stop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync keeps the secondary's replaced configs in today's config backup
The first copy of the day is kept, like a save, so a config backup import
undoes a sync's config changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Docker's Caddy logs every request
`docker compose logs -f web` shows each request with its status, size and
duration, e.g. a secondary's /media/ downloads during a sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync reports its progress as it downloads, and refuses quietly
The status updated only when a page of 500 FileGroups finished, so a page of
large files looked stuck; it now updates every 2 seconds. Requests refused
while syncing are answered by the middleware instead of raised, so the UI's
polling no longer logs an error each time. The primary's version is stored
without its trailing newline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A WROLPi can back up to a second drive; sync settings live in sync.yaml
The Sync tab's Backup Drive copies the media directory to a second mounted
drive with rsync during the sync window (hourly), or on "Back Up Now".
Nothing on the backup drive is deleted. The Tags and playlists directories
are left out (WROLPi rebuilds them from the configs), and the database is
written as an SQLite snapshot rather than copied as files, so the drive can be
mounted as the media directory and start. A backup is written only while the
chosen drive (by UUID) is mounted where it was chosen, and is not the media or
system drive; rsync is stopped if the drive disappears. Not in Docker, which
cannot reach other drives.
Role, primary, window and backup drive move from wrolpi.yaml to sync.yaml:
they belong to this machine, and are never copied from a primary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Sync tab shows the configured backup drive
getSettings copies each setting by name and did not list the backup drive,
so the Sync tab showed "None" while one was configured. It also dropped
`version`. A test now fails for any setting the API sends that getSettings
drops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A backup can go to a folder on its drive, and remembers its last success
The Backup Drive panel takes a folder on the chosen drive. The drive is
checked through the folder's mount point: the folder is written only when the
drive with the saved UUID is mounted there, so a folder left on the system
drive by an unplugged drive, or on another drive mounted inside the chosen
one, is refused. A missing folder is created once the drive is checked.
When the last backup succeeded is saved in sync.yaml, so it is known after a
restart. "Stop Backup" is disabled while the database snapshot is written (it
runs to the end), and a finished backup clears the request to stop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The backup drive has its own window
The sync window was shown in the Sync panel but also scheduled the backup
drive. The Backup Drive panel now has its own window (sync.yaml
backup_window_start/end), saved with the drive; the sync window is shown,
and used, only for a secondary's sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync and the backup drive can each exclude folders
A secondary or a backup drive may be smaller than the primary's library.
Each Sync tab panel has its own excluded folders in sync.yaml, chosen in
a folder browser: a secondary browses its primary's folders (through
Caddy), the backup drive browses this WROLPi's media directory. Excluded
folders are never copied; what was copied before is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The folder browser does not offer lost+found
Every ext4 drive has a root-only lost+found folder; a sync and a backup
never copy it, so it is not offered for exclusion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync never writes into config or tags, whatever their case
On a media drive that ignores case (exFAT, NTFS, macOS), a manifest path
like Config/wrolpi.db is the real config folder. The protected and
unlisted top-level folders are now compared case-insensitively in the
manifest, in resolve_destination, in excluded-folder validation, and in
both folder listings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A backup writes the database snapshot only when the database changed
Every backup rewrote the whole database snapshot (2.5 GB on a busy
WROLPi), even on the window's idle hourly runs. The snapshot is now
skipped when the live database (its size and mtime, and those of a
non-empty WAL) and the drive's copy are both unchanged since the last
snapshot. The record is in memory, so the first backup after a restart
writes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove the unused save_sync_config switch
Nothing activated it; SyncConfig saves directly. The test server's
log_message override is called by http.server, so it goes in the
vulture baseline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:53:34 -07:00
|
|
|
create_empty_config_files, import_file_configs, TRACE_LEVEL, is_secondary
|
2024-04-17 11:25:26 -06:00
|
|
|
from wrolpi.contexts import attach_shared_contexts, reset_shared_contexts, initialize_configs_contexts
|
2024-02-27 18:54:14 -07:00
|
|
|
from wrolpi.dates import Seconds
|
2026-02-20 14:37:58 -07:00
|
|
|
from modules.videos.cookies import lock_cookies
|
2026-02-14 17:21:39 -07:00
|
|
|
from wrolpi.downloader import download_manager, get_download_manager_config
|
2024-10-03 21:58:39 -06:00
|
|
|
from wrolpi.errors import WROLModeEnabled
|
2026-01-09 14:01:25 -07:00
|
|
|
from wrolpi.files.worker import file_worker
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
from wrolpi.perpetual import register_perpetual_process, perpetual_process_health_check
|
The Controller can make a secondary a copy of its primary with rsync (#992)
* The Controller can make a secondary a copy of its primary with rsync
The second part of replacing the sync. The Controller runs it, because the secondary's API is stopped while its
library is replaced: checks, a database snapshot on the primary, stop the API, rsync every file from the primary's
daemon (ignored directories included), fetch the snapshot and put it in the database's place, start the API, purge.
- Files the primary deleted or changed are moved to a dated folder in .sync-trash, and removed after
delete_after_days (a new sync.yaml setting, 30 by default; none never deletes), or sooner when the drive is nearly
full.
- A primary of another version or database migration, or with its media unmounted, is refused before anything is
touched. The database is replaced last, so a sync that stops earlier leaves one that works.
- On a Raspberry Pi and Debian rsync runs as the wrolpi user; in Docker in a helper container made from the api
container. A Controller that starts after an interrupted sync starts the API.
- When a secondary's API starts it takes the library settings of the primary's wrolpi.yaml, which the sync left.
Reached only through the Controller's /api/sync/run for now; the existing sync and its schedule are unchanged.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* The sync copies no link that leaves the library, and root follows none in the media directory
Security review of the Controller's sync:
- An exclusion that ends in a slash matches only a directory. A file or link named config/ssl, tags or
config/backup on the primary would have replaced this machine's directory of that name. The exclusions now match
whatever has the name.
- Links were copied as they are. One that points outside the primary's library would, on the secondary, point at
the secondary's own files, which the web server then serves. Only links that stay inside the library are copied,
and no device files.
- The Controller is root, in a media directory that belongs to another user. It no longer writes, reads, replaces
or purges through a symbolic link there.
Found on the way: the directory the Controller made for the incoming database belonged to root, so rsync, which
runs as the library's owner, could not have written into it. It now belongs to the media directory's owner.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* A sync ends when its primary stops answering, and makes room before it copies
Review found:
- rsync waits forever by default. A primary that lost power or its link during a copy left the secondary's API
stopped until someone cancelled. rsync now gives up after 30 seconds without a connection and 5 minutes without
data, which ends the run and starts the API.
- The trash was only purged after a run that succeeded. The trash is what fills the drive, and a full drive fails
every run, so it never got its turn. It is now also purged before the copy.
- A value in sync.yaml that could not be used raised after the run lock was taken and before the handlers that
release it, leaving "A sync is already running" until the Controller restarted.
- The playlists directory left out of the copy was this machine's, not the primary's, whose hard links would have
arrived as full copies. The primary's wrolpi.yaml is now fetched before the copy, and both are left out.
- A primary whose media directory is elsewhere is refused: its database names every file by its full path.
- In Docker rsync is its own container, which outlives a Controller that stops. A Controller that starts after an
interrupted sync now removes it before it starts the API.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 20:12:20 -07:00
|
|
|
from wrolpi.sync.lib import apply_primary_wrolpi_config
|
The Internet check is configurable and no longer stalls background work (#899)
* The Internet check is configurable and no longer stalls background work
The check connected to one host (one.one.one.one, port 80) with no timeout, in
the perpetual process's event loop. On a network that drops traffic to that
host, each attempt hung for about two minutes and froze every other perpetual
loop (config saves, downloads, the file worker, the media mount check), so
WROLPi reported "No Internet" and "No drive mounted" though both were fine.
- wrolpi.yaml's internet_check_hostname (default one.one.one.one, needs DNS) and
internet_check_ip (default 1.1.1.1) choose what to connect to; the Internet is
up when either answers. Either may be empty. The INTERNET_SERVER environment
variable is gone.
- A connection gives up after 5 seconds.
- The check runs in a thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Internet check tries its IP address first
socket.create_connection resolves a hostname before its timeout applies, so a
name lookup that hangs (a firewall dropping DNS) postponed the IP address, which
needs no lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:05:45 -06:00
|
|
|
from wrolpi.vars import PROJECT_DIR, DOCKERIZED
|
2022-01-24 16:41:03 -07:00
|
|
|
from wrolpi.version import get_version_string
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
logger = logger.getChild('wrolpi-main')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def db_main(args):
|
|
|
|
|
"""
|
2025-11-29 09:14:32 -07:00
|
|
|
Handle database migrations. This uses Alembic, supported commands are "upgrade", "downgrade", and "check".
|
2021-11-01 16:42:52 -06:00
|
|
|
"""
|
|
|
|
|
from alembic.config import Config
|
|
|
|
|
from alembic import command
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
from wrolpi.db import get_db_uri
|
2021-11-01 16:42:52 -06:00
|
|
|
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
uri = get_db_uri()
|
2021-11-01 16:42:52 -06:00
|
|
|
config = Config(PROJECT_DIR / 'alembic.ini')
|
|
|
|
|
# Overwrite the Alembic config, the is usually necessary when running in a docker container.
|
|
|
|
|
config.set_main_option('sqlalchemy.url', uri)
|
|
|
|
|
|
|
|
|
|
logger.warning(f'DB URI: {uri}')
|
|
|
|
|
|
|
|
|
|
if args.command == 'upgrade':
|
|
|
|
|
command.upgrade(config, 'head')
|
|
|
|
|
elif args.command == 'downgrade':
|
|
|
|
|
command.downgrade(config, '-1')
|
2025-11-29 09:14:32 -07:00
|
|
|
elif args.command == 'check':
|
|
|
|
|
command.check(config)
|
|
|
|
|
elif args.command == 'revision':
|
|
|
|
|
message = getattr(args, 'message', None) or 'auto migration'
|
|
|
|
|
command.revision(config, message=message, autogenerate=True)
|
2021-11-01 16:42:52 -06:00
|
|
|
else:
|
|
|
|
|
print(f'Unknown DB command: {args.command}')
|
|
|
|
|
return 2
|
|
|
|
|
|
|
|
|
|
return 0
|
|
|
|
|
|
|
|
|
|
|
2022-05-18 17:48:44 -06:00
|
|
|
INTERACTIVE_BANNER = '''
|
|
|
|
|
This is the interactive WROLPi shell. Use this to interact with the WROLPi API library.
|
|
|
|
|
|
|
|
|
|
Example (get the duration of every video file):
|
2024-05-01 18:42:15 -06:00
|
|
|
from modules.videos.models import Video
|
2024-11-02 15:14:02 -06:00
|
|
|
from modules.videos.common import extract_video_duration
|
2024-05-01 18:42:15 -06:00
|
|
|
|
|
|
|
|
videos = session.query(Video).all()
|
|
|
|
|
videos = list(videos)
|
|
|
|
|
for video in videos:
|
2024-11-02 15:14:02 -06:00
|
|
|
extract_video_duration(video.file_group.primary_path)
|
2022-05-18 17:48:44 -06:00
|
|
|
|
2022-05-18 18:36:12 -06:00
|
|
|
Check local variables:
|
|
|
|
|
locals().keys()
|
|
|
|
|
|
2022-05-18 17:48:44 -06:00
|
|
|
'''
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def launch_interactive_shell():
|
|
|
|
|
"""Launches an interactive shell with a DB session."""
|
|
|
|
|
import code
|
|
|
|
|
from wrolpi.db import get_db_session
|
|
|
|
|
|
|
|
|
|
with get_db_session() as session:
|
|
|
|
|
code.interact(banner=INTERACTIVE_BANNER, local=locals())
|
|
|
|
|
|
|
|
|
|
|
2022-10-09 18:42:23 -06:00
|
|
|
def main():
|
2021-11-01 16:42:52 -06:00
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
parser.add_argument('-v', '--verbose', action='count')
|
2022-01-05 17:07:54 -07:00
|
|
|
parser.add_argument('--version', action='store_true', default=False)
|
2022-01-24 18:04:23 -07:00
|
|
|
parser.add_argument('-c', '--check-media', action='store_true', default=False,
|
|
|
|
|
help='Check that the media directory is mounted and has the correct permissions.'
|
|
|
|
|
)
|
2022-05-18 17:48:44 -06:00
|
|
|
parser.add_argument('-i', '--interactive', action='store_true', default=False,
|
|
|
|
|
help='Enter an interactive shell with some WROLPi tools')
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
sub_commands = parser.add_subparsers(title='sub-commands', dest='sub_commands')
|
|
|
|
|
|
2024-05-02 10:56:43 -06:00
|
|
|
# Add the API parser, this will allow the user to specify host/port etc.
|
|
|
|
|
api_parser = sub_commands.add_parser('api')
|
|
|
|
|
root_api.init_parser(api_parser)
|
|
|
|
|
|
2021-11-01 16:42:52 -06:00
|
|
|
# DB Parser for running Alembic migrations
|
|
|
|
|
db_parser = sub_commands.add_parser('db')
|
2025-11-29 09:14:32 -07:00
|
|
|
db_parser.add_argument('command', help='Supported commands: upgrade, downgrade, check, revision')
|
|
|
|
|
db_parser.add_argument('-m', '--message', help='Migration message (for revision command)')
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
args = parser.parse_args()
|
|
|
|
|
|
2022-05-18 17:48:44 -06:00
|
|
|
if args.interactive:
|
|
|
|
|
launch_interactive_shell()
|
|
|
|
|
return 0
|
|
|
|
|
|
2022-01-05 17:07:54 -07:00
|
|
|
if args.version:
|
|
|
|
|
# Print out the relevant version information, then exit.
|
2022-01-24 16:41:03 -07:00
|
|
|
print(get_version_string())
|
2022-01-05 17:07:54 -07:00
|
|
|
return 0
|
|
|
|
|
|
2022-01-24 18:04:23 -07:00
|
|
|
if args.check_media:
|
|
|
|
|
# Run the media directory check. Exit with informative return code.
|
|
|
|
|
result = check_media_directory()
|
|
|
|
|
if result is False:
|
|
|
|
|
return 1
|
|
|
|
|
print('Media directory is correct.')
|
|
|
|
|
return 0
|
|
|
|
|
|
2023-01-31 17:11:31 -07:00
|
|
|
if not args.sub_commands:
|
|
|
|
|
parser.print_help()
|
|
|
|
|
return 1
|
|
|
|
|
|
2022-01-05 17:07:54 -07:00
|
|
|
logger.warning(f'Starting with: {sys.argv}')
|
2024-04-17 11:25:26 -06:00
|
|
|
if args.verbose == 1:
|
|
|
|
|
set_log_level(logging.INFO)
|
|
|
|
|
elif args.verbose and args.verbose == 2:
|
|
|
|
|
set_log_level(logging.DEBUG)
|
|
|
|
|
elif args.verbose and args.verbose >= 3:
|
|
|
|
|
# Log everything. Add SQLAlchemy debug logging.
|
2024-12-11 11:52:33 -07:00
|
|
|
set_log_level(TRACE_LEVEL)
|
2023-03-24 08:58:18 -06:00
|
|
|
logger.info(get_version_string())
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2022-01-15 18:18:17 -07:00
|
|
|
if DOCKERIZED:
|
|
|
|
|
logger.info('Running in Docker')
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
check_media_directory()
|
|
|
|
|
|
2021-11-01 16:42:52 -06:00
|
|
|
# Run DB migrations before anything else.
|
|
|
|
|
if args.sub_commands == 'db':
|
|
|
|
|
return db_main(args)
|
|
|
|
|
|
2024-05-02 10:56:43 -06:00
|
|
|
# Run the API.
|
|
|
|
|
if args.sub_commands == 'api':
|
|
|
|
|
return root_api.main(args)
|
|
|
|
|
|
2024-05-01 18:42:15 -06:00
|
|
|
return 1
|
2024-04-17 11:25:26 -06:00
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@api_app.main_process_ready
|
2024-09-26 09:13:29 -06:00
|
|
|
async def main_process_startup(app: Sanic):
|
2024-04-17 11:25:26 -06:00
|
|
|
"""
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
Initializes multiprocessing tools, flags, etc., then registers the perpetual process.
|
2024-04-17 11:25:26 -06:00
|
|
|
|
|
|
|
|
Performed only once when the server starts, this is done before server processes are forked.
|
|
|
|
|
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
@warning: This is NOT run after auto-reload! You must stop and start Sanic. (The perpetual process is
|
|
|
|
|
`transient`, so the manager does restart it on auto-reload.)
|
2024-04-17 11:25:26 -06:00
|
|
|
"""
|
2024-09-26 09:13:29 -06:00
|
|
|
logger.debug('main_process_startup')
|
2024-05-01 18:42:15 -06:00
|
|
|
|
|
|
|
|
check_media_directory()
|
|
|
|
|
|
2024-05-01 18:59:25 -06:00
|
|
|
if ('api',) not in app.router.routes_all:
|
|
|
|
|
logger.debug(f'{app.router.routes_all=}')
|
|
|
|
|
raise RuntimeError('WROLPi routes do not exist! Was root_api imported?')
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
# Initialize multiprocessing shared contexts before forking Sanic processes.
|
|
|
|
|
attach_shared_contexts(app)
|
2024-09-26 09:13:29 -06:00
|
|
|
logger.debug('main_process_startup done')
|
2024-04-17 11:25:26 -06:00
|
|
|
|
2024-11-28 13:28:23 -07:00
|
|
|
try:
|
2024-12-11 11:52:33 -07:00
|
|
|
create_empty_config_files()
|
2024-11-28 13:28:23 -07:00
|
|
|
except Exception as e:
|
2024-12-11 11:52:33 -07:00
|
|
|
logger.error('Failed to create initial config files', exc_info=e)
|
2024-11-28 13:28:23 -07:00
|
|
|
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
try:
|
|
|
|
|
# Create/migrate the SQLite database (if the media directory is available). If the media
|
|
|
|
|
# directory is mounted later, `perpetual_check_db_is_up_worker` retries this.
|
|
|
|
|
from wrolpi import db_bootstrap
|
|
|
|
|
db_bootstrap.ensure_db()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to create/migrate the database', exc_info=e)
|
|
|
|
|
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
# The singleton background loops run in ONE process the Worker Manager owns (not in a server worker), so no
|
|
|
|
|
# worker can claim or take them over. `shared_ctx` is populated above; the fork inherits it.
|
|
|
|
|
register_perpetual_process(app)
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@api_app.listener('after_server_start') # FileConfigs need to be initialized first.
|
2024-04-17 11:25:26 -06:00
|
|
|
async def initialize_configs(app: Sanic):
|
|
|
|
|
"""Each Sanic process runs this once."""
|
|
|
|
|
# Each process will have their own FileConfig object, but share the `app.shared_ctx.*config`
|
|
|
|
|
logger.debug('initialize_configs')
|
|
|
|
|
|
|
|
|
|
try:
|
|
|
|
|
initialize_configs_contexts(app)
|
2024-06-22 10:46:45 -06:00
|
|
|
await asyncio.sleep(0.5)
|
|
|
|
|
wrol_mode_enabled()
|
2024-09-15 15:33:07 -06:00
|
|
|
logger.info(f'initialize_configs succeeded pid={os.getpid()}')
|
2024-04-17 11:25:26 -06:00
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('initialize_configs failed with', exc_info=e)
|
|
|
|
|
raise
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@api_app.signal(Event.SERVER_SHUTDOWN_BEFORE)
|
|
|
|
|
@api_app.listener('reload_process_stop')
|
|
|
|
|
@limit_concurrent(1)
|
|
|
|
|
async def handle_server_shutdown(*args, **kwargs):
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
"""Stop downloads and securely clear sensitive data when server is shutting down.
|
|
|
|
|
|
|
|
|
|
Runs in each server worker. The perpetual process does the same for itself (wrolpi/perpetual.py)."""
|
2024-05-19 15:46:54 -06:00
|
|
|
logger.warning('Shutting down')
|
2025-12-22 17:56:35 -07:00
|
|
|
try:
|
|
|
|
|
download_manager.stop()
|
|
|
|
|
except AttributeError:
|
|
|
|
|
# Ignore AttributeError when context is empty, which can happen during server shutdown
|
|
|
|
|
pass
|
|
|
|
|
try:
|
|
|
|
|
await cancel_refresh_tasks()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('cancel_refresh_tasks failed. This is probably fine.', exc_info=e)
|
|
|
|
|
try:
|
|
|
|
|
await cancel_background_tasks()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('cancel_background_tasks failed. This is probably fine', exc_info=e)
|
2026-02-22 14:32:27 -07:00
|
|
|
# Clear cookies from shared memory before shutdown
|
2026-02-20 14:37:58 -07:00
|
|
|
try:
|
|
|
|
|
lock_cookies()
|
|
|
|
|
except Exception as e:
|
2026-02-22 14:32:27 -07:00
|
|
|
logger.error('Failed to clear cookies from shared memory', exc_info=e)
|
2024-04-17 11:25:26 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@api_app.signal(Event.SERVER_SHUTDOWN_AFTER)
|
|
|
|
|
async def handle_server_shutdown_reset(app: Sanic, loop):
|
|
|
|
|
"""Reset things after shutdown is complete, just in case server is going to start again."""
|
|
|
|
|
reset_shared_contexts(app)
|
|
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@api_app.after_server_start
|
2024-04-17 11:25:26 -06:00
|
|
|
async def start_single_tasks(app: Sanic):
|
|
|
|
|
"""Recurring/Single tasks that are started in only one Sanic process."""
|
|
|
|
|
# Only allow one child process to perform periodic tasks. See `handle_server_shutdown`
|
|
|
|
|
if app.shared_ctx.single_tasks_started.is_set():
|
|
|
|
|
return
|
|
|
|
|
app.shared_ctx.single_tasks_started.set()
|
|
|
|
|
|
|
|
|
|
logger.debug(f'start_single_tasks started')
|
2024-09-26 09:13:29 -06:00
|
|
|
|
2026-10-25 15:17:18 -06:00
|
|
|
# Import configs that don't require the database. Errors are logged so the service will start; configs refuse
|
|
|
|
|
# to save if they failed to import.
|
|
|
|
|
await import_file_configs()
|
2025-08-11 13:05:51 -06:00
|
|
|
wrolpi_config = get_wrolpi_config()
|
2024-09-26 09:13:29 -06:00
|
|
|
|
2026-09-12 15:18:13 -06:00
|
|
|
logger.debug('start_single_tasks waiting for db...')
|
|
|
|
|
async with flags.db_up.wait_for():
|
|
|
|
|
logger.debug('start_single_tasks db is up')
|
|
|
|
|
|
|
|
|
|
# The disk scan must overwrite the DB-restored `outdated_zims` value, so restore first (idempotent), then scan.
|
2024-09-26 09:13:29 -06:00
|
|
|
from modules.zim.lib import flag_outdated_zim_files
|
2024-04-17 11:25:26 -06:00
|
|
|
try:
|
2026-09-12 15:18:13 -06:00
|
|
|
flags.init_flags()
|
2024-04-17 11:25:26 -06:00
|
|
|
flag_outdated_zim_files()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to flag outdated Zims', exc_info=e)
|
|
|
|
|
|
2026-01-31 19:48:10 -07:00
|
|
|
# Import configs that require the database.
|
|
|
|
|
if wrolpi_config.successful_import and not wrolpi_config.wrol_mode:
|
The Controller can make a secondary a copy of its primary with rsync (#992)
* The Controller can make a secondary a copy of its primary with rsync
The second part of replacing the sync. The Controller runs it, because the secondary's API is stopped while its
library is replaced: checks, a database snapshot on the primary, stop the API, rsync every file from the primary's
daemon (ignored directories included), fetch the snapshot and put it in the database's place, start the API, purge.
- Files the primary deleted or changed are moved to a dated folder in .sync-trash, and removed after
delete_after_days (a new sync.yaml setting, 30 by default; none never deletes), or sooner when the drive is nearly
full.
- A primary of another version or database migration, or with its media unmounted, is refused before anything is
touched. The database is replaced last, so a sync that stops earlier leaves one that works.
- On a Raspberry Pi and Debian rsync runs as the wrolpi user; in Docker in a helper container made from the api
container. A Controller that starts after an interrupted sync starts the API.
- When a secondary's API starts it takes the library settings of the primary's wrolpi.yaml, which the sync left.
Reached only through the Controller's /api/sync/run for now; the existing sync and its schedule are unchanged.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* The sync copies no link that leaves the library, and root follows none in the media directory
Security review of the Controller's sync:
- An exclusion that ends in a slash matches only a directory. A file or link named config/ssl, tags or
config/backup on the primary would have replaced this machine's directory of that name. The exclusions now match
whatever has the name.
- Links were copied as they are. One that points outside the primary's library would, on the secondary, point at
the secondary's own files, which the web server then serves. Only links that stay inside the library are copied,
and no device files.
- The Controller is root, in a media directory that belongs to another user. It no longer writes, reads, replaces
or purges through a symbolic link there.
Found on the way: the directory the Controller made for the incoming database belonged to root, so rsync, which
runs as the library's owner, could not have written into it. It now belongs to the media directory's owner.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* A sync ends when its primary stops answering, and makes room before it copies
Review found:
- rsync waits forever by default. A primary that lost power or its link during a copy left the secondary's API
stopped until someone cancelled. rsync now gives up after 30 seconds without a connection and 5 minutes without
data, which ends the run and starts the API.
- The trash was only purged after a run that succeeded. The trash is what fills the drive, and a full drive fails
every run, so it never got its turn. It is now also purged before the copy.
- A value in sync.yaml that could not be used raised after the run lock was taken and before the handlers that
release it, leaving "A sync is already running" until the Controller restarted.
- The playlists directory left out of the copy was this machine's, not the primary's, whose hard links would have
arrived as full copies. The primary's wrolpi.yaml is now fetched before the copy, and both are left out.
- A primary whose media directory is elsewhere is refused: its database names every file by its full path.
- In Docker rsync is its own container, which outlives a Controller that stops. A Controller that starts after an
interrupted sync now removes it before it starts the API.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 20:12:20 -07:00
|
|
|
try:
|
|
|
|
|
# A secondary's library settings are its primary's; before the configs that depend on them.
|
|
|
|
|
apply_primary_wrolpi_config()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error("Failed to apply the primary's library settings", exc_info=e)
|
2026-02-14 17:21:39 -07:00
|
|
|
from wrolpi.common import import_all_db_configs
|
|
|
|
|
await import_all_db_configs()
|
2026-01-31 19:48:10 -07:00
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
if flags.refresh_complete.is_set():
|
|
|
|
|
# Set all downloads to new.
|
db: request-path library functions use the request's connection (#774)
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 11:05:20 -06:00
|
|
|
download_manager.retry_downloads_on_startup()
|
2026-10-13 16:07:04 -06:00
|
|
|
else:
|
|
|
|
|
file_worker.queue_first_refresh()
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2025-12-19 17:40:43 -07:00
|
|
|
# Hotspot/throttle on startup are now handled by the Controller service.
|
2022-02-16 21:57:27 -07:00
|
|
|
|
Sync and backup: a secondary WROLPi and a backup drive copy the library (#957)
* A secondary WROLPi copies its primary's library over HTTP
A WROLPi with role "secondary" pulls from its primary: a manifest of every
FileGroup's files (/api/sync/manifest), then each missing or changed file
from the primary's Caddy /media/ (resumable, written to a hidden partial and
renamed with the primary's mtime), a refresh, then the library configs,
imported as the primary wrote them. wrolpi.yaml contributes only its library
settings. Files deleted on the primary are kept.
A secondary never saves the synced configs and never downloads. Its API
answers 503 SYNCING while a sync runs, and every page warns that it is a
copy. It syncs in a daily window or on "Sync now"; WROL Mode stops it.
On a Pi 4 (no AES instructions) the client uses TLS 1.2 ChaCha20, which
pulls ~110 MB/s from Caddy versus ~41 MB/s for rsync over ssh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync has its own /admin tab; a sync's reads from the primary are bounded
The sync settings and progress move from Settings (and the component
gallery, which is for library components) to /admin/sync.
A download never writes past the size in the primary's manifest, and configs,
listings, and manifest pages are capped. A primary address may not carry a
user name or password (it is logged and shown). While a sync runs, the API
answers only the exact requests that show its progress or stop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync keeps the secondary's replaced configs in today's config backup
The first copy of the day is kept, like a save, so a config backup import
undoes a sync's config changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Docker's Caddy logs every request
`docker compose logs -f web` shows each request with its status, size and
duration, e.g. a secondary's /media/ downloads during a sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync reports its progress as it downloads, and refuses quietly
The status updated only when a page of 500 FileGroups finished, so a page of
large files looked stuck; it now updates every 2 seconds. Requests refused
while syncing are answered by the middleware instead of raised, so the UI's
polling no longer logs an error each time. The primary's version is stored
without its trailing newline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A WROLPi can back up to a second drive; sync settings live in sync.yaml
The Sync tab's Backup Drive copies the media directory to a second mounted
drive with rsync during the sync window (hourly), or on "Back Up Now".
Nothing on the backup drive is deleted. The Tags and playlists directories
are left out (WROLPi rebuilds them from the configs), and the database is
written as an SQLite snapshot rather than copied as files, so the drive can be
mounted as the media directory and start. A backup is written only while the
chosen drive (by UUID) is mounted where it was chosen, and is not the media or
system drive; rsync is stopped if the drive disappears. Not in Docker, which
cannot reach other drives.
Role, primary, window and backup drive move from wrolpi.yaml to sync.yaml:
they belong to this machine, and are never copied from a primary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Sync tab shows the configured backup drive
getSettings copies each setting by name and did not list the backup drive,
so the Sync tab showed "None" while one was configured. It also dropped
`version`. A test now fails for any setting the API sends that getSettings
drops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A backup can go to a folder on its drive, and remembers its last success
The Backup Drive panel takes a folder on the chosen drive. The drive is
checked through the folder's mount point: the folder is written only when the
drive with the saved UUID is mounted there, so a folder left on the system
drive by an unplugged drive, or on another drive mounted inside the chosen
one, is refused. A missing folder is created once the drive is checked.
When the last backup succeeded is saved in sync.yaml, so it is known after a
restart. "Stop Backup" is disabled while the database snapshot is written (it
runs to the end), and a finished backup clears the request to stop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The backup drive has its own window
The sync window was shown in the Sync panel but also scheduled the backup
drive. The Backup Drive panel now has its own window (sync.yaml
backup_window_start/end), saved with the drive; the sync window is shown,
and used, only for a secondary's sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Sync and the backup drive can each exclude folders
A secondary or a backup drive may be smaller than the primary's library.
Each Sync tab panel has its own excluded folders in sync.yaml, chosen in
a folder browser: a secondary browses its primary's folders (through
Caddy), the backup drive browses this WROLPi's media directory. Excluded
folders are never copied; what was copied before is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The folder browser does not offer lost+found
Every ext4 drive has a root-only lost+found folder; a sync and a backup
never copy it, so it is not offered for exclusion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A sync never writes into config or tags, whatever their case
On a media drive that ignores case (exFAT, NTFS, macOS), a manifest path
like Config/wrolpi.db is the real config folder. The protected and
unlisted top-level folders are now compared case-insensitively in the
manifest, in resolve_destination, in excluded-folder validation, and in
both folder listings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* A backup writes the database snapshot only when the database changed
Every backup rewrote the whole database snapshot (2.5 GB on a busy
WROLPi), even on the window's idle hourly runs. The snapshot is now
skipped when the live database (its size and mtime, and those of a
non-empty WAL) and the drive's copy are both unchanged since the last
snapshot. The record is in memory, so the first backup after a restart
writes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Remove the unused save_sync_config switch
Nothing activated it; SyncConfig saves directly. The test server's
log_message override is called by http.server, so it goes in the
vulture baseline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:53:34 -07:00
|
|
|
# A secondary never downloads; its library comes from its primary (see wrolpi.sync).
|
|
|
|
|
if get_wrolpi_config().download_on_startup and not wrol_mode_enabled() and not is_secondary():
|
2024-12-24 15:14:05 -07:00
|
|
|
# Only start downloading when prerequisites have been met.
|
|
|
|
|
try:
|
|
|
|
|
async with flags.have_internet.wait_for(timeout=30):
|
|
|
|
|
if get_download_manager_config().successful_import:
|
|
|
|
|
await download_manager.enable()
|
|
|
|
|
except TimeoutError as e:
|
|
|
|
|
logger.error('Failed to enable download', exc_info=e)
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
logger.debug(f'start_single_tasks done')
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
|
2026-01-09 14:01:25 -07:00
|
|
|
@perpetual_signal(sleep=1)
|
|
|
|
|
async def perpetual_file_worker_queue():
|
|
|
|
|
file_worker.transfer_queue()
|
2026-08-19 15:25:11 -06:00
|
|
|
processed = await file_worker.process_queue()
|
|
|
|
|
file_worker.check_queue_stall(processed=processed)
|
2026-01-09 14:01:25 -07:00
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@perpetual_signal(sleep=1)
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
@per_worker_task(sleep=1)
|
2024-09-26 09:13:29 -06:00
|
|
|
async def perpetual_check_log_level():
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
"""Copies the global log level into this process's logger. Registered for every server worker AND the
|
|
|
|
|
perpetual process (which does most of the interesting logging)."""
|
2024-04-17 11:25:26 -06:00
|
|
|
log_level = api_app.shared_ctx.log_level.value
|
|
|
|
|
if log_level != logger.getEffectiveLevel():
|
2024-05-19 17:32:37 -06:00
|
|
|
logger.info(f'changing log level from {logger.getEffectiveLevel()} to {log_level}')
|
|
|
|
|
set_log_level(log_level, warn_level=False)
|
2023-09-12 08:34:07 -06:00
|
|
|
|
2022-08-06 05:22:07 +00:00
|
|
|
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
# Every server worker watches the perpetual process and asks the manager to restart it if it dies.
|
|
|
|
|
per_worker_task(sleep=10)(perpetual_process_health_check)
|
|
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@perpetual_signal(sleep=10)
|
2024-04-17 11:25:26 -06:00
|
|
|
async def perpetual_check_db_is_up_worker():
|
|
|
|
|
try:
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
if not flags.db_up.is_set():
|
|
|
|
|
# The DB may not exist yet (media drive mounted after startup); try to create/migrate
|
|
|
|
|
# it. Only one worker bootstraps at a time.
|
|
|
|
|
from wrolpi import db_bootstrap
|
|
|
|
|
with db_bootstrap.bootstrap_lock() as acquired:
|
|
|
|
|
if acquired:
|
|
|
|
|
db_bootstrap.ensure_db()
|
2024-09-15 15:33:07 -06:00
|
|
|
flags.check_db_is_up()
|
|
|
|
|
flags.init_flags()
|
2024-06-12 16:39:04 -06:00
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to check db status', exc_info=e)
|
2022-08-06 05:22:07 +00:00
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@perpetual_signal(sleep=10)
|
2024-05-31 23:54:00 -06:00
|
|
|
async def perpetual_have_internet_worker():
|
|
|
|
|
try:
|
The Internet check is configurable and no longer stalls background work (#899)
* The Internet check is configurable and no longer stalls background work
The check connected to one host (one.one.one.one, port 80) with no timeout, in
the perpetual process's event loop. On a network that drops traffic to that
host, each attempt hung for about two minutes and froze every other perpetual
loop (config saves, downloads, the file worker, the media mount check), so
WROLPi reported "No Internet" and "No drive mounted" though both were fine.
- wrolpi.yaml's internet_check_hostname (default one.one.one.one, needs DNS) and
internet_check_ip (default 1.1.1.1) choose what to connect to; the Internet is
up when either answers. Either may be empty. The INTERNET_SERVER environment
variable is gone.
- A connection gives up after 5 seconds.
- The check runs in a thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Internet check tries its IP address first
socket.create_connection resolves a hostname before its timeout applies, so a
name lookup that hangs (a firewall dropping DNS) postponed the IP address, which
needs no lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:05:45 -06:00
|
|
|
# In a thread: the check blocks, and would stall every other perpetual loop.
|
|
|
|
|
if await asyncio.to_thread(have_internet_connection):
|
2024-09-15 15:33:07 -06:00
|
|
|
flags.have_internet.set()
|
|
|
|
|
# Check hourly once we have internet.
|
|
|
|
|
await asyncio.sleep(float(Seconds.hour))
|
|
|
|
|
else:
|
|
|
|
|
# Check more often until the internet is back.
|
|
|
|
|
flags.have_internet.clear()
|
2024-10-03 21:58:39 -06:00
|
|
|
except WROLModeEnabled:
|
|
|
|
|
flags.have_internet.clear()
|
2024-09-15 15:33:07 -06:00
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to check if internet is up', exc_info=e)
|
2024-05-31 23:54:00 -06:00
|
|
|
|
|
|
|
|
|
2026-05-14 15:51:32 -06:00
|
|
|
@perpetual_signal(sleep=30)
|
|
|
|
|
async def perpetual_media_mounted_worker():
|
|
|
|
|
try:
|
|
|
|
|
if flags._destinations_have_mounted_storage():
|
|
|
|
|
flags.media_mounted.set()
|
|
|
|
|
# Re-check less often once all destinations are covered.
|
|
|
|
|
await asyncio.sleep(float(Seconds.minute * 5))
|
|
|
|
|
else:
|
|
|
|
|
flags.media_mounted.clear()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to check media mount status', exc_info=e)
|
|
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@perpetual_signal(sleep=30)
|
2024-09-26 09:13:29 -06:00
|
|
|
async def perpetual_start_video_missing_comments_download():
|
2024-04-17 11:25:26 -06:00
|
|
|
from modules.videos.video.lib import get_missing_videos_comments
|
2021-12-20 16:08:13 -07:00
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
async with flags.refresh_complete.wait_for():
|
|
|
|
|
# We can't search for Videos missing comments until the refresh has completed.
|
|
|
|
|
pass
|
2021-12-20 16:08:13 -07:00
|
|
|
|
2026-02-19 16:15:18 -07:00
|
|
|
# Check if download missing video comments is enabled.
|
|
|
|
|
config = get_videos_downloader_config()
|
|
|
|
|
if not config.download_missing_video_comments:
|
|
|
|
|
logger.trace('Download missing video comments is disabled, skipping...')
|
|
|
|
|
return
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
# Wait for download manager to startup.
|
|
|
|
|
await asyncio.sleep(5)
|
2023-03-29 19:30:33 -06:00
|
|
|
|
2024-12-11 11:52:33 -07:00
|
|
|
if download_manager.can_download:
|
2024-05-31 23:54:00 -06:00
|
|
|
try:
|
|
|
|
|
await get_missing_videos_comments()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to get missing video comments', exc_info=e)
|
|
|
|
|
|
2024-12-11 11:52:33 -07:00
|
|
|
# Sleep one hour.
|
2024-04-17 11:25:26 -06:00
|
|
|
await asyncio.sleep(int(Seconds.hour))
|
2024-12-11 11:52:33 -07:00
|
|
|
else:
|
|
|
|
|
logger.debug('Waiting for downloads to be enabled before downloading comments...')
|
2023-03-29 19:30:33 -06:00
|
|
|
|
2022-07-09 18:51:45 -06:00
|
|
|
|
Apply the offset in a SingleFile saved date, and repair misread Archive dates (#862)
* Apply the offset in a SingleFile saved date
SingleFile writes the browser's local time with its offset ("... 18:36:23 GMT-0700"). The parser dropped
the offset and treated the local time as UTC, so an uploaded Archive's download_datetime was off by the
browser's UTC offset, and a Collection reorganize renamed its files with that shifted timestamp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Repair Archive download datetimes misread from a SingleFile saved date
Archives parsed before the offset fix keep the shifted download_datetime, because the parser only fills
an empty value. A one-time background repair (gated by a wrolpi_kv version, after the first refresh)
re-reads each Archive's saved date and corrects a download_datetime that equals the offset-dropped value.
Any other download_datetime is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read the whole SingleFile header; finish the date repair only when every Archive was read
The header was read in 1000 bytes, so a long URL cut the saved date's offset off, and the cut-off date was
stored as GMT. The header is now read through the end of its comment, and a malformed offset leaves the
date unset instead of guessing.
The repair is no longer marked done when an Archive could not be read (a missing file is a deleted Archive,
not a failure), and it waits for the media to be mounted, because `refresh_complete` is restored from the
database before the drives are checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:34:29 -06:00
|
|
|
@perpetual_signal(sleep=3600)
|
|
|
|
|
async def perpetual_repair_archive_saved_dates():
|
|
|
|
|
"""Correct Archive download datetimes that were misread from their SingleFile saved date. Runs once."""
|
|
|
|
|
async with flags.refresh_complete.wait_for():
|
|
|
|
|
# Archives must be in the database.
|
|
|
|
|
pass
|
|
|
|
|
async with flags.media_mounted.wait_for():
|
|
|
|
|
# Their files must be readable; `refresh_complete` is restored from the database before the drive is checked.
|
|
|
|
|
pass
|
|
|
|
|
|
|
|
|
|
from modules.archive.lib import repair_singlefile_saved_dates
|
|
|
|
|
try:
|
|
|
|
|
await asyncio.to_thread(repair_singlefile_saved_dates)
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to repair Archive download datetimes', exc_info=e)
|
|
|
|
|
|
|
|
|
|
|
2025-10-29 22:16:33 -06:00
|
|
|
@perpetual_signal(sleep=3600) # Check hourly
|
|
|
|
|
async def perpetual_check_for_updates():
|
|
|
|
|
"""
|
|
|
|
|
Check git remote for new commits on the current branch.
|
|
|
|
|
|
|
|
|
|
Updates shared_ctx with update info for the /api/status endpoint.
|
|
|
|
|
Only runs on native installs (not Docker).
|
|
|
|
|
"""
|
2025-12-02 16:07:32 -07:00
|
|
|
# Local import to avoid loading upgrade module at startup
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
from wrolpi.upgrade import refresh_update_status
|
2025-10-29 22:16:33 -06:00
|
|
|
|
|
|
|
|
# Don't check for updates in Docker environments
|
|
|
|
|
if DOCKERIZED:
|
|
|
|
|
return
|
|
|
|
|
|
2025-12-06 14:47:52 -07:00
|
|
|
# Don't check if setting is disabled
|
|
|
|
|
if not get_wrolpi_config().check_for_upgrades:
|
|
|
|
|
return
|
|
|
|
|
|
2025-10-29 22:16:33 -06:00
|
|
|
# Only check when we have internet
|
|
|
|
|
if not flags.have_internet.is_set():
|
|
|
|
|
return
|
|
|
|
|
|
|
|
|
|
try:
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
# Runs `git fetch` in a thread and stores the result in shared_ctx.status for all workers.
|
|
|
|
|
result = await refresh_update_status(api_app.shared_ctx.status)
|
2025-10-29 22:16:33 -06:00
|
|
|
|
|
|
|
|
if result.get('update_available'):
|
|
|
|
|
logger.info(f"Update available: {result['commits_behind']} commits behind on {result['branch']}")
|
2025-12-02 15:34:43 -07:00
|
|
|
except WROLModeEnabled:
|
|
|
|
|
logger.debug('Refusing to check for updates when WROL Mode is enabled.')
|
2025-10-29 22:16:33 -06:00
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to check for updates', exc_info=e)
|
|
|
|
|
|
|
|
|
|
|
2021-11-01 16:42:52 -06:00
|
|
|
if __name__ == '__main__':
|
2022-10-09 18:42:23 -06:00
|
|
|
sys.exit(main())
|