2021-11-01 16:42:52 -06:00
|
|
|
#! /usr/bin/env python3
|
|
|
|
|
import argparse
|
2023-01-31 17:11:31 -07:00
|
|
|
import asyncio
|
2021-11-01 16:42:52 -06:00
|
|
|
import logging
|
2024-08-15 21:45:06 -06:00
|
|
|
import os
|
2021-11-01 16:42:52 -06:00
|
|
|
import sys
|
|
|
|
|
|
2021-12-20 16:08:13 -07:00
|
|
|
from sanic import Sanic
|
2022-07-09 18:51:45 -06:00
|
|
|
from sanic.signals import Event
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2026-02-14 17:21:39 -07:00
|
|
|
from modules.videos.lib import get_videos_downloader_config
|
2025-12-19 17:40:43 -07:00
|
|
|
from wrolpi import flags
|
2024-05-01 18:59:25 -06:00
|
|
|
from wrolpi import root_api # noqa
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
from wrolpi.api_utils import api_app, perpetual_signal, per_worker_task
|
2024-04-17 11:25:26 -06:00
|
|
|
from wrolpi.common import logger, check_media_directory, set_log_level, limit_concurrent, \
|
The Internet check is configurable and no longer stalls background work (#899)
* The Internet check is configurable and no longer stalls background work
The check connected to one host (one.one.one.one, port 80) with no timeout, in
the perpetual process's event loop. On a network that drops traffic to that
host, each attempt hung for about two minutes and froze every other perpetual
loop (config saves, downloads, the file worker, the media mount check), so
WROLPi reported "No Internet" and "No drive mounted" though both were fine.
- wrolpi.yaml's internet_check_hostname (default one.one.one.one, needs DNS) and
internet_check_ip (default 1.1.1.1) choose what to connect to; the Internet is
up when either answers. Either may be empty. The INTERNET_SERVER environment
variable is gone.
- A connection gives up after 5 seconds.
- The check runs in a thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Internet check tries its IP address first
socket.create_connection resolves a hostname before its timeout applies, so a
name lookup that hangs (a firewall dropping DNS) postponed the IP address, which
needs no lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:05:45 -06:00
|
|
|
cancel_refresh_tasks, cancel_background_tasks, get_wrolpi_config, have_internet_connection, wrol_mode_enabled, \
|
2026-10-25 15:17:18 -06:00
|
|
|
create_empty_config_files, import_file_configs, TRACE_LEVEL
|
2024-04-17 11:25:26 -06:00
|
|
|
from wrolpi.contexts import attach_shared_contexts, reset_shared_contexts, initialize_configs_contexts
|
2024-02-27 18:54:14 -07:00
|
|
|
from wrolpi.dates import Seconds
|
2026-02-20 14:37:58 -07:00
|
|
|
from modules.videos.cookies import lock_cookies
|
2026-02-14 17:21:39 -07:00
|
|
|
from wrolpi.downloader import download_manager, get_download_manager_config
|
2024-10-03 21:58:39 -06:00
|
|
|
from wrolpi.errors import WROLModeEnabled
|
2026-01-09 14:01:25 -07:00
|
|
|
from wrolpi.files.worker import file_worker
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
from wrolpi.perpetual import register_perpetual_process, perpetual_process_health_check
|
The Internet check is configurable and no longer stalls background work (#899)
* The Internet check is configurable and no longer stalls background work
The check connected to one host (one.one.one.one, port 80) with no timeout, in
the perpetual process's event loop. On a network that drops traffic to that
host, each attempt hung for about two minutes and froze every other perpetual
loop (config saves, downloads, the file worker, the media mount check), so
WROLPi reported "No Internet" and "No drive mounted" though both were fine.
- wrolpi.yaml's internet_check_hostname (default one.one.one.one, needs DNS) and
internet_check_ip (default 1.1.1.1) choose what to connect to; the Internet is
up when either answers. Either may be empty. The INTERNET_SERVER environment
variable is gone.
- A connection gives up after 5 seconds.
- The check runs in a thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Internet check tries its IP address first
socket.create_connection resolves a hostname before its timeout applies, so a
name lookup that hangs (a firewall dropping DNS) postponed the IP address, which
needs no lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:05:45 -06:00
|
|
|
from wrolpi.vars import PROJECT_DIR, DOCKERIZED
|
2022-01-24 16:41:03 -07:00
|
|
|
from wrolpi.version import get_version_string
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
logger = logger.getChild('wrolpi-main')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def db_main(args):
|
|
|
|
|
"""
|
2025-11-29 09:14:32 -07:00
|
|
|
Handle database migrations. This uses Alembic, supported commands are "upgrade", "downgrade", and "check".
|
2021-11-01 16:42:52 -06:00
|
|
|
"""
|
|
|
|
|
from alembic.config import Config
|
|
|
|
|
from alembic import command
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
from wrolpi.db import get_db_uri
|
2021-11-01 16:42:52 -06:00
|
|
|
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
uri = get_db_uri()
|
2021-11-01 16:42:52 -06:00
|
|
|
config = Config(PROJECT_DIR / 'alembic.ini')
|
|
|
|
|
# Overwrite the Alembic config, the is usually necessary when running in a docker container.
|
|
|
|
|
config.set_main_option('sqlalchemy.url', uri)
|
|
|
|
|
|
|
|
|
|
logger.warning(f'DB URI: {uri}')
|
|
|
|
|
|
|
|
|
|
if args.command == 'upgrade':
|
|
|
|
|
command.upgrade(config, 'head')
|
|
|
|
|
elif args.command == 'downgrade':
|
|
|
|
|
command.downgrade(config, '-1')
|
2025-11-29 09:14:32 -07:00
|
|
|
elif args.command == 'check':
|
|
|
|
|
command.check(config)
|
|
|
|
|
elif args.command == 'revision':
|
|
|
|
|
message = getattr(args, 'message', None) or 'auto migration'
|
|
|
|
|
command.revision(config, message=message, autogenerate=True)
|
2021-11-01 16:42:52 -06:00
|
|
|
else:
|
|
|
|
|
print(f'Unknown DB command: {args.command}')
|
|
|
|
|
return 2
|
|
|
|
|
|
|
|
|
|
return 0
|
|
|
|
|
|
|
|
|
|
|
2022-05-18 17:48:44 -06:00
|
|
|
INTERACTIVE_BANNER = '''
|
|
|
|
|
This is the interactive WROLPi shell. Use this to interact with the WROLPi API library.
|
|
|
|
|
|
|
|
|
|
Example (get the duration of every video file):
|
2024-05-01 18:42:15 -06:00
|
|
|
from modules.videos.models import Video
|
2024-11-02 15:14:02 -06:00
|
|
|
from modules.videos.common import extract_video_duration
|
2024-05-01 18:42:15 -06:00
|
|
|
|
|
|
|
|
videos = session.query(Video).all()
|
|
|
|
|
videos = list(videos)
|
|
|
|
|
for video in videos:
|
2024-11-02 15:14:02 -06:00
|
|
|
extract_video_duration(video.file_group.primary_path)
|
2022-05-18 17:48:44 -06:00
|
|
|
|
2022-05-18 18:36:12 -06:00
|
|
|
Check local variables:
|
|
|
|
|
locals().keys()
|
|
|
|
|
|
2022-05-18 17:48:44 -06:00
|
|
|
'''
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def launch_interactive_shell():
|
|
|
|
|
"""Launches an interactive shell with a DB session."""
|
|
|
|
|
import code
|
|
|
|
|
from wrolpi.db import get_db_session
|
|
|
|
|
|
|
|
|
|
with get_db_session() as session:
|
|
|
|
|
code.interact(banner=INTERACTIVE_BANNER, local=locals())
|
|
|
|
|
|
|
|
|
|
|
2022-10-09 18:42:23 -06:00
|
|
|
def main():
|
2021-11-01 16:42:52 -06:00
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
parser.add_argument('-v', '--verbose', action='count')
|
2022-01-05 17:07:54 -07:00
|
|
|
parser.add_argument('--version', action='store_true', default=False)
|
2022-01-24 18:04:23 -07:00
|
|
|
parser.add_argument('-c', '--check-media', action='store_true', default=False,
|
|
|
|
|
help='Check that the media directory is mounted and has the correct permissions.'
|
|
|
|
|
)
|
2022-05-18 17:48:44 -06:00
|
|
|
parser.add_argument('-i', '--interactive', action='store_true', default=False,
|
|
|
|
|
help='Enter an interactive shell with some WROLPi tools')
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
sub_commands = parser.add_subparsers(title='sub-commands', dest='sub_commands')
|
|
|
|
|
|
2024-05-02 10:56:43 -06:00
|
|
|
# Add the API parser, this will allow the user to specify host/port etc.
|
|
|
|
|
api_parser = sub_commands.add_parser('api')
|
|
|
|
|
root_api.init_parser(api_parser)
|
|
|
|
|
|
2021-11-01 16:42:52 -06:00
|
|
|
# DB Parser for running Alembic migrations
|
|
|
|
|
db_parser = sub_commands.add_parser('db')
|
2025-11-29 09:14:32 -07:00
|
|
|
db_parser.add_argument('command', help='Supported commands: upgrade, downgrade, check, revision')
|
|
|
|
|
db_parser.add_argument('-m', '--message', help='Migration message (for revision command)')
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
args = parser.parse_args()
|
|
|
|
|
|
2022-05-18 17:48:44 -06:00
|
|
|
if args.interactive:
|
|
|
|
|
launch_interactive_shell()
|
|
|
|
|
return 0
|
|
|
|
|
|
2022-01-05 17:07:54 -07:00
|
|
|
if args.version:
|
|
|
|
|
# Print out the relevant version information, then exit.
|
2022-01-24 16:41:03 -07:00
|
|
|
print(get_version_string())
|
2022-01-05 17:07:54 -07:00
|
|
|
return 0
|
|
|
|
|
|
2022-01-24 18:04:23 -07:00
|
|
|
if args.check_media:
|
|
|
|
|
# Run the media directory check. Exit with informative return code.
|
|
|
|
|
result = check_media_directory()
|
|
|
|
|
if result is False:
|
|
|
|
|
return 1
|
|
|
|
|
print('Media directory is correct.')
|
|
|
|
|
return 0
|
|
|
|
|
|
2023-01-31 17:11:31 -07:00
|
|
|
if not args.sub_commands:
|
|
|
|
|
parser.print_help()
|
|
|
|
|
return 1
|
|
|
|
|
|
2022-01-05 17:07:54 -07:00
|
|
|
logger.warning(f'Starting with: {sys.argv}')
|
2024-04-17 11:25:26 -06:00
|
|
|
if args.verbose == 1:
|
|
|
|
|
set_log_level(logging.INFO)
|
|
|
|
|
elif args.verbose and args.verbose == 2:
|
|
|
|
|
set_log_level(logging.DEBUG)
|
|
|
|
|
elif args.verbose and args.verbose >= 3:
|
|
|
|
|
# Log everything. Add SQLAlchemy debug logging.
|
2024-12-11 11:52:33 -07:00
|
|
|
set_log_level(TRACE_LEVEL)
|
2023-03-24 08:58:18 -06:00
|
|
|
logger.info(get_version_string())
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2022-01-15 18:18:17 -07:00
|
|
|
if DOCKERIZED:
|
|
|
|
|
logger.info('Running in Docker')
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
check_media_directory()
|
|
|
|
|
|
2021-11-01 16:42:52 -06:00
|
|
|
# Run DB migrations before anything else.
|
|
|
|
|
if args.sub_commands == 'db':
|
|
|
|
|
return db_main(args)
|
|
|
|
|
|
2024-05-02 10:56:43 -06:00
|
|
|
# Run the API.
|
|
|
|
|
if args.sub_commands == 'api':
|
|
|
|
|
return root_api.main(args)
|
|
|
|
|
|
2024-05-01 18:42:15 -06:00
|
|
|
return 1
|
2024-04-17 11:25:26 -06:00
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@api_app.main_process_ready
|
2024-09-26 09:13:29 -06:00
|
|
|
async def main_process_startup(app: Sanic):
|
2024-04-17 11:25:26 -06:00
|
|
|
"""
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
Initializes multiprocessing tools, flags, etc., then registers the perpetual process.
|
2024-04-17 11:25:26 -06:00
|
|
|
|
|
|
|
|
Performed only once when the server starts, this is done before server processes are forked.
|
|
|
|
|
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
@warning: This is NOT run after auto-reload! You must stop and start Sanic. (The perpetual process is
|
|
|
|
|
`transient`, so the manager does restart it on auto-reload.)
|
2024-04-17 11:25:26 -06:00
|
|
|
"""
|
2024-09-26 09:13:29 -06:00
|
|
|
logger.debug('main_process_startup')
|
2024-05-01 18:42:15 -06:00
|
|
|
|
|
|
|
|
check_media_directory()
|
|
|
|
|
|
2024-05-01 18:59:25 -06:00
|
|
|
if ('api',) not in app.router.routes_all:
|
|
|
|
|
logger.debug(f'{app.router.routes_all=}')
|
|
|
|
|
raise RuntimeError('WROLPi routes do not exist! Was root_api imported?')
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
# Initialize multiprocessing shared contexts before forking Sanic processes.
|
|
|
|
|
attach_shared_contexts(app)
|
2024-09-26 09:13:29 -06:00
|
|
|
logger.debug('main_process_startup done')
|
2024-04-17 11:25:26 -06:00
|
|
|
|
2024-11-28 13:28:23 -07:00
|
|
|
try:
|
2024-12-11 11:52:33 -07:00
|
|
|
create_empty_config_files()
|
2024-11-28 13:28:23 -07:00
|
|
|
except Exception as e:
|
2024-12-11 11:52:33 -07:00
|
|
|
logger.error('Failed to create initial config files', exc_info=e)
|
2024-11-28 13:28:23 -07:00
|
|
|
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
try:
|
|
|
|
|
# Create/migrate the SQLite database (if the media directory is available). If the media
|
|
|
|
|
# directory is mounted later, `perpetual_check_db_is_up_worker` retries this.
|
|
|
|
|
from wrolpi import db_bootstrap
|
|
|
|
|
db_bootstrap.ensure_db()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to create/migrate the database', exc_info=e)
|
|
|
|
|
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
# The singleton background loops run in ONE process the Worker Manager owns (not in a server worker), so no
|
|
|
|
|
# worker can claim or take them over. `shared_ctx` is populated above; the fork inherits it.
|
|
|
|
|
register_perpetual_process(app)
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@api_app.listener('after_server_start') # FileConfigs need to be initialized first.
|
2024-04-17 11:25:26 -06:00
|
|
|
async def initialize_configs(app: Sanic):
|
|
|
|
|
"""Each Sanic process runs this once."""
|
|
|
|
|
# Each process will have their own FileConfig object, but share the `app.shared_ctx.*config`
|
|
|
|
|
logger.debug('initialize_configs')
|
|
|
|
|
|
|
|
|
|
try:
|
|
|
|
|
initialize_configs_contexts(app)
|
2024-06-22 10:46:45 -06:00
|
|
|
await asyncio.sleep(0.5)
|
|
|
|
|
wrol_mode_enabled()
|
2024-09-15 15:33:07 -06:00
|
|
|
logger.info(f'initialize_configs succeeded pid={os.getpid()}')
|
2024-04-17 11:25:26 -06:00
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('initialize_configs failed with', exc_info=e)
|
|
|
|
|
raise
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@api_app.signal(Event.SERVER_SHUTDOWN_BEFORE)
|
|
|
|
|
@api_app.listener('reload_process_stop')
|
|
|
|
|
@limit_concurrent(1)
|
|
|
|
|
async def handle_server_shutdown(*args, **kwargs):
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
"""Stop downloads and securely clear sensitive data when server is shutting down.
|
|
|
|
|
|
|
|
|
|
Runs in each server worker. The perpetual process does the same for itself (wrolpi/perpetual.py)."""
|
2024-05-19 15:46:54 -06:00
|
|
|
logger.warning('Shutting down')
|
2025-12-22 17:56:35 -07:00
|
|
|
try:
|
|
|
|
|
download_manager.stop()
|
|
|
|
|
except AttributeError:
|
|
|
|
|
# Ignore AttributeError when context is empty, which can happen during server shutdown
|
|
|
|
|
pass
|
|
|
|
|
try:
|
|
|
|
|
await cancel_refresh_tasks()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('cancel_refresh_tasks failed. This is probably fine.', exc_info=e)
|
|
|
|
|
try:
|
|
|
|
|
await cancel_background_tasks()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('cancel_background_tasks failed. This is probably fine', exc_info=e)
|
2026-02-22 14:32:27 -07:00
|
|
|
# Clear cookies from shared memory before shutdown
|
2026-02-20 14:37:58 -07:00
|
|
|
try:
|
|
|
|
|
lock_cookies()
|
|
|
|
|
except Exception as e:
|
2026-02-22 14:32:27 -07:00
|
|
|
logger.error('Failed to clear cookies from shared memory', exc_info=e)
|
2024-04-17 11:25:26 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@api_app.signal(Event.SERVER_SHUTDOWN_AFTER)
|
|
|
|
|
async def handle_server_shutdown_reset(app: Sanic, loop):
|
|
|
|
|
"""Reset things after shutdown is complete, just in case server is going to start again."""
|
|
|
|
|
reset_shared_contexts(app)
|
|
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@api_app.after_server_start
|
2024-04-17 11:25:26 -06:00
|
|
|
async def start_single_tasks(app: Sanic):
|
|
|
|
|
"""Recurring/Single tasks that are started in only one Sanic process."""
|
|
|
|
|
# Only allow one child process to perform periodic tasks. See `handle_server_shutdown`
|
|
|
|
|
if app.shared_ctx.single_tasks_started.is_set():
|
|
|
|
|
return
|
|
|
|
|
app.shared_ctx.single_tasks_started.set()
|
|
|
|
|
|
|
|
|
|
logger.debug(f'start_single_tasks started')
|
2024-09-26 09:13:29 -06:00
|
|
|
|
2026-10-25 15:17:18 -06:00
|
|
|
# Import configs that don't require the database. Errors are logged so the service will start; configs refuse
|
|
|
|
|
# to save if they failed to import.
|
|
|
|
|
await import_file_configs()
|
2025-08-11 13:05:51 -06:00
|
|
|
wrolpi_config = get_wrolpi_config()
|
2024-09-26 09:13:29 -06:00
|
|
|
|
2026-09-12 15:18:13 -06:00
|
|
|
logger.debug('start_single_tasks waiting for db...')
|
|
|
|
|
async with flags.db_up.wait_for():
|
|
|
|
|
logger.debug('start_single_tasks db is up')
|
|
|
|
|
|
|
|
|
|
# The disk scan must overwrite the DB-restored `outdated_zims` value, so restore first (idempotent), then scan.
|
2024-09-26 09:13:29 -06:00
|
|
|
from modules.zim.lib import flag_outdated_zim_files
|
2024-04-17 11:25:26 -06:00
|
|
|
try:
|
2026-09-12 15:18:13 -06:00
|
|
|
flags.init_flags()
|
2024-04-17 11:25:26 -06:00
|
|
|
flag_outdated_zim_files()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to flag outdated Zims', exc_info=e)
|
|
|
|
|
|
2026-01-31 19:48:10 -07:00
|
|
|
# Import configs that require the database.
|
|
|
|
|
if wrolpi_config.successful_import and not wrolpi_config.wrol_mode:
|
2026-02-14 17:21:39 -07:00
|
|
|
from wrolpi.common import import_all_db_configs
|
|
|
|
|
await import_all_db_configs()
|
2026-01-31 19:48:10 -07:00
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
if flags.refresh_complete.is_set():
|
|
|
|
|
# Set all downloads to new.
|
db: request-path library functions use the request's connection (#774)
* db: request-path library functions use the request's connection
Library functions an API request reaches now take the request's Session (first argument) instead of
opening their own session or cursor, so a request needs one SQLite connection. Raw SQL runs through the
new `wrolpi.db.session_curs(session)`, a cursor on the Session's connection; `RequestDB.curs()` uses it.
- Search: files, videos, archives, flasher, and docs search share the request connection for the total,
the page of ids, and the page's models (3 connections -> 1).
- Zim search, including tag-filtered search and estimates (2 -> 1).
- Channel detail (`Channel.get_statistics` reads on the Channel's own Session) and the channel list.
- Video detail, edit, description, captions, comments and transcode. `get_video_for_app` and
`update_video` no longer commit; their handlers use `request.ctx.db.write()`, so editing a Video and
reading it back is one transaction.
- Tags (list, overlaps, recent), file statistics and estimates, other estimates, bulk tag preview, the
download list, retry, kill and the status summary, and docs/videos/archive statistics.
Not converted: file operations that refresh files inside the request (delete, rename, upload, channel
and collection tag/move) commit page by page in code shared with the file worker; the cached zim entry
lookup; config import.
The connection-count tests now assert one connection for every endpoint they cover (no xfails left), and
cover the videos and archive statistics endpoints, which had no API test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* downloads: startup retries in a write session of its own
start_single_tasks still called retry_downloads() without a Session after it began requiring one, so any
start after a completed refresh raised TypeError there: incomplete downloads were not reset and the
download_on_startup enable after it never ran. Startup is not a request, so
DownloadManager.retry_downloads_on_startup() opens and commits its own short write transaction.
Tested through the method, not start_single_tasks: importing main registers its startup listeners on the
shared app, and every later test in that worker then runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 11:05:20 -06:00
|
|
|
download_manager.retry_downloads_on_startup()
|
2026-10-13 16:07:04 -06:00
|
|
|
else:
|
|
|
|
|
file_worker.queue_first_refresh()
|
2021-11-01 16:42:52 -06:00
|
|
|
|
2025-12-19 17:40:43 -07:00
|
|
|
# Hotspot/throttle on startup are now handled by the Controller service.
|
2022-02-16 21:57:27 -07:00
|
|
|
|
2024-12-24 15:14:05 -07:00
|
|
|
if get_wrolpi_config().download_on_startup and not wrol_mode_enabled():
|
|
|
|
|
# Only start downloading when prerequisites have been met.
|
|
|
|
|
try:
|
|
|
|
|
async with flags.have_internet.wait_for(timeout=30):
|
|
|
|
|
if get_download_manager_config().successful_import:
|
|
|
|
|
await download_manager.enable()
|
|
|
|
|
except TimeoutError as e:
|
|
|
|
|
logger.error('Failed to enable download', exc_info=e)
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
logger.debug(f'start_single_tasks done')
|
2021-11-01 16:42:52 -06:00
|
|
|
|
|
|
|
|
|
2026-01-09 14:01:25 -07:00
|
|
|
@perpetual_signal(sleep=1)
|
|
|
|
|
async def perpetual_file_worker_queue():
|
|
|
|
|
file_worker.transfer_queue()
|
2026-08-19 15:25:11 -06:00
|
|
|
processed = await file_worker.process_queue()
|
|
|
|
|
file_worker.check_queue_stall(processed=processed)
|
2026-01-09 14:01:25 -07:00
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@perpetual_signal(sleep=1)
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
@per_worker_task(sleep=1)
|
2024-09-26 09:13:29 -06:00
|
|
|
async def perpetual_check_log_level():
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
"""Copies the global log level into this process's logger. Registered for every server worker AND the
|
|
|
|
|
perpetual process (which does most of the interesting logging)."""
|
2024-04-17 11:25:26 -06:00
|
|
|
log_level = api_app.shared_ctx.log_level.value
|
|
|
|
|
if log_level != logger.getEffectiveLevel():
|
2024-05-19 17:32:37 -06:00
|
|
|
logger.info(f'changing log level from {logger.getEffectiveLevel()} to {log_level}')
|
|
|
|
|
set_log_level(log_level, warn_level=False)
|
2023-09-12 08:34:07 -06:00
|
|
|
|
2022-08-06 05:22:07 +00:00
|
|
|
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
# Every server worker watches the perpetual process and asks the manager to restart it if it dies.
|
|
|
|
|
per_worker_task(sleep=10)(perpetual_process_health_check)
|
|
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@perpetual_signal(sleep=10)
|
2024-04-17 11:25:26 -06:00
|
|
|
async def perpetual_check_db_is_up_worker():
|
|
|
|
|
try:
|
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
|
|
|
if not flags.db_up.is_set():
|
|
|
|
|
# The DB may not exist yet (media drive mounted after startup); try to create/migrate
|
|
|
|
|
# it. Only one worker bootstraps at a time.
|
|
|
|
|
from wrolpi import db_bootstrap
|
|
|
|
|
with db_bootstrap.bootstrap_lock() as acquired:
|
|
|
|
|
if acquired:
|
|
|
|
|
db_bootstrap.ensure_db()
|
2024-09-15 15:33:07 -06:00
|
|
|
flags.check_db_is_up()
|
|
|
|
|
flags.init_flags()
|
2024-06-12 16:39:04 -06:00
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to check db status', exc_info=e)
|
2022-08-06 05:22:07 +00:00
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@perpetual_signal(sleep=10)
|
2024-05-31 23:54:00 -06:00
|
|
|
async def perpetual_have_internet_worker():
|
|
|
|
|
try:
|
The Internet check is configurable and no longer stalls background work (#899)
* The Internet check is configurable and no longer stalls background work
The check connected to one host (one.one.one.one, port 80) with no timeout, in
the perpetual process's event loop. On a network that drops traffic to that
host, each attempt hung for about two minutes and froze every other perpetual
loop (config saves, downloads, the file worker, the media mount check), so
WROLPi reported "No Internet" and "No drive mounted" though both were fine.
- wrolpi.yaml's internet_check_hostname (default one.one.one.one, needs DNS) and
internet_check_ip (default 1.1.1.1) choose what to connect to; the Internet is
up when either answers. Either may be empty. The INTERNET_SERVER environment
variable is gone.
- A connection gives up after 5 seconds.
- The check runs in a thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* The Internet check tries its IP address first
socket.create_connection resolves a hostname before its timeout applies, so a
name lookup that hangs (a firewall dropping DNS) postponed the IP address, which
needs no lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:05:45 -06:00
|
|
|
# In a thread: the check blocks, and would stall every other perpetual loop.
|
|
|
|
|
if await asyncio.to_thread(have_internet_connection):
|
2024-09-15 15:33:07 -06:00
|
|
|
flags.have_internet.set()
|
|
|
|
|
# Check hourly once we have internet.
|
|
|
|
|
await asyncio.sleep(float(Seconds.hour))
|
|
|
|
|
else:
|
|
|
|
|
# Check more often until the internet is back.
|
|
|
|
|
flags.have_internet.clear()
|
2024-10-03 21:58:39 -06:00
|
|
|
except WROLModeEnabled:
|
|
|
|
|
flags.have_internet.clear()
|
2024-09-15 15:33:07 -06:00
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to check if internet is up', exc_info=e)
|
2024-05-31 23:54:00 -06:00
|
|
|
|
|
|
|
|
|
2026-05-14 15:51:32 -06:00
|
|
|
@perpetual_signal(sleep=30)
|
|
|
|
|
async def perpetual_media_mounted_worker():
|
|
|
|
|
try:
|
|
|
|
|
if flags._destinations_have_mounted_storage():
|
|
|
|
|
flags.media_mounted.set()
|
|
|
|
|
# Re-check less often once all destinations are covered.
|
|
|
|
|
await asyncio.sleep(float(Seconds.minute * 5))
|
|
|
|
|
else:
|
|
|
|
|
flags.media_mounted.clear()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to check media mount status', exc_info=e)
|
|
|
|
|
|
|
|
|
|
|
2024-09-15 15:33:07 -06:00
|
|
|
@perpetual_signal(sleep=30)
|
2024-09-26 09:13:29 -06:00
|
|
|
async def perpetual_start_video_missing_comments_download():
|
2024-04-17 11:25:26 -06:00
|
|
|
from modules.videos.video.lib import get_missing_videos_comments
|
2021-12-20 16:08:13 -07:00
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
async with flags.refresh_complete.wait_for():
|
|
|
|
|
# We can't search for Videos missing comments until the refresh has completed.
|
|
|
|
|
pass
|
2021-12-20 16:08:13 -07:00
|
|
|
|
2026-02-19 16:15:18 -07:00
|
|
|
# Check if download missing video comments is enabled.
|
|
|
|
|
config = get_videos_downloader_config()
|
|
|
|
|
if not config.download_missing_video_comments:
|
|
|
|
|
logger.trace('Download missing video comments is disabled, skipping...')
|
|
|
|
|
return
|
|
|
|
|
|
2024-04-17 11:25:26 -06:00
|
|
|
# Wait for download manager to startup.
|
|
|
|
|
await asyncio.sleep(5)
|
2023-03-29 19:30:33 -06:00
|
|
|
|
2024-12-11 11:52:33 -07:00
|
|
|
if download_manager.can_download:
|
2024-05-31 23:54:00 -06:00
|
|
|
try:
|
|
|
|
|
await get_missing_videos_comments()
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to get missing video comments', exc_info=e)
|
|
|
|
|
|
2024-12-11 11:52:33 -07:00
|
|
|
# Sleep one hour.
|
2024-04-17 11:25:26 -06:00
|
|
|
await asyncio.sleep(int(Seconds.hour))
|
2024-12-11 11:52:33 -07:00
|
|
|
else:
|
|
|
|
|
logger.debug('Waiting for downloads to be enabled before downloading comments...')
|
2023-03-29 19:30:33 -06:00
|
|
|
|
2022-07-09 18:51:45 -06:00
|
|
|
|
Apply the offset in a SingleFile saved date, and repair misread Archive dates (#862)
* Apply the offset in a SingleFile saved date
SingleFile writes the browser's local time with its offset ("... 18:36:23 GMT-0700"). The parser dropped
the offset and treated the local time as UTC, so an uploaded Archive's download_datetime was off by the
browser's UTC offset, and a Collection reorganize renamed its files with that shifted timestamp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Repair Archive download datetimes misread from a SingleFile saved date
Archives parsed before the offset fix keep the shifted download_datetime, because the parser only fills
an empty value. A one-time background repair (gated by a wrolpi_kv version, after the first refresh)
re-reads each Archive's saved date and corrects a download_datetime that equals the offset-dropped value.
Any other download_datetime is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Read the whole SingleFile header; finish the date repair only when every Archive was read
The header was read in 1000 bytes, so a long URL cut the saved date's offset off, and the cut-off date was
stored as GMT. The header is now read through the end of its comment, and a malformed offset leaves the
date unset instead of guessing.
The repair is no longer marked done when an Archive could not be read (a missing file is a deleted Archive,
not a failure), and it waits for the media to be mounted, because `refresh_complete` is restored from the
database before the drives are checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:34:29 -06:00
|
|
|
@perpetual_signal(sleep=3600)
|
|
|
|
|
async def perpetual_repair_archive_saved_dates():
|
|
|
|
|
"""Correct Archive download datetimes that were misread from their SingleFile saved date. Runs once."""
|
|
|
|
|
async with flags.refresh_complete.wait_for():
|
|
|
|
|
# Archives must be in the database.
|
|
|
|
|
pass
|
|
|
|
|
async with flags.media_mounted.wait_for():
|
|
|
|
|
# Their files must be readable; `refresh_complete` is restored from the database before the drive is checked.
|
|
|
|
|
pass
|
|
|
|
|
|
|
|
|
|
from modules.archive.lib import repair_singlefile_saved_dates
|
|
|
|
|
try:
|
|
|
|
|
await asyncio.to_thread(repair_singlefile_saved_dates)
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to repair Archive download datetimes', exc_info=e)
|
|
|
|
|
|
|
|
|
|
|
2025-10-29 22:16:33 -06:00
|
|
|
@perpetual_signal(sleep=3600) # Check hourly
|
|
|
|
|
async def perpetual_check_for_updates():
|
|
|
|
|
"""
|
|
|
|
|
Check git remote for new commits on the current branch.
|
|
|
|
|
|
|
|
|
|
Updates shared_ctx with update info for the /api/status endpoint.
|
|
|
|
|
Only runs on native installs (not Docker).
|
|
|
|
|
"""
|
2025-12-02 16:07:32 -07:00
|
|
|
# Local import to avoid loading upgrade module at startup
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
from wrolpi.upgrade import refresh_update_status
|
2025-10-29 22:16:33 -06:00
|
|
|
|
|
|
|
|
# Don't check for updates in Docker environments
|
|
|
|
|
if DOCKERIZED:
|
|
|
|
|
return
|
|
|
|
|
|
2025-12-06 14:47:52 -07:00
|
|
|
# Don't check if setting is disabled
|
|
|
|
|
if not get_wrolpi_config().check_for_upgrades:
|
|
|
|
|
return
|
|
|
|
|
|
2025-10-29 22:16:33 -06:00
|
|
|
# Only check when we have internet
|
|
|
|
|
if not flags.have_internet.is_set():
|
|
|
|
|
return
|
|
|
|
|
|
|
|
|
|
try:
|
api: run the perpetual loops in one Sanic-managed process (#718)
* api: run the perpetual loops in one Sanic-managed process
The singleton background loops (file worker, downloads, switches, jobs,
bulk tag, flag checks, update check) now run in a dedicated process that
Sanic's Worker Manager owns: `run_perpetual_process` is registered with
`app.manager.manage()` from `main_process_ready`. The manager guarantees
a single instance and SIGTERMs it on shutdown. No server worker runs the
loops, so the owner-pid / heartbeat / watchdog claim scheme is deleted
rather than hardened.
`perpetual_signal` keeps its name and is now a plain registrar. A new
`per_worker_task` registrar covers loops every server worker must run for
itself: syncing its log level (previously only the owner did, so the other
workers never picked up level changes) and a health check that asks the
manager to restart the perpetual process when it is FAILED (Sanic marks a
dead managed process but does not restart it).
Downloads are started as background tasks instead of Sanic signals; the
perpetual process has no server, so no finalized signal router. The
update check and the upgrade-check endpoint run git in a thread via
`refresh_update_status`. /api/status lists the perpetual process next
to the servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: survive a predecessor's death, exit cleanly, coordinate restarts
Review fixes for the managed perpetual process.
Shutdown: cancel_background_tasks / cancel_refresh_tasks gather with
return_exceptions, so a cancelled child's CancelledError no longer aborts
the shutdown that called them (the process exited 1 with a download in
flight).
Restart after a crash: reconcile_after_predecessor heals shared locks a
dead process left held, fails RUNNING Jobs owned by a dead pid, clears
processing_domains, renews pending Downloads, and resets the file-worker
status before the loops start. N server workers issue one restart
request per cooldown through a shared timestamp in shared_ctx.
The log-level sync now also runs in the perpetual process. Manage() is
registered through register_perpetual_process so it is testable; the
entry point refuses to run with no loops registered (spawn).
Tests exercise the real entry point in a forked process with a live
background task under SIGINT and SIGTERM, Sanic's real WorkerManager
marking a crashed managed process FAILED and reviving it on restart,
Sanic's process naming and ProcessState enum, the listener registration,
and the reconcile step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perpetual: import the jobs helper inside its reconcile step
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 21:52:55 -06:00
|
|
|
# Runs `git fetch` in a thread and stores the result in shared_ctx.status for all workers.
|
|
|
|
|
result = await refresh_update_status(api_app.shared_ctx.status)
|
2025-10-29 22:16:33 -06:00
|
|
|
|
|
|
|
|
if result.get('update_available'):
|
|
|
|
|
logger.info(f"Update available: {result['commits_behind']} commits behind on {result['branch']}")
|
2025-12-02 15:34:43 -07:00
|
|
|
except WROLModeEnabled:
|
|
|
|
|
logger.debug('Refusing to check for updates when WROL Mode is enabled.')
|
2025-10-29 22:16:33 -06:00
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Failed to check for updates', exc_info=e)
|
|
|
|
|
|
|
|
|
|
|
2021-11-01 16:42:52 -06:00
|
|
|
if __name__ == '__main__':
|
2022-10-09 18:42:23 -06:00
|
|
|
sys.exit(main())
|