wrolpi/docker-compose.yml

156 lines
4.6 KiB
YAML
Raw Normal View History

# Development containers, these should only be used if you are developing.
Postgres -> SQLite hard cutover: the database moves into the media directory. The database now lives at <media>/config/wrolpi.db so a library (files, configs, AND database) travels with the drive between WROLPi systems. Core: - wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s, foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3 adapters keep raw-SQL datetime formats identical to the ORM's. - wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN. Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines. Stemming contract pinned by dedicated tests (watch/watches/watched/ watching are interchangeable). - wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic baseline and test DBs; fixes the channel video_count DELETE/UPDATE bugs and a latent total_size bug (it used the video id as channel id). - wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor; network-filesystem refusal; never creates/migrates/trusts a database beneath an unmounted production media directory (shadow-DB guard). - Alembic: 85 Postgres-era migrations squashed into one baseline; revision ids are dates (2026_07_10_1350) so filenames self-sort and alembic_version says when a schema was created; render_as_batch=True. - All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> -> HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM, generate_series histograms->strftime GROUP BY, DISTINCT ON-> row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted to SQLAlchemy instead (flag upsert, download retry/summary/tag checks, channel-by-tag estimate, Directory delete). - Config imports count skipped entries; background dumps are blocked after a lossy import so partial imports can never overwrite YAML configs (they are the sole source of truth for rebuilds now). - Search tolerates explicit JSON nulls for limit/offset (Postgres treated OFFSET NULL as 0; SQLite errors). - file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite (BIGINT is not a rowid alias; FTS5 content_rowid requires it). Ops teardown: docker-compose db service, CI postgres sidecars, initialize_api_db.sh, controller postgresql service, psycopg2-binary all removed. upgrade.sh stops/disables postgresql (data is NOT deleted; `systemctl enable --now postgresql` restores it); systemd units no longer Want postgresql.service. Existing-user upgrade: first boot creates+migrates the DB, configs import (tags recreate their FileGroups from disk), a refresh rebuilds the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the Postgres baseline exactly; survived an uncontrolled hard reboot with clean integrity checks. Accepted behavior changes: FTS5 has no stopwords; bm25 length normalization shifts some rank orders; fast-path d_headline is un-highlighted; all-negative queries match nothing. Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked rebuild refresh with progress UX. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
# The database is SQLite inside the media directory (./test/config/wrolpi.db); no db service.
version: '3'
services:
controller: # WROLPi Controller Service
build:
context: .
dockerfile: docker/controller/Dockerfile
restart: unless-stopped
ports:
- "8080:80"
volumes:
# Controller code (for development hot-reload)
- ./controller:/app/controller
# Docker socket for container management
- /var/run/docker.sock:/var/run/docker.sock
# Host system info (read-only)
- /proc:/host/proc:ro
- /sys:/host/sys:ro
# Media directory (needed for Samba share path validation)
- ${MEDIA_DIRECTORY:-./test}:/media/wrolpi
environment:
- DOCKERIZED=true
- MEDIA_DIRECTORY=/media/wrolpi
# Override CMD to enable auto-reload for development
command: ["python", "-m", "controller.serve"]
healthcheck:
test: [ 'CMD-SHELL', 'curl -f http://127.0.0.1:80/api/health || exit 1' ]
interval: 30s
timeout: 5s
start_period: 10s
retries: 3
api:
depends_on:
controller:
condition: service_started
build:
context: .
dockerfile: docker/api/Dockerfile
environment:
# Sanic 25.x wants 'spawn' but our code uses 'fork' for multiprocessing.
# This tells Sanic to accept the existing start method.
- SANIC_START_METHOD_SET=1
volumes:
- './main.py:/opt/wrolpi/main.py'
- './wrolpi:/opt/wrolpi/wrolpi'
- './modules:/opt/wrolpi/modules'
- './scripts:/opt/wrolpi/scripts'
- './test:/opt/wrolpi/test'
- './test:/media/wrolpi'
- './alembic:/opt/wrolpi/alembic'
- './alembic.ini:/opt/wrolpi/alembic.ini'
ports:
- ${REACT_APP_API-0.0.0.0:8081}:8081
user: '${UID-1000}:${GID-1000}'
healthcheck:
test: [ 'CMD-SHELL', 'curl http://127.0.0.1:8081/api/echo' ]
interval: 2m
archive: # Python/Sanic wrapper around Singlefile and Readability.
build:
context: .
dockerfile: docker/archive/Dockerfile
volumes:
- './docker/archive/main.py:/app/main.py'
ports:
- '8083:8080'
healthcheck:
test: [ 'CMD-SHELL', 'curl http://127.0.0.1:8080/' ]
interval: 2m
app: # React App
depends_on:
- api
build:
context: .
dockerfile: docker/app/Dockerfile
volumes:
- './app/public:/app/public'
- './app/src:/app/src'
- './app/package.json:/app/package.json'
- '.env:/app/.env'
command: 'npm run start'
stdin_open: true
healthcheck:
test: [ 'CMD-SHELL', 'curl http://127.0.0.1:3000' ]
interval: 1m
user: '${UID-1000}:${GID-1000}'
Fixing changes before merge. Moving FilesRefreshButton and DirectoryRefreshButton out of browser file. Moving file fetching tino FileBrowser. Removing sub-folders from Files display when any parent folder is clicked. Using closed captioning for vtt/srt file icon. Using my own react-semantic-toasts-2 to use toasts in Node 18. Fixing bug where more than the last page was displayed. Upgrading yt-dlp. Fixing sub-folder file selection. Adding rowKey to SortableTable to give React a key for each row. Renaming key to path. Removing duplicate urls. Unifying file modals. Displaying modals for image and text files. Handling refresh loading in FileBrowser. Creating refreshing_directory flag. Removing accidental commit. Sorting by url_count. Using SortableTable on Domains page. Using SortableTable for Channels page. Created new FileBrowser which allows browsing of files in the media directory. Using Node 18 in pi-gen. Creating new FileBrowser to replace outdated react-keyed-file-browser. Returning mimetype with file list. Do not display 'Do not download' on Channel download in Dashboard. Allowing null channel download frequency so user can stop downloading a channel. Bumping version for Node 18. Using my own semantic-ui-offline to avoid pulling Google fonts. Upgrading app docker container to use Node 18. This requires the removal of several npm packages. File browser will require re-write. It now pulls fonts from Google, this will need to be fixed. Testing DownloadsToggle click. Testing trinary states of DownloadsToggle. Installing @testing-library/user-event for testing App.
2023-01-31 18:08:07 -07:00
environment:
# Force React to use Caddy port to connect to it's websocket.
Fixing changes before merge. Moving FilesRefreshButton and DirectoryRefreshButton out of browser file. Moving file fetching tino FileBrowser. Removing sub-folders from Files display when any parent folder is clicked. Using closed captioning for vtt/srt file icon. Using my own react-semantic-toasts-2 to use toasts in Node 18. Fixing bug where more than the last page was displayed. Upgrading yt-dlp. Fixing sub-folder file selection. Adding rowKey to SortableTable to give React a key for each row. Renaming key to path. Removing duplicate urls. Unifying file modals. Displaying modals for image and text files. Handling refresh loading in FileBrowser. Creating refreshing_directory flag. Removing accidental commit. Sorting by url_count. Using SortableTable on Domains page. Using SortableTable for Channels page. Created new FileBrowser which allows browsing of files in the media directory. Using Node 18 in pi-gen. Creating new FileBrowser to replace outdated react-keyed-file-browser. Returning mimetype with file list. Do not display 'Do not download' on Channel download in Dashboard. Allowing null channel download frequency so user can stop downloading a channel. Bumping version for Node 18. Using my own semantic-ui-offline to avoid pulling Google fonts. Upgrading app docker container to use Node 18. This requires the removal of several npm packages. File browser will require re-write. It now pulls fonts from Google, this will need to be fixed. Testing DownloadsToggle click. Testing trinary states of DownloadsToggle. Installing @testing-library/user-event for testing App.
2023-01-31 18:08:07 -07:00
- WDS_SOCKET_PORT=0
web: # Caddy reverse proxy - handles HTTPS for all services.
depends_on:
- app
- zim
- help
build:
context: .
dockerfile: docker/web/Dockerfile
ports:
- ${WEB_HOST-0.0.0.0}:${WEB_HTTPS_PORT-8443}:443
- '8084:8084'
- '8085:8085'
- '8086:8086'
volumes:
- './docker/web/Caddyfile:/etc/caddy/Caddyfile'
- './icon.ico:/etc/caddy/favicon.ico'
- './icon.png:/etc/caddy/icon.png'
- './test:/opt/media'
- './modules/map/static:/opt/wrolpi/modules/map/static:ro'
- './blobs:/opt/wrolpi-blobs:ro'
environment:
- MEDIA_DIRECTORY=/opt/media
# Set EXTRA_SANS to include the host's LAN IP in the TLS certificate.
# Example: EXTRA_SANS=IP:10.0.0.5
- EXTRA_SANS=${EXTRA_SANS:-}
healthcheck:
test: [ 'CMD-SHELL', 'wget --spider --no-check-certificate https://127.0.0.1:443 || exit 1' ]
interval: 1m
Squashed commit of the following: commit 26bde0fb48d59a86c2fb9f53a7218d49f22d4860 Author: Roland <roland@learningselfreliance.com> Date: Tue Mar 8 13:25:56 2022 -0700 Using osm2pgsql append to preserve other imported maps. commit c6c52623eaf02e691b1e75dcf39c3a8334ead55a Author: Roland <roland@learningselfreliance.com> Date: Tue Mar 8 08:20:38 2022 -0700 Using placeholder instead of loader when table is loading. commit 44fec6490e26603c88a98f6bc70fc18686e17f09 Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 20:58:34 2022 -0700 Simplifying tableRow function. commit 11b8b57f1fa59556397620f3cad9a33a0237501d Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 20:53:51 2022 -0700 Removing scientific humanFileSize. commit 0a09d0a6963afebed7f388aad71bbdd3151d2e9e Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 20:46:22 2022 -0700 Fixing active loader. commit 07f4deeac861351d7483ef57278eb2e0941e31fb Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 20:40:44 2022 -0700 Increasing recommended map file size. commit af3ac492c81f4c8f84563a448bc272ae9334d32d Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 20:35:13 2022 -0700 Reworking logic order for simplicity. Fetching downloads on interval. commit 3d9d31fcec87ed02ced7f67c34907faa5a3b54e5 Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 20:24:39 2022 -0700 Handling when pbf import status are loading. commit 3798961a48e7c9eb165ca4e4051f239dbb088d0c Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 19:53:50 2022 -0700 Fetching import status every 30 seconds. commit f7a8193b0955d14ead8090aa74ce3c5a681b57ec Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 19:33:17 2022 -0700 Stopping renderd during installation. commit 5bf050b59e0a6e391184596f5e6f0737dea1221d Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 19:26:59 2022 -0700 Removing duplciate serve. commit 892e3315eeeb5d2516996b901a32a73a66dae078 Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 19:22:14 2022 -0700 Setting bash flags in sub-install script. commit d5d2acd2cd85e089c78aa85a27c5d966222a1347 Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 19:20:19 2022 -0700 Moving bulk of installation script into separate script so the latest code will be used during initial installation. commit ab030d8af834a4a8772edd06df360c51e72fbb02 Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 18:40:34 2022 -0700 Removing mod_tile/ajt directory instead of entire contents of mod_tile. Showing import as relative to media directory. commit c05199db4ecf56f4d096051b58689457882af12a Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 18:39:59 2022 -0700 Remvoing marker from map. commit ba5df4cf62d61b55281277a6254836e973c766ae Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 11:57:41 2022 -0700 Creating import_map.sh script. commit 1a16e46130e775efe375adb22917ba12e44855cd Author: Roland <roland@learningselfreliance.com> Date: Mon Mar 7 11:11:01 2022 -0700 Increasing ram usage during import to avoid hanging import. commit 0f5ce1a9f806f1c0878dd804b04587b6340ffd16 Author: Roland <roland@learningselfreliance.com> Date: Sat Mar 5 19:19:47 2022 -0700 Increasing map_file.size size. commit eb6e01512af59fb4a18ab15643c7bc61f3d48e53 Author: Roland <roland@learningselfreliance.com> Date: Sat Mar 5 18:55:55 2022 -0700 Copying leaflet js/css files into docker container and to Apache directory so they will be available offline. commit 5b687324e119f021673f42e8d079b83034b70d53 Author: Roland <roland@learningselfreliance.com> Date: Sat Mar 5 18:36:06 2022 -0700 Matching tab height on Map page. commit 8031786d672e18366e9a2eb2d7bdfcd8611d27f3 Author: Roland <roland@learningselfreliance.com> Date: Sat Mar 5 18:34:20 2022 -0700 Shrinking HelpPopup Icon. commit 5ea94488e43bbc5c414e61c9173a3cfa908c7104 Author: Roland <roland@learningselfreliance.com> Date: Sat Mar 5 18:31:30 2022 -0700 Testing if no PBFs are requested to be imported. commit c4e86be1faed78bd3af1f3487471c01a1958a433 Author: Roland <roland@learningselfreliance.com> Date: Sat Mar 5 18:18:48 2022 -0700 Using Icon instead of Button for HelpPopup. commit 51d94f6d85885803813544fea16a56372babaa82 Author: Roland <roland@learningselfreliance.com> Date: Sat Mar 5 18:07:03 2022 -0700 PBF files are sorted by the api. commit 40f3a7ff04a83db0153f64a88a37f1d56ce043aa Author: Roland <roland@learningselfreliance.com> Date: Fri Mar 4 18:35:51 2022 -0700 Adding help popup. commit 1da8d9fa403ec2a3b3c16c11a3937de37c895a35 Author: Roland <roland@learningselfreliance.com> Date: Fri Mar 4 09:41:01 2022 -0700 Moving message up. Clearing cached tiles after import. commit 3192adf95192554c194c9e0a8aeb149822462682 Author: Roland <roland@learningselfreliance.com> Date: Thu Mar 3 21:37:13 2022 -0700 Removing osm2pgsql from api docker because it seems to be unsupported. commit 0e10522f3cdc7edd374c14f230bb2dbd68ab1a94 Author: Roland <roland@learningselfreliance.com> Date: Thu Mar 3 21:36:39 2022 -0700 Disabling import button for docker. Adding download instructions. commit e1d8f55d159fb4d4ef2ece859ef7cc1059885791 Author: Roland <roland@learningselfreliance.com> Date: Thu Mar 3 21:14:55 2022 -0700 Map files can be imported on RPi. commit 3128530e09968430a71b27cd2e75a567f85a708b Author: Roland <roland@learningselfreliance.com> Date: Thu Mar 3 15:31:54 2022 -0700 Adding shm_size to increase size to render large map tiles. commit f3cf80fae0ce46f44ea9ac5033f68a32d0197ae3 Author: Roland <roland@learningselfreliance.com> Date: Wed Mar 2 10:39:18 2022 -0700 Can get status of map imports. commit 3b7bbce004cd55a5d89980c2c09124b72c450bc6 Author: Roland <roland@learningselfreliance.com> Date: Tue Mar 1 20:51:13 2022 -0700 Adding map link to 50x.html. Swapping links using hostname to make it easier on the user. commit a8b6b60fe182233f40c2f1ecd825636c3d37fd23 Author: Roland <roland@learningselfreliance.com> Date: Tue Mar 1 12:01:50 2022 -0700 Moving Inventory link to end. commit c5346b8dc2ff83262d6fd153130912809dfd7574 Author: Roland <roland@learningselfreliance.com> Date: Tue Mar 1 12:00:47 2022 -0700 Squashed commit of the following: commit 39f0b8f3fc049fffd140d551cd91d7d9dba6d416 Author: Roland <roland@learningselfreliance.com> Date: Tue Mar 1 12:00:13 2022 -0700 Starting map refresh function. commit 2fac9ddf07e378389ddd85ca972f33b3323d30ac Author: Roland <roland@learningselfreliance.com> Date: Mon Feb 28 16:43:47 2022 -0700 Updating readme with map requirements. commit 8325e1ea89537128329e313ab4b5588f8809d1c9 Author: Roland <roland@learningselfreliance.com> Date: Mon Feb 28 16:05:35 2022 -0700 Fixing order of map db init scripts. Creating index.html to display map. Using :8084 index file to display the map. commit 8f81539dd9d25fc26607f472c156d41ddcfc11f5 Author: Roland <roland@learningselfreliance.com> Date: Sun Feb 27 19:17:14 2022 -0700 Installing apache2. commit c7aaef07f8291070aeaf5e7df553e34aca0719ec Author: Roland <roland@learningselfreliance.com> Date: Sun Feb 27 18:41:07 2022 -0700 Always configure postgres extensions. Initialize gis with dc. commit aa725ed4b8ec46d17accebd96dd41508ea2d9caf Author: Roland <roland@learningselfreliance.com> Date: Sun Feb 27 15:48:20 2022 -0700 Map is displayed as iframe. commit de9bb92dcaf44996e7727adce7774f01895ddf52 Author: Roland <roland@learningselfreliance.com> Date: Sun Feb 27 12:16:56 2022 -0700 Running script in openstreetmap directory. commit f277388095d2221aa6407643ab655f26dc349eb7 Author: Roland <roland@learningselfreliance.com> Date: Sun Feb 27 12:06:41 2022 -0700 Bumping version for map. commit afd4aaeffb2aced2dc5c1a8f94038885c8287265 Author: Roland <roland@learningselfreliance.com> Date: Sun Feb 27 11:59:31 2022 -0700 Checking that the installation script is on an rpi. commit a799382ceac3c201fc01d41fd63e37607282831a Author: Roland <roland@learningselfreliance.com> Date: Sun Feb 27 11:49:21 2022 -0700 Only running mod_tile install once. commit 0b98bb19cd4d86c9630a863cab4eb46410d1c9d1 Author: Roland <roland@learningselfreliance.com> Date: Fri Feb 25 11:31:21 2022 -0700 Building out map install script. commit 7c799775c11df573fa6cd65451adea28c0832cb0 Author: Roland <roland@learningselfreliance.com> Date: Thu Feb 24 12:16:53 2022 -0700 Creating OSM installation script. commit d3b93a3ee97d1ae4a00b6a0423ed59ea8b897ccd Author: Roland <roland@learningselfreliance.com> Date: Tue Feb 22 15:39:04 2022 -0700 Removing unused map module.
2022-03-08 13:26:03 -07:00
zim: # kiwix-serve
build:
context: .
dockerfile: docker/zim/Dockerfile
volumes:
- './test:/media/wrolpi'
- './docker/zim/entrypoint.sh:/entrypoint.sh'
healthcheck:
test: [ 'CMD-SHELL', 'curl http://127.0.0.1:80' ]
interval: 1m
samba: # Samba file sharing - started/stopped by the controller based on configured shares.
build:
context: .
dockerfile: docker/samba/Dockerfile
restart: "no"
depends_on:
controller:
condition: service_healthy
ports:
- "445:445"
volumes:
- ${MEDIA_DIRECTORY:-./test}:/media/wrolpi
help: # The WROLPi Help documentation.
build:
context: .
dockerfile: docker/help/Dockerfile
volumes:
- './docker/help/wrolpi-help:/opt/wrolpi-help'
healthcheck:
test: [ 'CMD-SHELL', 'curl http://127.0.0.1:8086' ]
interval: 1m