2021-11-01 16:42:52 -06:00
# WROLPi
2026-10-08 17:19:38 -06:00
[](https://github.com/lrnselfreliance/wrolpi/actions/workflows/backend.yml?query=branch%3Amaster)
2021-11-01 16:42:52 -06:00
[](https://github.com/lrnselfreliance/wrolpi/blob/master/LICENSE)
2022-03-03 09:55:38 -07:00
[](https://discord.gg/HrwFk7nqA2)
2023-02-03 00:02:02 +01:00
[](https://gitpod.io/#https://github .com/lrnselfreliance/wrolpi)
2021-11-01 16:42:52 -06:00
2022-11-13 14:20:48 -07:00
**Create your own off-grid library.**
2021-11-01 16:42:52 -06:00
< p align = "center" >
< img width = "256px" src = "https://raw.githubusercontent.com/lrnselfreliance/wrolpi/master/icon.png" alt = 'WROLPi Logo' >
< / p >
2022-11-13 14:36:13 -07:00
WROLPi is a self-contained collection of software to help you survive the world Without Rule of Law.
2021-11-01 16:42:52 -06:00
2022-11-13 14:36:13 -07:00
WROLPi is intended to be run on a Raspberry Pi with an optional external drive attached. It serves up it's own Wi-Fi
network so that any user with a laptop/tablet/phone can connect and use the library created by the maintainer.
2021-11-01 16:42:52 -06:00
2022-05-03 11:13:29 -06:00
# Table of Contents
* [Features ](#features )
* [Demo ](#demo )
2023-02-23 12:09:19 -07:00
* [Download ](#download )
2022-05-03 11:13:29 -06:00
* [Try WROLPi ](#try-wrolpi )
2026-05-28 21:33:22 -06:00
* [Install WROLPi ](INSTALL.md )
Replace nginx with Caddy for TLS termination and reverse proxying.
- Replace nginx with Caddy as the web server for both Docker and native
(RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
proxying to all services (API, React app, Controller, Kiwix, Help, Map),
and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
uses Name Constraints to restrict signing to local addresses only
(.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
certificate installation on iOS, Android, macOS, Windows, Linux, and
Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
/download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.
2026-03-07 12:08:27 -07:00
* [Docker Documentation ](docker/README.md )
2022-05-03 11:13:29 -06:00
* [Charter ](#charter )
* [Upgrading WROLPi ](UPGRADE.md )
2023-07-12 09:28:55 -06:00
* [Join ](#join )
2022-05-03 11:13:29 -06:00
2021-11-01 16:42:52 -06:00
# Features
2022-12-17 14:51:34 -07:00
- [x] Videos
2023-07-12 09:28:55 -06:00
- [x] Web Archives
2022-12-17 14:51:34 -07:00
- [x] Maps
2023-07-12 09:28:55 -06:00
- [x] Wikipedia
2022-12-29 11:25:43 -07:00
- [x] File search
2022-12-29 12:09:32 -07:00
- [x] eBooks (EPUB / PDFs)
2022-12-17 14:51:34 -07:00
- [x] Wi-Fi Hotspot
- [x] Universal search
- [x] Food Inventory management
- [x] One-Time pad generator, encrypter/decrypter
2023-07-12 09:28:55 -06:00
- [ ] Synchronizer/duplicator
2022-12-17 14:51:34 -07:00
- [ ] Food storage calculator
2023-02-23 12:09:19 -07:00
## Module features matrix
2022-12-17 14:51:34 -07:00
2023-07-12 09:28:55 -06:00
| **Module** | **View** | **Search** | **Download** | **Project** |
|----------------|----------|:-----------|--------------|------------------------------------------------------------|
| Videos | yes | yes | yes | [yt-dlp ](https://github.com/yt-dlp/yt-dlp ) |
| Web Archives | yes | yes | yes | [Singlefile ](https://github.com/gildas-lormeau/SingleFile ) |
| Wikipedia | yes | yes | yes | [Kiwix ](https://www.kiwix.org ) |
| eBooks | yes | yes | planned | EPUB/PDF |
2026-04-11 20:09:30 -06:00
| Map | yes | yes | yes | [Protomaps ](https://protomaps.com/ ) |
2023-07-12 09:28:55 -06:00
| Podcasts/Audio | yes | planned | planned | |
2022-12-17 11:28:29 -07:00
2022-01-21 10:19:56 -07:00
# Demo
2022-05-03 11:00:09 -06:00
2022-12-12 11:55:22 -07:00
< a href = "https://www.youtube.com/watch?v=Qz-FuenRylQ" > </ a >
2022-01-21 10:19:56 -07:00
2023-02-23 12:09:19 -07:00
# Download
2023-07-12 09:28:55 -06:00
Images are available at [wrolpi.org ](https://wrolpi.org )
2023-02-23 12:09:19 -07:00
2021-11-01 16:42:52 -06:00
# Try WROLPi!
You can try out WROLPi by running the docker containers.
1. [Install docker-ce ](https://docs.docker.com/install/linux/docker-ce/debian/ ) and
[docker-compose ](https://docs.docker.com/compose/install/ )
2023-07-12 09:28:55 -06:00
2. Copy the latest WROLPi code.
2022-03-19 17:49:29 -06:00
* `git clone https://github.com/lrnselfreliance/wrolpi.git`
2023-07-12 09:28:55 -06:00
3. Change directory into the code base.
2021-11-01 16:42:52 -06:00
* `cd wrolpi`
2024-08-05 17:06:49 -06:00
4. Initialize git submodules
* `git submodule update --init`
5. Build the containers.
2022-05-03 11:00:09 -06:00
* `docker-compose build --parallel`
2024-08-05 17:06:49 -06:00
6. Create volumes.
2022-05-03 11:00:09 -06:00
* `docker volume create --name=openstreetmap-data`
* `docker volume create --name=openstreetmap-rendered-tiles`
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
7. Initialize the database (SQLite in the media directory).
2022-01-03 16:36:58 -07:00
* `docker-compose run --rm api db upgrade`
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
8. Start the docker containers.
2021-11-01 16:42:52 -06:00
* `docker-compose up`
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.
Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
Stemming contract pinned by dedicated tests (watch/watches/watched/
watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
baseline and test DBs; fixes the channel video_count DELETE/UPDATE
bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
network-filesystem refusal; never creates/migrates/trusts a database
beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
revision ids are dates (2026_07_10_1350) so filenames self-sort and
alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
generate_series histograms->strftime GROUP BY, DISTINCT ON->
row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted
to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
after a lossy import so partial imports can never overwrite YAML
configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
(BIGINT is not a rowid alias; FTS5 content_rowid requires it).
Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed. upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.
Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.
Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.
Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:20 -06:00
9. Browse to WROLPi.
2024-06-12 18:44:59 -06:00
* https://0.0.0.0:8443
2021-11-01 16:42:52 -06:00
Replace nginx with Caddy for TLS termination and reverse proxying.
- Replace nginx with Caddy as the web server for both Docker and native
(RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
proxying to all services (API, React app, Controller, Kiwix, Help, Map),
and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
uses Name Constraints to restrict signing to local addresses only
(.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
certificate installation on iOS, Android, macOS, Windows, Linux, and
Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
/download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.
2026-03-07 12:08:27 -07:00
More Docker documentation is available in [docker/README.md ](docker/README.md )
2026-05-28 21:33:22 -06:00
# Install WROLPi
Full installation instructions are in [INSTALL.md ](INSTALL.md ). Pick the option that matches your hardware:
* **[WROLPi Portable (Live USB) ](INSTALL.md#wrolpi-portable-live-usb )** — boot a complete WROLPi from a USB stick on any x86 PC or laptop. Nothing is installed to the computer; the first boot creates a persistence partition on the USB so your library, database, and configuration survive reboots.
2026-10-13 15:57:10 -06:00
* **[Debian 13 ](INSTALL.md#debian-13-install )** — install WROLPi onto an x86 PC or laptop.
2026-05-28 21:33:22 -06:00
* **[Raspberry Pi ](INSTALL.md#raspberry-pi-install )** — install WROLPi onto a Raspberry Pi with an external drive.
* **[Docker ](docker/README.md )** — run the WROLPi containers on an existing Linux host.
2023-07-12 09:28:55 -06:00
2021-11-01 16:42:52 -06:00
# Charter
2023-05-29 18:04:03 -06:00
## Guiding Principals
1. Storage is cheaper than power.
2. Two is one, one is none.
2023-10-09 22:08:50 -06:00
3. Run silently.
2024-06-17 10:40:30 -06:00
4. Primary, secondary, tertiary.
2025-01-01 17:58:03 -07:00
5. Secure as a bookshelf in your home.
2023-05-29 18:04:03 -06:00
2021-11-01 16:42:52 -06:00
## Physical properties
1. A WROLPi instance should be capable of running with a minimal of hardware:
* Raspberry Pi
* SD card
* External USB hard drive
* Power supply and cables
2023-05-29 18:04:03 -06:00
* Wi-Fi USER device such as a phone, tablet, or laptop.
2023-02-23 12:09:19 -07:00
2. A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be
2021-11-01 16:42:52 -06:00
scarce when WROLPi is needed most.
## User expectations
1. A WROLPi instance should be run and maintained by a person (MAINTAINER) with a moderate amount of Linux and Raspberry
Pi experience. It is expected that they should be able to do this using only the documentation on their WROLPi.
## Software properties and capabilities
1. A WROLPi instance should be able to "factory-reset" itself without any outside services.
2023-02-23 12:09:19 -07:00
2. WROLPi should function completely without any internet services.
3. A user should have easy access to their data if WROLPi fails:
2021-11-01 16:42:52 -06:00
* For example, a user can watch their videos by navigating a short and intuitive directory structure and opening the
video in their preferred video player.
2023-02-23 12:09:19 -07:00
4. WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER.
2021-11-01 16:42:52 -06:00
* If the UI isn't functional, the code should be documented such that a user can restore functionality.
2023-02-23 12:09:19 -07:00
5. WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or
2021-11-01 16:42:52 -06:00
processing captions. This is to ensure that when a user adds content during non-WROL time, the processing for optimum
performance is already done for a WROL event.
# Join!
< p >
< img src = "https://raw.githubusercontent.com/lrnselfreliance/wrolpi/master/join_discord.png" alt = 'Discord QR Code' >
< / p >
[Join our Discord ](https://discord.gg/HrwFk7nqA2 )
