mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-26 19:01:39 -04:00
d9303a0ea8
The database now lives at <media>/config/wrolpi.db so a library (files, configs, AND database) travels with the drive between WROLPi systems. Core: - wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s, foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3 adapters keep raw-SQL datetime formats identical to the ORM's. - wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN. Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines. Stemming contract pinned by dedicated tests (watch/watches/watched/ watching are interchangeable). - wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic baseline and test DBs; fixes the channel video_count DELETE/UPDATE bugs and a latent total_size bug (it used the video id as channel id). - wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor; network-filesystem refusal; never creates/migrates/trusts a database beneath an unmounted production media directory (shadow-DB guard). - Alembic: 85 Postgres-era migrations squashed into one baseline; revision ids are dates (2026_07_10_1350) so filenames self-sort and alembic_version says when a schema was created; render_as_batch=True. - All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> -> HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM, generate_series histograms->strftime GROUP BY, DISTINCT ON-> row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit NULLS FIRST/LAST, LIMIT-before-OFFSET). Simple statements converted to SQLAlchemy instead (flag upsert, download retry/summary/tag checks, channel-by-tag estimate, Directory delete). - Config imports count skipped entries; background dumps are blocked after a lossy import so partial imports can never overwrite YAML configs (they are the sole source of truth for rebuilds now). - Search tolerates explicit JSON nulls for limit/offset (Postgres treated OFFSET NULL as 0; SQLite errors). - file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite (BIGINT is not a rowid alias; FTS5 content_rowid requires it). Ops teardown: docker-compose db service, CI postgres sidecars, initialize_api_db.sh, controller postgresql service, psycopg2-binary all removed. upgrade.sh stops/disables postgresql (data is NOT deleted; `systemctl enable --now postgresql` restores it); systemd units no longer Want postgresql.service. Existing-user upgrade: first boot creates+migrates the DB, configs import (tags recreate their FileGroups from disk), a refresh rebuilds the library index. QA'd on the 10.0.0.8 RPi: rebuild matched the Postgres baseline exactly; survived an uncontrolled hard reboot with clean integrity checks. Accepted behavior changes: FTS5 has no stopwords; bm25 length normalization shifts some rank orders; fast-path d_headline is un-highlighted; all-negative queries match nothing. Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked rebuild refresh with progress UX. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
145 lines
6.4 KiB
Markdown
145 lines
6.4 KiB
Markdown
# WROLPi
|
|
|
|
[](https://app.circleci.com/pipelines/github/lrnselfreliance/wrolpi?branch=release&filter=all)
|
|
[](https://github.com/lrnselfreliance/wrolpi/blob/master/LICENSE)
|
|
[](https://discord.gg/HrwFk7nqA2)
|
|
[](https://gitpod.io/#https://github.com/lrnselfreliance/wrolpi)
|
|
|
|
**Create your own off-grid library.**
|
|
|
|
<p align="center">
|
|
<img width="256px" src="https://raw.githubusercontent.com/lrnselfreliance/wrolpi/master/icon.png" alt='WROLPi Logo'>
|
|
</p>
|
|
|
|
WROLPi is a self-contained collection of software to help you survive the world Without Rule of Law.
|
|
|
|
WROLPi is intended to be run on a Raspberry Pi with an optional external drive attached. It serves up it's own Wi-Fi
|
|
network so that any user with a laptop/tablet/phone can connect and use the library created by the maintainer.
|
|
|
|
# Table of Contents
|
|
|
|
* [Features](#features)
|
|
* [Demo](#demo)
|
|
* [Download](#download)
|
|
* [Try WROLPi](#try-wrolpi)
|
|
* [Install WROLPi](INSTALL.md)
|
|
* [Docker Documentation](docker/README.md)
|
|
* [Charter](#charter)
|
|
* [Upgrading WROLPi](UPGRADE.md)
|
|
* [Join](#join)
|
|
|
|
# Features
|
|
|
|
- [x] Videos
|
|
- [x] Web Archives
|
|
- [x] Maps
|
|
- [x] Wikipedia
|
|
- [x] File search
|
|
- [x] eBooks (EPUB / PDFs)
|
|
- [x] Wi-Fi Hotspot
|
|
- [x] Universal search
|
|
- [x] Food Inventory management
|
|
- [x] One-Time pad generator, encrypter/decrypter
|
|
- [ ] Synchronizer/duplicator
|
|
- [ ] Food storage calculator
|
|
|
|
## Module features matrix
|
|
|
|
| **Module** | **View** | **Search** | **Download** | **Project** |
|
|
|----------------|----------|:-----------|--------------|------------------------------------------------------------|
|
|
| Videos | yes | yes | yes | [yt-dlp](https://github.com/yt-dlp/yt-dlp) |
|
|
| Web Archives | yes | yes | yes | [Singlefile](https://github.com/gildas-lormeau/SingleFile) |
|
|
| Wikipedia | yes | yes | yes | [Kiwix](https://www.kiwix.org) |
|
|
| eBooks | yes | yes | planned | EPUB/PDF |
|
|
| Map | yes | yes | yes | [Protomaps](https://protomaps.com/) |
|
|
| Podcasts/Audio | yes | planned | planned | |
|
|
|
|
# Demo
|
|
|
|
<a href="https://www.youtube.com/watch?v=Qz-FuenRylQ"> </a>
|
|
|
|
# Download
|
|
|
|
Images are available at [wrolpi.org](https://wrolpi.org)
|
|
|
|
# Try WROLPi!
|
|
|
|
You can try out WROLPi by running the docker containers.
|
|
|
|
1. [Install docker-ce](https://docs.docker.com/install/linux/docker-ce/debian/) and
|
|
[docker-compose](https://docs.docker.com/compose/install/)
|
|
2. Copy the latest WROLPi code.
|
|
* `git clone https://github.com/lrnselfreliance/wrolpi.git`
|
|
3. Change directory into the code base.
|
|
* `cd wrolpi`
|
|
4. Initialize git submodules
|
|
* `git submodule update --init`
|
|
5. Build the containers.
|
|
* `docker-compose build --parallel`
|
|
6. Create volumes.
|
|
* `docker volume create --name=openstreetmap-data`
|
|
* `docker volume create --name=openstreetmap-rendered-tiles`
|
|
7. Initialize the database (SQLite in the media directory).
|
|
* `docker-compose run --rm api db upgrade`
|
|
8. Start the docker containers.
|
|
* `docker-compose up`
|
|
9. Browse to WROLPi.
|
|
* https://0.0.0.0:8443
|
|
|
|
More Docker documentation is available in [docker/README.md](docker/README.md)
|
|
|
|
# Install WROLPi
|
|
|
|
Full installation instructions are in [INSTALL.md](INSTALL.md). Pick the option that matches your hardware:
|
|
|
|
* **[WROLPi Portable (Live USB)](INSTALL.md#wrolpi-portable-live-usb)** — boot a complete WROLPi from a USB stick on any x86 PC or laptop. Nothing is installed to the computer; the first boot creates a persistence partition on the USB so your library, database, and configuration survive reboots.
|
|
* **[Debian 12](INSTALL.md#debian-12-install)** — install WROLPi onto an x86 PC or laptop.
|
|
* **[Raspberry Pi](INSTALL.md#raspberry-pi-install)** — install WROLPi onto a Raspberry Pi with an external drive.
|
|
* **[Docker](docker/README.md)** — run the WROLPi containers on an existing Linux host.
|
|
|
|
# Charter
|
|
|
|
## Guiding Principals
|
|
|
|
1. Storage is cheaper than power.
|
|
2. Two is one, one is none.
|
|
3. Run silently.
|
|
4. Primary, secondary, tertiary.
|
|
5. Secure as a bookshelf in your home.
|
|
|
|
## Physical properties
|
|
|
|
1. A WROLPi instance should be capable of running with a minimal of hardware:
|
|
* Raspberry Pi
|
|
* SD card
|
|
* External USB hard drive
|
|
* Power supply and cables
|
|
* Wi-Fi USER device such as a phone, tablet, or laptop.
|
|
2. A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be
|
|
scarce when WROLPi is needed most.
|
|
|
|
## User expectations
|
|
|
|
1. A WROLPi instance should be run and maintained by a person (MAINTAINER) with a moderate amount of Linux and Raspberry
|
|
Pi experience. It is expected that they should be able to do this using only the documentation on their WROLPi.
|
|
|
|
## Software properties and capabilities
|
|
|
|
1. A WROLPi instance should be able to "factory-reset" itself without any outside services.
|
|
2. WROLPi should function completely without any internet services.
|
|
3. A user should have easy access to their data if WROLPi fails:
|
|
* For example, a user can watch their videos by navigating a short and intuitive directory structure and opening the
|
|
video in their preferred video player.
|
|
4. WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER.
|
|
* If the UI isn't functional, the code should be documented such that a user can restore functionality.
|
|
5. WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or
|
|
processing captions. This is to ensure that when a user adds content during non-WROL time, the processing for optimum
|
|
performance is already done for a WROL event.
|
|
|
|
# Join!
|
|
|
|
<p>
|
|
<img src="https://raw.githubusercontent.com/lrnselfreliance/wrolpi/master/join_discord.png" alt='Discord QR Code'>
|
|
</p>
|
|
|
|
[Join our Discord](https://discord.gg/HrwFk7nqA2)
|