rt/wrolpi
1
0
Fork 0
mirror of https://github.com/lrnselfreliance/wrolpi.git synced 2026-10-25 18:40:06 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
wrolpi/README.md
Roland d9303a0ea8
Postgres -> SQLite hard cutover: the database moves into the media directory. 
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.

Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
  foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
  sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
  adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
  Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
  A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
  Stemming contract pinned by dedicated tests (watch/watches/watched/
  watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
  baseline and test DBs; fixes the channel video_count DELETE/UPDATE
  bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
  written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
  network-filesystem refusal; never creates/migrates/trusts a database
  beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
  revision ids are dates (2026_07_10_1350) so filenames self-sort and
  alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
  HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
  generate_series histograms->strftime GROUP BY, DISTINCT ON->
  row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
  NULLS FIRST/LAST, LIMIT-before-OFFSET).  Simple statements converted
  to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
  channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
  after a lossy import so partial imports can never overwrite YAML
  configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
  treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
  (BIGINT is not a rowid alias; FTS5 content_rowid requires it).

Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed.  upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.

Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index.  QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.

Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.

Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:09:37 -06:00

6.4 KiB
Raw Blame History

WROLPi

Build Status License: GPL v3 Discord Gitpod

Create your own off-grid library.

WROLPi Logo

WROLPi is a self-contained collection of software to help you survive the world Without Rule of Law.

WROLPi is intended to be run on a Raspberry Pi with an optional external drive attached. It serves up it's own Wi-Fi network so that any user with a laptop/tablet/phone can connect and use the library created by the maintainer.

Table of Contents

Features

  • Videos
  • Web Archives
  • Maps
  • Wikipedia
  • File search
  • eBooks (EPUB / PDFs)
  • Wi-Fi Hotspot
  • Universal search
  • Food Inventory management
  • One-Time pad generator, encrypter/decrypter
  • Synchronizer/duplicator
  • Food storage calculator

Module features matrix

Module View Search Download Project
Videos yes yes yes yt-dlp
Web Archives yes yes yes Singlefile
Wikipedia yes yes yes Kiwix
eBooks yes yes planned EPUB/PDF
Map yes yes yes Protomaps
Podcasts/Audio yes planned planned

Demo

YouTube Demo Video

Download

Images are available at wrolpi.org

Try WROLPi!

You can try out WROLPi by running the docker containers.

  1. Install docker-ce and docker-compose
  2. Copy the latest WROLPi code.
    • git clone https://github.com/lrnselfreliance/wrolpi.git
  3. Change directory into the code base.
    • cd wrolpi
  4. Initialize git submodules
    • git submodule update --init
  5. Build the containers.
    • docker-compose build --parallel
  6. Create volumes.
    • docker volume create --name=openstreetmap-data
    • docker volume create --name=openstreetmap-rendered-tiles
  7. Initialize the database (SQLite in the media directory).
    • docker-compose run --rm api db upgrade
  8. Start the docker containers.
    • docker-compose up
  9. Browse to WROLPi.

More Docker documentation is available in docker/README.md

Install WROLPi

Full installation instructions are in INSTALL.md. Pick the option that matches your hardware:

  • WROLPi Portable (Live USB) — boot a complete WROLPi from a USB stick on any x86 PC or laptop. Nothing is installed to the computer; the first boot creates a persistence partition on the USB so your library, database, and configuration survive reboots.
  • Debian 12 — install WROLPi onto an x86 PC or laptop.
  • Raspberry Pi — install WROLPi onto a Raspberry Pi with an external drive.
  • Docker — run the WROLPi containers on an existing Linux host.

Charter

Guiding Principals

  1. Storage is cheaper than power.
  2. Two is one, one is none.
  3. Run silently.
  4. Primary, secondary, tertiary.
  5. Secure as a bookshelf in your home.

Physical properties

  1. A WROLPi instance should be capable of running with a minimal of hardware:
    • Raspberry Pi
    • SD card
    • External USB hard drive
    • Power supply and cables
    • Wi-Fi USER device such as a phone, tablet, or laptop.
  2. A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be scarce when WROLPi is needed most.

User expectations

  1. A WROLPi instance should be run and maintained by a person (MAINTAINER) with a moderate amount of Linux and Raspberry Pi experience. It is expected that they should be able to do this using only the documentation on their WROLPi.

Software properties and capabilities

  1. A WROLPi instance should be able to "factory-reset" itself without any outside services.
  2. WROLPi should function completely without any internet services.
  3. A user should have easy access to their data if WROLPi fails:
    • For example, a user can watch their videos by navigating a short and intuitive directory structure and opening the video in their preferred video player.
  4. WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER.
    • If the UI isn't functional, the code should be documented such that a user can restore functionality.
  5. WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or processing captions. This is to ensure that when a user adds content during non-WROL time, the processing for optimum performance is already done for a WROL event.

Join!

Discord QR Code

Join our Discord