mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-10-26 10:51:36 -04:00
Create your own off-grid library
https://wrolpi.org
9310fe801f
A refresh on a large library aborted the whole doc_modeler run on the
first lock error, so the ~20k stranded docs never got modeled.
doc_modeler held one write transaction across a whole batch of 10 docs
while doing slow per-doc extraction (a PDF/epub can be tens of MB). When
another connection committed mid-batch, the deferred transaction failed
to upgrade to a write with SQLITE_BUSY_SNAPSHOT ("database is locked",
which busy_timeout does not absorb -- the same trap that motivated the
short-transaction rewrite of apply_indexers). The per-item error handler
then did `logger.error(f'... {file_group}')`, which lazy-loaded an
attribute on the now-rolled-back session and raised a *second* exception,
escaping the handler and aborting the entire modeler.
Restructure:
- Pick each batch of candidate ids in a short read-only transaction, then
model each doc in its own short write transaction. A lock/error on one
doc rolls back only that doc and cannot poison the rest of the run.
- Retry a doc a few times on "database is locked" -- a fresh transaction
gets a fresh snapshot and succeeds.
- Log the plain file_group id on failure; never touch the ORM object on a
possibly rolled-back session.
- Track only *failed* ids for the termination guard (successes are
excluded by the Doc.id IS NULL gate), keeping the notin_ set small and
well under SQLite's bound-variable limit.
Adds test_doc_modeler_survives_database_locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .circleci | ||
| .github | ||
| .run | ||
| alembic | ||
| app | ||
| controller | ||
| debian-live-config | ||
| docker | ||
| etc | ||
| modules | ||
| pi-gen | ||
| release | ||
| scripts | ||
| test | ||
| wrolpi | ||
| .coveragerc | ||
| .dockerignore | ||
| .gitignore | ||
| .gitmodules | ||
| .gitpod.yml | ||
| .mcp.json.example | ||
| alembic.ini | ||
| conftest.py | ||
| demo_link.jpg | ||
| docker-compose.yml | ||
| help.sh | ||
| icon.ico | ||
| icon.png | ||
| INSTALL.md | ||
| install.sh | ||
| join_discord.png | ||
| LICENSE | ||
| main.py | ||
| pytest.ini | ||
| README.md | ||
| repair.sh | ||
| requirements.txt | ||
| test_all.sh | ||
| UPGRADE.md | ||
| upgrade.sh | ||
WROLPi
Create your own off-grid library.
WROLPi is a self-contained collection of software to help you survive the world Without Rule of Law.
WROLPi is intended to be run on a Raspberry Pi with an optional external drive attached. It serves up it's own Wi-Fi network so that any user with a laptop/tablet/phone can connect and use the library created by the maintainer.
Table of Contents
Features
- Videos
- Web Archives
- Maps
- Wikipedia
- File search
- eBooks (EPUB / PDFs)
- Wi-Fi Hotspot
- Universal search
- Food Inventory management
- One-Time pad generator, encrypter/decrypter
- Synchronizer/duplicator
- Food storage calculator
Module features matrix
| Module | View | Search | Download | Project |
|---|---|---|---|---|
| Videos | yes | yes | yes | yt-dlp |
| Web Archives | yes | yes | yes | Singlefile |
| Wikipedia | yes | yes | yes | Kiwix |
| eBooks | yes | yes | planned | EPUB/PDF |
| Map | yes | yes | yes | Protomaps |
| Podcasts/Audio | yes | planned | planned |
Demo
Download
Images are available at wrolpi.org
Try WROLPi!
You can try out WROLPi by running the docker containers.
- Install docker-ce and docker-compose
- Copy the latest WROLPi code.
git clone https://github.com/lrnselfreliance/wrolpi.git
- Change directory into the code base.
cd wrolpi
- Initialize git submodules
git submodule update --init
- Build the containers.
docker-compose build --parallel
- Create volumes.
docker volume create --name=openstreetmap-datadocker volume create --name=openstreetmap-rendered-tiles
- Initialize the database (SQLite in the media directory).
docker-compose run --rm api db upgrade
- Start the docker containers.
docker-compose up
- Browse to WROLPi.
More Docker documentation is available in docker/README.md
Install WROLPi
Full installation instructions are in INSTALL.md. Pick the option that matches your hardware:
- WROLPi Portable (Live USB) — boot a complete WROLPi from a USB stick on any x86 PC or laptop. Nothing is installed to the computer; the first boot creates a persistence partition on the USB so your library, database, and configuration survive reboots.
- Debian 12 — install WROLPi onto an x86 PC or laptop.
- Raspberry Pi — install WROLPi onto a Raspberry Pi with an external drive.
- Docker — run the WROLPi containers on an existing Linux host.
Charter
Guiding Principals
- Storage is cheaper than power.
- Two is one, one is none.
- Run silently.
- Primary, secondary, tertiary.
- Secure as a bookshelf in your home.
Physical properties
- A WROLPi instance should be capable of running with a minimal of hardware:
- Raspberry Pi
- SD card
- External USB hard drive
- Power supply and cables
- Wi-Fi USER device such as a phone, tablet, or laptop.
- A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be scarce when WROLPi is needed most.
User expectations
- A WROLPi instance should be run and maintained by a person (MAINTAINER) with a moderate amount of Linux and Raspberry Pi experience. It is expected that they should be able to do this using only the documentation on their WROLPi.
Software properties and capabilities
- A WROLPi instance should be able to "factory-reset" itself without any outside services.
- WROLPi should function completely without any internet services.
- A user should have easy access to their data if WROLPi fails:
- For example, a user can watch their videos by navigating a short and intuitive directory structure and opening the video in their preferred video player.
- WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER.
- If the UI isn't functional, the code should be documented such that a user can restore functionality.
- WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or processing captions. This is to ensure that when a user adds content during non-WROL time, the processing for optimum performance is already done for a WROL event.
Join!
