mirror of
https://github.com/lrnselfreliance/wrolpi.git
synced 2026-11-19 14:36:22 -05:00
Create your own off-grid library
https://wrolpi.org
* Zim search estimates run in a separate process so a cold estimate cannot stall the API worker The global search estimates every auto-search Zim on each keystroke. On a large library that estimate took tens of seconds when the Zims' pages were not cached, and every other request routed to the same Sanic worker, such as the channel suggestions, waited behind it. Two things held the worker: opening a large Zim cold takes seconds (ten for a 75 GB Zim) and `Zims.get_all` did that on the event loop; and python-libzim never releases the GIL, so the estimate offloaded with asyncio.to_thread still froze the worker. Everything libzim does for an estimate now runs in a standalone script started as a subprocess (Sanic workers are daemonic, so a multiprocessing pool cannot be forked from them). It reads the Zim paths and search string on stdin and writes the counts on stdout. A Zim it cannot read is reported and left out, as `Zims.get_all` would leave it out. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Zim searches run in the libzim worker process too The /api/zim/search endpoints had the same stall as the estimate: they opened every Zim through Zims.get_all and searched them with libzim, which holds the GIL, so a thread offload still froze the Sanic worker. The estimate script is now a general libzim worker with estimate, search and entries jobs. A search of all Zims sends every Zim's job to one worker process; the API worker only headlines the returned HTML and looks up tags. The worker also returns each Zim's metadata so Zim.__json__ does not open the Zim again on the event loop. Zim.entries_with_tags is split so the tagged paths come from the database without opening the Zim. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Tagged Zim estimates count from the database, and a cancelled request kills the libzim worker Review of the earlier commits found two gaps. Estimates with tags selected still went through Zims.entries_with_tags, which opened every Zim in the API process, and after the thread offload was removed it did so on the event loop; they now count the tagged paths in SQL and ask the worker only for metadata. And Sanic cancels a handler at its response timeout or on disconnect, which left the worker process running until its stdout pipe filled; the runner now kills and reaps it on cancellation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|---|---|---|
| .github | ||
| .run | ||
| alembic | ||
| app | ||
| controller | ||
| debian-live-config | ||
| docker | ||
| etc | ||
| modules | ||
| npm-tools | ||
| pi-gen | ||
| release | ||
| scripts | ||
| test | ||
| wrolpi | ||
| .coveragerc | ||
| .dockerignore | ||
| .gitignore | ||
| .gitmodules | ||
| .gitpod.yml | ||
| .mcp.json.example | ||
| alembic.ini | ||
| conftest.py | ||
| demo_link.jpg | ||
| docker-compose.yml | ||
| help.sh | ||
| icon.ico | ||
| icon.png | ||
| INSTALL.md | ||
| install.sh | ||
| join_discord.png | ||
| LICENSE | ||
| main.py | ||
| pytest.ini | ||
| README.md | ||
| repair.sh | ||
| requirements.lock | ||
| requirements.txt | ||
| test_all.sh | ||
| UPGRADE.md | ||
| upgrade.sh | ||
WROLPi
Create your own off-grid library.
WROLPi is a self-contained collection of software to help you survive the world Without Rule of Law.
WROLPi is intended to be run on a Raspberry Pi with an optional external drive attached. It serves up it's own Wi-Fi network so that any user with a laptop/tablet/phone can connect and use the library created by the maintainer.
Table of Contents
Features
- Videos
- Web Archives
- Maps
- Wikipedia
- File search
- eBooks (EPUB / PDFs)
- Wi-Fi Hotspot
- Universal search
- Food Inventory management
- One-Time pad generator, encrypter/decrypter
- Synchronizer/duplicator
- Food storage calculator
Module features matrix
| Module | View | Search | Download | Project |
|---|---|---|---|---|
| Videos | yes | yes | yes | yt-dlp |
| Web Archives | yes | yes | yes | Singlefile |
| Wikipedia | yes | yes | yes | Kiwix |
| eBooks | yes | yes | planned | EPUB/PDF |
| Map | yes | yes | yes | Protomaps |
| Podcasts/Audio | yes | planned | planned |
Demo
Download
Images are available at wrolpi.org
Try WROLPi!
You can try out WROLPi by running the docker containers.
- Install docker-ce and docker-compose
- Copy the latest WROLPi code.
git clone https://github.com/lrnselfreliance/wrolpi.git
- Change directory into the code base.
cd wrolpi
- Initialize git submodules
git submodule update --init
- Build the containers.
docker-compose build --parallel
- Create volumes.
docker volume create --name=openstreetmap-datadocker volume create --name=openstreetmap-rendered-tiles
- Initialize the database (SQLite in the media directory).
docker-compose run --rm api db upgrade
- Start the docker containers.
docker-compose up
- Browse to WROLPi.
More Docker documentation is available in docker/README.md
Install WROLPi
Full installation instructions are in INSTALL.md. Pick the option that matches your hardware:
- WROLPi Portable (Live USB) — boot a complete WROLPi from a USB stick on any x86 PC or laptop. Nothing is installed to the computer; the first boot creates a persistence partition on the USB so your library, database, and configuration survive reboots.
- Debian 13 — install WROLPi onto an x86 PC or laptop.
- Raspberry Pi — install WROLPi onto a Raspberry Pi with an external drive.
- Docker — run the WROLPi containers on an existing Linux host.
Charter
Guiding Principals
- Storage is cheaper than power.
- Two is one, one is none.
- Run silently.
- Primary, secondary, tertiary.
- Secure as a bookshelf in your home.
Physical properties
- A WROLPi instance should be capable of running with a minimal of hardware:
- Raspberry Pi
- SD card
- External USB hard drive
- Power supply and cables
- Wi-Fi USER device such as a phone, tablet, or laptop.
- A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be scarce when WROLPi is needed most.
User expectations
- A WROLPi instance should be run and maintained by a person (MAINTAINER) with a moderate amount of Linux and Raspberry Pi experience. It is expected that they should be able to do this using only the documentation on their WROLPi.
Software properties and capabilities
- A WROLPi instance should be able to "factory-reset" itself without any outside services.
- WROLPi should function completely without any internet services.
- A user should have easy access to their data if WROLPi fails:
- For example, a user can watch their videos by navigating a short and intuitive directory structure and opening the video in their preferred video player.
- WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER.
- If the UI isn't functional, the code should be documented such that a user can restore functionality.
- WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or processing captions. This is to ensure that when a user adds content during non-WROL time, the processing for optimum performance is already done for a WROL event.
Join!
