Commit graph

155 commits

Author SHA1 Message Date
Roland
960eb81880
Documenting WROLPi Portable USB, splitting install docs into INSTALL.md.
- New INSTALL.md consolidates all install paths: WROLPi Portable (Live
  USB) as the headline option, plus Debian 12, Raspberry Pi, and Docker.
- README install sections collapsed into a single Install section that
  links to INSTALL.md.
- UPGRADE.md gains an "Upgrading a WROLPi Portable USB" section covering
  scripts/wrolpi-usb.sh upgrade, and documents that in-place conversion
  of an existing data drive is not supported (backup -> flash -> restore).
- wrolpi-usb.sh install now prints that supported procedure instead of a
  "not implemented" stub; debian-live-config/README.md documents the same.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-28 21:33:22 -06:00
Roland
9edfb0de25 Addressing Greptile P1s on wrolpi-usb.sh.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
numfmt with a negative argument:
  Under `set -euo pipefail` the upgrade subcommand would die at
  `numfmt --to=iec ... $headroom_bytes` whenever the new ISO was larger
  than the headroom — numfmt treats the leading `-` as an option flag,
  exits non-zero, and the script aborts before printing the descriptive
  back-up-and-reflash error message just below.  Splitting positive
  and negative cases and feeding numfmt the absolute value fixes it;
  the eventual error message now actually shows up.

Silent umount during sanity check:
  `umount "$child" 2>/dev/null || true` was swallowing both stderr and
  the exit code.  A busy partition (open file handle from a daemon
  that auto-opened the USB) would survive the loop, and `cmd_upgrade`
  would proceed straight to `dd if=<iso> of=<dev>` — writing to a
  block device whose partition is still mounted, corrupting the
  filesystem on that partition.  Adds a post-loop assertion: re-read
  MOUNTPOINTS and die with an `lsof` hint if anything is still mounted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
17e26fab97 Adding scripts/wrolpi-usb.sh — upgrade subcommand.
`wrolpi-usb.sh upgrade <iso> <device>` replaces the boot partition of an
existing WROLPi USB with a new ISO while preserving the persistence
partition.

Flow:
  1. Locate the persistence partition by label.
  2. Read its sector range from sfdisk -d.
  3. Confirm the new ISO fits in the headroom (i.e. its byte count <=
     persistence_start_sector * 512).  If not, bail with instructions
     telling the user to back up /media/wrolpi, re-flash from scratch,
     and restore.  This is the case for USBs flashed before the 8 GiB-
     headroom commit.
  4. Save the partition table to /tmp/ for emergency recovery.
  5. Prompt for "yes".
  6. dd the new ISO onto the device.  The persistence partition's
     sectors are past the ISO's end, so its data is untouched.
  7. sfdisk --append to re-add the persistence partition entry at the
     saved sector range.  The data on disk is still there; we are just
     fixing up the partition table the new ISO wrote.
  8. partx -u to refresh the kernel view; verify the label is reachable.

`install` subcommand is wired up to a die() for now; will land in a
follow-up commit.

The script refuses to operate on non-removable disks (foot-gun guard).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 20:16:02 -06:00
Roland
f555855b15
Updating raspberry pi package list file name.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-04-14 16:52:33 -06:00
Roland
c726654de9
Map search added to global search. Preview cards are displayed for all results in new Map tab.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Map viewer has simple search.

Map search databases can be downloaded from CDN, or built locally. Map searches can be built offline, but will not have enriched things like state names downloaded from Wikidata.
2026-04-13 12:17:08 -06:00
Roland
e0324ce0fc Migrating Controller from port 8087 to 80 so that it is more discoverable. Reformatting Controller dashboard to make header at the top. Reformatting navigation bar to handle the new links which prevented Admin from being visible in some screens. 2026-04-09 15:01:05 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
ce0cdd3a2e Samba is installed and can be configured in the Controller. 2026-03-29 10:19:28 -06:00
Roland
66a891c759
Refactoring videos and archives UI paths for consistent scheme. 2026-03-20 15:38:32 -06:00
Roland
1109dd3933
Restarting Controller as quickly as possible during upgrade to make upgrade monitoring smoother. 2026-03-19 15:02:12 -06:00
Roland
fbb71e012d Replace nginx with Caddy for TLS termination and reverse proxying.
- Replace nginx with Caddy as the web server for both Docker and native
  (RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
  proxying to all services (API, React app, Controller, Kiwix, Help, Map),
  and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
  uses Name Constraints to restrict signing to local addresses only
  (.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
  risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
  certificate installation on iOS, Android, macOS, Windows, Linux, and
  Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
  /download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
  status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.
2026-03-07 18:48:44 -07:00
Roland
4008312077
File permissions fix script checks for exfat and other file systems before trying to repair, displays a warning and gives advice on how to fix the issue.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-05 18:18:53 -07:00
Roland
71c2b6cd6a
Creating script that fixes media directory file and directory permissions, adding it to the Controller UI. 2026-03-05 17:58:21 -07:00
Roland
5f0ec9b79a
Creating script to clean up orphaned tag files.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-28 17:05:02 -07:00
Roland
1deaeb3144
Clearing cached python files in repair.sh. Creating reset_virtual_environments.sh script which deletes the virtual environments and rebuilds them (requires internet)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-28 16:11:12 -07:00
Roland
f375e4c955
Generating nginx certs on first startup. 2026-02-24 17:14:56 -07:00
Roland
adb8fefd67
Including Deno and curl-cffi which are now required for yt-dlp functionality. Installing both of these packages during upgrade. 2026-02-16 08:08:46 -07:00
Roland
50782f47cc
Checking that map dump is valid before building image. Protecting map
dump file in built image so that it cannot be modified.
2026-02-15 14:49:19 -07:00
Roland
48acfc62bc
Creating script to reset tags. 2026-02-14 15:37:09 -07:00
Roland
1d9b794fca
Converting JSON columns to JSONB for performance.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-01-06 06:23:32 -07:00
Roland
628b9f3096
Adding disk mount handling to React UI.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-16 17:04:46 -07:00
Roland
b9158b60fe Creating Controller service which will control WROLPi services, admin, SMART, disks, and status. nginx will fallback to Controller UI when React UI is unavailable (such as during upgrades). 2025-12-15 15:48:40 -07:00
Roland
f61eee1289
Upgrade script trap preserves exit code so that it does not report upgrade is complete when it actually fails. 2025-12-07 17:05:20 -07:00
Roland
3e3ab04f4d upgrade.sh will restart help service, even on failure. 2025-12-06 15:11:37 -07:00
Roland
41e1ed6be6
Using template0 when creating map db to avoid utf8 bug.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Fixed test_new_archive to handle platform-specific HTML formatting from BeautifulSoup.
Bumped version to 0.16.1-beta.
2025-11-24 07:50:50 -07:00
Roland
3725f0f39d Adding edit button so user can edit downloads, currently supports videos and archives. 2025-08-06 13:03:32 -06:00
Roland
d2ae135e5b
Updating maps import so previously imported pbf files are reset on import. Ignoring map tile initialization script failure when importing. 2025-03-13 15:52:40 -06:00
Roland
581d5eaf6e
Adding toggle to Channel page to stop downloading comments for certain channels. 2025-03-01 18:23:29 -07:00
Roland
47d4d72bf8
Creating script which takes lat/long coordinates to download map tiles into the current directory. 2025-02-28 14:15:51 -07:00
Roland
531ec3d65e
Creating new /api/archive/upload which accepts a file from the SingleFile browser extension and converts it to an Archive. 2025-01-30 18:41:32 -07:00
Roland
036b12382e
Creating script which updates yt-dlp in the wrolpi venv to the latest available. 2025-01-23 17:46:37 -07:00
Roland
b49cedd62a
Forcing utf8 for map database. 2025-01-22 08:33:33 -07:00
Roland
bd599e1b2b
Fixing upgrade which deletes the wrong map blob. Fixing the port so the old map database is deleted. 2025-01-12 11:06:51 -07:00
Roland
af5d4dfd51
Disabling JIT for map database as recommended by mod_tile because of excess memory usgae. 2025-01-11 11:06:01 -07:00
Roland
8e4d173791
Converting map blob to use "custom" postgres dump format so restore can use multiple threads. Renaming map db blob from gis-map.dump.gz to map-db-gis.dump. 2025-01-02 12:04:03 -07:00
Roland
7324e83626
Creating script which migrates map database to new database for custom niceness. 2025-01-01 16:41:21 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
2b8ef79561
Initializing git submodules for dev environment. 2024-10-16 18:56:18 -06:00
Roland
a98aebfdda
A video that is uploaded is attached to its Channel. Saving Channels config with both download url and frequency in case downloads config is corrupted. 2024-10-16 18:56:11 -06:00
Roland
218fb3e760
Replacing Tag names with escaped tag names to avoid errors in Tags Directory. 2024-10-16 18:56:10 -06:00
Roland
a068273beb
Updating wrolpi-help commit. Forcing re-install of mkdocs because of navbar bug. 2024-10-16 18:56:07 -06:00
Roland
cdc73d9d16
Trimming video file name to avoid error when file name is too long. 2024-10-16 18:55:52 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
38de2f9b40
Moving HTTP kiwix/help services to 9085/9086 ports. Nginx will serve HTTPS proxy for kiwix/help services at the old ports. Modifying apache2 to use HTTPS to serve map.
Using new https_proxy docker container to serve HTTPS for zim, map and help.
2024-06-12 21:38:51 -06:00
Roland
86355b664d
Using older version of single-file-cli to avoid error. 2024-05-14 20:29:57 -06:00
Roland
d0c9bec7f4
Modifying singlefile_to_archive to accept file list in stdin. 2024-05-07 09:41:48 -06:00
Roland
2924867f55
Bumping single-file-cli version for all systems. 2024-05-02 09:09:21 -06:00
Roland
ba2dd6dd11
Many fixes split out from the feature/unified-query branch. Installing diagnostic tools in the built images. Displaying video comment statistics. Echoing error when install.sh fails. 2024-04-14 13:13:30 -06:00
Roland
08f8454540
Simplifying scripts to find similarlly named videos. 2024-02-05 14:29:54 -07:00