Commit graph

26 commits

Author SHA1 Message Date
lrnselfreliance
1acb148c90
Single file deno (#478)
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.

single-file-cli is designed for Deno.  Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.

The native path and the Archive docker container now run single-file with
"deno run".  Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum).  Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.

Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind.  The Archive
downloader now has a ten minute timeout.  Install scripts install Deno
(previously only upgrade.sh and the image builds did).

* Singlefile writes to a file instead of dumping to stdout.

Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives).  single-file now writes its result to a
temporary file which WROLPi reads back.

As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.

* Compressed Archive screenshots and readability images are no longer broken.

The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths.  Screenshots are now taken from the fully
extracted ZIP so those references resolve.  Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).

* Compressed Archives are created by converting the uncompressed capture.

The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed).  When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file.  The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages.  If the
conversion fails, the uncompressed singlefile is stored instead.

* Consolidating duplicate compressed-singlefile tests.

The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one.  The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.

* Addressing review comments: removeprefix, PermissionError when killing.

* Bumping version to 0.24.1-beta.

* Install scripts assert the deno.zip hash to avoid malicious or invalid files.

The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.

* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:07:28 -06:00
Roland
65513eab82
Upgrade script upgrades single-file-cli to the pinned version.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Old single-file versions (1.x) create compressed Archives without the
SingleFile comment header in the self-extracting prelude, which WROLPi
rejects as invalid.  The install scripts only install single-file when it
is missing, so existing machines were never upgraded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 20:05:10 -06:00
Roland
e0324ce0fc Migrating Controller from port 8087 to 80 so that it is more discoverable. Reformatting Controller dashboard to make header at the top. Reformatting navigation bar to handle the new links which prevented Admin from being visible in some screens. 2026-04-09 15:01:05 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
ce0cdd3a2e Samba is installed and can be configured in the Controller. 2026-03-29 10:19:28 -06:00
Roland
1109dd3933
Restarting Controller as quickly as possible during upgrade to make upgrade monitoring smoother. 2026-03-19 15:02:12 -06:00
Roland
fbb71e012d Replace nginx with Caddy for TLS termination and reverse proxying.
- Replace nginx with Caddy as the web server for both Docker and native
  (RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
  proxying to all services (API, React app, Controller, Kiwix, Help, Map),
  and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
  uses Name Constraints to restrict signing to local addresses only
  (.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
  risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
  certificate installation on iOS, Android, macOS, Windows, Linux, and
  Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
  /download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
  status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.
2026-03-07 18:48:44 -07:00
Roland
1deaeb3144
Clearing cached python files in repair.sh. Creating reset_virtual_environments.sh script which deletes the virtual environments and rebuilds them (requires internet)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-28 16:11:12 -07:00
Roland
adb8fefd67
Including Deno and curl-cffi which are now required for yt-dlp functionality. Installing both of these packages during upgrade. 2026-02-16 08:08:46 -07:00
Roland
628b9f3096
Adding disk mount handling to React UI.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-16 17:04:46 -07:00
Roland
b9158b60fe Creating Controller service which will control WROLPi services, admin, SMART, disks, and status. nginx will fallback to Controller UI when React UI is unavailable (such as during upgrades). 2025-12-15 15:48:40 -07:00
Roland
f61eee1289
Upgrade script trap preserves exit code so that it does not report upgrade is complete when it actually fails. 2025-12-07 17:05:20 -07:00
Roland
3e3ab04f4d upgrade.sh will restart help service, even on failure. 2025-12-06 15:11:37 -07:00
Roland
bd599e1b2b
Fixing upgrade which deletes the wrong map blob. Fixing the port so the old map database is deleted. 2025-01-12 11:06:51 -07:00
Roland
8e4d173791
Converting map blob to use "custom" postgres dump format so restore can use multiple threads. Renaming map db blob from gis-map.dump.gz to map-db-gis.dump. 2025-01-02 12:04:03 -07:00
Roland
7324e83626
Creating script which migrates map database to new database for custom niceness. 2025-01-01 16:41:21 -07:00
Roland
218fb3e760
Replacing Tag names with escaped tag names to avoid errors in Tags Directory. 2024-10-16 18:56:10 -06:00
Roland
4425b4d8a3
Adding wrolpi-help to git safe directory. Help install failure should not break upgrade. 2023-11-23 16:22:46 -07:00
Roland
792e0c15cd
Upgrading help service when doing upgrade. 2023-11-23 16:20:28 -07:00
Roland
6e0f9a999c
Combining install scripts into a new repair script. Removing create_db script in favor of repair script. 2023-10-03 12:15:29 -06:00
Roland
454891be15
Changing project ownership after upgrade. 2023-07-11 17:23:56 -06:00
Roland
f764cee44e
Upgrade systemd file before building api. 2023-05-23 16:41:48 -06:00
Roland
ccb28db95c
Enabling Python optimization when running in Docker and systemd. 2023-05-23 16:31:10 -06:00
Roland
7ff2351dc5
Updating nginx configs during upgrade. 2023-05-01 13:00:23 -06:00
Roland
a14ebf1909
Making upgrade executable. 2023-02-24 17:33:07 -07:00
Roland
dc9384ba4b
Calling second upgrade script so new upgrades can be pulled from git. 2023-02-24 17:31:08 -07:00