The contour label layer named no font, so MapLibre asked for its default
("Open Sans Regular,Arial Unicode MS Regular"), which the map does not ship.
The request failed and the labels were drawn with glyphs the browser made up
locally. The layer now names Noto Sans Regular, in the map viewer and in the
standalone viewer.
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
When a MapLibre style (and optionally a sprite sheet) sits beside a PMTiles file, the overlay is drawn with it instead of the generic colors: `<name>.style.json` and `<name>.sprite.json/.png`, or `style.json` and `sprite.json/.png` shared by the files of a directory.
Only the style's layers which draw that file are used; its sources, sprite and glyph URLs never are. Fonts are replaced by the ones the map has, images are pointed at the overlay's own sprite (as strings, expressions or property functions), and each layer is validated so one the map could not draw is skipped instead of stopping the map.
The sprite is served by the API, which falls back to the single resolution sheet when there is no double resolution one.
Any PMTiles file under the map directory, at any depth, can be shown. The API reads each file's header and metadata and says what it is (basemap, terrain, overlay, unreadable). Overlays are offered in the layer control under the name the file gives itself, hidden until turned on, with a checkbox per vector layer, and are drawn generically.
Map files are named by their path within the map directory everywhere, and unfinished downloads are no longer listed. A file can be ignored from the Manage page; ignored files are kept in a new maps.yaml config.
PMTiles metadata is not decompressed past its size limit, and the standalone viewer encodes map file paths.
* Map downloads are checked against the hash in the signed manifest
A meta4 signs only a bare hash, so any genuinely signed meta4 (an older version of a region, another region) passed
its own check. The manifest now lists each file's sha256; it is signed as a whole, so the hash is tied to the URL.
The catalog copies each hash into its download's settings, and aria2c checks the file against it. Downloads from a
manifest without hashes still require the file's signed meta4.
build_map_regions.py writes the hashes, and hash_map_manifest.py adds them to the published manifest from the
verified meta4s without rebuilding any maps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* hash_map_manifest.py hashes each file's bytes instead of trusting its meta4's name
Only the bare hash in a meta4 is signed, so an older version's meta4 with its name rewritten (or terrain's, whose name
never changes) passed the name and duplicate-hash checks. Each file is now hashed from a local copy of the published
files (--files) or downloaded from the CDN, and must match its signed meta4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Map files are never downloaded without a signed hash
A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later)
instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it
downloaded with no hash whenever the .meta4 was missing.
A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when
the meta4's unsigned name differs from the URL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* aria2c checks map files only against the signed hash, and a rejected file is never kept
Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered
file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the
signed hash and the download URL alone.
A rejected download was left in the map directory under its final name, where the next attempt took it as already
downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that
fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Only the map file aria2c checked is moved into place
download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c
saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as
an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the
URL, must be a plain file name, and any other reported path fails the download.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`Download.settings['region'].astext` is PostgreSQL's JSONB operator. On SQLite it raises AttributeError, so the
map catalog failed before queuing any region Download, and an extract never checked for a CDN search index. Use
`.as_string()`.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Deletes functions, methods, classes, schema dataclasses, error classes, Events
senders, and module-level constants that nothing references, including tests.
Also drops the unused `doc_cleanup(test_directory)` and
`_wait_for_sub_job_with_progress(collection_idx)` parameters, and the imports
that only served removed code.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The base import for a config with no sections now reads its file once, reports a 0-byte file as empty, treats a
missing file as the defaults, and records success or failure itself (sending the failure Event on request). The
per-config overrides that only set successful_import are removed.
wrolpi.yaml and the downloader settings import in a thread at startup (import_file_configs), and the Import, Save
and backup Import buttons run their config work in a thread.
The pre-diff import path (import_decision, record_import_version) has no callers left and is removed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
PUT/DELETE /api/videos/channels/<id>, POST /api/videos/channels/<id>/tag,
and POST/DELETE /api/map/subscribe read, then wrote, in the request's
deferred transaction, so they failed with a 500 ("database is locked")
when another connection wrote in between. They now write inside
request.ctx.db.write().
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ui: a preview map that waited for a container it could never see
Both map preview modals -- the per-region eye button and "View All Regions"
-- opened to an empty box. Each renders a div into a Modal and hands it to
MapLibre from an effect gated on `open`, reading the node out of a `useRef`:
if (!open || !mapContainer.current) return;
Mantine's Modal mounts its body one commit after `opened` flips true, so on
the render that opens the modal the ref is still null. The effect returns,
nothing it depends on changes again, and it never runs a second time. No
console error, no rejected promise, no canvas -- just an empty div.
Semantic UI's Portal mounted the body in the same commit that opened it, so
the ref was attached by the time effects ran. The effect bodies are
byte-identical across that migration; the modal library is the only variable,
confirmed against a device still running the older build where both previews
render.
A callback ref instead, so the effect re-runs when the node arrives and this
stops depending on any modal's mounting schedule. The async source lookup
also gets a cancelled flag: it could resolve after the cleanup ran, building
a map nothing would ever remove and leaking a WebGL context per open.
The tests drive the real page and assert a map was built against a node that
is in the document -- the outcome, rather than the mechanism that quietly
changed underneath it. Seven of the eight fail on the code before this.
* build: docker had no world underneath its map regions
Caddy serves /blobs/map-overview.pmtiles out of /opt/wrolpi-blobs, and that
blob is the world at zoom 0-6 -- what a regional extract sits on top of.
Without it a user who subscribes to a few regions gets those regions floating
on an empty background everywhere their data does not reach. Only a full
planet file, at 134GB, makes it unnecessary.
Raspberry Pi and Debian installs get the blob from pi-gen, the live-build
config, and upgrade.sh. Docker got nothing: the compose file bind-mounted a
git-ignored host directory over /opt/wrolpi-blobs, and a bind mount replaces
the image's directory rather than merging with it, so the path existed and was
empty on every Docker WROLPi. Verified against a running deployment, where
that URL returned 404 while the same URL on hardware returned 206.
So the web image fetches the blob at build time, as the other three builders
do, and the mount that would hide it is gone. `curl -f` so an HTTP error
fails the layer -- an image that builds without the blob is this same bug,
except silent at build time too.
The tests read the deployment files rather than a built container: building
to check costs a CDN download and several minutes, and both things that broke
-- nothing fetches it, a mount hides it -- are visible in the source.
* ui: cancel the load handler too, and make the cancel test falsifiable
Review follow-ups on the preview modals.
The `cancelled` flag was checked before constructing the map but not inside
the `load` handler it installs, so cleanup landing between the two left that
body drawing into a map `remove()` had already torn down. Guarded, mirroring
the `destroyed` pattern in MapViewer.js. The source promise also gets a
`.catch`: the helper cannot reject today, but nothing says it never will, and
an unhandled rejection here would be as quiet as the bug this all started as.
Two test corrections, both cases of an assertion claiming more than it read:
The region-drawing test spied on `addSource` AFTER construction and then
re-fired `load` by hand, so its count came entirely from that manual call and
said nothing about the production path. FakeMap now records the sources it is
given, and the assertion reads what the component's own `load` drew -- each
region once, in order, and none for the entry with no bbox.
The new cancel test passed with the guard deleted, which is worse than not
having it: Mantine had not mounted the modal body when it closed, so the
effect had never run and there was nothing to cancel. It now waits for the
effect to reach the source lookup before closing, and ships beside a control
that releases the same held promise WITHOUT closing and requires a map. The
control is what makes the zero meaningful; verified that removing the guard
fails the cancel test while the control still passes.
Map viewer has simple search.
Map search databases can be downloaded from CDN, or built locally. Map searches can be built offline, but will not have enriched things like state names downloaded from Wikidata.
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
Python 3.11 is now required, old versions were causing unpredictability in tests. (Sanic does not yet
support 3.12)
Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
https://github.com/sanic-org/sanic/issues/2921
New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes. The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation. Now all multiprocessing tools are attached to the
app.shared_ctx. This means `api_app` is imported in many, many places.
This forced a change in how the DownloadManager works. Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue. Now, a single worker checks
for new downloads and sends a Sanic signal.
Flags have been reworked to use the `api_app`. I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.