wrolpi/modules/map/downloader.py

602 lines
25 KiB
Python
Raw Normal View History

2026-05-06 19:46:24 -06:00
import pathlib
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
import re
import tempfile
2026-05-06 19:46:24 -06:00
from dataclasses import dataclass
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
from pathlib import Path, PurePosixPath
2026-05-06 19:46:24 -06:00
from typing import Any, List, Optional
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
from urllib.parse import urlparse
from xml.etree import ElementTree as ET
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
from xml.sax.saxutils import escape, quoteattr
2026-05-06 19:46:24 -06:00
from sqlalchemy.orm import Session
from modules.map import lib
from wrolpi.common import logger, verify_gpg_signature
2026-05-06 19:46:24 -06:00
from wrolpi.downloader import (
Download,
DownloadContext,
Downloader,
DownloadResult,
download_manager,
)
__all__ = [
'MapCatalogDownloader', 'MapExtractDownloader', 'MapSearchIndexDownloader',
'PreparedMapCatalog', 'ExecutedMapCatalog',
'PreparedMapExtract', 'ExecutedMapExtract',
'PreparedMapSearchIndex', 'ExecutedMapSearchIndex',
'map_catalog_downloader', 'map_extract_downloader', 'map_search_index_downloader',
]
logger = logger.getChild(__name__)
METALINK_NS = {'ml': 'urn:ietf:params:xml:ns:metalink'}
SHA256_PATTERN = re.compile(r'[0-9a-f]{64}')
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
# aria2c's exit code when a downloaded file does not match the meta4's hash.
ARIA2C_CHECKSUM_ERROR = 32
# Map files are downloaded here, and moved into the map directory only once aria2c has checked them against the signed
# hash. A failed download must never leave a file where `prepare_download` would take it as already downloaded.
PARTIAL_DIRECTORY = '.partial'
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
async def verified_meta4_sha256(meta4_contents: bytes) -> Optional[str]:
"""Parse meta4 XML bytes, extract the sha-256 hash and its GPG signature, and verify.
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
Returns the hash if its signature is valid, None if invalid or missing. Only this hash is signed; nothing else in
the meta4 is trusted."""
try:
root = ET.fromstring(meta4_contents)
file_el = root.find('ml:file', METALINK_NS)
if file_el is None:
logger.warning('meta4 missing <file> element')
return False
hash_el = file_el.find('ml:hash[@type="sha-256"]', METALINK_NS)
sig_el = file_el.find('ml:signature', METALINK_NS)
if hash_el is None or hash_el.text is None:
logger.warning('meta4 missing <hash type="sha-256"> element')
return False
if sig_el is None or sig_el.text is None:
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
logger.warning('meta4 missing <signature> element')
return False
sha256_hash = hash_el.text.strip()
signature = sig_el.text.strip()
except ET.ParseError as e:
logger.warning(f'Failed to parse meta4 XML: {e}')
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
return None
if not SHA256_PATTERN.fullmatch(sha256_hash):
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
logger.warning('meta4 sha-256 hash is malformed')
return None
# Write hash and signature to temp files for GPG verification.
with tempfile.TemporaryDirectory() as tmpdir:
hash_path = Path(tmpdir) / 'hash'
sig_path = Path(tmpdir) / 'hash.sig'
hash_path.write_text(sha256_hash)
sig_path.write_text(signature)
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
if await verify_gpg_signature(hash_path, sig_path):
return sha256_hash
return None
async def get_signed_sha256(downloader: Downloader, url: str, manifest_sha256: Optional[str],
label: str) -> tuple[Optional[str], Optional[str]]:
"""The signed sha-256 hash a map file must match, or an error explaining why there is none.
The hash from the signed manifest is used when it has one: a meta4 signs only a bare hash, so any genuinely signed
meta4 (an older version, another region) would pass its own check. Downloads from a manifest without hashes use
the file's signed meta4."""
if manifest_sha256:
if SHA256_PATTERN.fullmatch(manifest_sha256):
return manifest_sha256, None
return None, f'Malformed hash in the manifest for {label}'
meta4_contents = await downloader.get_meta4_contents(url)
if not meta4_contents:
return None, f'No signed hash (meta4) available for {label}, not downloading'
if not (sha256 := await verified_meta4_sha256(meta4_contents)):
return None, f'meta4 hash signature verification failed for {label}'
logger.info(f'meta4 signature verified for {label}')
return sha256, None
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
def meta4_file_name(url: str) -> Optional[str]:
"""The name aria2c saves `url` as, or None if the URL's file name is not a plain one."""
name = PurePosixPath(urlparse(url).path).name
return name if re.fullmatch(r'\w[\w.-]*', name) else None
def make_meta4(url: str, name: str, sha256: str) -> bytes:
"""A meta4 for aria2c holding only `url`, the `name` to save it as, and its signed `sha256` hash.
The downloaded meta4 is never given to aria2c: it would also obey any other hash (e.g. a sha-512 of a tampered
file), pieces, files, or URLs in it, none of which are signed."""
return (
'<?xml version="1.0" encoding="UTF-8"?>\n'
'<metalink xmlns="urn:ietf:params:xml:ns:metalink">\n'
f' <file name={quoteattr(name)}>\n'
f' <hash type="sha-256">{sha256}</hash>\n'
f' <url>{escape(url)}</url>\n'
' </file>\n'
'</metalink>\n'
).encode()
def remove_rejected_partial(error: Exception, partial_path: pathlib.Path):
"""Delete a partial download that aria2c rejected for not matching the signed hash; a retry would only reject it
again. Other failures keep it, so the next attempt resumes."""
if f'return code {ARIA2C_CHECKSUM_ERROR}' in str(error):
logger.warning(f'Deleting download that did not match its signed hash: {partial_path}')
partial_path.unlink(missing_ok=True)
partial_path.with_name(f'{partial_path.name}.aria2').unlink(missing_ok=True)
2026-05-06 19:46:24 -06:00
# ---------------------------------------------------------------------------
# MapCatalogDownloader
# ---------------------------------------------------------------------------
@dataclass
class PreparedMapCatalog:
"""Plan produced by MapCatalogDownloader.prepare_download.
`subscribed_regions` is the list of {region: ...} dicts the user has chosen to track.
The manifest itself is fetched in execute_download.
"""
url: str
subscribed_regions: List[dict]
@dataclass
class ExecutedMapCatalog:
"""Output of MapCatalogDownloader.execute_download.
Either `manifest` is populated (success) or `error` is set (fetch failed). Child
download creation happens in finalize_download where the manager's session is in
scope.
"""
manifest: Optional[dict]
subscribed_regions: List[dict]
error: Optional[str] = None
class MapCatalogDownloader(Downloader):
"""Fetches the manifest and creates a child Download for each subscribed region."""
name = 'map_catalog'
listable = False
2026-05-06 19:46:24 -06:00
def prepare_download(self, session: Session, download: Download) -> PreparedMapCatalog:
subscribed = (download.settings or {}).get('regions', [])
return PreparedMapCatalog(url=download.url, subscribed_regions=subscribed)
async def execute_download(self, prepared: PreparedMapCatalog, ctx: DownloadContext,
download: Download = None) -> ExecutedMapCatalog:
try:
2026-05-06 19:46:24 -06:00
manifest = await lib.fetch_manifest(prepared.url)
except Exception as e:
logger.error(f'Failed to fetch map manifest: {e}')
2026-05-06 19:46:24 -06:00
return ExecutedMapCatalog(
manifest=None,
subscribed_regions=prepared.subscribed_regions,
error=f'Failed to fetch manifest: {e}',
)
return ExecutedMapCatalog(manifest=manifest,
subscribed_regions=prepared.subscribed_regions)
2026-05-06 19:46:24 -06:00
def finalize_download(self, session: Session, download: Download,
executed: ExecutedMapCatalog) -> DownloadResult:
if executed.error:
return DownloadResult(success=False, error=executed.error)
manifest = executed.manifest or {}
version = manifest.get('version', '')
manifest_regions = manifest.get('regions', {})
search_indexes = manifest.get('search_indexes', {})
hashes = manifest.get('sha256', {})
if not manifest_regions:
return DownloadResult(success=False, error='Manifest does not contain regions')
2026-05-06 19:46:24 -06:00
if not executed.subscribed_regions:
return DownloadResult(success=True)
map_directory = lib.get_map_directory()
created = 0
# Defer search-index downloads so they are all inserted AFTER every pmtiles
# download. Downloads process in insertion order; this prevents a search-index
# download from running before its companion pmtiles file exists on disk.
pending_search_downloads = []
2026-05-06 19:46:24 -06:00
for r in executed.subscribed_regions:
region = r.get('region', '')
region_url = manifest_regions.get(region)
if not region_url:
logger.warning(f'Subscribed region {region!r} not found in manifest — skipping')
continue
# Compute the expected PMTiles filename.
if version:
pmtiles_name = f'{region}-{version}.pmtiles'
else:
pmtiles_name = f'{region}.pmtiles'
# Delete stale download for this region if the URL changed (new version).
old = session.query(Download).filter(
Download.downloader == 'map_extract',
Download.settings['region'].as_string() == region,
2026-05-06 19:46:24 -06:00
Download.url != region_url,
).one_or_none()
if old:
logger.info(f'Deleting stale download for {region}: {old.url}')
session.delete(old)
session.flush()
child = download_manager.get_or_create_download(session, region_url, reset_attempts=True,
override_skip=True)
child.downloader = 'map_extract'
child.location = '/map/manage'
child.settings = {
'region': region,
'version': version,
'search_index_url': search_indexes.get(region, ''),
'sha256': hashes.get(region_url),
2026-05-06 19:46:24 -06:00
}
child.renew(reset_attempts=True)
created += 1
# Queue search index download if the CDN provides one and the
# .search.db file does not already exist on disk.
search_url = search_indexes.get(region)
search_db_path = (map_directory / pmtiles_name).with_suffix('.search.db')
if search_url and not search_db_path.is_file():
pending_search_downloads.append((region, search_url, pmtiles_name))
# Insert all search-index downloads after every pmtiles download, so they
# process in insertion order behind their companion pmtiles.
for region, search_url, pmtiles_name in pending_search_downloads:
search_child = download_manager.get_or_create_download(
session, search_url, reset_attempts=True, override_skip=True)
search_child.downloader = 'map_search_index'
search_child.location = '/map/manage'
search_child.settings = {'region': region, 'pmtiles_name': pmtiles_name,
'sha256': hashes.get(search_url)}
2026-05-06 19:46:24 -06:00
search_child.renew(reset_attempts=True)
logger.info(f'Created search index download for {region}')
if not created:
return DownloadResult(success=False, error='No matching region URLs found in manifest')
logger.info(f'Created {created} map region downloads')
return DownloadResult(success=True, location='/map/manage')
map_catalog_downloader = MapCatalogDownloader()
2026-05-06 19:46:24 -06:00
# ---------------------------------------------------------------------------
# MapExtractDownloader
# ---------------------------------------------------------------------------
@dataclass
class PreparedMapExtract:
"""Plan produced by MapExtractDownloader.prepare_download.
Filenames are computed from settings; `already_done` short-circuits execute when
the output already exists. `error` carries an early validation failure (e.g. no
region in settings).
"""
region: str
version: str
map_directory: pathlib.Path
output_name: str
output_path: pathlib.Path
tmp_path: pathlib.Path
sha256: Optional[str] = None # From the signed manifest.
2026-05-06 19:46:24 -06:00
already_done: bool = False
error: Optional[str] = None
@dataclass
class ExecutedMapExtract:
"""Output of MapExtractDownloader.execute_download.
`skipped` covers the already-done short-circuit; `error` covers download/verify
failures. Otherwise the file is at output_path and finalize decides whether to
rebuild the search index locally.
"""
region: str
output_path: pathlib.Path
map_directory: pathlib.Path
skipped: bool = False
error: Optional[str] = None
class MapExtractDownloader(Downloader):
"""Downloads a pre-extracted region file from the CDN via aria2c.
Settings contain: region, version — set by MapCatalogDownloader.
aria2c checks the file against a signed hash (see `get_signed_sha256`), given to it in a meta4 that WROLPi writes.
"""
name = 'map_extract'
listable = False
2026-05-06 19:46:24 -06:00
def prepare_download(self, session: Session, download: Download) -> PreparedMapExtract:
settings = download.settings or {}
region = settings.get('region')
version = settings.get('version', '')
if not region:
2026-05-06 19:46:24 -06:00
return PreparedMapExtract(
region='', version='', map_directory=pathlib.Path('.'),
output_name='', output_path=pathlib.Path('.'), tmp_path=pathlib.Path('.'),
error='No region in download settings',
)
map_directory = lib.get_map_directory()
# Versioned filename.
if version:
output_name = f'{region}-{version}.pmtiles'
else:
output_name = f'{region}.pmtiles'
output_path = map_directory / output_name
tmp_path = map_directory / f'{output_name}.tmp'
2026-05-06 19:46:24 -06:00
prepared = PreparedMapExtract(
region=region, version=version, map_directory=map_directory,
output_name=output_name, output_path=output_path, tmp_path=tmp_path,
sha256=settings.get('sha256'),
2026-05-06 19:46:24 -06:00
)
# Skip if versioned file already exists.
if output_path.is_file():
logger.info(f'{output_name} already exists, skipping download')
2026-05-06 19:46:24 -06:00
prepared.already_done = True
return prepared
# Clean up any leftover temp file.
if tmp_path.is_file():
tmp_path.unlink()
2026-05-06 19:46:24 -06:00
return prepared
async def execute_download(self, prepared: PreparedMapExtract, ctx: DownloadContext,
download: Download = None) -> ExecutedMapExtract:
if prepared.error:
return ExecutedMapExtract(region=prepared.region, output_path=prepared.output_path,
map_directory=prepared.map_directory, error=prepared.error)
if prepared.already_done:
return ExecutedMapExtract(region=prepared.region, output_path=prepared.output_path,
map_directory=prepared.map_directory, skipped=True)
download = download if download is not None else Download(url=prepared.output_path.name)
# The original download URL is needed for download_file; the manager passes the real
# Download in production, so we read .url from it. In unit tests, callers must supply
# `download` with a URL.
url = download.url
# A map file is never downloaded without a signed hash.
sha256, error = await get_signed_sha256(self, url, prepared.sha256, prepared.region)
if error:
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
return ExecutedMapExtract(
region=prepared.region, output_path=prepared.output_path,
map_directory=prepared.map_directory, error=error,
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
)
if not (name := meta4_file_name(url)):
return ExecutedMapExtract(
region=prepared.region, output_path=prepared.output_path,
map_directory=prepared.map_directory,
error=f'No usable file name in {url}',
)
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
# Download via aria2c, which refuses a file that does not match the signed hash.
2026-05-06 19:46:24 -06:00
logger.warning(f'Downloading {prepared.region} from {url}')
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
partial_directory = prepared.map_directory / PARTIAL_DIRECTORY
partial_directory.mkdir(parents=True, exist_ok=True)
try:
downloaded_path = await self.download_file(
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
download, url, partial_directory,
check_for_meta4=False, concurrent=1, meta4_xml=make_meta4(url, name, sha256),
2026-05-06 19:46:24 -06:00
ctx=ctx,
)
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
# Only the file aria2c saved under the meta4's name was checked against the signed hash. The path
# download_file reports comes from the server's response, so it may name some other file.
if downloaded_path != partial_directory / name:
raise RuntimeError(f'Downloaded {downloaded_path.name!r}, expected {name!r}')
2026-05-06 19:46:24 -06:00
downloaded_path.rename(prepared.tmp_path)
except Exception as e:
2026-05-06 19:46:24 -06:00
logger.error(f'Download failed for {prepared.region}: {e}')
prepared.tmp_path.unlink(missing_ok=True)
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
remove_rejected_partial(e, partial_directory / name)
2026-05-06 19:46:24 -06:00
return ExecutedMapExtract(
region=prepared.region, output_path=prepared.output_path,
map_directory=prepared.map_directory,
error=f'Download failed: {str(e)[:500]}',
)
2026-05-06 19:46:24 -06:00
if not prepared.tmp_path.is_file():
return ExecutedMapExtract(
region=prepared.region, output_path=prepared.output_path,
map_directory=prepared.map_directory,
error=f'Output file was not created: {prepared.tmp_path}',
)
# Replace old file atomically.
2026-05-06 19:46:24 -06:00
prepared.tmp_path.rename(prepared.output_path)
# Delete older versions of this region.
2026-05-06 19:46:24 -06:00
for old_file in prepared.map_directory.glob(f'{prepared.region}-*.pmtiles'):
if old_file.name != prepared.output_name:
logger.warning(f'Deleting old map version: {old_file}')
old_file.unlink()
# Also delete unversioned file if it exists.
2026-05-06 19:46:24 -06:00
unversioned = prepared.map_directory / f'{prepared.region}.pmtiles'
if unversioned.is_file() and unversioned.name != prepared.output_name:
logger.warning(f'Deleting unversioned map file: {unversioned}')
unversioned.unlink()
2026-05-06 19:46:24 -06:00
size = prepared.output_path.stat().st_size
logger.warning(f'Successfully downloaded {prepared.region} ({size} bytes) to {prepared.output_path}')
return ExecutedMapExtract(region=prepared.region, output_path=prepared.output_path,
map_directory=prepared.map_directory)
def finalize_download(self, session: Session, download: Download,
executed: ExecutedMapExtract) -> DownloadResult:
if executed.error:
return DownloadResult(success=False, error=executed.error)
if executed.skipped:
return DownloadResult(success=True, location='/map')
# Build search index locally if tippecanoe is available and no CDN
# search index download already exists for this region.
2026-05-06 19:46:24 -06:00
has_search_download = session.query(Download).filter(
Download.downloader == 'map_search_index',
Download.settings['region'].as_string() == executed.region,
2026-05-06 19:46:24 -06:00
).count() > 0
if not has_search_download:
try:
from modules.map.search import rebuild_search_index
2026-05-06 19:46:24 -06:00
rebuild_search_index(executed.output_path.name)
except Exception as e:
2026-05-06 19:46:24 -06:00
logger.error(f'Failed to start search index build for {executed.region}: {e}')
return DownloadResult(success=True, location='/map')
map_extract_downloader = MapExtractDownloader()
2026-05-06 19:46:24 -06:00
# ---------------------------------------------------------------------------
# MapSearchIndexDownloader
# ---------------------------------------------------------------------------
@dataclass
class PreparedMapSearchIndex:
"""Plan produced by MapSearchIndexDownloader.prepare_download.
`error` covers missing settings or a missing PMTiles file (the search index can't
exist without one). `already_done` short-circuits when the .search.db is on disk.
"""
region: str
pmtiles_name: str
map_directory: pathlib.Path
pmtiles_path: pathlib.Path
search_db_path: pathlib.Path
sha256: Optional[str] = None # From the signed manifest.
2026-05-06 19:46:24 -06:00
already_done: bool = False
error: Optional[str] = None
@dataclass
class ExecutedMapSearchIndex:
region: str
skipped: bool = False
error: Optional[str] = None
class MapSearchIndexDownloader(Downloader):
"""Downloads a pre-built .search.db file from the CDN via aria2c with meta4 verification.
Created by MapCatalogDownloader alongside map_extract downloads for each
subscribed region. Will wait (fail + retry) until the PMTiles file exists.
"""
name = 'map_search_index'
listable = False
2026-05-06 19:46:24 -06:00
def prepare_download(self, session: Session, download: Download) -> PreparedMapSearchIndex:
settings = download.settings or {}
region = settings.get('region', '')
pmtiles_name = settings.get('pmtiles_name', '')
map_directory = lib.get_map_directory()
2026-05-06 19:46:24 -06:00
prepared = PreparedMapSearchIndex(
region=region, pmtiles_name=pmtiles_name, map_directory=map_directory,
pmtiles_path=map_directory / pmtiles_name if pmtiles_name else map_directory,
search_db_path=(map_directory / pmtiles_name).with_suffix('.search.db')
if pmtiles_name else map_directory,
sha256=settings.get('sha256'),
2026-05-06 19:46:24 -06:00
)
if not region or not pmtiles_name:
prepared.error = 'Missing region or pmtiles_name in settings'
return prepared
2026-05-06 19:46:24 -06:00
if not prepared.pmtiles_path.is_file():
prepared.error = f'PMTiles file not found: {pmtiles_name}'
return prepared
2026-05-06 19:46:24 -06:00
if prepared.search_db_path.is_file():
logger.info(f'Search index already exists for {region}, skipping')
2026-05-06 19:46:24 -06:00
prepared.already_done = True
return prepared
async def execute_download(self, prepared: PreparedMapSearchIndex, ctx: DownloadContext,
download: Download = None) -> ExecutedMapSearchIndex:
if prepared.error:
return ExecutedMapSearchIndex(region=prepared.region, error=prepared.error)
if prepared.already_done:
return ExecutedMapSearchIndex(region=prepared.region, skipped=True)
download = download if download is not None else Download(url=prepared.pmtiles_name)
url = download.url
# A search index is never downloaded without a signed hash.
sha256, error = await get_signed_sha256(self, url, prepared.sha256, f'{prepared.region} search index')
if error:
return ExecutedMapSearchIndex(region=prepared.region, error=error)
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
if not (name := meta4_file_name(url)):
return ExecutedMapSearchIndex(
region=prepared.region,
error=f'No usable file name in {url}',
)
# Download via aria2c, which refuses a file that does not match the signed hash.
2026-05-06 19:46:24 -06:00
logger.warning(f'Downloading search index for {prepared.region} from {url}')
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
partial_directory = prepared.map_directory / PARTIAL_DIRECTORY
partial_directory.mkdir(parents=True, exist_ok=True)
try:
downloaded_path = await self.download_file(
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
download, url, partial_directory,
check_for_meta4=False, concurrent=1, meta4_xml=make_meta4(url, name, sha256),
2026-05-06 19:46:24 -06:00
ctx=ctx,
)
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
# Only the file aria2c saved under the meta4's name was checked against the signed hash. The path
# download_file reports comes from the server's response, so it may name some other file.
if downloaded_path != partial_directory / name:
raise RuntimeError(f'Downloaded {downloaded_path.name!r}, expected {name!r}')
# Rename to match the local PMTiles filename.
2026-05-06 19:46:24 -06:00
downloaded_path.rename(prepared.search_db_path)
except Exception as e:
2026-05-06 19:46:24 -06:00
logger.error(f'Search index download failed for {prepared.region}: {e}')
Map files are never downloaded without a signed hash (#930) * Map files are never downloaded without a signed hash A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later) instead of downloading with TLS as its only check. The search index downloader had no fallback branch at all, so it downloaded with no hash whenever the .meta4 was missing. A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when the meta4's unsigned name differs from the URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * aria2c checks map files only against the signed hash, and a rejected file is never kept Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4. A sha-512 of a tampered file added beside the signed sha-256 made aria2c accept the tampered file. aria2c now gets a meta4 built from the signed hash and the download URL alone. A rejected download was left in the map directory under its final name, where the next attempt took it as already downloaded. Map files now download into map/.partial and move into place only once aria2c accepts them; a file that fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Only the map file aria2c checked is moved into place download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c saves and checks the file under the name in the meta4 WROLPi writes. When they differed, an unchecked file (such as an interrupted download's leftover in map/.partial) could be moved into place. The meta4 name now comes from the URL, must be a plain file name, and any other reported path fails the download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
remove_rejected_partial(e, partial_directory / name)
2026-05-06 19:46:24 -06:00
return ExecutedMapSearchIndex(
region=prepared.region,
error=f'Download failed: {str(e)[:500]}',
)
size = prepared.search_db_path.stat().st_size
logger.warning(f'Downloaded search index for {prepared.region} ({size} bytes)')
return ExecutedMapSearchIndex(region=prepared.region)
2026-05-06 19:46:24 -06:00
def finalize_download(self, session: Session, download: Download,
executed: ExecutedMapSearchIndex) -> DownloadResult:
if executed.error:
return DownloadResult(success=False, error=executed.error)
return DownloadResult(success=True, location='/map')
map_search_index_downloader = MapSearchIndexDownloader()