Commit graph

70 commits

Author SHA1 Message Date
lrnselfreliance
faf8e735d1
A map is exported as a PDF: a 1:24,000 quadrangle, in layers, on a sheet, which says where it is (#1006)
* The map can choose a 1:24,000 quadrangle to export (#971)

The first part of exporting a map as a PDF: a grid of 7.5-minute quadrangles is drawn over the
map, and a click chooses one.  The grid is arithmetic, so it needs no data.  Exporting itself is
not here yet; the button is disabled.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A quadrangle of the map is exported as a PDF with a layer for each map layer (#972)

* A quadrangle of the map is exported as a PDF with a layer for each map layer

The browser draws the quadrangle off the screen at the size of a 1:24,000 sheet, once for each
layer which is on, and the API puts the images in a PDF, each in a layer of its own which a
reader can show and hide.  A layer in which nothing is drawn is left out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The exported map's canvas keeps what was drawn, and an export's memory is bounded

- The drawing buffer is asked for where this MapLibre reads it (canvasContextAttributes); as an
  option of the map it was ignored, so the canvas could be read after it was cleared.
- Each layer's pixels are compressed in the PDF as the layer is added.  PyMuPDF kept them
  uncompressed until the PDF was written: about 90 MB a layer of a quadrangle, all at once.
- The pixels of all the layers together are limited, before any image is decoded.
- pymupdf is imported when a PDF is made, not when the API starts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where on the Earth it is (#973)

* An exported map says where on the Earth it is

The PDF's page has a geospatial viewport (ISO 32000-2): the page is the map in Web Mercator,
with the latitude and longitude of each corner.  The corners are the edges of the picture the
browser drew, read from the map which drew it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A map whose edge is the antimeridian can be exported

The map's arithmetic puts the east edge of the last quadrangle before the antimeridian a rounding error past
180 degrees, and the export was refused.  An edge within 1e-8 of a degree of 180 is at 180 (it is not wrapped
to -180); a map which is really across the antimeridian is still refused.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map is on a sheet: a margin, a grid, scale bars and notes (#974)

* An exported map is on a sheet: a margin, a grid, scale bars and notes

The map is put on a larger page with what a topographic map has around it: a title, the latitude and longitude
of its corners and of ticks between them, scale bars (kilometers, miles, feet), and notes of the projection, the
grid and where the map's data is from.  Over the map is a 1,000 meter UTM grid, each line named in the margin.

The grid and the margin are PDF layers of their own.  The PDF says where the map is, not where the page is.  The
picker has a box to export the map alone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The scale of a narrow map is written under its name, not over it

A quadrangle near a pole is two inches wide, which is not room for its name and its scale on one line.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where a compass points (#975)

* An exported map says where a compass points

The sheet has a diagram of the three norths (true, grid and magnetic) beside its scale bars, and a note of how
far magnetic north is from true north on the day the map was made, and how fast it moves.

The declination is from the World Magnetic Model 2025 (NOAA NCEI and the British Geological Survey; public
domain), whose coefficient file is shipped: nothing is downloaded.  The model is good until 2030; outside its
years nothing is said of a compass.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A corner of an exported map is not also a tick

The edges of the picture the browser draws are a hair outside the quadrangle's, so the quadrangle's own edges
were taken for ticks inside the map, and each was written over the coordinate of its corner.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A compass which passes due south has moved a little, not most of the way around

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A map can be exported at 400 DPI (#976)

The picker has a choice of 300 or 400 DPI.  A map is drawn less sharp than was asked for, rather than refused,
when it would be more pixels than the API makes a PDF of: a quadrangle near the equator, or one with many layers
on.  The picker says so while it draws.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported quadrangle goes by the name the USGS gives it (#977)

* An exported quadrangle goes by the name the USGS gives it

The picker says what the chosen quadrangle is called, and the sheet's title, the PDF's title and the file's
name have it.  Where the USGS has no name, which is most of the world, a quadrangle goes by where it is, as
before.

The names are an index of the 65,600 7.5 minute quadrangles of The National Map (U.S. Geological Survey; public
domain), which is shipped: nothing is downloaded.  In Alaska a USGS quadrangle is wider than 7.5 minutes, so a
quadrangle of the picker's grid there is "Part of" the one which has its middle.

The places the tests of the map export are of are changed for others.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An area half of which is the last quadrangle of the land is a part of it

The middle of such an area is on the quadrangle's east or north edge, with no quadrangle beyond it, and it was
given no name.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The PDF of a map is made by a Job, which the browser watches (#978)

* The PDF of a map is made by a Job, which the browser watches

A PDF of a quadrangle is a minute or two of work on a Raspberry Pi 4, and the request which waited for it was
cut off after a minute: only the smallest quadrangles could be exported there.

Now the browser describes the map, sends the image of each layer in a request of its own, and starts a Job; it
watches the Job, shows how far along it is, and fetches the PDF when the Job is done.  Cancel stops the Job.
The images are kept on disk until the PDF is made of them, and an export is removed when it is an hour old.

The credits on the sheet are no longer cut off: an attribution which names many sources is each of them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A cancelled map export stops, and is removed

- A cancelled Job is not over until the thread making the PDF has stopped, so the next Job does not start beside
  it; the work can also be stopped while the sheet is drawn and the PDF written.
- DELETE /api/map/export/<id> stops an export at any step and removes what was sent; the browser calls it when
  the person exporting cancels, while layers are sent as well as while the PDF is made.
- An export whose Job is waiting or running is not removed for being old; one whose Job was cancelled is removed
  at once.
- Cancel while the map is drawn stops the drawing between two layers.
- Exports are kept in a directory made for this user alone, which is refused when it is a link or is another
  user's; only what is an export there is removed.
- A request for the finished PDF which is lost is made again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An overlay's areas and lines go under the map's labels, and its own labels over them (#979)

An overlay was drawn whole over the basemap, so an area it filled covered the names of what is in it.  Labels are
now always on top: the areas, lines and pictures of every overlay are under the basemap's labels, and the text,
icons and points of every overlay are over them.  It is so in the map, the standalone viewer, and an exported PDF,
where an overlay with both is two layers ("Towers" and "Towers (labels)").

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An overlay's ground is over the ground of every basemap file (#980)

* An overlay's ground is over the ground of every basemap file

On a map of several basemap files, the ground of an overlay was put before the first label of the first file,
which is under the land, water and roads of every file after it: an overlay's areas and lines were hidden, and
in an exported PDF its layer was just above the background.  It goes where the basemap's labels begin: the first
of the labels which are over all of the basemap's ground.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The labels of every basemap file are over the ground of every file

A map of several basemap files was their layers one file after another, so only the labels of the last file were
on top: the labels of the others were under the land, water and roads of the files after them, and under the
ground of an overlay.  Both viewers now draw the ground of every file and then the labels of every file.

The contour lines are drawn after the basemap's labels and are not of the basemap: an overlay's ground goes under
the basemap's labels, not under the contours instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* Contours are drawn from as deep as the terrain file goes (#981)

The map read a terrain file to zoom 8, which is how deep the terrain WROLPi offers goes, however deep the file
was.  It now reads a file as deep as the file says it goes, in both viewers, and draws finer contour lines from a
deeper file: every 40 feet (or 20 meters) from zoom 11, as a 1:24,000 sheet has.  A file which goes to zoom 8 is
drawn as it was.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map names its small roads (#982)

* An exported map names its small roads

A quadrangle is drawn at about zoom 13.  The basemap's style names small roads
from zoom 15, and its tiles have no names for them below zoom 14, so the streets
of a town were printed as lines with no names.

Before the map is drawn, the tiles of zoom 14 which the quadrangle is in are
read, and the roads in them which have a name are given to the export map as a
source of its own.  The layer which names small roads draws from that source, as
its style says to.

At the scale of a sheet most streets are shorter than their names, and a map
writes a name along a line only where the line is longer.  Such a road is named
over its middle, turned as the road goes; a road in pieces is named once.

A source whose tiles can not be read is exported as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The names of small roads are placed by the export, at every latitude

A quadrangle nearer the equator is drawn at a little over zoom 14, where the
names were not read and the style still kept them for zoom 15: such a map had
no names of small roads.  They are now read wherever a layer would not name
roads.

Every name of a small road is now a point: over the middle of its road, turned
as the road goes under it, again every 400 pixels of a long road.  Before, a
road long enough was left for the map to name along its line, which the map
often does not (the line bends, or is in pieces), and a short piece beside it
was named too.  One place decides where a road is named, so a road in pieces is
named once, and a piece of another kind of road does not take the name.

Reading the names can be cancelled.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet has a bar of meters, its norths in mils, and how far apart its contour lines are (#983)

Under the map are bars of kilometers, meters, miles and feet.

How far the grid's north and a compass's north are from true north is written in
mils as well as in degrees, in the diagram and in the notes: a compass may be
marked in either.

A map which is exported with its contour lines says how far apart they are
(CONTOUR INTERVAL 40 FEET), under the bars.  The browser knows: it is the
interval of the tiles the lines are made for at the zoom of the sheet, in the
unit the map measures in.  It says so with the sheet, and what is not an
interval is not written.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet says what the quadrangles around it are called (#984)

* A sheet says what the quadrangles around it are called

Under the scale bars is a diagram of nine squares, the map the middle one, and
beside it what the USGS calls each of the eight around it: the sheets to print
to carry on off the edge of this one.

The names are from the index of quadrangles which names the sheet itself.  A
quadrangle with no name (out to sea, or over a border) has no number; a map of
where the USGS names nothing, and a map too narrow for the diagram, go without.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet is never beside itself

Where a quadrangle of the USGS is wider than the map (Alaska), the area the
size of the map to its east or west is often more of the quadrangle the map is
in, and the diagram named the sheet as its own neighbor.  The quadrangle on a
side is now the next one which is not the map's own.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet has a legend of the lines of its map (#985)

* A sheet has a legend of the lines of its map

Under the scale bars is what each kind of line of the map is: highway, major
road, minor road, path, railway, boundary, contour and index contour, each a
piece of the line as the map draws it, on the color of the map's land.

The map's style says how a line is drawn at each zoom, and a sheet is drawn at
one.  The browser, which has the style, says what each line's color, width,
dashes and edges are at that zoom; the API draws them.  A line is in the legend
when its layer is exported and draws something at that zoom.

What the browser sends is made into lines before it is kept: what is not a line
is left out, and the map is exported.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A legend has no line too thin to see, and no name longer than its room

A style brings a line in from nothing as the map is zoomed in: a path is a few
hundredths of a point wide on a quadrangle near the equator.  It was in the
legend, drawn several times wider than on the map.  A line under a tenth of a
point is now left out, by the browser and by the API.

What a line is called is cut to the room beside its line, as the name of an
adjoining quadrangle is.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* Fixes to the map PDF export: wrong neighbors, leftover exports, exports which outlive the map (#989)

* A sheet names what is around it, and its edges, by where they are and not a hair from there

The edges the browser gives of a map are a hair outside its quadrangle.  The quadrangles around a map whose
middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of
them was the map itself.  An edge a hair south of the equator was 0 degrees south, a map which ended where the
UTM zones do had no grid, and a sheet dated in the last year there is could not be made.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export is removed when it is over, and its work stops when the API does

The images and the PDF of an export were kept until the next export began, however long that was.  The browser
now has the export removed however it ends, and old exports are removed when a PDF is made too.  The work of a
Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it.  What
is said of an export, and each image, is written whole.  Text which is no text is taken out when an export
begins, not found by the Job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export stops with its map, and waits for a page which can not be seen

Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did
closing the picker, which is now left open.  A PDF which arrives after the export was stopped is not saved.
The time a page is behind another tab is not counted against a map which is drawing.  The name of a road is
one line.  A photograph which ends a little inside the map is not made a JPEG with a black strip.  An accent
in the name of a quadrangle is not a gap in the name of its file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A PDF is not kept from who made it by a server slow to remove the export

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its water over its parks (#990)

* A sheet names what is around it, and its edges, by where they are and not a hair from there

The edges the browser gives of a map are a hair outside its quadrangle.  The quadrangles around a map whose
middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of
them was the map itself.  An edge a hair south of the equator was 0 degrees south, a map which ended where the
UTM zones do had no grid, and a sheet dated in the last year there is could not be made.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export is removed when it is over, and its work stops when the API does

The images and the PDF of an export were kept until the next export began, however long that was.  The browser
now has the export removed however it ends, and old exports are removed when a PDF is made too.  The work of a
Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it.  What
is said of an export, and each image, is written whole.  Text which is no text is taken out when an export
begins, not found by the Job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export stops with its map, and waits for a page which can not be seen

Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did
closing the picker, which is now left open.  A PDF which arrives after the export was stopped is not saved.
The time a page is behind another tab is not counted against a map which is drawing.  The name of a road is
one line.  A photograph which ends a little inside the map is not made a JPEG with a black strip.  An accent
in the name of a quadrangle is not a gap in the name of its file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its water over its parks

A layer of the PDF was one image, drawn where the topmost of its layers is.  The parks of Land Cover are under
the water on the map and its piers are over it, so all of Land Cover was drawn over the water: a lake in a
park, or among houses, was not on the PDF.  A second basemap file put its water over the first's buildings.

A layer is now as many images as it must be, each where it is on the map, in one layer of the PDF: a part
which fills an area stays where it is, and a part which is only lines is drawn with the next.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A PDF is not kept from who made it by a server slow to remove the export

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A map file which is of somewhere else is not drawn when a map is exported

Each basemap file is several images of an exported map, and most files of a library are of other places: each
was an image of nothing, counted against how many images an export may be and how sharp it may be.  A file
whose own edges are outside the quadrangle is now left out before the layers are counted.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its streams (#991)

A basemap draws streams from zoom 14, and service roads and taxiways from 13.  A quadrangle is drawn a little
short of that (zoom 13.8 at 40 degrees, 13.2 at 61), from tiles which have them: none were on the PDF.  What
is drawn from a zoom no deeper than a quadrangle's detail is drawn.  Names are left as they are.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An export which is cancelled stops at once, in the middle of a request (#993)

* An export which is cancelled stops at once, in the middle of a request

An export was stopped between two requests.  The image of a layer is tens of megabytes, which is a long request
on a slow connection, and a request to a server which has stopped answering never ends: the export said it was
stopping until then.  Cancel, and leaving the map, now end the request.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The requests an export makes after its layers are sent are seen to be stopped too

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A map is exported as it is printed: on white, in dark lines (#996)

* A map is exported as it is printed: on white, in dark lines

The light and dark maps are made for a screen.  On paper their roads are white on pale gray and their streams
half a pixel of pale blue.  The printed map is a third look, of the map's own: white ground, dark road edges
and names, blue streams wide enough to see, paths drawn, brown contours.

It is what a map is exported in unless Light or Dark is chosen where the quadrangle is, and the choice is
remembered in the browser.  While a quadrangle is chosen the map is shown in the look it will be exported in.
The light and dark maps are as they were.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Changing the look of the map does not move it

What is done when a new style is in place was asked for after the style was set.  A style which only differs
from the one the map has is in place before that, so it was done at the next change of style instead, with
where the map had been at this one: the map went back there when a quadrangle was chosen, and again when it was
not.  It is asked for first, and the map is not moved at all.  An export no longer waits for the map to be
still, which it need not and which nothing ended.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The colors of the printed map are chosen where its layers are (#997)

* The colors of the printed map are chosen where its layers are

While a quadrangle is chosen and the map is shown as it is printed, each layer of the layer control has its
color beside it, to be changed: one for most, three for the land (what grows, what was built, sand and ice)
and two for names (the name, and what is around it).  A basemap has several colors where one is chosen, which
are shades of it.  The edge of a road stays dark.  What is chosen is remembered in the browser, and can be put
back.  The map on the screen is as it was.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A ramp on a bridge is the color of a major road, as it is off one

A basemap draws a ramp on a bridge in the color of a minor road on a bridge: on the printed map it was white
where the rest of the ramp is yellow, and followed the color chosen for minor roads.  What puts the colors
back is a button, which a keyboard reaches.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A layer with nothing in it does not make an exported map less sharp (#1001)

* A layer with nothing in it does not make an exported map less sharp

How sharp a map may be depends on how many images it is, and every group of layers was counted: a basemap
file with no runway, highway or boundary in the quadrangle was images of nothing, drawn, sent, and counted
against the rest.  A map of 23 images asked for at 400 DPI was made at 330.  The groups are now looked at
first, on a small map, and only those with something in them are drawn, as sharp as that many may be.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An image which arrives after its export was removed is not kept, and one mil is a mil

Stopping an export while an image was on its way left the image behind it, in a directory nothing would
remove for an hour.  An angle of one mil was "1 mils" on the sheet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The map which is exported from is made as sharp as it is to be (#1002)

* The map which is exported from is made as sharp as it is to be

The groups of a map are looked at on a small map, which was then made larger to draw them.  A browser lets
a canvas be larger when it is made than it lets one be made larger afterwards: a quadrangle asked for at
400 DPI was 33 million pixels, 288 DPI at 40 degrees, less than before the groups were looked at first.  The
small map is now removed, and the map which is exported from is made in its place.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A canvas the browser stopped drawing on is not a layer with nothing in it

A canvas whose drawing was taken away by the browser reads as clear, and a group looked at then was left out
of the PDF without a word: the export fails instead.  An export stopped as the last group is looked at does
not make the map it would have been drawn on.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where it is in a projection Adobe Acrobat knows (#1003)

* An exported map says where it is in a projection Adobe Acrobat knows

The page was said to be in Web Mercator, which it is drawn in.  Acrobat does not know Web Mercator, in the
words of the standard or in Esri's: it took the page for a map, and put every place on it at 0, 0.  The page
is now said to be Mercator on the WGS 84 ellipsoid, which it reads, with the same corners.  Between the corners
the two differ by under three centimeters on the ground.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* How well a page is said to be where it is, is measured from what the PDF says

The test of it measured two formulas of its own.  It now reads the corners and the projection from the PDF and
does a reader's sum with them.  The few centimeters are of a quadrangle: a taller page is less well said, by
the square of how tall it is, which the comment now says and a test measures.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The help says how a map is exported as a PDF (#1004)

The help of the map is five pages, one of them the PDF export.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 17:22:31 -07:00
lrnselfreliance
0389e4659c
An overlay's layers which the basemap already draws start hidden (#969)
A file of features often carries a basemap's layers too (roads, water, buildings, place names).
Drawn over the basemap they double every road and label.  Those layers, by the names of the
Protomaps and OpenMapTiles schemas, are now off until they are checked; every other layer is on
as before, and a choice already made is kept.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 11:17:31 -07:00
lrnselfreliance
b2b8e2075c
The map draws its own icons on an overlay (#967)
An overlay drawn with a style of its own names icons, which needed the sprite sheet that style was made for.  The map now ships icons for overlays, the Maki and Temaki sets (both CC0), and always draws those, in both viewers.

The sprite sheet is the translation: its index lists every icon under its own name, under each other name it answers to (modules/map/icons/aliases.json), and in both spellings of each.  A name with no icon is drawn as a plain dot.  The icons are signed distance fields, so a style's icon color and halo apply.

A pattern is not an icon: what it filled (a fill, a line or an extrusion) is drawn in a translucent color, where removing a pattern used to leave MapLibre's opaque black.

A sprite sheet beside a map file is no longer used, served or reported.  wrolpi/scripts/build_map_icons.py builds the sheet from the SVGs; the sheet is committed.
2026-11-13 17:59:21 -07:00
lrnselfreliance
177d0dba4b
The standalone map viewer draws overlays, as the map does (#966)
The standalone viewer (port 8084) drew basemaps only: a map file which is
not a basemap was skipped.  It now offers each overlay in its layer panel,
hidden until turned on, with a checkbox per layer, draws it with the style
beside it when there is one, and shows a feature's attributes on a click.

The overlay logic is the map viewer's own.  It moved to dependency-free
modules (app/src/map-shared) which a browser can load as they are; the
standalone viewer, which has no build step, loads copies of them from
modules/map/static/js.  A test fails while a copy differs from its module
(`npm run sync:map-shared` makes the copies).  What differs between the
viewers is passed in: where a map file and its style are fetched from, and
whether a layer can be checked before MapLibre is given it.

The standalone viewer cannot check a layer first, so it adds an overlay's
layers to the map one at a time once the style has loaded: MapLibre refuses a
whole style over one layer it cannot draw, but only that layer when it is
added by itself.

The API also names a style by its path within the map directory, which is
what the standalone viewer is served.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 17:28:41 -07:00
lrnselfreliance
113e70a402
Contour labels are drawn in a font the map has (#965)
The contour label layer named no font, so MapLibre asked for its default
("Open Sans Regular,Arial Unicode MS Regular"), which the map does not ship.
The request failed and the labels were drawn with glyphs the browser made up
locally.  The layer now names Noto Sans Regular, in the map viewer and in the
standalone viewer.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 15:53:01 -07:00
lrnselfreliance
4f58d2862f
A style beside a PMTiles overlay says how the map draws it (#963)
When a MapLibre style (and optionally a sprite sheet) sits beside a PMTiles file, the overlay is drawn with it instead of the generic colors: `<name>.style.json` and `<name>.sprite.json/.png`, or `style.json` and `sprite.json/.png` shared by the files of a directory.

Only the style's layers which draw that file are used; its sources, sprite and glyph URLs never are.  Fonts are replaced by the ones the map has, images are pointed at the overlay's own sprite (as strings, expressions or property functions), and each layer is validated so one the map could not draw is skipped instead of stopping the map.

The sprite is served by the API, which falls back to the single resolution sheet when there is no double resolution one.
2026-11-13 15:41:43 -07:00
lrnselfreliance
adb0adbd94
The map draws any PMTiles file as an overlay, in any sub-directory (#962)
Any PMTiles file under the map directory, at any depth, can be shown.  The API reads each file's header and metadata and says what it is (basemap, terrain, overlay, unreadable).  Overlays are offered in the layer control under the name the file gives itself, hidden until turned on, with a checkbox per vector layer, and are drawn generically.

Map files are named by their path within the map directory everywhere, and unfinished downloads are no longer listed.  A file can be ignored from the Manage page; ignored files are kept in a new maps.yaml config.

PMTiles metadata is not decompressed past its size limit, and the standalone viewer encodes map file paths.
2026-11-13 15:11:01 -07:00
lrnselfreliance
e6b7b9e5dd
Map downloads are checked against the hash in the signed manifest (#934)
* Map downloads are checked against the hash in the signed manifest

A meta4 signs only a bare hash, so any genuinely signed meta4 (an older version of a region, another region) passed
its own check.  The manifest now lists each file's sha256; it is signed as a whole, so the hash is tied to the URL.
The catalog copies each hash into its download's settings, and aria2c checks the file against it.  Downloads from a
manifest without hashes still require the file's signed meta4.

build_map_regions.py writes the hashes, and hash_map_manifest.py adds them to the published manifest from the
verified meta4s without rebuilding any maps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hash_map_manifest.py hashes each file's bytes instead of trusting its meta4's name

Only the bare hash in a meta4 is signed, so an older version's meta4 with its name rewritten (or terrain's, whose name
never changes) passed the name and duplicate-hash checks.  Each file is now hashed from a local copy of the published
files (--files) or downloaded from the CDN, and must match its signed meta4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 20:38:57 -07:00
lrnselfreliance
21006eae09
Map files are never downloaded without a signed hash (#930)
* Map files are never downloaded without a signed hash

A region, terrain, or search index download whose .meta4 is missing or unfetchable now fails (and retries later)
instead of downloading with TLS as its only check.  The search index downloader had no fallback branch at all, so it
downloaded with no hash whenever the .meta4 was missing.

A real aria2c test pins what this relies on: aria2c refuses a file that does not match the signed hash, even when
the meta4's unsigned name differs from the URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* aria2c checks map files only against the signed hash, and a rejected file is never kept

Only the sha-256 hash in a meta4 is signed, but aria2c was given the whole downloaded meta4.  A sha-512 of a tampered
file added beside the signed sha-256 made aria2c accept the tampered file.  aria2c now gets a meta4 built from the
signed hash and the download URL alone.

A rejected download was left in the map directory under its final name, where the next attempt took it as already
downloaded.  Map files now download into map/.partial and move into place only once aria2c accepts them; a file that
fails the hash check is deleted so a retry starts over, other failures keep it so the download resumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Only the map file aria2c checked is moved into place

download_file reports the path named by the server's HEAD response (Content-Disposition, redirects), while aria2c
saves and checks the file under the name in the meta4 WROLPi writes.  When they differed, an unchecked file (such as
an interrupted download's leftover in map/.partial) could be moved into place.  The meta4 name now comes from the
URL, must be a plain file name, and any other reported path fails the download.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:03:16 -07:00
lrnselfreliance
9352002d07
Map downloads compare a Download's region with SQLite's JSON operator (#894)
`Download.settings['region'].astext` is PostgreSQL's JSONB operator.  On SQLite it raises AttributeError, so the
map catalog failed before queuing any region Download, and an extract never checked for a CDN search index.  Use
`.as_string()`.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 18:58:19 -06:00
lrnselfreliance
22ada7eb57
Remove unused code found by vulture (#885)
Deletes functions, methods, classes, schema dataclasses, error classes, Events
senders, and module-level constants that nothing references, including tests.
Also drops the unused `doc_cleanup(test_directory)` and
`_wait_for_sub_job_with_progress(collection_idx)` parameters, and the imports
that only served removed code.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 15:50:09 -06:00
lrnselfreliance
c3e25b84d6
Configs without database rows read and validate only; imports run off the event loop (#874)
The base import for a config with no sections now reads its file once, reports a 0-byte file as empty, treats a
missing file as the defaults, and records success or failure itself (sending the failure Event on request).  The
per-config overrides that only set successful_import are removed.

wrolpi.yaml and the downloader settings import in a thread at startup (import_file_configs), and the Import, Save
and backup Import buttons run their config work in a thread.

The pre-diff import path (import_decision, record_import_version) has no callers left and is removed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 15:17:31 -06:00
lrnselfreliance
3946e315fc
Channel and map subscription changes wait for a concurrent writer (#851)
PUT/DELETE /api/videos/channels/<id>, POST /api/videos/channels/<id>/tag,
and POST/DELETE /api/map/subscribe read, then wrote, in the request's
deferred transaction, so they failed with a 500 ("database is locked")
when another connection wrote in between.  They now write inside
request.ctx.db.write().

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 16:48:52 -06:00
lrnselfreliance
ebdf9dcc4b
Map previews that never drew, and a Docker map with no world under it (#658)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* ui: a preview map that waited for a container it could never see

Both map preview modals -- the per-region eye button and "View All Regions"
-- opened to an empty box.  Each renders a div into a Modal and hands it to
MapLibre from an effect gated on `open`, reading the node out of a `useRef`:

    if (!open || !mapContainer.current) return;

Mantine's Modal mounts its body one commit after `opened` flips true, so on
the render that opens the modal the ref is still null.  The effect returns,
nothing it depends on changes again, and it never runs a second time.  No
console error, no rejected promise, no canvas -- just an empty div.

Semantic UI's Portal mounted the body in the same commit that opened it, so
the ref was attached by the time effects ran.  The effect bodies are
byte-identical across that migration; the modal library is the only variable,
confirmed against a device still running the older build where both previews
render.

A callback ref instead, so the effect re-runs when the node arrives and this
stops depending on any modal's mounting schedule.  The async source lookup
also gets a cancelled flag: it could resolve after the cleanup ran, building
a map nothing would ever remove and leaking a WebGL context per open.

The tests drive the real page and assert a map was built against a node that
is in the document -- the outcome, rather than the mechanism that quietly
changed underneath it.  Seven of the eight fail on the code before this.

* build: docker had no world underneath its map regions

Caddy serves /blobs/map-overview.pmtiles out of /opt/wrolpi-blobs, and that
blob is the world at zoom 0-6 -- what a regional extract sits on top of.
Without it a user who subscribes to a few regions gets those regions floating
on an empty background everywhere their data does not reach.  Only a full
planet file, at 134GB, makes it unnecessary.

Raspberry Pi and Debian installs get the blob from pi-gen, the live-build
config, and upgrade.sh.  Docker got nothing: the compose file bind-mounted a
git-ignored host directory over /opt/wrolpi-blobs, and a bind mount replaces
the image's directory rather than merging with it, so the path existed and was
empty on every Docker WROLPi.  Verified against a running deployment, where
that URL returned 404 while the same URL on hardware returned 206.

So the web image fetches the blob at build time, as the other three builders
do, and the mount that would hide it is gone.  `curl -f` so an HTTP error
fails the layer -- an image that builds without the blob is this same bug,
except silent at build time too.

The tests read the deployment files rather than a built container: building
to check costs a CDN download and several minutes, and both things that broke
-- nothing fetches it, a mount hides it -- are visible in the source.

* ui: cancel the load handler too, and make the cancel test falsifiable

Review follow-ups on the preview modals.

The `cancelled` flag was checked before constructing the map but not inside
the `load` handler it installs, so cleanup landing between the two left that
body drawing into a map `remove()` had already torn down.  Guarded, mirroring
the `destroyed` pattern in MapViewer.js.  The source promise also gets a
`.catch`: the helper cannot reject today, but nothing says it never will, and
an unhandled rejection here would be as quiet as the bug this all started as.

Two test corrections, both cases of an assertion claiming more than it read:

The region-drawing test spied on `addSource` AFTER construction and then
re-fired `load` by hand, so its count came entirely from that manual call and
said nothing about the production path.  FakeMap now records the sources it is
given, and the assertion reads what the component's own `load` drew -- each
region once, in order, and none for the entry with no bbox.

The new cancel test passed with the guard deleted, which is worse than not
having it: Mantine had not mounted the modal body when it closed, so the
effect had never run and there was nothing to cancel.  It now waits for the
effect to reach the source lookup before closing, and ships beside a control
that releases the same held promise WITHOUT closing and requires a map.  The
control is what makes the zero meaningful; verified that removing the guard
fails the cancel test while the control still passes.
2026-08-11 20:37:23 -06:00
Roland
c644d83a5b Migrating map downloaders. 2026-05-07 15:54:53 -06:00
Roland
c726654de9
Map search added to global search. Preview cards are displayed for all results in new Map tab.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Map viewer has simple search.

Map search databases can be downloaded from CDN, or built locally. Map searches can be built offline, but will not have enriched things like state names downloaded from Wikidata.
2026-04-13 12:17:08 -06:00
Roland
bbc23dbf90
Adding map default view option to the map context menu. 2026-04-04 14:56:25 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
2a27d354a5
Creating AutoAwaitTestClient for tests which automatically runs background tasks and expires db objects.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-01-05 18:26:49 -07:00
Roland
344fe9035f
Removing optional_session, passing session down from request middleware. Moving session param to the front of functions.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-11 16:03:18 -07:00
Roland
1d8738ace2 Running alembic check in cicd. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
00c86e1a1a
Searching wrolpi user's home directory for browser profiles. 2025-05-02 17:24:51 -06:00
Roland
25cdbd7ba2
Updating map import times using 3.5GB pbf. 2025-03-25 15:52:43 -06:00
Roland
f4fee874fb
Updating map import time estimates. 2025-03-23 09:44:31 -06:00
Roland
2728e1bf50
Removing map import timeout so the map import is not cancelled. 2025-03-21 17:10:00 -06:00
Roland
d2ae135e5b
Updating maps import so previously imported pbf files are reset on import. Ignoring map tile initialization script failure when importing. 2025-03-13 15:52:40 -06:00
Roland
b43677e674
Testing importing multiple map pbf files. 2025-03-13 14:54:27 -06:00
Roland
64543b5ba6
Fixing importing multiple map files. 2025-03-03 19:53:16 -07:00
Roland
531ec3d65e
Creating new /api/archive/upload which accepts a file from the SingleFile browser extension and converts it to an Archive. 2025-01-30 18:41:32 -07:00
Roland
d34301a085
Configs are imported manually in main. Configs now have dump/save/import methods. Configs will refuse to save if they have not been imported. 2024-10-16 18:56:44 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
30a3ef1cac
Converting seconds_to_import to use param instead of global variable. 2024-06-17 14:49:20 -06:00
Roland
9ee73af61d
Converting APIError to use SanicException as base. 2024-05-01 18:04:25 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
b05b2e7a6d
Allowing maintainer to change special directories. 2024-03-09 16:51:26 -07:00
Roland
00fe799b8f
User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
Roland
f44ac91319
Use exponential curve for large map files. 2023-09-07 13:01:19 -06:00
Roland
3d6ad99f5b
Moving map function test into lib tests. 2023-08-19 20:38:23 -06:00
Roland
6791e88baf
Stopping/starting renderd during map import. Moving tile cache clearing to import script. 2023-08-17 11:49:19 -06:00
Roland
6376e6539b
Using flags.map_importing to handle map importing control. Updating RPi 4 map import estimate using create import tests, rather than slower append import tests. 2023-08-17 08:27:49 -06:00
Roland
1e0e72fa24
Using clear_map_cache.sh script to handle clearing cache and restarting renderd after map import. 2023-08-16 14:34:20 -06:00
Roland
503f9e90ae
Removing secondary renderd map cache directory, if it exists. 2023-08-16 13:06:05 -06:00
Roland
fdaaa510c4
Splitting errors into their modules. Removing API_ERRORS dict in favor of storing error data on the class. 2023-07-17 14:21:27 -06:00
Roland
93f0e7494f
Creating Event and Flag systems. Events are used to send and display a toast to the App. Flags are used to signal if some action has been performed, or if a package is installed. 2022-12-23 16:04:09 -07:00
Roland
3751ed7999
Reformatting files. 2022-12-14 19:11:11 -07:00
Roland
02124307e4
Downloads can be deleted. 2022-11-12 14:17:21 -07:00
Roland
d2fbf12e26
Using worst-case-scenario for calculating map time-to-import. 2022-11-05 14:38:33 -06:00
Roland
66336e7ca5
Using worst-case-scenario for calculating map time-to-import. 2022-11-05 13:28:40 -06:00