Commit graph

126 commits

Author SHA1 Message Date
lrnselfreliance
faf8e735d1
A map is exported as a PDF: a 1:24,000 quadrangle, in layers, on a sheet, which says where it is (#1006)
* The map can choose a 1:24,000 quadrangle to export (#971)

The first part of exporting a map as a PDF: a grid of 7.5-minute quadrangles is drawn over the
map, and a click chooses one.  The grid is arithmetic, so it needs no data.  Exporting itself is
not here yet; the button is disabled.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A quadrangle of the map is exported as a PDF with a layer for each map layer (#972)

* A quadrangle of the map is exported as a PDF with a layer for each map layer

The browser draws the quadrangle off the screen at the size of a 1:24,000 sheet, once for each
layer which is on, and the API puts the images in a PDF, each in a layer of its own which a
reader can show and hide.  A layer in which nothing is drawn is left out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The exported map's canvas keeps what was drawn, and an export's memory is bounded

- The drawing buffer is asked for where this MapLibre reads it (canvasContextAttributes); as an
  option of the map it was ignored, so the canvas could be read after it was cleared.
- Each layer's pixels are compressed in the PDF as the layer is added.  PyMuPDF kept them
  uncompressed until the PDF was written: about 90 MB a layer of a quadrangle, all at once.
- The pixels of all the layers together are limited, before any image is decoded.
- pymupdf is imported when a PDF is made, not when the API starts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where on the Earth it is (#973)

* An exported map says where on the Earth it is

The PDF's page has a geospatial viewport (ISO 32000-2): the page is the map in Web Mercator,
with the latitude and longitude of each corner.  The corners are the edges of the picture the
browser drew, read from the map which drew it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A map whose edge is the antimeridian can be exported

The map's arithmetic puts the east edge of the last quadrangle before the antimeridian a rounding error past
180 degrees, and the export was refused.  An edge within 1e-8 of a degree of 180 is at 180 (it is not wrapped
to -180); a map which is really across the antimeridian is still refused.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map is on a sheet: a margin, a grid, scale bars and notes (#974)

* An exported map is on a sheet: a margin, a grid, scale bars and notes

The map is put on a larger page with what a topographic map has around it: a title, the latitude and longitude
of its corners and of ticks between them, scale bars (kilometers, miles, feet), and notes of the projection, the
grid and where the map's data is from.  Over the map is a 1,000 meter UTM grid, each line named in the margin.

The grid and the margin are PDF layers of their own.  The PDF says where the map is, not where the page is.  The
picker has a box to export the map alone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The scale of a narrow map is written under its name, not over it

A quadrangle near a pole is two inches wide, which is not room for its name and its scale on one line.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where a compass points (#975)

* An exported map says where a compass points

The sheet has a diagram of the three norths (true, grid and magnetic) beside its scale bars, and a note of how
far magnetic north is from true north on the day the map was made, and how fast it moves.

The declination is from the World Magnetic Model 2025 (NOAA NCEI and the British Geological Survey; public
domain), whose coefficient file is shipped: nothing is downloaded.  The model is good until 2030; outside its
years nothing is said of a compass.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A corner of an exported map is not also a tick

The edges of the picture the browser draws are a hair outside the quadrangle's, so the quadrangle's own edges
were taken for ticks inside the map, and each was written over the coordinate of its corner.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A compass which passes due south has moved a little, not most of the way around

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A map can be exported at 400 DPI (#976)

The picker has a choice of 300 or 400 DPI.  A map is drawn less sharp than was asked for, rather than refused,
when it would be more pixels than the API makes a PDF of: a quadrangle near the equator, or one with many layers
on.  The picker says so while it draws.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported quadrangle goes by the name the USGS gives it (#977)

* An exported quadrangle goes by the name the USGS gives it

The picker says what the chosen quadrangle is called, and the sheet's title, the PDF's title and the file's
name have it.  Where the USGS has no name, which is most of the world, a quadrangle goes by where it is, as
before.

The names are an index of the 65,600 7.5 minute quadrangles of The National Map (U.S. Geological Survey; public
domain), which is shipped: nothing is downloaded.  In Alaska a USGS quadrangle is wider than 7.5 minutes, so a
quadrangle of the picker's grid there is "Part of" the one which has its middle.

The places the tests of the map export are of are changed for others.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An area half of which is the last quadrangle of the land is a part of it

The middle of such an area is on the quadrangle's east or north edge, with no quadrangle beyond it, and it was
given no name.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The PDF of a map is made by a Job, which the browser watches (#978)

* The PDF of a map is made by a Job, which the browser watches

A PDF of a quadrangle is a minute or two of work on a Raspberry Pi 4, and the request which waited for it was
cut off after a minute: only the smallest quadrangles could be exported there.

Now the browser describes the map, sends the image of each layer in a request of its own, and starts a Job; it
watches the Job, shows how far along it is, and fetches the PDF when the Job is done.  Cancel stops the Job.
The images are kept on disk until the PDF is made of them, and an export is removed when it is an hour old.

The credits on the sheet are no longer cut off: an attribution which names many sources is each of them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A cancelled map export stops, and is removed

- A cancelled Job is not over until the thread making the PDF has stopped, so the next Job does not start beside
  it; the work can also be stopped while the sheet is drawn and the PDF written.
- DELETE /api/map/export/<id> stops an export at any step and removes what was sent; the browser calls it when
  the person exporting cancels, while layers are sent as well as while the PDF is made.
- An export whose Job is waiting or running is not removed for being old; one whose Job was cancelled is removed
  at once.
- Cancel while the map is drawn stops the drawing between two layers.
- Exports are kept in a directory made for this user alone, which is refused when it is a link or is another
  user's; only what is an export there is removed.
- A request for the finished PDF which is lost is made again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An overlay's areas and lines go under the map's labels, and its own labels over them (#979)

An overlay was drawn whole over the basemap, so an area it filled covered the names of what is in it.  Labels are
now always on top: the areas, lines and pictures of every overlay are under the basemap's labels, and the text,
icons and points of every overlay are over them.  It is so in the map, the standalone viewer, and an exported PDF,
where an overlay with both is two layers ("Towers" and "Towers (labels)").

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An overlay's ground is over the ground of every basemap file (#980)

* An overlay's ground is over the ground of every basemap file

On a map of several basemap files, the ground of an overlay was put before the first label of the first file,
which is under the land, water and roads of every file after it: an overlay's areas and lines were hidden, and
in an exported PDF its layer was just above the background.  It goes where the basemap's labels begin: the first
of the labels which are over all of the basemap's ground.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The labels of every basemap file are over the ground of every file

A map of several basemap files was their layers one file after another, so only the labels of the last file were
on top: the labels of the others were under the land, water and roads of the files after them, and under the
ground of an overlay.  Both viewers now draw the ground of every file and then the labels of every file.

The contour lines are drawn after the basemap's labels and are not of the basemap: an overlay's ground goes under
the basemap's labels, not under the contours instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* Contours are drawn from as deep as the terrain file goes (#981)

The map read a terrain file to zoom 8, which is how deep the terrain WROLPi offers goes, however deep the file
was.  It now reads a file as deep as the file says it goes, in both viewers, and draws finer contour lines from a
deeper file: every 40 feet (or 20 meters) from zoom 11, as a 1:24,000 sheet has.  A file which goes to zoom 8 is
drawn as it was.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map names its small roads (#982)

* An exported map names its small roads

A quadrangle is drawn at about zoom 13.  The basemap's style names small roads
from zoom 15, and its tiles have no names for them below zoom 14, so the streets
of a town were printed as lines with no names.

Before the map is drawn, the tiles of zoom 14 which the quadrangle is in are
read, and the roads in them which have a name are given to the export map as a
source of its own.  The layer which names small roads draws from that source, as
its style says to.

At the scale of a sheet most streets are shorter than their names, and a map
writes a name along a line only where the line is longer.  Such a road is named
over its middle, turned as the road goes; a road in pieces is named once.

A source whose tiles can not be read is exported as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The names of small roads are placed by the export, at every latitude

A quadrangle nearer the equator is drawn at a little over zoom 14, where the
names were not read and the style still kept them for zoom 15: such a map had
no names of small roads.  They are now read wherever a layer would not name
roads.

Every name of a small road is now a point: over the middle of its road, turned
as the road goes under it, again every 400 pixels of a long road.  Before, a
road long enough was left for the map to name along its line, which the map
often does not (the line bends, or is in pieces), and a short piece beside it
was named too.  One place decides where a road is named, so a road in pieces is
named once, and a piece of another kind of road does not take the name.

Reading the names can be cancelled.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet has a bar of meters, its norths in mils, and how far apart its contour lines are (#983)

Under the map are bars of kilometers, meters, miles and feet.

How far the grid's north and a compass's north are from true north is written in
mils as well as in degrees, in the diagram and in the notes: a compass may be
marked in either.

A map which is exported with its contour lines says how far apart they are
(CONTOUR INTERVAL 40 FEET), under the bars.  The browser knows: it is the
interval of the tiles the lines are made for at the zoom of the sheet, in the
unit the map measures in.  It says so with the sheet, and what is not an
interval is not written.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet says what the quadrangles around it are called (#984)

* A sheet says what the quadrangles around it are called

Under the scale bars is a diagram of nine squares, the map the middle one, and
beside it what the USGS calls each of the eight around it: the sheets to print
to carry on off the edge of this one.

The names are from the index of quadrangles which names the sheet itself.  A
quadrangle with no name (out to sea, or over a border) has no number; a map of
where the USGS names nothing, and a map too narrow for the diagram, go without.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet is never beside itself

Where a quadrangle of the USGS is wider than the map (Alaska), the area the
size of the map to its east or west is often more of the quadrangle the map is
in, and the diagram named the sheet as its own neighbor.  The quadrangle on a
side is now the next one which is not the map's own.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet has a legend of the lines of its map (#985)

* A sheet has a legend of the lines of its map

Under the scale bars is what each kind of line of the map is: highway, major
road, minor road, path, railway, boundary, contour and index contour, each a
piece of the line as the map draws it, on the color of the map's land.

The map's style says how a line is drawn at each zoom, and a sheet is drawn at
one.  The browser, which has the style, says what each line's color, width,
dashes and edges are at that zoom; the API draws them.  A line is in the legend
when its layer is exported and draws something at that zoom.

What the browser sends is made into lines before it is kept: what is not a line
is left out, and the map is exported.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A legend has no line too thin to see, and no name longer than its room

A style brings a line in from nothing as the map is zoomed in: a path is a few
hundredths of a point wide on a quadrangle near the equator.  It was in the
legend, drawn several times wider than on the map.  A line under a tenth of a
point is now left out, by the browser and by the API.

What a line is called is cut to the room beside its line, as the name of an
adjoining quadrangle is.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* Fixes to the map PDF export: wrong neighbors, leftover exports, exports which outlive the map (#989)

* A sheet names what is around it, and its edges, by where they are and not a hair from there

The edges the browser gives of a map are a hair outside its quadrangle.  The quadrangles around a map whose
middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of
them was the map itself.  An edge a hair south of the equator was 0 degrees south, a map which ended where the
UTM zones do had no grid, and a sheet dated in the last year there is could not be made.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export is removed when it is over, and its work stops when the API does

The images and the PDF of an export were kept until the next export began, however long that was.  The browser
now has the export removed however it ends, and old exports are removed when a PDF is made too.  The work of a
Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it.  What
is said of an export, and each image, is written whole.  Text which is no text is taken out when an export
begins, not found by the Job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export stops with its map, and waits for a page which can not be seen

Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did
closing the picker, which is now left open.  A PDF which arrives after the export was stopped is not saved.
The time a page is behind another tab is not counted against a map which is drawing.  The name of a road is
one line.  A photograph which ends a little inside the map is not made a JPEG with a black strip.  An accent
in the name of a quadrangle is not a gap in the name of its file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A PDF is not kept from who made it by a server slow to remove the export

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its water over its parks (#990)

* A sheet names what is around it, and its edges, by where they are and not a hair from there

The edges the browser gives of a map are a hair outside its quadrangle.  The quadrangles around a map whose
middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of
them was the map itself.  An edge a hair south of the equator was 0 degrees south, a map which ended where the
UTM zones do had no grid, and a sheet dated in the last year there is could not be made.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export is removed when it is over, and its work stops when the API does

The images and the PDF of an export were kept until the next export began, however long that was.  The browser
now has the export removed however it ends, and old exports are removed when a PDF is made too.  The work of a
Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it.  What
is said of an export, and each image, is written whole.  Text which is no text is taken out when an export
begins, not found by the Job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export stops with its map, and waits for a page which can not be seen

Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did
closing the picker, which is now left open.  A PDF which arrives after the export was stopped is not saved.
The time a page is behind another tab is not counted against a map which is drawing.  The name of a road is
one line.  A photograph which ends a little inside the map is not made a JPEG with a black strip.  An accent
in the name of a quadrangle is not a gap in the name of its file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its water over its parks

A layer of the PDF was one image, drawn where the topmost of its layers is.  The parks of Land Cover are under
the water on the map and its piers are over it, so all of Land Cover was drawn over the water: a lake in a
park, or among houses, was not on the PDF.  A second basemap file put its water over the first's buildings.

A layer is now as many images as it must be, each where it is on the map, in one layer of the PDF: a part
which fills an area stays where it is, and a part which is only lines is drawn with the next.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A PDF is not kept from who made it by a server slow to remove the export

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A map file which is of somewhere else is not drawn when a map is exported

Each basemap file is several images of an exported map, and most files of a library are of other places: each
was an image of nothing, counted against how many images an export may be and how sharp it may be.  A file
whose own edges are outside the quadrangle is now left out before the layers are counted.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its streams (#991)

A basemap draws streams from zoom 14, and service roads and taxiways from 13.  A quadrangle is drawn a little
short of that (zoom 13.8 at 40 degrees, 13.2 at 61), from tiles which have them: none were on the PDF.  What
is drawn from a zoom no deeper than a quadrangle's detail is drawn.  Names are left as they are.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An export which is cancelled stops at once, in the middle of a request (#993)

* An export which is cancelled stops at once, in the middle of a request

An export was stopped between two requests.  The image of a layer is tens of megabytes, which is a long request
on a slow connection, and a request to a server which has stopped answering never ends: the export said it was
stopping until then.  Cancel, and leaving the map, now end the request.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The requests an export makes after its layers are sent are seen to be stopped too

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A map is exported as it is printed: on white, in dark lines (#996)

* A map is exported as it is printed: on white, in dark lines

The light and dark maps are made for a screen.  On paper their roads are white on pale gray and their streams
half a pixel of pale blue.  The printed map is a third look, of the map's own: white ground, dark road edges
and names, blue streams wide enough to see, paths drawn, brown contours.

It is what a map is exported in unless Light or Dark is chosen where the quadrangle is, and the choice is
remembered in the browser.  While a quadrangle is chosen the map is shown in the look it will be exported in.
The light and dark maps are as they were.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Changing the look of the map does not move it

What is done when a new style is in place was asked for after the style was set.  A style which only differs
from the one the map has is in place before that, so it was done at the next change of style instead, with
where the map had been at this one: the map went back there when a quadrangle was chosen, and again when it was
not.  It is asked for first, and the map is not moved at all.  An export no longer waits for the map to be
still, which it need not and which nothing ended.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The colors of the printed map are chosen where its layers are (#997)

* The colors of the printed map are chosen where its layers are

While a quadrangle is chosen and the map is shown as it is printed, each layer of the layer control has its
color beside it, to be changed: one for most, three for the land (what grows, what was built, sand and ice)
and two for names (the name, and what is around it).  A basemap has several colors where one is chosen, which
are shades of it.  The edge of a road stays dark.  What is chosen is remembered in the browser, and can be put
back.  The map on the screen is as it was.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A ramp on a bridge is the color of a major road, as it is off one

A basemap draws a ramp on a bridge in the color of a minor road on a bridge: on the printed map it was white
where the rest of the ramp is yellow, and followed the color chosen for minor roads.  What puts the colors
back is a button, which a keyboard reaches.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A layer with nothing in it does not make an exported map less sharp (#1001)

* A layer with nothing in it does not make an exported map less sharp

How sharp a map may be depends on how many images it is, and every group of layers was counted: a basemap
file with no runway, highway or boundary in the quadrangle was images of nothing, drawn, sent, and counted
against the rest.  A map of 23 images asked for at 400 DPI was made at 330.  The groups are now looked at
first, on a small map, and only those with something in them are drawn, as sharp as that many may be.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An image which arrives after its export was removed is not kept, and one mil is a mil

Stopping an export while an image was on its way left the image behind it, in a directory nothing would
remove for an hour.  An angle of one mil was "1 mils" on the sheet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The map which is exported from is made as sharp as it is to be (#1002)

* The map which is exported from is made as sharp as it is to be

The groups of a map are looked at on a small map, which was then made larger to draw them.  A browser lets
a canvas be larger when it is made than it lets one be made larger afterwards: a quadrangle asked for at
400 DPI was 33 million pixels, 288 DPI at 40 degrees, less than before the groups were looked at first.  The
small map is now removed, and the map which is exported from is made in its place.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A canvas the browser stopped drawing on is not a layer with nothing in it

A canvas whose drawing was taken away by the browser reads as clear, and a group looked at then was left out
of the PDF without a word: the export fails instead.  An export stopped as the last group is looked at does
not make the map it would have been drawn on.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where it is in a projection Adobe Acrobat knows (#1003)

* An exported map says where it is in a projection Adobe Acrobat knows

The page was said to be in Web Mercator, which it is drawn in.  Acrobat does not know Web Mercator, in the
words of the standard or in Esri's: it took the page for a map, and put every place on it at 0, 0.  The page
is now said to be Mercator on the WGS 84 ellipsoid, which it reads, with the same corners.  Between the corners
the two differ by under three centimeters on the ground.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* How well a page is said to be where it is, is measured from what the PDF says

The test of it measured two formulas of its own.  It now reads the corners and the projection from the PDF and
does a reader's sum with them.  The few centimeters are of a quadrangle: a taller page is less well said, by
the square of how tall it is, which the comment now says and a test measures.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The help says how a map is exported as a PDF (#1004)

The help of the map is five pages, one of them the PDF export.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 17:22:31 -07:00
lrnselfreliance
d157057d81
Help: Sync (a secondary WROLPi) and the backup drive's trash (#1000)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Dead Code / vulture (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
The final part of replacing the sync: the help documents it.  A new Sync page; Backup rewritten around the backup
drive and its delayed delete (it taught a hand-made rsync job over SSH); sync.yaml and sync_state.yaml in How
Configs Work.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:16:38 -07:00
lrnselfreliance
2ac4a3a294
A WROLPi serves its library over rsync, and makes a database snapshot on request (#988)
* A WROLPi serves its library over rsync, and makes a database snapshot on request

The first part of replacing the sync: a secondary will copy its primary's whole media directory with rsync and
replace its database with the primary's, instead of downloading the files of FileGroups over HTTP (which never
copies an ignored directory, such as the map's).

- An rsync daemon serves the media directory read-only on port 8873, on every WROLPi.  It never serves the TLS keys,
  the cookies, or the database in use.  On a Raspberry Pi and Debian it is the wrolpi-rsyncd unit; in Docker the
  Controller runs it in a helper container made from the api container (its image, user and volumes), so the media
  directory is still configured in one place.  All three read controller/rsyncd.conf.
- POST /api/sync/snapshot asks for a snapshot of the database, which the daemon serves from config/.sync.  An
  unchanged database is not written again.
- /api/sync/info also says the database's migration and whether the media directory is mounted.

Nothing uses these yet; the existing sync is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The rsync daemon follows no links, and no snapshot is made in WROL Mode

- A client that asked the daemon to follow symbolic links could read what a link in the library points at.  A
  relative link to config/ssl was read this way although that directory is left out of the module.  The daemon now
  refuses the three options that follow links; a link is sent as a link.
- A database snapshot is a new file on the primary's drive, so WROL Mode refuses it, as it does a backup.  A snapshot
  made earlier is still served.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* No rsync helper from an image without rsync, and the helper follows the api container

Review found:

- A helper made from an api image built before rsync was added exits at once, and Docker restarted it forever while
  the Controller reported it created.  The image is now looked at first; no helper is made from one without rsync,
  an older helper is left serving, and the reason is logged once.
- The helper was only checked when the Controller started, so a recreated or rebuilt api container left it stale.
  It is now checked every minute.
- A snapshot cut off by its process being killed stayed marked as running and held back every later request.
- The snapshot's half-written temporary file was served by the daemon.
- The comment on the daemon's refused options claimed more than they give: a link that is part of the requested
  path is still followed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 16:47:02 -07:00
lrnselfreliance
0389e4659c
An overlay's layers which the basemap already draws start hidden (#969)
A file of features often carries a basemap's layers too (roads, water, buildings, place names).
Drawn over the basemap they double every road and label.  Those layers, by the names of the
Protomaps and OpenMapTiles schemas, are now off until they are checked; every other layer is on
as before, and a choice already made is kept.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 11:17:31 -07:00
lrnselfreliance
38832870c9
Help: map overlays, their styles and icons, and ignoring a map file (#968)
Bump the help docs to document overlays, the style which can sit beside a
map file, the map's own overlay icons, the Type and Visible columns of the
Map's Manage tab, and maps.yaml.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 11:09:20 -07:00
lrnselfreliance
6cd233dce6
Sync and backup: a secondary WROLPi and a backup drive copy the library (#957)
* A secondary WROLPi copies its primary's library over HTTP

A WROLPi with role "secondary" pulls from its primary: a manifest of every
FileGroup's files (/api/sync/manifest), then each missing or changed file
from the primary's Caddy /media/ (resumable, written to a hidden partial and
renamed with the primary's mtime), a refresh, then the library configs,
imported as the primary wrote them.  wrolpi.yaml contributes only its library
settings.  Files deleted on the primary are kept.

A secondary never saves the synced configs and never downloads.  Its API
answers 503 SYNCING while a sync runs, and every page warns that it is a
copy.  It syncs in a daily window or on "Sync now"; WROL Mode stops it.

On a Pi 4 (no AES instructions) the client uses TLS 1.2 ChaCha20, which
pulls ~110 MB/s from Caddy versus ~41 MB/s for rsync over ssh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync has its own /admin tab; a sync's reads from the primary are bounded

The sync settings and progress move from Settings (and the component
gallery, which is for library components) to /admin/sync.

A download never writes past the size in the primary's manifest, and configs,
listings, and manifest pages are capped.  A primary address may not carry a
user name or password (it is logged and shown).  While a sync runs, the API
answers only the exact requests that show its progress or stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync keeps the secondary's replaced configs in today's config backup

The first copy of the day is kept, like a save, so a config backup import
undoes a sync's config changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Docker's Caddy logs every request

`docker compose logs -f web` shows each request with its status, size and
duration, e.g. a secondary's /media/ downloads during a sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync reports its progress as it downloads, and refuses quietly

The status updated only when a page of 500 FileGroups finished, so a page of
large files looked stuck; it now updates every 2 seconds.  Requests refused
while syncing are answered by the middleware instead of raised, so the UI's
polling no longer logs an error each time.  The primary's version is stored
without its trailing newline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A WROLPi can back up to a second drive; sync settings live in sync.yaml

The Sync tab's Backup Drive copies the media directory to a second mounted
drive with rsync during the sync window (hourly), or on "Back Up Now".
Nothing on the backup drive is deleted.  The Tags and playlists directories
are left out (WROLPi rebuilds them from the configs), and the database is
written as an SQLite snapshot rather than copied as files, so the drive can be
mounted as the media directory and start.  A backup is written only while the
chosen drive (by UUID) is mounted where it was chosen, and is not the media or
system drive; rsync is stopped if the drive disappears.  Not in Docker, which
cannot reach other drives.

Role, primary, window and backup drive move from wrolpi.yaml to sync.yaml:
they belong to this machine, and are never copied from a primary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Sync tab shows the configured backup drive

getSettings copies each setting by name and did not list the backup drive,
so the Sync tab showed "None" while one was configured.  It also dropped
`version`.  A test now fails for any setting the API sends that getSettings
drops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A backup can go to a folder on its drive, and remembers its last success

The Backup Drive panel takes a folder on the chosen drive.  The drive is
checked through the folder's mount point: the folder is written only when the
drive with the saved UUID is mounted there, so a folder left on the system
drive by an unplugged drive, or on another drive mounted inside the chosen
one, is refused.  A missing folder is created once the drive is checked.

When the last backup succeeded is saved in sync.yaml, so it is known after a
restart.  "Stop Backup" is disabled while the database snapshot is written (it
runs to the end), and a finished backup clears the request to stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The backup drive has its own window

The sync window was shown in the Sync panel but also scheduled the backup
drive.  The Backup Drive panel now has its own window (sync.yaml
backup_window_start/end), saved with the drive; the sync window is shown,
and used, only for a secondary's sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync and the backup drive can each exclude folders

A secondary or a backup drive may be smaller than the primary's library.
Each Sync tab panel has its own excluded folders in sync.yaml, chosen in
a folder browser: a secondary browses its primary's folders (through
Caddy), the backup drive browses this WROLPi's media directory.  Excluded
folders are never copied; what was copied before is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The folder browser does not offer lost+found

Every ext4 drive has a root-only lost+found folder; a sync and a backup
never copy it, so it is not offered for exclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync never writes into config or tags, whatever their case

On a media drive that ignores case (exFAT, NTFS, macOS), a manifest path
like Config/wrolpi.db is the real config folder. The protected and
unlisted top-level folders are now compared case-insensitively in the
manifest, in resolve_destination, in excluded-folder validation, and in
both folder listings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A backup writes the database snapshot only when the database changed

Every backup rewrote the whole database snapshot (2.5 GB on a busy
WROLPi), even on the window's idle hourly runs. The snapshot is now
skipped when the live database (its size and mtime, and those of a
non-empty WAL) and the drive's copy are both unchanged since the last
snapshot. The record is in memory, so the first backup after a restart
writes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove the unused save_sync_config switch

Nothing activated it; SyncConfig saves directly.  The test server's
log_message override is called by http.server, so it goes in the
vulture baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:53:45 -07:00
lrnselfreliance
851b12b060
A Zim's Search toggle is saved in zims.yaml so a rebuilt database or a newer Zim keeps it (#931)
The toggle was only the zim.auto_search column, which defaults to on.  The
database is rebuilt from files and configs, and a Zim's newer version is a new
row, so every rebuild and every subscription update turned search back on for
every Zim.  On a large library the global search then searched all of them.

zims.yaml records auto_search for each Zim, keyed by the Zim's name without
its date, so a newer version of a Zim is the same Zim.  A Zim the file does not
name is searched (the default stays on).  The config is imported on startup
with the usual version gating, saved when the toggle changes, and a newly
modeled Zim takes its setting from it.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:09:52 -07:00
lrnselfreliance
8edace08bd
Python packages install from hash-locked requirements; yt-dlp still floats (#913)
* Python packages install from hash-locked requirements; yt-dlp still floats

Every pip install trusted PyPI over TLS: requirements.txt pins only top-level versions
(some with >=), the controller's are almost all >=, and dependencies were never pinned, so
a forged certificate (as the eMudhra key compromise allowed) could change code in the API,
the controller, or the archive container.  scripts/lock_requirements.sh generates a
universal, hash-locked .lock beside each requirements.txt, and
install_python_requirements.sh installs it with --require-hashes at every site.  yt-dlp is
left out of the lock and upgraded afterwards so devices still get YouTube fixes; its
dependencies are locked.  The unpinned `pip install --upgrade pip` steps are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The unhashed yt-dlp install cannot replace locked packages, and nothing builds from source

Review of #913 found two holes.  The yt-dlp step ran without --no-deps, so pip's
only-if-needed strategy would still fetch an unhashed dependency whenever yt-dlp's metadata
asked for a version the lock did not install.  And srt is published only as a source
archive, whose build installed setuptools with hash checking off on every install.

The yt-dlp step now uses --no-deps (its dependencies come only from the lock), both steps
are retried as the image builds did before, and installs and locks are wheels-only.  srt
3.5.3 is vendored unmodified as wrolpi/vendor/srt.py (MIT), verified against the sdist
hash the lock pinned; webvtt-py's SRT reader rejects WROLPi's own test file, so it is not
a replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:00:39 -07:00
lrnselfreliance
d69f8317aa
serve, single-file and readability-extractor install from a committed lockfile (#912)
They were installed with `npm i -g`, which pinned only the three top-level versions and
resolved their 156 dependencies fresh from the registry, so a forged TLS certificate (as
the eMudhra key compromise allowed) could change code WROLPi runs on every archive.
npm-tools/package-lock.json pins every package with its sha512, and install_npm_tools.sh
runs `npm ci` and links the commands into /usr/local/bin for every install path, upgrades,
both images and the archive container.  The app container's unpinned, unused global serve
is removed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 10:39:08 -06:00
lrnselfreliance
9ccf19b82b
Distrust compromised eMudhra intermediates in certifi and Deno (#854)
* Distrust compromised eMudhra intermediates in certifi and Deno

The private keys of eight eMudhra intermediate CAs can be recovered from
their certificates, so anyone can mint a trusted certificate for any host.

* Require certifi>=2026.10.19; yt-dlp, requests and curl-cffi use its bundle,
  and 2026.10.19 removes the emSign ECC Root CA - G3 root they chain to.
* Set DENO_TLS_CA_STORE=system for the API (systemd + Docker) and the archive
  container so Deno follows the apt-patched system store instead of roots
  that only change with a Deno upgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep Deno's bundled roots in the archive container

browserless/chrome:1.61.1 is Ubuntu 22.04 whose newest ca-certificates is
from 2023, older than Deno 2.8.3's bundled roots.  The system store only
becomes the better choice once the archive base image is upgraded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 08:54:57 -06:00
lrnselfreliance
1f07b5f944
Readability extraction no longer takes minutes on image-heavy pages (#838)
* Readability extraction no longer takes minutes on image-heavy pages

SingleFile inlines every image as a base64 data URI, and readability-extractor
(jsdom) slows to minutes on a page carrying megabytes of them, so these pages
hit the readability timeout and were saved without readability text.

Replace each base64 payload with a short placeholder before running
readability-extractor, then restore the payloads in its output.  Applied in
both the archive container and the native (Raspberry Pi/Debian) path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Data URI placeholders cannot be forged by page text

The placeholder was a fixed string, so a page containing placeholder-like
text could index past the payloads (IndexError, failing the archive on the
native path) or repeat a placeholder to copy a large image into the output
many times.  Each extraction now uses a random token, and text that is not a
known placeholder is left as-is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Data URI placeholders survive Readability's lazy-image cleanup

Readability's _fixLazyImages removes an <img> src whose base64 payload is
shorter than 133 characters when another attribute names an image file
(alt="cat.png", title, a leftover data-src), treating it as a lazy-load
placeholder.  The ~50 character placeholder triggered this, so those images
lost their inline data.  The token is now 160 hex characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 18:50:21 -06:00
lrnselfreliance
7aa7481a56
The hotspot starts on boot (#829)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Nothing started the hotspot since hotspot_on_startup moved out of the
API, and NetworkManager does not autoconnect the Hotspot connection.
The Controller now starts it once per boot unless hotspot.on_startup is
false; a legacy wrolpi.yaml hotspot_on_startup: false is migrated.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:29:15 -06:00
lrnselfreliance
5ec85c3e04
Help: document resuming where you left off (#799)
Bumps the help docs for the new "Resume Where You Left Off" page and the
recently_viewed.yaml config.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:25:29 -06:00
lrnselfreliance
93d878952c
Add a Repos module that mirrors git repositories (#783)
Repos keeps local mirrors of git repositories, cloned over https and kept up to date on a schedule.

- A Repo is a Collection (kind='repo') plus a `repository` row.  The new `git` downloader clones it, then updates it
  (weekly by default, or Never: the clone is kept as it is and only updated on request).
- Repos are saved in `repos/<tag>/<name>` (the Repos Directory, ignored by default) or a directory the user chooses.
  An existing clone can be imported where it is.  repos.yaml is the source of truth.
- A GitHub-like page shows each repo's README, files, and history, with a ZIP download.  READMEs are searchable
  under the global search's Other tab.
- git is hardened against hostile repos and configs: https only, pinned addresses with no private addresses (for the
  repo and its submodules), sanitized configs, no hooks or links, and only clones WROLPi made or imported are
  updated.
- Tagging a Repo moves its clone as one directory.  Repo changes wait for repos.yaml to be imported at startup.
- The file mover groups unindexed files with no primary file as one FileGroup, and never replaces a file at the
  destination.  A failed tag move restores the Collection's tag.  A Collection's directory cannot leave the media
  directory.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:16:58 -06:00
lrnselfreliance
cc8afa5370
Add a Solar calculator (#778)
* Add a Solar calculator with offline NASA POWER data

Estimates a solar array's typical-year production from a location and the
panels; everything else has defaults under an optional "Fine-tune" section.

The model works from 12 monthly averages: Klein's average day per month, the
Erbs monthly diffuse fraction, Collares-Pereira & Rabl / Liu & Jordan hourly
split, HDKR transposition for fixed, single-axis and dual-axis arrays, then
PVWatts' ASHRAE incidence loss, SAPM cell temperature, DC power, system
losses and optional inverter.  "Best for the year" and "Best for winter" sweep
the tilt.

Sunlight, diffuse, albedo and temperature come from NASA POWER's 2001-2020
climatology on a 1 degree grid, packed into app/public/data (3.1 MB, ~1 MB
gzipped) by scripts/build_solar_climatology.py and fetched only when the
calculator opens.  Users can replace it with their own monthly values.

Every setting lives in the URL, with defaults left out, so the Share button
shares the whole configuration.

Against PVWatts v8 for a 1 kW fixed array: Phoenix +0.6%, Seattle +8.8%,
Anchorage +8.0% annually; these are recorded as tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Open the Solar calculator from the map, pins and current location

The map's right-click (or long-press) menu gains "Calculate Solar
Performance", and each saved pin a sun button; both open the calculator at
that location.  A button beside Latitude fills in the browser's current
location.  Locations are rounded to 4 decimals, and longitudes from a map
panned around the world are wrapped back to -180..180.

The map removed lat/lon/z from the URL when it closed, even after navigating
to another page, which stripped the calculator's location from its URL (and
from links shared with the Share button).  It now only cleans up while the
URL is still a map page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add off-grid sizing to the Solar calculator

A new "Off-grid sizing" tab sizes a battery system for the darkest month
from a list of devices (watts x hours per day):

- Battery bank for a number of days without sun, for LiFePO4 or lead-acid,
  at 12, 24 or 48 V.  Outdoor lead-acid is enlarged for the cold; outdoor
  lithium warns against charging below 0 C.
- Panels needed, from the worst month's DC production per kW at the chosen
  angle, with a button to apply the count.  A PWM controller's loss (battery
  voltage over the panel's Vmp) raises the count.
- Charge controller current, and the string voltage on the coldest morning
  against the controller's limit, since cold panels exceed their rated Voc.
  The coldest morning defaults to 25 C below the coldest monthly average.

All inputs live in the URL like the rest of the calculator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bump wrolpi-help for the Solar calculator help page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Credit NASA POWER as it asks, and say how the data was changed

- The calculator gains a "Data attribution" link under the location with
  NASA POWER's requested acknowledgment, the data reference (Climatology API
  v2.10.0, downloaded 2026/10/04), the CC BY 4.0 license, and how WROLPi
  modified the data (rounded to one byte per value, temperature averaged
  into 1 degree cells, interpolated between cells), which CC BY 4.0 requires.
- The build script writes the same text to solar-climatology.LICENSE.txt
  beside the data, with the API version taken from NASA's responses and the
  date the tiles were downloaded.  A test keeps the page and file in step.
- Bumps wrolpi-help for the updated attribution in the help page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: fix PWM sizing, Voc sign, unit switching and loading on review

- PWM usable power now uses the string's voltage (Vmp x panels in series),
  delivers nothing when the charging voltage is at or above the string's Voc,
  and falls off between Vmp and Voc.  The "too little voltage" warning covers
  PWM as well as MPPT.  Previously two panels in series counted double, and a
  panel that could not charge a 48 V bank counted as fully usable.
- A Voc temperature coefficient entered without its minus sign is treated as
  negative, in the math, the URL and the input, so the cold-morning check
  cannot be made too generous.
- Switching Fahrenheit on or off converts the entered temperatures instead
  of reinterpreting them.
- While the sunlight data downloads, the off-grid tab shows a loader rather
  than asking for a location, and holds the cold-morning verdict until the
  site's temperatures are known or one has been entered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: move the data attribution to the bottom of the page

The full NASA POWER acknowledgment, license and changes now sit in a small
"Data attribution" section at the end of the calculator, always visible,
instead of behind a link under the location.  The source line under the
location points to it.  Bumps wrolpi-help to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: drop the data source line under the location

The attribution at the bottom of the page is enough.  The line under the
location now only appears to report the data's state: loading, failed to
load, or replaced by the user's own values.  Bumps wrolpi-help for the
refreshed screenshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:36:45 -06:00
dependabot[bot]
c973a4106f
build(deps): bump sanic from 24.12.0 to 24.12.1 in /docker/archive (#714)
Bumps [sanic](https://github.com/sanic-org/sanic) from 24.12.0 to 24.12.1.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Commits](https://github.com/sanic-org/sanic/compare/v24.12.0...v24.12.1)

---
updated-dependencies:
- dependency-name: sanic
  dependency-version: 24.12.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-03 10:07:17 -06:00
lrnselfreliance
57e647c5b4
docker: web container reaps zombies; zim and help healthchecks can pass (#744)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Caddy is PID 1 in the web container and reaps nothing.  The healthcheck's
busybox `wget --spider https://...` forks an `ssl_client` helper that is
orphaned to caddy and stays a zombie, one per check.  `init: true` runs
tini as PID 1 so they are reaped.

The zim healthcheck calls curl, which the debian:13 kiwix image does not
install, so every check exited 127.  The help healthcheck probed port
8086, which is Caddy's HTTPS proxy on the web container; mkdocs serves
on port 80 inside its own container.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 15:49:34 -06:00
lrnselfreliance
86d4bbab98
Bookmarks: a nav menu of user links, stored in bookmarks.yaml (#729)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* bookmarks: user bookmarks in the navigation bar, stored in bookmarks.yaml

A "Bookmarks" tab between Calculators and Flasher opens on hover or click and
lists the user's bookmarks, with directories as flyout submenus (Menu.Sub).
The mobile hamburger shows directories as labelled groups instead.  The tab
always shows, with an "Edit bookmarks" item, so the feature can be found
before any bookmark exists.

Bookmarks live in a new config-only bookmarks.yaml: a nested tree where a
node with `children` is a directory and a node with `url` is a bookmark.
Every node carries an integer id so the API can rename, move and delete
without relying on names.  A URL may be a path on this WROLPi, a port on this
host (`:8096/` or `http://:8096/`, resolved against whatever hostname the
browser used, so it works over LAN IP, mDNS and the hotspot alike), or an
absolute URL.  Each bookmark chooses whether to open in a new tab.

The editor at /more/bookmarks follows the playlist items table: numbered rows,
up/down/edit/delete, and inline "Add a bookmark" / "Add a directory" forms
with a Location picker.  Deleting a directory asks first, since its contents
go with it.  The menu and the editor are in the component gallery.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* bookmarks: mark More current when the editor is folded inside it; rem indents in the mobile menu

Found on the QA Pi at a width where Bookmarks overflows into More: the More
button carried no mark on /more/bookmarks, because the bookmarks link has no
`to` for isLinkActive to match.  isLinkActive now knows the bookmarks link owns
its editor page, and the Bookmarks tab uses the same rule.

The mobile menu's directory labels sat a few pixels left of the labels around
them: the indent was in em, and a Menu.Label is set smaller than a Menu.Item.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* bookmarks: parse URLs instead of matching a prefix; validate the whole tree on every write

The URL contract was checked by prefix on both sides, and a prefix is not a
form: `//evil.com` starts like a path and `:8096@evil.com` starts like a port,
and a browser takes both off this host.  A path is now a single leading slash
that resolves to this origin, a port form is an optional scheme, `:`, digits
and then only a path, query or fragment (no `@`, no backslash, no control
characters), and anything else must parse as http(s).  The same rules live in
wrolpi/bookmarks.py and BookmarksContext.js, with the same bypass strings in
both test suites.

The config's dataclass validator only saw that `bookmarks` was a list, so a
hand-edited file or the generic config API could store a `javascript:` URL
that the bookmarks API would refuse, and the nav rendered it.  The config now
validates the tree itself on every write and import: URL forms, exactly one of
url/children, unique integer ids, boolean new_tab, and no cycles (a YAML
anchor can build one).  An invalid file is not imported and, since it was never
successfully imported, is never overwritten.

On the page, a stored URL that does not resolve is rendered as text, never as
an href.  A refused add keeps what was typed instead of clearing the form, and
an edit whose move is refused still refreshes the nav (the rename was
committed) and keeps the modal open.  An IPv6 host is bracketed before a port.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* help: document Bookmarks

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 13:11:36 -06:00
lrnselfreliance
d701846254
Hotspot captive portal: welcome page with the WROLPi's address (#709)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Hotspot captive portal: welcome page with the WROLPi's address

Devices that join the hotspot probe a fixed plain-HTTP URL to check for
Internet access.  NetworkManager's shared-mode dnsmasq now resolves those
probe hostnames (and `wrolpi`) to the hotspot address via an include the
Controller writes when the hotspot starts, the probes land on the Controller
(port 80), and are redirected to /portal: "This is your WROLPi, its address is
10.42.0.1, open WROLPi / open the Controller".

Only the probe hostnames are hijacked so the hotspot keeps forwarding
everything else when the WROLPi has an uplink.  The include is corrected and
the connection re-activated once if NetworkManager picked a non-default
subnet.  Writing the include is best-effort; the hotspot starts regardless.

`hotspot.captive_portal` (controller.yaml, default true) toggles the probe
redirects and the include; like the other hotspot settings it applies on the
next hotspot start, so saving while the hotspot is up reports
restart_required and both Controller UIs say so.  The status endpoint gains
the hotspot address and portal URL, shown in both UIs.  repair.sh only
creates the include directory so a disabled setting survives upgrades.
Native only: Docker has no hotspot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: address review

* React: re-read the hotspot address whenever the HotspotToggle starts or
  stops the hotspot, matching the fallback UI's refresh-on-status.
* A WPA3 start that reverts to WPA2 leaves the hotspot up, so it now gets
  the captive portal address correction too.
* The corrective `nmcli connection up` logs a warning on a non-zero exit;
  dnsmasq would otherwise keep the old address silently.
* HotspotStatusResponse.captive_portal defaults to true like everywhere else.
* Trim design narrative from the module docstring and React comments.
* The include-before-activate test now observes the include at the moment
  of the `nmcli ... hotspot` call instead of inferring from call order.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: never forward any record type for probe hosts

`address=` only synthesizes A/AAAA answers.  iOS also asks for the HTTPS
record type; dnsmasq forwarded that upstream, Apple's answer aliased to
captive.g.aaplimg.com, and the phone fetched the genuine "Success" from
Apple instead of the portal.  Each probe host now also gets `local=`, so
dnsmasq answers every type itself, and the CDN alias is a probe host too.

Observed with dnsmasq query logging on a hotspot with an eth0 uplink.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: let acknowledged clients finish so the OS offers "Done"

A phone's sign-in sheet only shows "Done" once the OS re-probes and gets
the genuine success reply.  The Controller now remembers each client (by
hotspot IP) that has loaded the portal page and answers its later probes
with the reply each OS expects (Apple "Success" HTML, Android 204, Windows
"Microsoft Connect Test", ...).  The set is cleared when the hotspot
starts so a new session shows the page again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: acknowledge on Continue, not on view

iOS fetches the portal page in the background before opening the sheet,
then re-probes after the sheet's own navigation.  Acknowledging on view
made that re-probe return "Success" as the sheet's content.  Phones only
re-check the network (and offer "Done") after a full-page navigation, so
the welcome page now has a Continue button leading to /portal/continue,
which is the only place a client is acknowledged.  The Continue page says
to tap Done and links to WROLPi and the Controller.

Also hijack clients3.google.com (older Android probe host).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* help: bump wrolpi-help to the captive portal and Speed Test docs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: comments describe the Continue design; clear acks on stop

Three comments still described acknowledging on view, which c72a7f7b
removed.  Acknowledged clients are now also cleared when the hotspot
stops, so a lease reused in a later session sees the portal page.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:02 -06:00
lrnselfreliance
91f92683ba
Speed Test calculator: ping, download, and upload against the API (#708)
* Speed Test calculator: ping, download, and upload against the API

Users asking whether the hotspot is fast enough, or how many people can
stream video at once, had no objective tool that works offline.  This
adds one under Calculators > Tools & Reference.

Backend (wrolpi/speedtest.py, /api/speedtest):
* ping: tiny no-store JSON the browser times for round trip latency.
* download: streams slices of a pre-built random buffer for a clamped
  number of seconds (0.1-15).  Random so no proxy or compression can
  flatter the result; slices so no bytes are generated per request.
* upload: streaming-body handler that reads and discards the body and
  reports bytes and elapsed time.  Requests over 8 MiB are refused.
* info: the client address Caddy forwarded plus how many tests are in
  flight.  Sanic now trusts one proxy hop (PROXIES_COUNT = 1) so
  request.remote_addr is the browser, not Caddy.  Nothing else read
  that field before.

There is deliberately no one-test-at-a-time lock.  Several people
testing at once is how a user learns how the link shares between
streamers, so the server only counts in-flight tests (a shared_ctx
Value created pre-fork so every worker sees the same number) and the
UI reports how many others ran alongside.

Both Caddyfiles route /api/speedtest/* unbuffered, as the SSE chat
route already is, so throughput is measured live rather than in bursts.

Frontend:
* speedtest.js holds the logic with no React in it (bucketed
  throughput, steady-state average that drops the first second, ping
  stats, stream-capacity tiers, hotspot address check) so Jest drives
  it with fake fetch and streams.
* SpeedTestCalculator runs 20 pings, a 10 s download, and a 10 s upload
  with a live figure and rolling trace, then shows Mbps and MB/s, ping
  verdict, the client address with a hotspot badge, and a table of how
  many 480p..4K streams the download speed supports.  Upload is
  sequential POSTs of one 4 MiB Blob because Safari cannot stream a
  request body.  Nothing is saved.
* Sparkline is a new inline-SVG ui component (no chart dependency),
  added to /theme-sample and checked in all four themes.  A missing
  sample is a gap, not a plunge to zero.

The page states that it measures the network path, not the drive that
actually serves video files.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* speedtest: bound uploads in time, close the Start race, address review

Review on #708 found three bugs and some polish:

* Start could be clicked again while the first fetchInfo() was still
  pending, because the phase only left `idle` after that await.  A
  second click made a second AbortController and a second concurrent
  run, and Cancel only reached the later one.  start() now refuses when
  a controller already exists and enters a `starting` phase
  synchronously, so the button is Cancel before the first await.

* The upload phase could overrun the advertised 10 s by however long
  the in-flight 4 MiB POST took: on a slow hotspot, the very link this
  tool exists for, that is tens of seconds.  runUpload() now aborts the
  in-flight request at the phase deadline (counting only completed
  requests) and uses 1 MiB bodies so a few-Mbps link still completes
  several inside the phase.

* The server upload sink was capped by size only, so a slow or paused
  body could hold a worker until 8 MiB eventually arrived.  It now also
  enforces MAX_UPLOAD_SECONDS (15 s, matching download) and answers
  408 when exceeded.  Both caps together are what "cannot hold a
  worker" actually requires.

Also: a NaN `seconds` now gets a NaN message instead of "must not be
negative"; the module, Sparkline, and gallery docstrings are trimmed
to purpose; and the upload counter is covered for the mid-stream 413,
the 408, and a broken body.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 18:47:18 -06:00
lrnselfreliance
f1270b4e6c
archive: raise archive service request limit to 500MB (#692)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
Uploaded singlefiles are base64-encoded into JSON before being sent to the
archive container, so an 88MB upload becomes a ~118MB request.  Sanic's default
REQUEST_MAX_SIZE of 100MB rejected it with a 413 before the handler ran, and the
API surfaced this as KeyError: 'singlefile'.

Only the Docker deployment is affected; Debian/RPi extract in-process.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-01 20:09:35 -06:00
lrnselfreliance
ebdf9dcc4b
Map previews that never drew, and a Docker map with no world under it (#658)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* ui: a preview map that waited for a container it could never see

Both map preview modals -- the per-region eye button and "View All Regions"
-- opened to an empty box.  Each renders a div into a Modal and hands it to
MapLibre from an effect gated on `open`, reading the node out of a `useRef`:

    if (!open || !mapContainer.current) return;

Mantine's Modal mounts its body one commit after `opened` flips true, so on
the render that opens the modal the ref is still null.  The effect returns,
nothing it depends on changes again, and it never runs a second time.  No
console error, no rejected promise, no canvas -- just an empty div.

Semantic UI's Portal mounted the body in the same commit that opened it, so
the ref was attached by the time effects ran.  The effect bodies are
byte-identical across that migration; the modal library is the only variable,
confirmed against a device still running the older build where both previews
render.

A callback ref instead, so the effect re-runs when the node arrives and this
stops depending on any modal's mounting schedule.  The async source lookup
also gets a cancelled flag: it could resolve after the cleanup ran, building
a map nothing would ever remove and leaking a WebGL context per open.

The tests drive the real page and assert a map was built against a node that
is in the document -- the outcome, rather than the mechanism that quietly
changed underneath it.  Seven of the eight fail on the code before this.

* build: docker had no world underneath its map regions

Caddy serves /blobs/map-overview.pmtiles out of /opt/wrolpi-blobs, and that
blob is the world at zoom 0-6 -- what a regional extract sits on top of.
Without it a user who subscribes to a few regions gets those regions floating
on an empty background everywhere their data does not reach.  Only a full
planet file, at 134GB, makes it unnecessary.

Raspberry Pi and Debian installs get the blob from pi-gen, the live-build
config, and upgrade.sh.  Docker got nothing: the compose file bind-mounted a
git-ignored host directory over /opt/wrolpi-blobs, and a bind mount replaces
the image's directory rather than merging with it, so the path existed and was
empty on every Docker WROLPi.  Verified against a running deployment, where
that URL returned 404 while the same URL on hardware returned 206.

So the web image fetches the blob at build time, as the other three builders
do, and the mount that would hide it is gone.  `curl -f` so an HTTP error
fails the layer -- an image that builds without the blob is this same bug,
except silent at build time too.

The tests read the deployment files rather than a built container: building
to check costs a CDN download and several minutes, and both things that broke
-- nothing fetches it, a mount hides it -- are visible in the source.

* ui: cancel the load handler too, and make the cancel test falsifiable

Review follow-ups on the preview modals.

The `cancelled` flag was checked before constructing the map but not inside
the `load` handler it installs, so cleanup landing between the two left that
body drawing into a map `remove()` had already torn down.  Guarded, mirroring
the `destroyed` pattern in MapViewer.js.  The source promise also gets a
`.catch`: the helper cannot reject today, but nothing says it never will, and
an unhandled rejection here would be as quiet as the bug this all started as.

Two test corrections, both cases of an assertion claiming more than it read:

The region-drawing test spied on `addSource` AFTER construction and then
re-fired `load` by hand, so its count came entirely from that manual call and
said nothing about the production path.  FakeMap now records the sources it is
given, and the assertion reads what the component's own `load` drew -- each
region once, in order, and none for the entry with no bbox.

The new cancel test passed with the guard deleted, which is worse than not
having it: Mantine had not mounted the modal body when it closed, so the
effect had never run and there was nothing to cancel.  It now waits for the
effect to reach the source lookup before closing, and ships beside a control
that releases the same held promise WITHOUT closing and requires a map.  The
control is what makes the zero meaningful; verified that removing the guard
fails the cancel test while the control still passes.
2026-08-11 20:37:23 -06:00
lrnselfreliance
616424b223
fix: repair CI after the dependency bumps, and align it with the Trixie build (#651)
* fix(app): depend on @testing-library/dom directly

jest-dom 7 moved @testing-library/dom from a bundled dependency to a peer
dependency. Nothing else pulls the top-level copy in — @testing-library/react
13 carries its own nested v8 — so it was present only as a peer, and CI
installs with --legacy-peer-deps, which skips peers. Every suite then failed
to load setupTests.js with "Cannot find module '@testing-library/dom'".

Declare it as a devDependency so it installs under either resolution mode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(app): import mount from cypress/react

Cypress 14 removed the cypress/react18 subpath; cypress/react mounts React
18 itself. The e2e support file loads commands.js, so the stale import broke
bundling for every spec, not just component tests:

  Module not found: Package path ./react18 is not exported from package cypress

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* ci: install and cache Cypress the way the bump requires

The Cypress binary lives in ~/.cache/Cypress, outside node_modules, but only
node_modules was cached. A restored cache satisfied npm, so npm skipped the
postinstall download and the 15.x binary was never fetched:

  The cypress npm package is installed, but the Cypress binary is missing.

Cache both paths and install the binary explicitly.

Also align the JS toolchain with what actually ships. Debian 13 devices run
nodejs 20.19.2 and install with a plain `npm install`; the Docker app image
does the same. CI pinned older Node and passed --legacy-peer-deps, which
skips peer dependencies -- that is why jest-dom 7's peer requirement passed
review and broke CI. Match the devices instead.

Pin caddy to 2.11.4, the version Debian 13 devices run, rather than latest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* ci: test on Python 3.13, the version that ships

Debian 13 devices run Python 3.13.5 and the api/controller/help images are
python:3.13-trixie, but CI ran 3.11 (and one job on 3.12), so the runtime
under test was two minors behind the runtime in production.

Verified on 3.13 before switching: backend 1712 passed, 5 skipped; controller
883 passed. The only new warning is SQLAlchemy 1.3.22 reaching Python's
deprecated datetime.utcfromtimestamp(), which is noise rather than a break.

Drop api-tests-3-12: no workflow referenced it, it duplicated api-tests, and
its "Sanic does not yet support 3.12" note is stale now that devices run 3.13.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 18:01:52 -06:00
Roland
19f4958fb5 zim: ship a default fallback Zim and honor the configured Zim directory
Two related gaps in how WROLPi serves Kiwix content:

1. A fresh WROLPi (or one whose zims are all corrupt/absent) showed a blank
   Kiwix.  Every image now ships a small built-in default Zim that Kiwix falls
   back to, telling the user to download real Zim files.

2. The startup scripts hardcoded the `/zims` directory, ignoring wrolpi.yaml's
   `zims_destination` (which the Python side already honors).

Changes:
- New wrolpi/scripts/read_config_value.sh: a zero-dependency (awk) reader so
  non-Python scripts can reliably read top-level wrolpi.yaml scalars.  Also a
  config_value() wrapper in wrolpi/scripts/lib.sh.
- New wrolpi/scripts/default_zim/: HTML source + a reproducible builder
  (ghcr.io/openzim/zim-tools) that outputs the committed wrolpi/blobs/default.zim.
- New wrolpi/blobs/: any committed blob here is copied to /opt/wrolpi-blobs by
  the pi-gen and debian-live builds; the Docker zim image bakes it in.
- start_kiwix_serve.sh honors zims_destination and falls back to the default
  Zim when no functioning zim is found.
- docker/zim/entrypoint.sh gains the same default-Zim fallback.  It needs no
  config read: it already scans the whole mounted media dir, and
  zims_destination is always a subdirectory of that mount.
- Changing zims_destination via the settings API now restarts Kiwix so it
  serves the new directory without a reboot.

Tests: new test_read_config_value.py, extended test_start_kiwix_serve.py
(configured dir + default-Zim fallback), and a root_api test for the restart.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 19:09:57 -06:00
Roland
80cc758e6c
zim: drop --address 0.0.0.0 in the Docker entrypoint so kiwix-serve works on trixie
The zim Docker image is FROM debian:13, whose kiwix-tools 3.7.0 rejects the
literal "--address 0.0.0.0" ("IP address is not available on this system"),
so the container failed to start after loading the library. Omit --address;
kiwix-serve then defaults to binding all interfaces.

This mirrors 9e92732a, which applied the same fix to the systemd
start_kiwix_serve.sh path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 13:21:08 -06:00
Roland
698e61be34 Help: bump wrolpi-help submodule for SQLite docs.
Update the vendored help docs after documenting the SQLite database and
removing outdated PostgreSQL references.
2026-07-18 13:28:28 -06:00
Roland
e01795ad7e
Debian 13 (trixie) rebase: docker, debian-live-config, and pi-gen builds.
All three build targets move from Debian 12/bookworm to Debian 13/trixie
for fresh installs; existing Debian 12 systems remain supported in place.

Docker: api/help on python:3.13-trixie, controller on 3.13-slim-trixie,
app on node:20-trixie (matches trixie's nodejs 20), samba/zim on
debian:13.  Full backend suite passes on Python 3.13 in the api container.

debian-live-config: --distribution trixie; postgres teardown for the
SQLite cutover (pg_dropcluster hook block removed, wrolpi-bootstrap.sh no
longer creates a cluster or calls the deleted initialize_api_db.sh — the
API creates wrolpi.db itself on first start).  README documents the
SQLite layout and drops Etcher recommendations.

pi-gen: upstream has no trixie branch — the arm64 branch now defaults to
RELEASE=trixie; KEYBOARD_KEYMAP=en_US is rejected by trixie's
console-setup (ckbcomp validates), use 'us'; stop stamping a Postgres
.pgpass into /etc/skel.

Package lists (both targets): postgresql-15/postgresql-doc/libpq-dev
deleted; libtiff5-dev renamed to libtiff-dev; software-properties-common
and cpufrequtils removed from trixie (Controller CPU throttle falls back
to sysfs, dead cpufreq-set sudoers rule removed); locales and gnupg added
(no longer arrive transitively under --apt-recommends false); sqlite3 CLI
added for reset_tags.sh.

help.sh: replace the Postgres service/port/socket/cluster/psql checks
(all false FAILEDs after the SQLite cutover) with SQLite checks:
wrolpi.db exists, PRAGMA quick_check, file_group table + row count, run
as the wrolpi user so root never creates the -shm sidecar.

Both build scripts take a WROLPI_BUILD_DIR override so development builds
can run beside the release pipeline's build trees; the sudo re-exec
passes it through explicitly (a plain sudo would drop it).

test_common.py: the truncate tests hard-coded sys.getsizeof results from
the 3.11 object layout (str/list shrank in 3.12); assert the contract
(fits budget, prefix, monotonic in budget) so they pass on 3.11 and 3.13.

Verified: trixie ISO (3.7G) and pi-gen desktop image (4.3G) both built
from scratch and passed their sanity checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 15:43:08 -06:00
Roland
d9303a0ea8
Postgres -> SQLite hard cutover: the database moves into the media directory.
The database now lives at <media>/config/wrolpi.db so a library (files,
configs, AND database) travels with the drive between WROLPi systems.

Core:
- wrolpi/db.py: lazy SQLite engine factory (WAL, busy_timeout=30s,
  foreign_keys=ON); get_db_curs yields a plain sqlite3.Cursor; native
  sqlite3 paramstyle everywhere (no compatibility shims); global sqlite3
  adapters keep raw-SQL datetime formats identical to the ORM's.
- wrolpi/fts.py: FTS5 external-content search replacing tsvector/GIN.
  Injection-proof websearch->FTS5 translator, bm25(10,4,2,1) reproducing
  A/B/C/D weighting, fast(abc)/deep column filtering, snippet headlines.
  Stemming contract pinned by dedicated tests (watch/watches/watched/
  watching are interchangeable).
- wrolpi/schema_ddl.py: SQLite triggers shared verbatim by the alembic
  baseline and test DBs; fixes the channel video_count DELETE/UPDATE
  bugs and a latent total_size bug (it used the video id as channel id).
- wrolpi/db_bootstrap.py: startup auto-create/migrate; refuses databases
  written by a newer WROLPi (db_version_mismatch); SQLite 3.40 floor;
  network-filesystem refusal; never creates/migrates/trusts a database
  beneath an unmounted production media directory (shadow-DB guard).
- Alembic: 85 Postgres-era migrations squashed into one baseline;
  revision ids are dates (2026_07_10_1350) so filenames self-sort and
  alembic_version says when a schema was created; render_as_batch=True.
- All raw Postgres SQL rewritten (ANY->json_each IN, array_agg @> ->
  HAVING COUNT(DISTINCT), unnest->executemany/UPDATE..FROM,
  generate_series histograms->strftime GROUP BY, DISTINCT ON->
  row_number(), writable CTEs->two statements, jsonb_*->json_*, explicit
  NULLS FIRST/LAST, LIMIT-before-OFFSET).  Simple statements converted
  to SQLAlchemy instead (flag upsert, download retry/summary/tag checks,
  channel-by-tag estimate, Directory delete).
- Config imports count skipped entries; background dumps are blocked
  after a lossy import so partial imports can never overwrite YAML
  configs (they are the sole source of truth for rebuilds now).
- Search tolerates explicit JSON nulls for limit/offset (Postgres
  treated OFFSET NULL as 0; SQLite errors).
- file_group.id/doc_section.id render as INTEGER PRIMARY KEY on SQLite
  (BIGINT is not a rowid alias; FTS5 content_rowid requires it).

Ops teardown: docker-compose db service, CI postgres sidecars,
initialize_api_db.sh, controller postgresql service, psycopg2-binary all
removed.  upgrade.sh stops/disables postgresql (data is NOT deleted;
`systemctl enable --now postgresql` restores it); systemd units no
longer Want postgresql.service.

Existing-user upgrade: first boot creates+migrates the DB, configs
import (tags recreate their FileGroups from disk), a refresh rebuilds
the library index.  QA'd on the 10.0.0.8 RPi: rebuild matched the
Postgres baseline exactly; survived an uncontrolled hard reboot with
clean integrity checks.

Accepted behavior changes: FTS5 has no stopwords; bm25 length
normalization shifts some rank orders; fast-path d_headline is
un-highlighted; all-negative queries match nothing.

Deferred: Debian/trixie package lists (incl. sqlite3 CLI), Cypress
re-baseline, VACUUM INTO snapshots, exFAT/NTFS warning, auto-kicked
rebuild refresh with progress UX.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:09:37 -06:00
Roland
75cc0bbe6c
Help: update wrolpi-help submodule with doc screenshots.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 18:58:09 -06:00
Roland
ab6b0fae12
Help: update wrolpi-help submodule with new docs.
Adds Tags, Configs, FileGroups, Downloads, and Upgrade docs, the
WROLPi Portable section in Getting Started, and a synced TOC.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 18:35:00 -06:00
Roland
bf1ec42cc7
Bumping deno version for latest yt-dlp. 2026-06-24 15:12:27 -06:00
Roland
5f42b28c2d
Bump wrolpi-help submodule to latest (ship Inventory docs)
PR #494 re-pointed the submodule at 80648d3 to preserve the previously-pinned
snapshot, but that commit predates the Inventory documentation, so the Docker
help image had no Inventory pages.  Bump to 002e747 (current help master tip),
which includes the Inventory docs and the Restore section.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 17:03:40 -06:00
Roland
b29dc3e11f
Bump wrolpi-help submodule to re-signed history
The wrolpi-help history was rewritten to correct the author email
(rolobio@rolobio.com -> roland@learningselfreliance.com) and GPG-sign all
commits, which changed every SHA from the affected commits onward and orphaned
the previously-pinned 2c75fbb.  Re-point the submodule at 80648d3, the
content-identical (same tree), signed equivalent of the old pin, so
`git submodule update --init` and the Docker help build keep working.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 16:46:54 -06:00
lrnselfreliance
1acb148c90
Single file deno (#478)
* Singlefile runs under Deno 2.8.3; browser leftovers are reaped.

single-file-cli is designed for Deno.  Under node 18 (Debian 12/RPi OS) it
cannot reach the browser CDP on IPv6-enabled hosts ("localhost" resolves to
::1 first with no IPv4 fallback) and it never exits after a successful save,
blocking the download worker forever.

The native path and the Archive docker container now run single-file with
"deno run".  Deno is pinned at v2.8.3 everywhere because yt-dlp 2026.06.09
requires at least 2.3.0 (our node is Debian's v18, below yt-dlp's v22
minimum).  Deno >= 2.3 kills the browser single-file spawns (piped stderr
that single-file never reads), so the browser is run through
singlefile_browser_wrapper.sh which discards its stderr.

Commands are run in their own process group which is killed when they
finish, reaping the browser single-file leaves behind.  The Archive
downloader now has a ten minute timeout.  Install scripts install Deno
(previously only upgrade.sh and the image builds did).

* Singlefile writes to a file instead of dumping to stdout.

Deno >= 2.3 truncates large --dump-content output at exit; compressed
(SingleFileZ) singlefiles of large pages lost their ZIP tail, failed the
compression check in write_archive_files, and were destroyed by the HTML
prettifier (blank white archives).  single-file now writes its result to a
temporary file which WROLPi reads back.

As insurance, a singlefile marked compressed (<html data-sfz) whose ZIP is
unreadable is now stored verbatim instead of being prettified.

* Compressed Archive screenshots and readability images are no longer broken.

The index.html inside a compressed (SingleFileZ) singlefile references its
resources by relative ZIP paths.  Screenshots are now taken from the fully
extracted ZIP so those references resolve.  Readability images are inlined
as data URIs read from the ZIP (readability-extractor absolutizes relative
srcs against the page URL, so both forms are matched).

* Compressed Archives are created by converting the uncompressed capture.

The page is always captured uncompressed; readability, screenshot, and
title come from that canonical page (its images are inline data URIs, so
no ZIP extraction or image inlining is needed).  When compress_singlefile
is set, a second single-file run converts the captured file into a
compressed (SingleFileZ) self-extracting file.  The conversion is a local
file:// operation - the page is not fetched again, and single-file
preserves the original URL header when re-saving its own pages.  If the
conversion fails, the uncompressed singlefile is stored instead.

* Consolidating duplicate compressed-singlefile tests.

The write_archive_files tests (verbatim write, title fallback, broken-ZIP
guard, readability inlining) are combined into one test; the helper probes
and the setting-plumbing tests are each merged into one.  The do_singlefile
compression-failure scenario and the timeout constant are folded into the
main do_singlefile test, which shared their setup.

* Addressing review comments: removeprefix, PermissionError when killing.

* Bumping version to 0.24.1-beta.

* Install scripts assert the deno.zip hash to avoid malicious or invalid files.

The hashes are the official sha256sum assets of the Deno v2.8.3 release,
per architecture.

* Archive Dockerfile asserts the deno.zip hash to avoid malicious or invalid files.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:07:28 -06:00
lrnselfreliance
0a776d030f
Archives can be compressed (SingleFileZ self-extracting HTML) (#477)
* Archives can be compressed (SingleFileZ self-extracting HTML).

A new compress_singlefile download setting runs single-file with
--compress-content, creating a self-extracting HTML file which is also a
valid ZIP.  Only the singlefile .html is compressed; readability files are
always written uncompressed.  Readability, screenshots, titles, and URLs
are extracted from the index.html inside the ZIP.  A Compress checkbox in
the Archive and RSS download forms remembers the user's choice in local
storage.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 19:46:35 -06:00
Roland
c726654de9
Map search added to global search. Preview cards are displayed for all results in new Map tab.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Map viewer has simple search.

Map search databases can be downloaded from CDN, or built locally. Map searches can be built offline, but will not have enriched things like state names downloaded from Wikidata.
2026-04-13 12:17:08 -06:00
Roland
e0324ce0fc Migrating Controller from port 8087 to 80 so that it is more discoverable. Reformatting Controller dashboard to make header at the top. Reformatting navigation bar to handle the new links which prevented Admin from being visible in some screens. 2026-04-09 15:01:05 -06:00
Roland
3d58b25b04
Serving ca.crt from Caddy instead of React app. 2026-04-05 13:14:32 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
1e2e59ec2e
Adding docker support for samba shares.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-03-29 11:14:38 -06:00
Roland
d61b9d2ef9
Bumping version to include updated docs.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2026-03-22 19:49:53 -06:00
Roland
aefc7b1def
Add Docs module for searching and organizing PDFs and EPUBs
Create a new Docs module that consolidates PDF and eBook handling into a
unified system with subjects and authors support. This replaces the
separate PDF/eBook modelers with a single Doc model backed by an
alembic migration (rename ebook to doc).

Key changes:
- New modules/docs package with models, API, extractors, and lib
- Subject and author metadata extraction with case-insensitive
  normalization for cleaner PDF metadata
- Frontend Docs pages with AuthorsPage, SubjectsPage using
  CollectionTable, and doc card deletion
- Global search integration for subjects and authors
- Generic collection item count support
- Frontend file card metadata adjusted to match collection sort
- Bump yt-dlp version
2026-03-20 14:17:06 -06:00
Roland
da8a9bb923
Caddy must serve files inline on /media and download on /download. 2026-03-08 09:54:00 -06:00
Roland
8261df5bef
Redirecting to Controller when main react app is down. 2026-03-07 19:17:38 -07:00
Roland
fbb71e012d Replace nginx with Caddy for TLS termination and reverse proxying.
- Replace nginx with Caddy as the web server for both Docker and native
  (RPi/Debian) deployments. Caddy handles HTTPS termination, reverse
  proxying to all services (API, React app, Controller, Kiwix, Help, Map),
  and serves the media/download file browsers.
- Generate a Root CA and signed leaf certificates for HTTPS. The Root CA
  uses Name Constraints to restrict signing to local addresses only
  (.local, localhost, and RFC 1918 private IP ranges), mitigating MITM
  risk if the CA key is compromised.
- Add an HTTP landing page that guides users through one-time CA
  certificate installation on iOS, Android, macOS, Windows, Linux, and
  Firefox, with auto-redirect when the cert is already trusted.
- Add trailing-slash redirects for /controller, /docs, /media, and
  /download so bare paths don't fall through to the React catch-all.
- Add a Root CA download link on the Settings page under Configs.
- Update help.sh diagnostics for IPv6 listener detection and HTTP/2
  status codes.
- Remove all nginx config files and the https_proxy Docker container.
- Bump wrolpi-help submodule with certificate documentation.
2026-03-07 18:48:44 -07:00
Roland
9f395cd631
Bumping wrolpi-help to include new docs. 2026-02-23 17:30:24 -07:00
Roland
ca3589b555
Bumping WROLPi Help to include Collection docs. 2026-02-23 15:39:32 -07:00
Roland
0c059865e2
Updating download process to prevent rate limiting.
* Added encrypted cookies feature for use with yt-dlp. Decrypted cookies never touch the disk.
* Adding sleep between all downloads to prevent rapid requests.
* Channels are fully refreshed before downloading.
* Removing superseded video browser profile.
2026-02-21 14:08:00 -07:00
Roland
0e6dcd280d
Adding special handling for yaml files so they don't try to download.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-02-02 16:13:09 -07:00
Roland
35853f2a59
Enhancing trace logging by switching to dev api service. 2025-12-19 17:55:06 -07:00