Commit graph

2702 commits

Author SHA1 Message Date
lrnselfreliance
faf8e735d1
A map is exported as a PDF: a 1:24,000 quadrangle, in layers, on a sheet, which says where it is (#1006)
* The map can choose a 1:24,000 quadrangle to export (#971)

The first part of exporting a map as a PDF: a grid of 7.5-minute quadrangles is drawn over the
map, and a click chooses one.  The grid is arithmetic, so it needs no data.  Exporting itself is
not here yet; the button is disabled.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A quadrangle of the map is exported as a PDF with a layer for each map layer (#972)

* A quadrangle of the map is exported as a PDF with a layer for each map layer

The browser draws the quadrangle off the screen at the size of a 1:24,000 sheet, once for each
layer which is on, and the API puts the images in a PDF, each in a layer of its own which a
reader can show and hide.  A layer in which nothing is drawn is left out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The exported map's canvas keeps what was drawn, and an export's memory is bounded

- The drawing buffer is asked for where this MapLibre reads it (canvasContextAttributes); as an
  option of the map it was ignored, so the canvas could be read after it was cleared.
- Each layer's pixels are compressed in the PDF as the layer is added.  PyMuPDF kept them
  uncompressed until the PDF was written: about 90 MB a layer of a quadrangle, all at once.
- The pixels of all the layers together are limited, before any image is decoded.
- pymupdf is imported when a PDF is made, not when the API starts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where on the Earth it is (#973)

* An exported map says where on the Earth it is

The PDF's page has a geospatial viewport (ISO 32000-2): the page is the map in Web Mercator,
with the latitude and longitude of each corner.  The corners are the edges of the picture the
browser drew, read from the map which drew it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A map whose edge is the antimeridian can be exported

The map's arithmetic puts the east edge of the last quadrangle before the antimeridian a rounding error past
180 degrees, and the export was refused.  An edge within 1e-8 of a degree of 180 is at 180 (it is not wrapped
to -180); a map which is really across the antimeridian is still refused.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map is on a sheet: a margin, a grid, scale bars and notes (#974)

* An exported map is on a sheet: a margin, a grid, scale bars and notes

The map is put on a larger page with what a topographic map has around it: a title, the latitude and longitude
of its corners and of ticks between them, scale bars (kilometers, miles, feet), and notes of the projection, the
grid and where the map's data is from.  Over the map is a 1,000 meter UTM grid, each line named in the margin.

The grid and the margin are PDF layers of their own.  The PDF says where the map is, not where the page is.  The
picker has a box to export the map alone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The scale of a narrow map is written under its name, not over it

A quadrangle near a pole is two inches wide, which is not room for its name and its scale on one line.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where a compass points (#975)

* An exported map says where a compass points

The sheet has a diagram of the three norths (true, grid and magnetic) beside its scale bars, and a note of how
far magnetic north is from true north on the day the map was made, and how fast it moves.

The declination is from the World Magnetic Model 2025 (NOAA NCEI and the British Geological Survey; public
domain), whose coefficient file is shipped: nothing is downloaded.  The model is good until 2030; outside its
years nothing is said of a compass.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A corner of an exported map is not also a tick

The edges of the picture the browser draws are a hair outside the quadrangle's, so the quadrangle's own edges
were taken for ticks inside the map, and each was written over the coordinate of its corner.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A compass which passes due south has moved a little, not most of the way around

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A map can be exported at 400 DPI (#976)

The picker has a choice of 300 or 400 DPI.  A map is drawn less sharp than was asked for, rather than refused,
when it would be more pixels than the API makes a PDF of: a quadrangle near the equator, or one with many layers
on.  The picker says so while it draws.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported quadrangle goes by the name the USGS gives it (#977)

* An exported quadrangle goes by the name the USGS gives it

The picker says what the chosen quadrangle is called, and the sheet's title, the PDF's title and the file's
name have it.  Where the USGS has no name, which is most of the world, a quadrangle goes by where it is, as
before.

The names are an index of the 65,600 7.5 minute quadrangles of The National Map (U.S. Geological Survey; public
domain), which is shipped: nothing is downloaded.  In Alaska a USGS quadrangle is wider than 7.5 minutes, so a
quadrangle of the picker's grid there is "Part of" the one which has its middle.

The places the tests of the map export are of are changed for others.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An area half of which is the last quadrangle of the land is a part of it

The middle of such an area is on the quadrangle's east or north edge, with no quadrangle beyond it, and it was
given no name.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The PDF of a map is made by a Job, which the browser watches (#978)

* The PDF of a map is made by a Job, which the browser watches

A PDF of a quadrangle is a minute or two of work on a Raspberry Pi 4, and the request which waited for it was
cut off after a minute: only the smallest quadrangles could be exported there.

Now the browser describes the map, sends the image of each layer in a request of its own, and starts a Job; it
watches the Job, shows how far along it is, and fetches the PDF when the Job is done.  Cancel stops the Job.
The images are kept on disk until the PDF is made of them, and an export is removed when it is an hour old.

The credits on the sheet are no longer cut off: an attribution which names many sources is each of them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A cancelled map export stops, and is removed

- A cancelled Job is not over until the thread making the PDF has stopped, so the next Job does not start beside
  it; the work can also be stopped while the sheet is drawn and the PDF written.
- DELETE /api/map/export/<id> stops an export at any step and removes what was sent; the browser calls it when
  the person exporting cancels, while layers are sent as well as while the PDF is made.
- An export whose Job is waiting or running is not removed for being old; one whose Job was cancelled is removed
  at once.
- Cancel while the map is drawn stops the drawing between two layers.
- Exports are kept in a directory made for this user alone, which is refused when it is a link or is another
  user's; only what is an export there is removed.
- A request for the finished PDF which is lost is made again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An overlay's areas and lines go under the map's labels, and its own labels over them (#979)

An overlay was drawn whole over the basemap, so an area it filled covered the names of what is in it.  Labels are
now always on top: the areas, lines and pictures of every overlay are under the basemap's labels, and the text,
icons and points of every overlay are over them.  It is so in the map, the standalone viewer, and an exported PDF,
where an overlay with both is two layers ("Towers" and "Towers (labels)").

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An overlay's ground is over the ground of every basemap file (#980)

* An overlay's ground is over the ground of every basemap file

On a map of several basemap files, the ground of an overlay was put before the first label of the first file,
which is under the land, water and roads of every file after it: an overlay's areas and lines were hidden, and
in an exported PDF its layer was just above the background.  It goes where the basemap's labels begin: the first
of the labels which are over all of the basemap's ground.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The labels of every basemap file are over the ground of every file

A map of several basemap files was their layers one file after another, so only the labels of the last file were
on top: the labels of the others were under the land, water and roads of the files after them, and under the
ground of an overlay.  Both viewers now draw the ground of every file and then the labels of every file.

The contour lines are drawn after the basemap's labels and are not of the basemap: an overlay's ground goes under
the basemap's labels, not under the contours instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* Contours are drawn from as deep as the terrain file goes (#981)

The map read a terrain file to zoom 8, which is how deep the terrain WROLPi offers goes, however deep the file
was.  It now reads a file as deep as the file says it goes, in both viewers, and draws finer contour lines from a
deeper file: every 40 feet (or 20 meters) from zoom 11, as a 1:24,000 sheet has.  A file which goes to zoom 8 is
drawn as it was.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map names its small roads (#982)

* An exported map names its small roads

A quadrangle is drawn at about zoom 13.  The basemap's style names small roads
from zoom 15, and its tiles have no names for them below zoom 14, so the streets
of a town were printed as lines with no names.

Before the map is drawn, the tiles of zoom 14 which the quadrangle is in are
read, and the roads in them which have a name are given to the export map as a
source of its own.  The layer which names small roads draws from that source, as
its style says to.

At the scale of a sheet most streets are shorter than their names, and a map
writes a name along a line only where the line is longer.  Such a road is named
over its middle, turned as the road goes; a road in pieces is named once.

A source whose tiles can not be read is exported as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The names of small roads are placed by the export, at every latitude

A quadrangle nearer the equator is drawn at a little over zoom 14, where the
names were not read and the style still kept them for zoom 15: such a map had
no names of small roads.  They are now read wherever a layer would not name
roads.

Every name of a small road is now a point: over the middle of its road, turned
as the road goes under it, again every 400 pixels of a long road.  Before, a
road long enough was left for the map to name along its line, which the map
often does not (the line bends, or is in pieces), and a short piece beside it
was named too.  One place decides where a road is named, so a road in pieces is
named once, and a piece of another kind of road does not take the name.

Reading the names can be cancelled.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet has a bar of meters, its norths in mils, and how far apart its contour lines are (#983)

Under the map are bars of kilometers, meters, miles and feet.

How far the grid's north and a compass's north are from true north is written in
mils as well as in degrees, in the diagram and in the notes: a compass may be
marked in either.

A map which is exported with its contour lines says how far apart they are
(CONTOUR INTERVAL 40 FEET), under the bars.  The browser knows: it is the
interval of the tiles the lines are made for at the zoom of the sheet, in the
unit the map measures in.  It says so with the sheet, and what is not an
interval is not written.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet says what the quadrangles around it are called (#984)

* A sheet says what the quadrangles around it are called

Under the scale bars is a diagram of nine squares, the map the middle one, and
beside it what the USGS calls each of the eight around it: the sheets to print
to carry on off the edge of this one.

The names are from the index of quadrangles which names the sheet itself.  A
quadrangle with no name (out to sea, or over a border) has no number; a map of
where the USGS names nothing, and a map too narrow for the diagram, go without.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet is never beside itself

Where a quadrangle of the USGS is wider than the map (Alaska), the area the
size of the map to its east or west is often more of the quadrangle the map is
in, and the diagram named the sheet as its own neighbor.  The quadrangle on a
side is now the next one which is not the map's own.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet has a legend of the lines of its map (#985)

* A sheet has a legend of the lines of its map

Under the scale bars is what each kind of line of the map is: highway, major
road, minor road, path, railway, boundary, contour and index contour, each a
piece of the line as the map draws it, on the color of the map's land.

The map's style says how a line is drawn at each zoom, and a sheet is drawn at
one.  The browser, which has the style, says what each line's color, width,
dashes and edges are at that zoom; the API draws them.  A line is in the legend
when its layer is exported and draws something at that zoom.

What the browser sends is made into lines before it is kept: what is not a line
is left out, and the map is exported.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A legend has no line too thin to see, and no name longer than its room

A style brings a line in from nothing as the map is zoomed in: a path is a few
hundredths of a point wide on a quadrangle near the equator.  It was in the
legend, drawn several times wider than on the map.  A line under a tenth of a
point is now left out, by the browser and by the API.

What a line is called is cut to the room beside its line, as the name of an
adjoining quadrangle is.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* Fixes to the map PDF export: wrong neighbors, leftover exports, exports which outlive the map (#989)

* A sheet names what is around it, and its edges, by where they are and not a hair from there

The edges the browser gives of a map are a hair outside its quadrangle.  The quadrangles around a map whose
middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of
them was the map itself.  An edge a hair south of the equator was 0 degrees south, a map which ended where the
UTM zones do had no grid, and a sheet dated in the last year there is could not be made.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export is removed when it is over, and its work stops when the API does

The images and the PDF of an export were kept until the next export began, however long that was.  The browser
now has the export removed however it ends, and old exports are removed when a PDF is made too.  The work of a
Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it.  What
is said of an export, and each image, is written whole.  Text which is no text is taken out when an export
begins, not found by the Job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export stops with its map, and waits for a page which can not be seen

Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did
closing the picker, which is now left open.  A PDF which arrives after the export was stopped is not saved.
The time a page is behind another tab is not counted against a map which is drawing.  The name of a road is
one line.  A photograph which ends a little inside the map is not made a JPEG with a black strip.  An accent
in the name of a quadrangle is not a gap in the name of its file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A PDF is not kept from who made it by a server slow to remove the export

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its water over its parks (#990)

* A sheet names what is around it, and its edges, by where they are and not a hair from there

The edges the browser gives of a map are a hair outside its quadrangle.  The quadrangles around a map whose
middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of
them was the map itself.  An edge a hair south of the equator was 0 degrees south, a map which ended where the
UTM zones do had no grid, and a sheet dated in the last year there is could not be made.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export is removed when it is over, and its work stops when the API does

The images and the PDF of an export were kept until the next export began, however long that was.  The browser
now has the export removed however it ends, and old exports are removed when a PDF is made too.  The work of a
Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it.  What
is said of an export, and each image, is written whole.  Text which is no text is taken out when an export
begins, not found by the Job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export stops with its map, and waits for a page which can not be seen

Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did
closing the picker, which is now left open.  A PDF which arrives after the export was stopped is not saved.
The time a page is behind another tab is not counted against a map which is drawing.  The name of a road is
one line.  A photograph which ends a little inside the map is not made a JPEG with a black strip.  An accent
in the name of a quadrangle is not a gap in the name of its file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its water over its parks

A layer of the PDF was one image, drawn where the topmost of its layers is.  The parks of Land Cover are under
the water on the map and its piers are over it, so all of Land Cover was drawn over the water: a lake in a
park, or among houses, was not on the PDF.  A second basemap file put its water over the first's buildings.

A layer is now as many images as it must be, each where it is on the map, in one layer of the PDF: a part
which fills an area stays where it is, and a part which is only lines is drawn with the next.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A PDF is not kept from who made it by a server slow to remove the export

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A map file which is of somewhere else is not drawn when a map is exported

Each basemap file is several images of an exported map, and most files of a library are of other places: each
was an image of nothing, counted against how many images an export may be and how sharp it may be.  A file
whose own edges are outside the quadrangle is now left out before the layers are counted.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its streams (#991)

A basemap draws streams from zoom 14, and service roads and taxiways from 13.  A quadrangle is drawn a little
short of that (zoom 13.8 at 40 degrees, 13.2 at 61), from tiles which have them: none were on the PDF.  What
is drawn from a zoom no deeper than a quadrangle's detail is drawn.  Names are left as they are.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An export which is cancelled stops at once, in the middle of a request (#993)

* An export which is cancelled stops at once, in the middle of a request

An export was stopped between two requests.  The image of a layer is tens of megabytes, which is a long request
on a slow connection, and a request to a server which has stopped answering never ends: the export said it was
stopping until then.  Cancel, and leaving the map, now end the request.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The requests an export makes after its layers are sent are seen to be stopped too

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A map is exported as it is printed: on white, in dark lines (#996)

* A map is exported as it is printed: on white, in dark lines

The light and dark maps are made for a screen.  On paper their roads are white on pale gray and their streams
half a pixel of pale blue.  The printed map is a third look, of the map's own: white ground, dark road edges
and names, blue streams wide enough to see, paths drawn, brown contours.

It is what a map is exported in unless Light or Dark is chosen where the quadrangle is, and the choice is
remembered in the browser.  While a quadrangle is chosen the map is shown in the look it will be exported in.
The light and dark maps are as they were.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Changing the look of the map does not move it

What is done when a new style is in place was asked for after the style was set.  A style which only differs
from the one the map has is in place before that, so it was done at the next change of style instead, with
where the map had been at this one: the map went back there when a quadrangle was chosen, and again when it was
not.  It is asked for first, and the map is not moved at all.  An export no longer waits for the map to be
still, which it need not and which nothing ended.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The colors of the printed map are chosen where its layers are (#997)

* The colors of the printed map are chosen where its layers are

While a quadrangle is chosen and the map is shown as it is printed, each layer of the layer control has its
color beside it, to be changed: one for most, three for the land (what grows, what was built, sand and ice)
and two for names (the name, and what is around it).  A basemap has several colors where one is chosen, which
are shades of it.  The edge of a road stays dark.  What is chosen is remembered in the browser, and can be put
back.  The map on the screen is as it was.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A ramp on a bridge is the color of a major road, as it is off one

A basemap draws a ramp on a bridge in the color of a minor road on a bridge: on the printed map it was white
where the rest of the ramp is yellow, and followed the color chosen for minor roads.  What puts the colors
back is a button, which a keyboard reaches.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A layer with nothing in it does not make an exported map less sharp (#1001)

* A layer with nothing in it does not make an exported map less sharp

How sharp a map may be depends on how many images it is, and every group of layers was counted: a basemap
file with no runway, highway or boundary in the quadrangle was images of nothing, drawn, sent, and counted
against the rest.  A map of 23 images asked for at 400 DPI was made at 330.  The groups are now looked at
first, on a small map, and only those with something in them are drawn, as sharp as that many may be.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An image which arrives after its export was removed is not kept, and one mil is a mil

Stopping an export while an image was on its way left the image behind it, in a directory nothing would
remove for an hour.  An angle of one mil was "1 mils" on the sheet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The map which is exported from is made as sharp as it is to be (#1002)

* The map which is exported from is made as sharp as it is to be

The groups of a map are looked at on a small map, which was then made larger to draw them.  A browser lets
a canvas be larger when it is made than it lets one be made larger afterwards: a quadrangle asked for at
400 DPI was 33 million pixels, 288 DPI at 40 degrees, less than before the groups were looked at first.  The
small map is now removed, and the map which is exported from is made in its place.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A canvas the browser stopped drawing on is not a layer with nothing in it

A canvas whose drawing was taken away by the browser reads as clear, and a group looked at then was left out
of the PDF without a word: the export fails instead.  An export stopped as the last group is looked at does
not make the map it would have been drawn on.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where it is in a projection Adobe Acrobat knows (#1003)

* An exported map says where it is in a projection Adobe Acrobat knows

The page was said to be in Web Mercator, which it is drawn in.  Acrobat does not know Web Mercator, in the
words of the standard or in Esri's: it took the page for a map, and put every place on it at 0, 0.  The page
is now said to be Mercator on the WGS 84 ellipsoid, which it reads, with the same corners.  Between the corners
the two differ by under three centimeters on the ground.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* How well a page is said to be where it is, is measured from what the PDF says

The test of it measured two formulas of its own.  It now reads the corners and the projection from the PDF and
does a reader's sum with them.  The few centimeters are of a quadrangle: a taller page is less well said, by
the square of how tall it is, which the comment now says and a test measures.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The help says how a map is exported as a PDF (#1004)

The help of the map is five pages, one of them the PDF export.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 17:22:31 -07:00
lrnselfreliance
d157057d81
Help: Sync (a secondary WROLPi) and the backup drive's trash (#1000)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Dead Code / vulture (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
The final part of replacing the sync: the help documents it.  A new Sync page; Backup rewritten around the backup
drive and its delayed delete (it taught a hand-made rsync job over SSH); sync.yaml and sync_state.yaml in How
Configs Work.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:16:38 -07:00
lrnselfreliance
a5ab01198b
The backup drive keeps deleted files for 30 days, then removes them (#999)
* The backup drive keeps deleted files for 30 days, then removes them

The fifth part of replacing the sync.  A backup drive never deleted anything, so it grew forever and kept every file
the media directory had ever held.  Now, like a secondary:

- A file deleted from the media directory, or the old version of one that changed, is moved to a dated folder in
  .backup-trash on the backup drive.  Folders older than backup_delete_after_days (sync.yaml; 30 by default, also
  for drives set up before this; none never deletes) are removed, before and after each backup, and the oldest go
  sooner when the drive is nearly full.
- A backup refuses a media directory that is not mounted, or has no config/wrolpi.yaml: copied with deletes, an
  empty folder would move the whole backup drive to the trash.
- The Backup Drive panel has the same "Delete after N days / Never delete them" control as the Sync panel, now one
  component, and refuses an empty number of days.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A nearly full drive keeps the trash it just filled, and makes room for the database

Review found, in the backup drive's trash and in the secondary's sync, which share its shape:

- Below 1 GiB free, the purge after a copy removed the newest trash folder too.  Moving a file to the trash is a
  rename, which frees nothing, so on a nearly full drive a deleted file was gone at the end of the same run instead
  of 30 days later.  The newest folder is now only removed by age (0 days still deletes at once).
- The purge made room only down to 1 GiB, so a database larger than the free space failed to be copied every time
  while the trash was full.  Before the database is written (the backup's snapshot, the secondary's copy of its
  primary's), the trash now makes room for the whole database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:50:21 -07:00
lrnselfreliance
38885990f6
The old sync engine is removed (#998)
* The old sync engine is removed

Part 4 of replacing the sync.  Since the previous part a secondary copies its primary with rsync, run by the
Controller, so nothing used the HTTP engine any more:

- the manifest (/api/sync/manifest), the download of a FileGroup's files over HTTP, the copy and import of each
  synced config, the moving of a replaced file's links, the request, cancel and tick of the old sync;
- the API lock that answered 503 SYNCING while that engine ran, its Event, the sync_running flag, the old status
  record in /api/status and its schema, and the frontend's handling of 503 SYNCING;
- their tests.

What a primary says about itself (/api/sync/info), the secondary's settings (its primary, the folders it does not
copy, Test Connection, the folder chooser), taking the primary's library settings when the API starts, and the
daily window of the backup drive stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The primary's sync info no longer reports the largest FileGroup id

Only the old engine read it.  Review also found a docstring that still spoke of a manifest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync is announced again

Since the Controller runs a sync, nothing sent its events: the old engine did, and the Controller cannot send the
API's events (the API is stopped for most of a sync).  The API's loop now reads the Controller's record of the last
sync and sends one event per finished sync: "Synced N files from the primary" (with the files moved to the trash,
or a warning) or "Sync from the primary failed".  As before, a sync that changed nothing is not announced (within
the window that is every 15 minutes), nor one that was stopped on purpose.  What was announced is kept in
config/.sync, which a sync never touches, so the restart every sync makes neither repeats nor loses one.

Found by review, and by the dead-code check, which saw the two event methods unused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:37:44 -07:00
lrnselfreliance
8cfe36b3ea
A secondary syncs by rsync: the Controller runs it on its schedule, and both pages follow it (#995)
* A secondary syncs by rsync: the Controller runs it on its schedule, and both pages follow it

The third part of replacing the sync.

- The Controller syncs within the daily window (every 15 minutes after the last sync finished, in WROLPi's
  timezone), as the API did.  The API's loop no longer runs the old engine, and its /api/sync/run and
  /api/sync/cancel ask the Controller.
- The Sync page and the overlay that covers the app read the sync's status from the Controller, which is the only
  one that can answer while the API is stopped.  The overlay is also shown when the settings cannot be read, as
  happens while the API is stopped.  Sync Now and Stop go to the Controller.
- The Controller's fallback page, where Caddy sends the browser while the API is down, shows the sync's progress
  and a Stop button.
- New on the Sync page: how long files the primary deletes are kept (30 days by default, or never), and a note that
  an excluded folder's files are still listed on a secondary.

The old engine's code is removed in the next part.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove SyncRunRequest, unused since a sync is asked of the Controller

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* In Docker, a sync's progress moves while a large file is copied

Docker's logs hold a line until it ends, and rsync's progress for one large file is a single line rewritten with
carriage returns, so the page showed 0% until the file was done (hours, for a large map).  The rsync helper's
output is now read by attaching to it, before it is started so that nothing is missed.

Found by a sync between two Docker WROLPis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync is recovered before the schedule can start another, and an empty days value is refused

Review found:

- The schedule and the recovery of an interrupted sync started side by side when the Controller started.  The state
  an interrupted sync leaves is due, so inside the window a new sync could start while the recovery restarted the
  API and wrote over the new sync's status.  Every tick of the schedule now recovers first, and only a status of
  "running" with no sync holding the lock counts as interrupted.  A sync records its outcome before it frees the
  lock, so a tick between the two cannot mistake it for an interrupted one.
- The Controller may start before the drive is mounted.  The empty record it read then was kept and saved over the
  real one, losing the last successful sync.  The state file is now read again until it can be.
- A cleared number of days was saved as 0, which empties the trash at the next sync.  It is now refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:06:25 -07:00
lrnselfreliance
6772762ce9
A secondary deletes its own stale write-ahead logs, and the sync leaves no stray files (#994)
* A secondary deletes its own stale write-ahead logs, and the sync leaves no stray files

Found by the first sync between two real machines:

- The primary's unfinished files (SQLite logs, partial downloads) were excluded from the copy, and an excluded path
  is never deleted on the receiving side.  So a secondary kept its own write-ahead log beside a map search
  database the primary had replaced, and SQLite would apply that log to the new database.  These patterns are now
  only hidden on the primary's side, so the secondary's stale ones are deleted (to the trash).
- Reading the incoming snapshot to check it made a -wal and -shm beside it, which stayed behind when the snapshot
  was moved into place.  It is now opened immutable.
- The primary's info, written by the Controller (root), belonged to root.  Files the Controller writes in the media
  directory now belong to its owner, like the directories it makes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* With deletes turned off, a secondary still drops its stale unfinished files

Review found that the hide rules of the previous commit only take effect through --delete, which is not given when
delete_after_days is null (never delete).  A secondary then kept its own write-ahead log beside a database the
primary had replaced, and SQLite applied it to the new database.

Without a trash, --delete is now given with every path protected except the unfinished patterns (and what is
inside an unfinished folder), so only those are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:19:09 -07:00
lrnselfreliance
ea7cc4dec2
The Controller can make a secondary a copy of its primary with rsync (#992)
Some checks are pending
Backend Tests / api-tests (core) (push) Waiting to run
Backend Tests / api-tests (files) (push) Waiting to run
Backend Tests / api-tests (media) (push) Waiting to run
Backend Tests / api-tests (rest) (push) Waiting to run
Backend Tests / api-contract-tests (push) Waiting to run
Backend Tests / api-migrations-and-start (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Cypress Tests / cypress-component-tests (push) Waiting to run
Cypress Tests / cypress-e2e-tests (push) Waiting to run
Dead Code / vulture (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* The Controller can make a secondary a copy of its primary with rsync

The second part of replacing the sync.  The Controller runs it, because the secondary's API is stopped while its
library is replaced: checks, a database snapshot on the primary, stop the API, rsync every file from the primary's
daemon (ignored directories included), fetch the snapshot and put it in the database's place, start the API, purge.

- Files the primary deleted or changed are moved to a dated folder in .sync-trash, and removed after
  delete_after_days (a new sync.yaml setting, 30 by default; none never deletes), or sooner when the drive is nearly
  full.
- A primary of another version or database migration, or with its media unmounted, is refused before anything is
  touched.  The database is replaced last, so a sync that stops earlier leaves one that works.
- On a Raspberry Pi and Debian rsync runs as the wrolpi user; in Docker in a helper container made from the api
  container.  A Controller that starts after an interrupted sync starts the API.
- When a secondary's API starts it takes the library settings of the primary's wrolpi.yaml, which the sync left.

Reached only through the Controller's /api/sync/run for now; the existing sync and its schedule are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The sync copies no link that leaves the library, and root follows none in the media directory

Security review of the Controller's sync:

- An exclusion that ends in a slash matches only a directory.  A file or link named config/ssl, tags or
  config/backup on the primary would have replaced this machine's directory of that name.  The exclusions now match
  whatever has the name.
- Links were copied as they are.  One that points outside the primary's library would, on the secondary, point at
  the secondary's own files, which the web server then serves.  Only links that stay inside the library are copied,
  and no device files.
- The Controller is root, in a media directory that belongs to another user.  It no longer writes, reads, replaces
  or purges through a symbolic link there.

Found on the way: the directory the Controller made for the incoming database belonged to root, so rsync, which
runs as the library's owner, could not have written into it.  It now belongs to the media directory's owner.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A sync ends when its primary stops answering, and makes room before it copies

Review found:

- rsync waits forever by default.  A primary that lost power or its link during a copy left the secondary's API
  stopped until someone cancelled.  rsync now gives up after 30 seconds without a connection and 5 minutes without
  data, which ends the run and starts the API.
- The trash was only purged after a run that succeeded.  The trash is what fills the drive, and a full drive fails
  every run, so it never got its turn.  It is now also purged before the copy.
- A value in sync.yaml that could not be used raised after the run lock was taken and before the handlers that
  release it, leaving "A sync is already running" until the Controller restarted.
- The playlists directory left out of the copy was this machine's, not the primary's, whose hard links would have
  arrived as full copies.  The primary's wrolpi.yaml is now fetched before the copy, and both are left out.
- A primary whose media directory is elsewhere is refused: its database names every file by its full path.
- In Docker rsync is its own container, which outlives a Controller that stops.  A Controller that starts after an
  interrupted sync now removes it before it starts the API.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 20:21:55 -07:00
lrnselfreliance
2ac4a3a294
A WROLPi serves its library over rsync, and makes a database snapshot on request (#988)
* A WROLPi serves its library over rsync, and makes a database snapshot on request

The first part of replacing the sync: a secondary will copy its primary's whole media directory with rsync and
replace its database with the primary's, instead of downloading the files of FileGroups over HTTP (which never
copies an ignored directory, such as the map's).

- An rsync daemon serves the media directory read-only on port 8873, on every WROLPi.  It never serves the TLS keys,
  the cookies, or the database in use.  On a Raspberry Pi and Debian it is the wrolpi-rsyncd unit; in Docker the
  Controller runs it in a helper container made from the api container (its image, user and volumes), so the media
  directory is still configured in one place.  All three read controller/rsyncd.conf.
- POST /api/sync/snapshot asks for a snapshot of the database, which the daemon serves from config/.sync.  An
  unchanged database is not written again.
- /api/sync/info also says the database's migration and whether the media directory is mounted.

Nothing uses these yet; the existing sync is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The rsync daemon follows no links, and no snapshot is made in WROL Mode

- A client that asked the daemon to follow symbolic links could read what a link in the library points at.  A
  relative link to config/ssl was read this way although that directory is left out of the module.  The daemon now
  refuses the three options that follow links; a link is sent as a link.
- A database snapshot is a new file on the primary's drive, so WROL Mode refuses it, as it does a backup.  A snapshot
  made earlier is still served.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* No rsync helper from an image without rsync, and the helper follows the api container

Review found:

- A helper made from an api image built before rsync was added exits at once, and Docker restarted it forever while
  the Controller reported it created.  The image is now looked at first; no helper is made from one without rsync,
  an older helper is left serving, and the reason is logged once.
- The helper was only checked when the Controller started, so a recreated or rebuilt api container left it stale.
  It is now checked every minute.
- A snapshot cut off by its process being killed stayed marked as running and held back every later request.
- The snapshot's half-written temporary file was served by the daemon.
- The comment on the daemon's refused options claimed more than they give: a link that is part of the requested
  path is still followed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 16:47:02 -07:00
lrnselfreliance
bb7a66b0d6
The calculators have no Temperature calculator (#987)
The Celsius/Fahrenheit/Kelvin converter is removed from the Calculators
page and from the search suggestions.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 15:49:45 -07:00
lrnselfreliance
d11b72bbd1
A file the sync replaces keeps its Tags Directory and playlist links (#986)
* A file the sync replaces keeps its Tags Directory and playlist links

A secondary replaces a changed file with a new inode, so the file's hard links in the Tags Directory and the
playlists directory were left holding the old version.  Both directories repair a link that is still wanted,
but refuse to delete one with no other link.  When the primary also changed the file's tags, or reordered the
playlist, in the same sync, the old link was no longer wanted and could never be deleted.

After each page's downloads, every link to a replaced version in those two directories now becomes a link to
the new version.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A replaced file's links move before the new version takes its place

Review of the first version found three gaps:

- A playlist with its own directory keeps its links outside the playlists directory; those were not looked at.
- The links were repaired after a whole page of downloads, so a sync that stopped mid-page left the files it had
  already replaced with their old links, and a later sync saw nothing to do.
- Two library files that are hard links of each other share an inode, so the link of the one that did not change
  was pointed at the one that did.

The links of a page's files are now found before its downloads, in the Tags Directory, the playlists directory
and each playlist's own directory, and only by the name those give a link.  Each file's links are moved to the
new version just before it is renamed into place.  The temporary name is short, so a long file name still fits.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-17 17:06:22 -07:00
lrnselfreliance
dc219e0775
Only offer an update when the remote tip is signed by the release key (#970)
check_for_update reported update_available whenever origin/<branch> was ahead, but upgrade.sh refuses an
unsigned tip (exit 5).  On master, merges land before they are re-signed, so users saw the update icon and
then an upgrade error.  The remote tip is now checked with verify_wrolpi_commit.sh (pinned fingerprint, key
from the running checkout) before an update is offered.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:36:02 -07:00
dependabot[bot]
3e23022418
build(deps): bump node-forge from 1.4.0 to 1.4.3 in /app (#964)
Bumps [node-forge](https://github.com/digitalbazaar/forge) from 1.4.0 to 1.4.3.
- [Changelog](https://github.com/digitalbazaar/forge/blob/main/CHANGELOG.md)
- [Commits](https://github.com/digitalbazaar/forge/compare/v1.4.0...v1.4.3)

---
updated-dependencies:
- dependency-name: node-forge
  dependency-version: 1.4.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-14 11:18:16 -07:00
lrnselfreliance
0389e4659c
An overlay's layers which the basemap already draws start hidden (#969)
A file of features often carries a basemap's layers too (roads, water, buildings, place names).
Drawn over the basemap they double every road and label.  Those layers, by the names of the
Protomaps and OpenMapTiles schemas, are now off until they are checked; every other layer is on
as before, and a choice already made is kept.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 11:17:31 -07:00
lrnselfreliance
38832870c9
Help: map overlays, their styles and icons, and ignoring a map file (#968)
Bump the help docs to document overlays, the style which can sit beside a
map file, the map's own overlay icons, the Type and Visible columns of the
Map's Manage tab, and maps.yaml.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 11:09:20 -07:00
lrnselfreliance
b2b8e2075c
The map draws its own icons on an overlay (#967)
An overlay drawn with a style of its own names icons, which needed the sprite sheet that style was made for.  The map now ships icons for overlays, the Maki and Temaki sets (both CC0), and always draws those, in both viewers.

The sprite sheet is the translation: its index lists every icon under its own name, under each other name it answers to (modules/map/icons/aliases.json), and in both spellings of each.  A name with no icon is drawn as a plain dot.  The icons are signed distance fields, so a style's icon color and halo apply.

A pattern is not an icon: what it filled (a fill, a line or an extrusion) is drawn in a translucent color, where removing a pattern used to leave MapLibre's opaque black.

A sprite sheet beside a map file is no longer used, served or reported.  wrolpi/scripts/build_map_icons.py builds the sheet from the SVGs; the sheet is committed.
2026-11-13 17:59:21 -07:00
lrnselfreliance
177d0dba4b
The standalone map viewer draws overlays, as the map does (#966)
The standalone viewer (port 8084) drew basemaps only: a map file which is
not a basemap was skipped.  It now offers each overlay in its layer panel,
hidden until turned on, with a checkbox per layer, draws it with the style
beside it when there is one, and shows a feature's attributes on a click.

The overlay logic is the map viewer's own.  It moved to dependency-free
modules (app/src/map-shared) which a browser can load as they are; the
standalone viewer, which has no build step, loads copies of them from
modules/map/static/js.  A test fails while a copy differs from its module
(`npm run sync:map-shared` makes the copies).  What differs between the
viewers is passed in: where a map file and its style are fetched from, and
whether a layer can be checked before MapLibre is given it.

The standalone viewer cannot check a layer first, so it adds an overlay's
layers to the map one at a time once the style has loaded: MapLibre refuses a
whole style over one layer it cannot draw, but only that layer when it is
added by itself.

The API also names a style by its path within the map directory, which is
what the standalone viewer is served.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 17:28:41 -07:00
lrnselfreliance
113e70a402
Contour labels are drawn in a font the map has (#965)
The contour label layer named no font, so MapLibre asked for its default
("Open Sans Regular,Arial Unicode MS Regular"), which the map does not ship.
The request failed and the labels were drawn with glyphs the browser made up
locally.  The layer now names Noto Sans Regular, in the map viewer and in the
standalone viewer.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 15:53:01 -07:00
lrnselfreliance
4f58d2862f
A style beside a PMTiles overlay says how the map draws it (#963)
When a MapLibre style (and optionally a sprite sheet) sits beside a PMTiles file, the overlay is drawn with it instead of the generic colors: `<name>.style.json` and `<name>.sprite.json/.png`, or `style.json` and `sprite.json/.png` shared by the files of a directory.

Only the style's layers which draw that file are used; its sources, sprite and glyph URLs never are.  Fonts are replaced by the ones the map has, images are pointed at the overlay's own sprite (as strings, expressions or property functions), and each layer is validated so one the map could not draw is skipped instead of stopping the map.

The sprite is served by the API, which falls back to the single resolution sheet when there is no double resolution one.
2026-11-13 15:41:43 -07:00
lrnselfreliance
adb0adbd94
The map draws any PMTiles file as an overlay, in any sub-directory (#962)
Any PMTiles file under the map directory, at any depth, can be shown.  The API reads each file's header and metadata and says what it is (basemap, terrain, overlay, unreadable).  Overlays are offered in the layer control under the name the file gives itself, hidden until turned on, with a checkbox per vector layer, and are drawn generically.

Map files are named by their path within the map directory everywhere, and unfinished downloads are no longer listed.  A file can be ignored from the Manage page; ignored files are kept in a new maps.yaml config.

PMTiles metadata is not decompressed past its size limit, and the standalone viewer encodes map file paths.
2026-11-13 15:11:01 -07:00
lrnselfreliance
4072ac30ed
Docs sort by their title, ignoring case (#961)
The title sort ordered by `a_text` (the indexed words) with binary collation:
uppercase titles sorted before all lowercase ones ("AWLS" before "Alps"), and
Docs with no indexed words sorted last though they show a title.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:59 -07:00
lrnselfreliance
30b2bf6df0
Any Tag alone searches the library, and Zim search does not offer it (#960)
On the search page, choosing Any Tag without a search string left the
Dashboard showing: the page, the Files results and the Files count all waited
for a search string or tag names.

Zim search has no "any tag" search, and its tag picker's Any button left no
filter applied, so the picker no longer offers it.  Zim and Other counts are
only requested when there is a search string or tag names to count by.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:44:14 -07:00
lrnselfreliance
f1d344d8a2
Search results sort as the filter dialog says (#959)
* Search results sort as the filter dialog says

Docs sorted ascending orders backwards and fell back to id order for every
descending order (`-size`, `-published_datetime`); it now takes the same `-`
prefixed orders as the other searches and rejects unknown ones with a 400.

Archives and Docs searches without an order now put the most relevant results
first, as Videos does and as the dialog shows.

Rank has no direction in the dialog: choosing it sends `rank` (most relevant
first) instead of inheriting a descending `-rank` (least relevant first).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An old -rank link becomes rank when the filter dialog closes

The dialog shows Rank without a direction, but it seeded its draft from the URL
unchanged, so Done left a saved `-rank` (least relevant first) in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:42:23 -07:00
lrnselfreliance
2800112f3e
The Any Tag filter works on the Videos, Archives and Docs searches (#958)
The filter modal on these pages offered "Any Tag", but only the Files search
sent it; the module searches had no `any_tag` field.  Docs also matched any of
the listed tag_names, while every other search requires all of them.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:21:06 -07:00
Roland
a0f30a0a60
Bump version to 0.31.0-beta 2026-11-11 13:15:42 -07:00
lrnselfreliance
6cd233dce6
Sync and backup: a secondary WROLPi and a backup drive copy the library (#957)
* A secondary WROLPi copies its primary's library over HTTP

A WROLPi with role "secondary" pulls from its primary: a manifest of every
FileGroup's files (/api/sync/manifest), then each missing or changed file
from the primary's Caddy /media/ (resumable, written to a hidden partial and
renamed with the primary's mtime), a refresh, then the library configs,
imported as the primary wrote them.  wrolpi.yaml contributes only its library
settings.  Files deleted on the primary are kept.

A secondary never saves the synced configs and never downloads.  Its API
answers 503 SYNCING while a sync runs, and every page warns that it is a
copy.  It syncs in a daily window or on "Sync now"; WROL Mode stops it.

On a Pi 4 (no AES instructions) the client uses TLS 1.2 ChaCha20, which
pulls ~110 MB/s from Caddy versus ~41 MB/s for rsync over ssh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync has its own /admin tab; a sync's reads from the primary are bounded

The sync settings and progress move from Settings (and the component
gallery, which is for library components) to /admin/sync.

A download never writes past the size in the primary's manifest, and configs,
listings, and manifest pages are capped.  A primary address may not carry a
user name or password (it is logged and shown).  While a sync runs, the API
answers only the exact requests that show its progress or stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync keeps the secondary's replaced configs in today's config backup

The first copy of the day is kept, like a save, so a config backup import
undoes a sync's config changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Docker's Caddy logs every request

`docker compose logs -f web` shows each request with its status, size and
duration, e.g. a secondary's /media/ downloads during a sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync reports its progress as it downloads, and refuses quietly

The status updated only when a page of 500 FileGroups finished, so a page of
large files looked stuck; it now updates every 2 seconds.  Requests refused
while syncing are answered by the middleware instead of raised, so the UI's
polling no longer logs an error each time.  The primary's version is stored
without its trailing newline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A WROLPi can back up to a second drive; sync settings live in sync.yaml

The Sync tab's Backup Drive copies the media directory to a second mounted
drive with rsync during the sync window (hourly), or on "Back Up Now".
Nothing on the backup drive is deleted.  The Tags and playlists directories
are left out (WROLPi rebuilds them from the configs), and the database is
written as an SQLite snapshot rather than copied as files, so the drive can be
mounted as the media directory and start.  A backup is written only while the
chosen drive (by UUID) is mounted where it was chosen, and is not the media or
system drive; rsync is stopped if the drive disappears.  Not in Docker, which
cannot reach other drives.

Role, primary, window and backup drive move from wrolpi.yaml to sync.yaml:
they belong to this machine, and are never copied from a primary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Sync tab shows the configured backup drive

getSettings copies each setting by name and did not list the backup drive,
so the Sync tab showed "None" while one was configured.  It also dropped
`version`.  A test now fails for any setting the API sends that getSettings
drops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A backup can go to a folder on its drive, and remembers its last success

The Backup Drive panel takes a folder on the chosen drive.  The drive is
checked through the folder's mount point: the folder is written only when the
drive with the saved UUID is mounted there, so a folder left on the system
drive by an unplugged drive, or on another drive mounted inside the chosen
one, is refused.  A missing folder is created once the drive is checked.

When the last backup succeeded is saved in sync.yaml, so it is known after a
restart.  "Stop Backup" is disabled while the database snapshot is written (it
runs to the end), and a finished backup clears the request to stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The backup drive has its own window

The sync window was shown in the Sync panel but also scheduled the backup
drive.  The Backup Drive panel now has its own window (sync.yaml
backup_window_start/end), saved with the drive; the sync window is shown,
and used, only for a secondary's sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync and the backup drive can each exclude folders

A secondary or a backup drive may be smaller than the primary's library.
Each Sync tab panel has its own excluded folders in sync.yaml, chosen in
a folder browser: a secondary browses its primary's folders (through
Caddy), the backup drive browses this WROLPi's media directory.  Excluded
folders are never copied; what was copied before is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The folder browser does not offer lost+found

Every ext4 drive has a root-only lost+found folder; a sync and a backup
never copy it, so it is not offered for exclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync never writes into config or tags, whatever their case

On a media drive that ignores case (exFAT, NTFS, macOS), a manifest path
like Config/wrolpi.db is the real config folder. The protected and
unlisted top-level folders are now compared case-insensitively in the
manifest, in resolve_destination, in excluded-folder validation, and in
both folder listings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A backup writes the database snapshot only when the database changed

Every backup rewrote the whole database snapshot (2.5 GB on a busy
WROLPi), even on the window's idle hourly runs. The snapshot is now
skipped when the live database (its size and mtime, and those of a
non-empty WAL) and the drive's copy are both unchanged since the last
snapshot. The record is in memory, so the first backup after a restart
writes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove the unused save_sync_config switch

Nothing activated it; SyncConfig saves directly.  The test server's
log_message override is called by http.server, so it goes in the
vulture baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:53:45 -07:00
lrnselfreliance
86f168d2c1
A script's result reads the journal when systemd has cleared its timestamps (#956)
The upgrade reloads systemd, which clears the run's exit timestamp, so a
successful upgrade looked like a script that never ran: finished false and
success null. The journal keeps systemd's unit success/failure record of the
run's end, so it now decides whether a run finished, and when.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:25:38 -07:00
lrnselfreliance
19ff59d5e0
A failed upgrade is shown on the upgrade page instead of redirecting home (#955)
* A failed upgrade is shown on the upgrade page instead of redirecting home

A failed upgrade restarts the services it stopped, so the upgrade page saw
them come back and redirected home as if the upgrade had worked.  The
Controller now reports how the last run of a script ended
(GET /api/scripts/{name}/result, from systemd's Result and exit status),
and the upgrade page reads it before redirecting.  A failure turns the
banner red, gives the exit code, opens the output, and offers a link back.

Only a run the page has seen start counts, because the result of an
earlier run is still reported until the new run begins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An upgrade's result survives the daemon-reload it does

The upgrade reinstalls its unit and reloads systemd, which clears the run's
timestamps but keeps Result and the exit status.  A failure is now known
from those alone, and the result reports whether the script is running, so
the upgrade page waits on that instead of on a timestamp that may never come.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:11:59 -07:00
lrnselfreliance
d88db90ce9
The Settings upgrade can switch to a chosen branch (#954)
* The Settings upgrade can switch to a chosen branch

The Upgrade panel gains a Branch input: a dropdown listing the current
branch, master, and release that also takes a typed branch name.
/api/upgrade/start accepts an optional {"branch": ...}; without it the
current branch is upgraded as before.  A branch is written into the systemd
env file and passed to `upgrade.sh -b`, so only plain git branch names are
accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Controller upgrade refuses a branch that is not a plain branch name

The Controller's upgrade script wrote its branch straight into the systemd
env file, so a newline in it added a variable, and a leading dash became an
upgrade.sh option.  It now accepts the same branch names as the API, and no
script parameter may span lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:49:26 -07:00
lrnselfreliance
f3f762ef1b
Controller logs: an unparseable since is a 400, not empty logs (#953)
journalctl exits 1 with nothing on stdout when it cannot parse --since
(a bare "1h" is one such value); the controller returned that empty
stdout as the logs, with a 200.  A failed journalctl is now an error:
400 when it could not parse since, 500 otherwise.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:02:48 -07:00
Roland
d25ebb9e63
Bump version to 0.30.6-beta 2026-11-07 17:14:13 -07:00
lrnselfreliance
2e0e50c3aa
3D previews load ASCII STLs, stand upright, and fit the preview modal (#952)
* STL files preview with three's own STLLoader

react-stl-viewer inlines a copy of three-stdlib 2.17.2's STLLoader, which reads
ASCII STL text with LoaderUtils.decodeText.  three removed that in r175, so
every ASCII STL preview failed.  The three-stdlib override never reached that
copy, and react-stl-viewer has had no release since January 2024.

The 3MF preview's three.js viewer now previews STL files too, loading each
format with the loader that ships with three, so the loader always matches the
installed three.  react-stl-viewer and the three-stdlib override are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An e2e spec previews ASCII and binary STL files in a real browser

The STL preview only fails in a browser with WebGL, so the spec opens both
encodings on the live stack and checks the dimensions line appears without a
load error.  fb:reset can now write binary files ({base64: ...}).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* STL previews stand Z-up models upright

STLs are Z-up (the 3D printing convention) and three is Y-up, so the
preview rotates STL geometry to stand the model upright, as react-stl-viewer
did.  Model Z reads as Height again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* 3MF previews stand Z-up models upright too

3MF is Z-up like STL, so the rotation moves from the STL loader into the
shared viewer and applies to every model.  Model Z reads as Height.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* File previews show their buttons without scrolling

.preview-fit reserved a fixed 18.75rem for the rest of the preview modal,
which was short: every framed preview overflowed a little, and a 3D model's
dimensions line pushed the action buttons below the fold.  The preview modal
now fills its height as a flex column, and the .preview-fit takes whatever
height is left.  Previews without a .preview-fit keep their content's height.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* 3D previews frame the model and light it brightly

- The camera fits the model's bounding sphere into the narrower field of view,
  from a raised three-quarter front view, instead of backing off 2.2x along Z.
- A hemisphere light and a headlight that follows the camera (from a little
  above and left) replace lights whose intensities were set before three's
  r155 lighting change and read dim.
- Geometry without normals (3MF) gets creased normals, so flat faces no longer
  shade into their neighbors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 15:33:58 -07:00
lrnselfreliance
6c24d7d762
A move neither scans every FileGroup nor stalls the downloader's writes (#951)
* A move neither scans every FileGroup nor stalls the downloader's writes

Two stalls made a 109 file move take three minutes on a large library:

- Planning joined the move_sources temp table to file_group.  The temp
  table has no statistics, so with a real library's statistics SQLite
  scanned all of file_group (233k rows, 2.4 GB) instead of looking up 109
  paths in the primary_path index.  CROSS JOIN fixes the join order.

- The move's write transaction yielded to the event loop between chunks.
  The process that moves files also downloads; a download's write then
  waited on the move's lock synchronously, so the event loop could not
  resume the move: each write waited out the 30 s busy_timeout and failed
  ("Unable to renew downloads!").  The transaction now runs in a thread
  (inline in tests, which share one connection), so another writer waits
  only until the move commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A cancelled move waits for its thread however often it is cancelled

Review fix: the move waited for its thread on the first cancellation by
awaiting the thread's future again, unshielded.  Shutdown cancels twice
(run_loops cancels the file worker loop and stops waiting after 15 s, then
asyncio.run cancels every pending task, including the thread's future), so
the second cancellation raised while the thread was still in its
transaction, and handle_move reverted the files the thread then committed.

The move now waits on a threading.Event the thread sets when it leaves its
transaction, absorbing every cancellation meanwhile.  A cancelled future
reads "done" while its thread still runs, so it cannot be that signal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:47:12 -07:00
lrnselfreliance
aff5f1eb1f
A directory rename is finished by the file worker, not the request (#950)
* A directory rename is finished by the file worker, not the request

A directory rename queued a move, waited for it, then removed the old
directory and updated nested Collections itself.  A rename that outlasted
Sanic's RESPONSE_TIMEOUT (60 s) was cancelled while waiting: the move still
finished, but the old directory and its emptied subdirectories were left
behind, nested Collections kept their old directories, and the browser was
told the rename failed.

The move job now finishes the rename (nested Collections, then removing the
old directory, empty directories only), and the request returns once the
move is queued.  A failure is reported by the job's event.  An empty
directory can be renamed (it raised before).  The worker status carries
`rename_from`, so the Files page progress bar labels the rename, and the
file browser fetches again when the file worker is done.

A directory rename no longer queues a refresh of its parent: the file
worker runs the newest task first, so it scanned the directory before the
rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A queued rename returns its job, and the file browser waits for it

Review fixes:

- The cancellation test patched `wait_for_job`, which this path no longer
  calls, so it cancelled nothing.  It now asserts the invariant: the
  request only queues the rename (it never waits, and nothing has moved
  when it returns), and the file worker alone finishes it.

- The file browser inferred the end of a rename from the file worker's
  polled status.  A short job can start and finish between two polls, and
  the first idle poll (before the job started) spent the "fetch once"
  token, so the table could keep the old name.  POST /api/files/rename now
  returns `{"job_id": ...}` (null when the rename is already done), GET
  /api/files/jobs/<job_id> reports pending/complete/failed, and the file
  browser polls that job before fetching.

- useBrowseFiles ignores a fetch that settles after a newer one, so a
  listing requested before a rename cannot replace the one after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:09:33 -07:00
lrnselfreliance
c7bafaa7fb
Starting the app again adds no signal routes (#949)
Sanic's SignalRouter.finalize appends a `sanic.__signal__.__init__` handler
before it checks whether the router is already finalized.  Every start of the
app finalizes it again (the test client, Sanic's startup, and
finalize_signals_after_auto_routes), so each test left two more routes.  After
a few hundred tests in one process the compiled router was too deep for the
compiler (RecursionError in sanic_routing, intermittently failing whichever
tests a worker happened to run late).

api_app's signal router now skips a finalize when it is already finalized, and
registers that handler once.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:00:41 -07:00
lrnselfreliance
e5987fdd7e
ASCII STL files preview again (#948)
react-stl-viewer pins three-stdlib 2.17.2, whose STLLoader decodes ASCII STL
files with three's LoaderUtils.decodeText.  three removed that function, so
every ASCII STL preview failed while binary STLs (which never decode text)
still worked.  Override three-stdlib to 2.36.1, which ships its own decodeText.

Jest can now load three: a small transform converts three/three-stdlib ES
modules to CommonJS without babel-preset-react-app, whose class-properties
plugin rejects three's static class blocks at our pinned Babel version.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:55:55 -07:00
lrnselfreliance
20439437f8
Subtitle language-region suffixes are checked against the codes, not a 336,000-entry set (#947)
Every API process built a set of every language-region and language-auto
.srt/.vtt suffix at import: about 1 s and 73 MiB per process on a Pi 4.  A
suffix is now split into its language and region, and each is checked against
its own code set.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 17:19:14 -07:00
lrnselfreliance
ba9b47edf3
API startup no longer imports pymupdf (#946)
pymupdf 2.0 imports pymupdf.layout and onnxruntime, which every API start
paid for (about 2.5 s on a Pi 4) although only PDF extraction uses it.  It is
now imported when a PDF is extracted.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:01:45 -07:00
lrnselfreliance
cafa2a2279
Workers finalize the signal router once at startup, not once per route (#945)
sanic-ext adds the automatic HEAD and OPTIONS routes one at a time in a
before_server_start listener.  Each route re-finalized the already-finalized
signal router (Sanic.amend), and each finalize registered one more
`sanic.__signal__` route, so a worker rebuilt an ever-growing router about 245
times.  The signal router is now left unfinalized while sanic-ext adds those
routes and finalized once afterwards.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:50:10 -07:00
Roland
89b84c9765
Bump version to 0.30.5-beta 2026-11-02 20:12:14 -07:00
lrnselfreliance
479aa2254e
A Zim search without a search string or tag names is a 400, not a 500 (#944)
Both Zim search endpoints handed an empty request to the worker-job builder,
whose RuntimeError surfaced as a 500 "Unexpected error" with a traceback.  The
request is the client's mistake: the handlers now raise ValidationError, as the
other endpoints do, before any work is scheduled.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 19:55:47 -07:00
lrnselfreliance
87137cc40c
Zim names with a re-release letter are parsed, and an undated Zim no longer warns at every start (#943)
The kiwix naming scheme is {name}_{YYYY-MM}.zim, and a re-release in the same
month carries a letter: 3dprinting.stackexchange.com_en_all_2026-07a.zim.
parse_name rejected the letter, so such a Zim was never checked for newer
releases and zims.yaml keyed it by its full stem instead of its name.  The
letter is now accepted and folded into the day of the month-precision date, so
a re-release orders after the plain release wherever dates are compared.  The
pattern also anchors the ".zim" extension instead of matching any character.

A Zim that is not kiwix-named (edibleplantdb.zim) has no date to compare and is
simply not checked.  find_outdated_zim_files logged that at WARNING for every
such file at every start; it is a DEBUG line now.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 19:36:07 -07:00
lrnselfreliance
9797af157a
Removing a duplicate video keeps the Download that is finishing (#942)
* A refresh that finishes before tags.yaml is imported no longer reports a failed save (#941)

Every refresh fires save_tags_config at its cleanup.  In the first seconds
after a start, a refresh can finish before the startup import of tags.yaml,
which on a rebuilt database happens while the first refresh runs.  The dump
then asked ConfigFile.save to write a database that held no tags over the
file; the save guard refused, as it must, but it did so with an ERROR traceback
and a "Failed to save config: config/tags.yaml" event for the user, although
nothing was lost: the import that followed read the file and recreated the tags.

save_tags_config now skips the dump while the config has not been imported and
says so at WARNING.  The file stays the source of truth until the import.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* Removing a duplicate video keeps the Download that is finishing

After a video download, Video.delete_duplicate_videos removes the other copies
of the same video, which happens when the file name changed (a renamed file, a
new title).  Each copy was deleted through FileGroup.delete, which also deletes
the Download of the copy's URL and, without one, adds the URL to the skip list.
The duplicates share that URL with the video just downloaded, so the Download
still running was deleted from under its worker: the worker read its row again
to record the result and failed with NoResultFound ("Unexpected error"), and the
download's record was gone.

FileGroup.delete and Video.delete take keep_download; the duplicate cleanup
passes it, so the URL's Download and the skip list are left alone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Only a duplicate at the downloaded URL keeps its Download

delete_duplicate_videos also finds copies by source_id, and those can sit under
another URL.  Such a copy is deleted as any video is, with that URL's Download
or a skip-list entry; only a copy that shares the URL being downloaded leaves
the finishing Download alone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 19:18:46 -07:00
lrnselfreliance
0aa4019f86
A refresh that finishes before tags.yaml is imported no longer reports a failed save (#941)
Every refresh fires save_tags_config at its cleanup.  In the first seconds
after a start, a refresh can finish before the startup import of tags.yaml,
which on a rebuilt database happens while the first refresh runs.  The dump
then asked ConfigFile.save to write a database that held no tags over the
file; the save guard refused, as it must, but it did so with an ERROR traceback
and a "Failed to save config: config/tags.yaml" event for the user, although
nothing was lost: the import that followed read the file and recreated the tags.

save_tags_config now skips the dump while the config has not been imported and
says so at WARNING.  The file stays the source of truth until the import.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 19:12:41 -07:00
lrnselfreliance
871a26324d
Startup no longer renews failed once-downloads (#940)
retry_downloads_on_startup set pending, deferred and failed Downloads back to
"new".  A failed once-download has already given up: its downloader raised
UnrecoverableDownloadError or it reached its attempt limit.  Renewed at every
boot, it only failed again at once and logged a traceback-level warning each
time.  Startup now renews only the pending and deferred Downloads, which were
interrupted or are waiting for their next try.  The user's Retry button still
renews failed Downloads and resets their attempts.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 18:36:57 -07:00
lrnselfreliance
c78378d4f5
The Zim list reads each Zim's metadata in the libzim worker instead of opening it in the API (#939)
GET /api/zim went through Zims.get_all, which opened every Zim's Archive in the
API process, and then Zim.__json__ opened each one again for its metadata.
python-libzim holds the GIL and a cold open of a large Zim takes seconds, so
loading the Zims page stalled that API worker for every other request (27.8 s
cold against 0.75 s warm for 44 Zims).  #928 moved estimates and searches to the
worker process; the list now uses the same worker for metadata, and a Zim the
worker cannot read is left out as before.  count_tagged_zim_entries shares the
new readable_zims helper.  Zims.get_all had no other callers and is removed.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 18:19:29 -07:00
lrnselfreliance
b8122a4637
The tail-decode check ignores the h264 decoder's "mmco: unref short failure" note (#938)
After a download, ffmpeg decodes the last 30 seconds of the video and any stderr
line not on the ignore list marks the video incomplete.  Seeking into an h264
stream can make the decoder print "mmco: unref short failure": a note about
reference-picture marking for frames before the seek point.  The frames decode
and ffmpeg exits 0, but the verdict was "Video was incomplete", so the download
was deferred and the same sound video was downloaded again on every retry.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 18:00:03 -07:00
Roland
6c8ea61eaa
Bump version to 0.30.4-beta 2026-11-02 16:55:04 -07:00
lrnselfreliance
2991f737da
test_get_custom_videos_directory attaches the shared context it needs (#937)
Setting a config field calls ConfigFile.update(), which takes
api_app.shared_ctx.config_update_lock.  The test did not use async_client,
so the lock only existed when another test in the same process had already
attached the shared context; running first in a worker, it failed with
"'SharedContext' object has no attribute 'config_update_lock'".

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 16:31:55 -07:00
dependabot[bot]
662a41afc3
build(deps): update pytest-asyncio requirement in /controller (#926)
Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
- [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases)
- [Commits](https://github.com/pytest-dev/pytest-asyncio/compare/v1.4.0...v1.4.1)

---
updated-dependencies:
- dependency-name: pytest-asyncio
  dependency-version: 1.4.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:05:36 -07:00
lrnselfreliance
31b47dacf8
The inventory legacy-date repair no longer takes the save lock during import (#915)
MultiFileConfig.initialize() imports every file while holding the shared save
lock, which is not reentrant. The repair added in #901 took that lock again from
inside import_all, so any WROLPi whose inventory still held epoch-second dates
hung every API process at startup.

The repair now only rewrites the in-memory entities during import and records
which inventories changed. import_inventories_config() writes those files once
it runs, outside the lock. A regression test runs initialize() against an
epoch-date file with a timeout.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-02 11:58:42 -07:00