Commit graph

826 commits

Author SHA1 Message Date
lrnselfreliance
faf8e735d1
A map is exported as a PDF: a 1:24,000 quadrangle, in layers, on a sheet, which says where it is (#1006)
* The map can choose a 1:24,000 quadrangle to export (#971)

The first part of exporting a map as a PDF: a grid of 7.5-minute quadrangles is drawn over the
map, and a click chooses one.  The grid is arithmetic, so it needs no data.  Exporting itself is
not here yet; the button is disabled.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A quadrangle of the map is exported as a PDF with a layer for each map layer (#972)

* A quadrangle of the map is exported as a PDF with a layer for each map layer

The browser draws the quadrangle off the screen at the size of a 1:24,000 sheet, once for each
layer which is on, and the API puts the images in a PDF, each in a layer of its own which a
reader can show and hide.  A layer in which nothing is drawn is left out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The exported map's canvas keeps what was drawn, and an export's memory is bounded

- The drawing buffer is asked for where this MapLibre reads it (canvasContextAttributes); as an
  option of the map it was ignored, so the canvas could be read after it was cleared.
- Each layer's pixels are compressed in the PDF as the layer is added.  PyMuPDF kept them
  uncompressed until the PDF was written: about 90 MB a layer of a quadrangle, all at once.
- The pixels of all the layers together are limited, before any image is decoded.
- pymupdf is imported when a PDF is made, not when the API starts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where on the Earth it is (#973)

* An exported map says where on the Earth it is

The PDF's page has a geospatial viewport (ISO 32000-2): the page is the map in Web Mercator,
with the latitude and longitude of each corner.  The corners are the edges of the picture the
browser drew, read from the map which drew it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A map whose edge is the antimeridian can be exported

The map's arithmetic puts the east edge of the last quadrangle before the antimeridian a rounding error past
180 degrees, and the export was refused.  An edge within 1e-8 of a degree of 180 is at 180 (it is not wrapped
to -180); a map which is really across the antimeridian is still refused.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map is on a sheet: a margin, a grid, scale bars and notes (#974)

* An exported map is on a sheet: a margin, a grid, scale bars and notes

The map is put on a larger page with what a topographic map has around it: a title, the latitude and longitude
of its corners and of ticks between them, scale bars (kilometers, miles, feet), and notes of the projection, the
grid and where the map's data is from.  Over the map is a 1,000 meter UTM grid, each line named in the margin.

The grid and the margin are PDF layers of their own.  The PDF says where the map is, not where the page is.  The
picker has a box to export the map alone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The scale of a narrow map is written under its name, not over it

A quadrangle near a pole is two inches wide, which is not room for its name and its scale on one line.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where a compass points (#975)

* An exported map says where a compass points

The sheet has a diagram of the three norths (true, grid and magnetic) beside its scale bars, and a note of how
far magnetic north is from true north on the day the map was made, and how fast it moves.

The declination is from the World Magnetic Model 2025 (NOAA NCEI and the British Geological Survey; public
domain), whose coefficient file is shipped: nothing is downloaded.  The model is good until 2030; outside its
years nothing is said of a compass.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A corner of an exported map is not also a tick

The edges of the picture the browser draws are a hair outside the quadrangle's, so the quadrangle's own edges
were taken for ticks inside the map, and each was written over the coordinate of its corner.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A compass which passes due south has moved a little, not most of the way around

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A map can be exported at 400 DPI (#976)

The picker has a choice of 300 or 400 DPI.  A map is drawn less sharp than was asked for, rather than refused,
when it would be more pixels than the API makes a PDF of: a quadrangle near the equator, or one with many layers
on.  The picker says so while it draws.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported quadrangle goes by the name the USGS gives it (#977)

* An exported quadrangle goes by the name the USGS gives it

The picker says what the chosen quadrangle is called, and the sheet's title, the PDF's title and the file's
name have it.  Where the USGS has no name, which is most of the world, a quadrangle goes by where it is, as
before.

The names are an index of the 65,600 7.5 minute quadrangles of The National Map (U.S. Geological Survey; public
domain), which is shipped: nothing is downloaded.  In Alaska a USGS quadrangle is wider than 7.5 minutes, so a
quadrangle of the picker's grid there is "Part of" the one which has its middle.

The places the tests of the map export are of are changed for others.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An area half of which is the last quadrangle of the land is a part of it

The middle of such an area is on the quadrangle's east or north edge, with no quadrangle beyond it, and it was
given no name.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The PDF of a map is made by a Job, which the browser watches (#978)

* The PDF of a map is made by a Job, which the browser watches

A PDF of a quadrangle is a minute or two of work on a Raspberry Pi 4, and the request which waited for it was
cut off after a minute: only the smallest quadrangles could be exported there.

Now the browser describes the map, sends the image of each layer in a request of its own, and starts a Job; it
watches the Job, shows how far along it is, and fetches the PDF when the Job is done.  Cancel stops the Job.
The images are kept on disk until the PDF is made of them, and an export is removed when it is an hour old.

The credits on the sheet are no longer cut off: an attribution which names many sources is each of them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A cancelled map export stops, and is removed

- A cancelled Job is not over until the thread making the PDF has stopped, so the next Job does not start beside
  it; the work can also be stopped while the sheet is drawn and the PDF written.
- DELETE /api/map/export/<id> stops an export at any step and removes what was sent; the browser calls it when
  the person exporting cancels, while layers are sent as well as while the PDF is made.
- An export whose Job is waiting or running is not removed for being old; one whose Job was cancelled is removed
  at once.
- Cancel while the map is drawn stops the drawing between two layers.
- Exports are kept in a directory made for this user alone, which is refused when it is a link or is another
  user's; only what is an export there is removed.
- A request for the finished PDF which is lost is made again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An overlay's areas and lines go under the map's labels, and its own labels over them (#979)

An overlay was drawn whole over the basemap, so an area it filled covered the names of what is in it.  Labels are
now always on top: the areas, lines and pictures of every overlay are under the basemap's labels, and the text,
icons and points of every overlay are over them.  It is so in the map, the standalone viewer, and an exported PDF,
where an overlay with both is two layers ("Towers" and "Towers (labels)").

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An overlay's ground is over the ground of every basemap file (#980)

* An overlay's ground is over the ground of every basemap file

On a map of several basemap files, the ground of an overlay was put before the first label of the first file,
which is under the land, water and roads of every file after it: an overlay's areas and lines were hidden, and
in an exported PDF its layer was just above the background.  It goes where the basemap's labels begin: the first
of the labels which are over all of the basemap's ground.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The labels of every basemap file are over the ground of every file

A map of several basemap files was their layers one file after another, so only the labels of the last file were
on top: the labels of the others were under the land, water and roads of the files after them, and under the
ground of an overlay.  Both viewers now draw the ground of every file and then the labels of every file.

The contour lines are drawn after the basemap's labels and are not of the basemap: an overlay's ground goes under
the basemap's labels, not under the contours instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* Contours are drawn from as deep as the terrain file goes (#981)

The map read a terrain file to zoom 8, which is how deep the terrain WROLPi offers goes, however deep the file
was.  It now reads a file as deep as the file says it goes, in both viewers, and draws finer contour lines from a
deeper file: every 40 feet (or 20 meters) from zoom 11, as a 1:24,000 sheet has.  A file which goes to zoom 8 is
drawn as it was.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map names its small roads (#982)

* An exported map names its small roads

A quadrangle is drawn at about zoom 13.  The basemap's style names small roads
from zoom 15, and its tiles have no names for them below zoom 14, so the streets
of a town were printed as lines with no names.

Before the map is drawn, the tiles of zoom 14 which the quadrangle is in are
read, and the roads in them which have a name are given to the export map as a
source of its own.  The layer which names small roads draws from that source, as
its style says to.

At the scale of a sheet most streets are shorter than their names, and a map
writes a name along a line only where the line is longer.  Such a road is named
over its middle, turned as the road goes; a road in pieces is named once.

A source whose tiles can not be read is exported as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The names of small roads are placed by the export, at every latitude

A quadrangle nearer the equator is drawn at a little over zoom 14, where the
names were not read and the style still kept them for zoom 15: such a map had
no names of small roads.  They are now read wherever a layer would not name
roads.

Every name of a small road is now a point: over the middle of its road, turned
as the road goes under it, again every 400 pixels of a long road.  Before, a
road long enough was left for the map to name along its line, which the map
often does not (the line bends, or is in pieces), and a short piece beside it
was named too.  One place decides where a road is named, so a road in pieces is
named once, and a piece of another kind of road does not take the name.

Reading the names can be cancelled.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet has a bar of meters, its norths in mils, and how far apart its contour lines are (#983)

Under the map are bars of kilometers, meters, miles and feet.

How far the grid's north and a compass's north are from true north is written in
mils as well as in degrees, in the diagram and in the notes: a compass may be
marked in either.

A map which is exported with its contour lines says how far apart they are
(CONTOUR INTERVAL 40 FEET), under the bars.  The browser knows: it is the
interval of the tiles the lines are made for at the zoom of the sheet, in the
unit the map measures in.  It says so with the sheet, and what is not an
interval is not written.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet says what the quadrangles around it are called (#984)

* A sheet says what the quadrangles around it are called

Under the scale bars is a diagram of nine squares, the map the middle one, and
beside it what the USGS calls each of the eight around it: the sheets to print
to carry on off the edge of this one.

The names are from the index of quadrangles which names the sheet itself.  A
quadrangle with no name (out to sea, or over a border) has no number; a map of
where the USGS names nothing, and a map too narrow for the diagram, go without.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet is never beside itself

Where a quadrangle of the USGS is wider than the map (Alaska), the area the
size of the map to its east or west is often more of the quadrangle the map is
in, and the diagram named the sheet as its own neighbor.  The quadrangle on a
side is now the next one which is not the map's own.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A sheet has a legend of the lines of its map (#985)

* A sheet has a legend of the lines of its map

Under the scale bars is what each kind of line of the map is: highway, major
road, minor road, path, railway, boundary, contour and index contour, each a
piece of the line as the map draws it, on the color of the map's land.

The map's style says how a line is drawn at each zoom, and a sheet is drawn at
one.  The browser, which has the style, says what each line's color, width,
dashes and edges are at that zoom; the API draws them.  A line is in the legend
when its layer is exported and draws something at that zoom.

What the browser sends is made into lines before it is kept: what is not a line
is left out, and the map is exported.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A legend has no line too thin to see, and no name longer than its room

A style brings a line in from nothing as the map is zoomed in: a path is a few
hundredths of a point wide on a quadrangle near the equator.  It was in the
legend, drawn several times wider than on the map.  A line under a tenth of a
point is now left out, by the browser and by the API.

What a line is called is cut to the room beside its line, as the name of an
adjoining quadrangle is.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* Fixes to the map PDF export: wrong neighbors, leftover exports, exports which outlive the map (#989)

* A sheet names what is around it, and its edges, by where they are and not a hair from there

The edges the browser gives of a map are a hair outside its quadrangle.  The quadrangles around a map whose
middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of
them was the map itself.  An edge a hair south of the equator was 0 degrees south, a map which ended where the
UTM zones do had no grid, and a sheet dated in the last year there is could not be made.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export is removed when it is over, and its work stops when the API does

The images and the PDF of an export were kept until the next export began, however long that was.  The browser
now has the export removed however it ends, and old exports are removed when a PDF is made too.  The work of a
Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it.  What
is said of an export, and each image, is written whole.  Text which is no text is taken out when an export
begins, not found by the Job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export stops with its map, and waits for a page which can not be seen

Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did
closing the picker, which is now left open.  A PDF which arrives after the export was stopped is not saved.
The time a page is behind another tab is not counted against a map which is drawing.  The name of a road is
one line.  A photograph which ends a little inside the map is not made a JPEG with a black strip.  An accent
in the name of a quadrangle is not a gap in the name of its file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A PDF is not kept from who made it by a server slow to remove the export

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its water over its parks (#990)

* A sheet names what is around it, and its edges, by where they are and not a hair from there

The edges the browser gives of a map are a hair outside its quadrangle.  The quadrangles around a map whose
middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of
them was the map itself.  An edge a hair south of the equator was 0 degrees south, a map which ended where the
UTM zones do had no grid, and a sheet dated in the last year there is could not be made.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export is removed when it is over, and its work stops when the API does

The images and the PDF of an export were kept until the next export began, however long that was.  The browser
now has the export removed however it ends, and old exports are removed when a PDF is made too.  The work of a
Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it.  What
is said of an export, and each image, is written whole.  Text which is no text is taken out when an export
begins, not found by the Job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An export stops with its map, and waits for a page which can not be seen

Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did
closing the picker, which is now left open.  A PDF which arrives after the export was stopped is not saved.
The time a page is behind another tab is not counted against a map which is drawing.  The name of a road is
one line.  A photograph which ends a little inside the map is not made a JPEG with a black strip.  An accent
in the name of a quadrangle is not a gap in the name of its file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its water over its parks

A layer of the PDF was one image, drawn where the topmost of its layers is.  The parks of Land Cover are under
the water on the map and its piers are over it, so all of Land Cover was drawn over the water: a lake in a
park, or among houses, was not on the PDF.  A second basemap file put its water over the first's buildings.

A layer is now as many images as it must be, each where it is on the map, in one layer of the PDF: a part
which fills an area stays where it is, and a part which is only lines is drawn with the next.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A PDF is not kept from who made it by a server slow to remove the export

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A map file which is of somewhere else is not drawn when a map is exported

Each basemap file is several images of an exported map, and most files of a library are of other places: each
was an image of nothing, counted against how many images an export may be and how sharp it may be.  A file
whose own edges are outside the quadrangle is now left out before the layers are counted.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map has its streams (#991)

A basemap draws streams from zoom 14, and service roads and taxiways from 13.  A quadrangle is drawn a little
short of that (zoom 13.8 at 40 degrees, 13.2 at 61), from tiles which have them: none were on the PDF.  What
is drawn from a zoom no deeper than a quadrangle's detail is drawn.  Names are left as they are.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An export which is cancelled stops at once, in the middle of a request (#993)

* An export which is cancelled stops at once, in the middle of a request

An export was stopped between two requests.  The image of a layer is tens of megabytes, which is a long request
on a slow connection, and a request to a server which has stopped answering never ends: the export said it was
stopping until then.  Cancel, and leaving the map, now end the request.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* The requests an export makes after its layers are sent are seen to be stopped too

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A map is exported as it is printed: on white, in dark lines (#996)

* A map is exported as it is printed: on white, in dark lines

The light and dark maps are made for a screen.  On paper their roads are white on pale gray and their streams
half a pixel of pale blue.  The printed map is a third look, of the map's own: white ground, dark road edges
and names, blue streams wide enough to see, paths drawn, brown contours.

It is what a map is exported in unless Light or Dark is chosen where the quadrangle is, and the choice is
remembered in the browser.  While a quadrangle is chosen the map is shown in the look it will be exported in.
The light and dark maps are as they were.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Changing the look of the map does not move it

What is done when a new style is in place was asked for after the style was set.  A style which only differs
from the one the map has is in place before that, so it was done at the next change of style instead, with
where the map had been at this one: the map went back there when a quadrangle was chosen, and again when it was
not.  It is asked for first, and the map is not moved at all.  An export no longer waits for the map to be
still, which it need not and which nothing ended.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The colors of the printed map are chosen where its layers are (#997)

* The colors of the printed map are chosen where its layers are

While a quadrangle is chosen and the map is shown as it is printed, each layer of the layer control has its
color beside it, to be changed: one for most, three for the land (what grows, what was built, sand and ice)
and two for names (the name, and what is around it).  A basemap has several colors where one is chosen, which
are shades of it.  The edge of a road stays dark.  What is chosen is remembered in the browser, and can be put
back.  The map on the screen is as it was.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A ramp on a bridge is the color of a major road, as it is off one

A basemap draws a ramp on a bridge in the color of a minor road on a bridge: on the printed map it was white
where the rest of the ramp is yellow, and followed the color chosen for minor roads.  What puts the colors
back is a button, which a keyboard reaches.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* A layer with nothing in it does not make an exported map less sharp (#1001)

* A layer with nothing in it does not make an exported map less sharp

How sharp a map may be depends on how many images it is, and every group of layers was counted: a basemap
file with no runway, highway or boundary in the quadrangle was images of nothing, drawn, sent, and counted
against the rest.  A map of 23 images asked for at 400 DPI was made at 330.  The groups are now looked at
first, on a small map, and only those with something in them are drawn, as sharp as that many may be.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* An image which arrives after its export was removed is not kept, and one mil is a mil

Stopping an export while an image was on its way left the image behind it, in a directory nothing would
remove for an hour.  An angle of one mil was "1 mils" on the sheet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The map which is exported from is made as sharp as it is to be (#1002)

* The map which is exported from is made as sharp as it is to be

The groups of a map are looked at on a small map, which was then made larger to draw them.  A browser lets
a canvas be larger when it is made than it lets one be made larger afterwards: a quadrangle asked for at
400 DPI was 33 million pixels, 288 DPI at 40 degrees, less than before the groups were looked at first.  The
small map is now removed, and the map which is exported from is made in its place.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* A canvas the browser stopped drawing on is not a layer with nothing in it

A canvas whose drawing was taken away by the browser reads as clear, and a group looked at then was left out
of the PDF without a word: the export fails instead.  An export stopped as the last group is looked at does
not make the map it would have been drawn on.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* An exported map says where it is in a projection Adobe Acrobat knows (#1003)

* An exported map says where it is in a projection Adobe Acrobat knows

The page was said to be in Web Mercator, which it is drawn in.  Acrobat does not know Web Mercator, in the
words of the standard or in Esri's: it took the page for a map, and put every place on it at 0, 0.  The page
is now said to be Mercator on the WGS 84 ellipsoid, which it reads, with the same corners.  Between the corners
the two differ by under three centimeters on the ground.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* How well a page is said to be where it is, is measured from what the PDF says

The test of it measured two formulas of its own.  It now reads the corners and the projection from the PDF and
does a reader's sum with them.  The few centimeters are of a quadrangle: a taller page is less well said, by
the square of how tall it is, which the comment now says and a test measures.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

* The help says how a map is exported as a PDF (#1004)

The help of the map is five pages, one of them the PDF export.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 17:22:31 -07:00
lrnselfreliance
a5ab01198b
The backup drive keeps deleted files for 30 days, then removes them (#999)
* The backup drive keeps deleted files for 30 days, then removes them

The fifth part of replacing the sync.  A backup drive never deleted anything, so it grew forever and kept every file
the media directory had ever held.  Now, like a secondary:

- A file deleted from the media directory, or the old version of one that changed, is moved to a dated folder in
  .backup-trash on the backup drive.  Folders older than backup_delete_after_days (sync.yaml; 30 by default, also
  for drives set up before this; none never deletes) are removed, before and after each backup, and the oldest go
  sooner when the drive is nearly full.
- A backup refuses a media directory that is not mounted, or has no config/wrolpi.yaml: copied with deletes, an
  empty folder would move the whole backup drive to the trash.
- The Backup Drive panel has the same "Delete after N days / Never delete them" control as the Sync panel, now one
  component, and refuses an empty number of days.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A nearly full drive keeps the trash it just filled, and makes room for the database

Review found, in the backup drive's trash and in the secondary's sync, which share its shape:

- Below 1 GiB free, the purge after a copy removed the newest trash folder too.  Moving a file to the trash is a
  rename, which frees nothing, so on a nearly full drive a deleted file was gone at the end of the same run instead
  of 30 days later.  The newest folder is now only removed by age (0 days still deletes at once).
- The purge made room only down to 1 GiB, so a database larger than the free space failed to be copied every time
  while the trash was full.  Before the database is written (the backup's snapshot, the secondary's copy of its
  primary's), the trash now makes room for the whole database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:50:21 -07:00
lrnselfreliance
8cfe36b3ea
A secondary syncs by rsync: the Controller runs it on its schedule, and both pages follow it (#995)
* A secondary syncs by rsync: the Controller runs it on its schedule, and both pages follow it

The third part of replacing the sync.

- The Controller syncs within the daily window (every 15 minutes after the last sync finished, in WROLPi's
  timezone), as the API did.  The API's loop no longer runs the old engine, and its /api/sync/run and
  /api/sync/cancel ask the Controller.
- The Sync page and the overlay that covers the app read the sync's status from the Controller, which is the only
  one that can answer while the API is stopped.  The overlay is also shown when the settings cannot be read, as
  happens while the API is stopped.  Sync Now and Stop go to the Controller.
- The Controller's fallback page, where Caddy sends the browser while the API is down, shows the sync's progress
  and a Stop button.
- New on the Sync page: how long files the primary deletes are kept (30 days by default, or never), and a note that
  an excluded folder's files are still listed on a secondary.

The old engine's code is removed in the next part.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove SyncRunRequest, unused since a sync is asked of the Controller

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* In Docker, a sync's progress moves while a large file is copied

Docker's logs hold a line until it ends, and rsync's progress for one large file is a single line rewritten with
carriage returns, so the page showed 0% until the file was done (hours, for a large map).  The rsync helper's
output is now read by attaching to it, before it is started so that nothing is missed.

Found by a sync between two Docker WROLPis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync is recovered before the schedule can start another, and an empty days value is refused

Review found:

- The schedule and the recovery of an interrupted sync started side by side when the Controller started.  The state
  an interrupted sync leaves is due, so inside the window a new sync could start while the recovery restarted the
  API and wrote over the new sync's status.  Every tick of the schedule now recovers first, and only a status of
  "running" with no sync holding the lock counts as interrupted.  A sync records its outcome before it frees the
  lock, so a tick between the two cannot mistake it for an interrupted one.
- The Controller may start before the drive is mounted.  The empty record it read then was kept and saved over the
  real one, losing the last successful sync.  The state file is now read again until it can be.
- A cleared number of days was saved as 0, which empties the trash at the next sync.  It is now refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:06:25 -07:00
lrnselfreliance
bb7a66b0d6
The calculators have no Temperature calculator (#987)
The Celsius/Fahrenheit/Kelvin converter is removed from the Calculators
page and from the search suggestions.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 15:49:45 -07:00
lrnselfreliance
0389e4659c
An overlay's layers which the basemap already draws start hidden (#969)
A file of features often carries a basemap's layers too (roads, water, buildings, place names).
Drawn over the basemap they double every road and label.  Those layers, by the names of the
Protomaps and OpenMapTiles schemas, are now off until they are checked; every other layer is on
as before, and a choice already made is kept.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 11:17:31 -07:00
lrnselfreliance
b2b8e2075c
The map draws its own icons on an overlay (#967)
An overlay drawn with a style of its own names icons, which needed the sprite sheet that style was made for.  The map now ships icons for overlays, the Maki and Temaki sets (both CC0), and always draws those, in both viewers.

The sprite sheet is the translation: its index lists every icon under its own name, under each other name it answers to (modules/map/icons/aliases.json), and in both spellings of each.  A name with no icon is drawn as a plain dot.  The icons are signed distance fields, so a style's icon color and halo apply.

A pattern is not an icon: what it filled (a fill, a line or an extrusion) is drawn in a translucent color, where removing a pattern used to leave MapLibre's opaque black.

A sprite sheet beside a map file is no longer used, served or reported.  wrolpi/scripts/build_map_icons.py builds the sheet from the SVGs; the sheet is committed.
2026-11-13 17:59:21 -07:00
lrnselfreliance
177d0dba4b
The standalone map viewer draws overlays, as the map does (#966)
The standalone viewer (port 8084) drew basemaps only: a map file which is
not a basemap was skipped.  It now offers each overlay in its layer panel,
hidden until turned on, with a checkbox per layer, draws it with the style
beside it when there is one, and shows a feature's attributes on a click.

The overlay logic is the map viewer's own.  It moved to dependency-free
modules (app/src/map-shared) which a browser can load as they are; the
standalone viewer, which has no build step, loads copies of them from
modules/map/static/js.  A test fails while a copy differs from its module
(`npm run sync:map-shared` makes the copies).  What differs between the
viewers is passed in: where a map file and its style are fetched from, and
whether a layer can be checked before MapLibre is given it.

The standalone viewer cannot check a layer first, so it adds an overlay's
layers to the map one at a time once the style has loaded: MapLibre refuses a
whole style over one layer it cannot draw, but only that layer when it is
added by itself.

The API also names a style by its path within the map directory, which is
what the standalone viewer is served.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 17:28:41 -07:00
lrnselfreliance
113e70a402
Contour labels are drawn in a font the map has (#965)
The contour label layer named no font, so MapLibre asked for its default
("Open Sans Regular,Arial Unicode MS Regular"), which the map does not ship.
The request failed and the labels were drawn with glyphs the browser made up
locally.  The layer now names Noto Sans Regular, in the map viewer and in the
standalone viewer.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 15:53:01 -07:00
lrnselfreliance
4f58d2862f
A style beside a PMTiles overlay says how the map draws it (#963)
When a MapLibre style (and optionally a sprite sheet) sits beside a PMTiles file, the overlay is drawn with it instead of the generic colors: `<name>.style.json` and `<name>.sprite.json/.png`, or `style.json` and `sprite.json/.png` shared by the files of a directory.

Only the style's layers which draw that file are used; its sources, sprite and glyph URLs never are.  Fonts are replaced by the ones the map has, images are pointed at the overlay's own sprite (as strings, expressions or property functions), and each layer is validated so one the map could not draw is skipped instead of stopping the map.

The sprite is served by the API, which falls back to the single resolution sheet when there is no double resolution one.
2026-11-13 15:41:43 -07:00
lrnselfreliance
adb0adbd94
The map draws any PMTiles file as an overlay, in any sub-directory (#962)
Any PMTiles file under the map directory, at any depth, can be shown.  The API reads each file's header and metadata and says what it is (basemap, terrain, overlay, unreadable).  Overlays are offered in the layer control under the name the file gives itself, hidden until turned on, with a checkbox per vector layer, and are drawn generically.

Map files are named by their path within the map directory everywhere, and unfinished downloads are no longer listed.  A file can be ignored from the Manage page; ignored files are kept in a new maps.yaml config.

PMTiles metadata is not decompressed past its size limit, and the standalone viewer encodes map file paths.
2026-11-13 15:11:01 -07:00
lrnselfreliance
30b2bf6df0
Any Tag alone searches the library, and Zim search does not offer it (#960)
On the search page, choosing Any Tag without a search string left the
Dashboard showing: the page, the Files results and the Files count all waited
for a search string or tag names.

Zim search has no "any tag" search, and its tag picker's Any button left no
filter applied, so the picker no longer offers it.  Zim and Other counts are
only requested when there is a search string or tag names to count by.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:44:14 -07:00
lrnselfreliance
f1d344d8a2
Search results sort as the filter dialog says (#959)
* Search results sort as the filter dialog says

Docs sorted ascending orders backwards and fell back to id order for every
descending order (`-size`, `-published_datetime`); it now takes the same `-`
prefixed orders as the other searches and rejects unknown ones with a 400.

Archives and Docs searches without an order now put the most relevant results
first, as Videos does and as the dialog shows.

Rank has no direction in the dialog: choosing it sends `rank` (most relevant
first) instead of inheriting a descending `-rank` (least relevant first).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An old -rank link becomes rank when the filter dialog closes

The dialog shows Rank without a direction, but it seeded its draft from the URL
unchanged, so Done left a saved `-rank` (least relevant first) in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:42:23 -07:00
lrnselfreliance
6cd233dce6
Sync and backup: a secondary WROLPi and a backup drive copy the library (#957)
* A secondary WROLPi copies its primary's library over HTTP

A WROLPi with role "secondary" pulls from its primary: a manifest of every
FileGroup's files (/api/sync/manifest), then each missing or changed file
from the primary's Caddy /media/ (resumable, written to a hidden partial and
renamed with the primary's mtime), a refresh, then the library configs,
imported as the primary wrote them.  wrolpi.yaml contributes only its library
settings.  Files deleted on the primary are kept.

A secondary never saves the synced configs and never downloads.  Its API
answers 503 SYNCING while a sync runs, and every page warns that it is a
copy.  It syncs in a daily window or on "Sync now"; WROL Mode stops it.

On a Pi 4 (no AES instructions) the client uses TLS 1.2 ChaCha20, which
pulls ~110 MB/s from Caddy versus ~41 MB/s for rsync over ssh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync has its own /admin tab; a sync's reads from the primary are bounded

The sync settings and progress move from Settings (and the component
gallery, which is for library components) to /admin/sync.

A download never writes past the size in the primary's manifest, and configs,
listings, and manifest pages are capped.  A primary address may not carry a
user name or password (it is logged and shown).  While a sync runs, the API
answers only the exact requests that show its progress or stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync keeps the secondary's replaced configs in today's config backup

The first copy of the day is kept, like a save, so a config backup import
undoes a sync's config changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Docker's Caddy logs every request

`docker compose logs -f web` shows each request with its status, size and
duration, e.g. a secondary's /media/ downloads during a sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync reports its progress as it downloads, and refuses quietly

The status updated only when a page of 500 FileGroups finished, so a page of
large files looked stuck; it now updates every 2 seconds.  Requests refused
while syncing are answered by the middleware instead of raised, so the UI's
polling no longer logs an error each time.  The primary's version is stored
without its trailing newline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A WROLPi can back up to a second drive; sync settings live in sync.yaml

The Sync tab's Backup Drive copies the media directory to a second mounted
drive with rsync during the sync window (hourly), or on "Back Up Now".
Nothing on the backup drive is deleted.  The Tags and playlists directories
are left out (WROLPi rebuilds them from the configs), and the database is
written as an SQLite snapshot rather than copied as files, so the drive can be
mounted as the media directory and start.  A backup is written only while the
chosen drive (by UUID) is mounted where it was chosen, and is not the media or
system drive; rsync is stopped if the drive disappears.  Not in Docker, which
cannot reach other drives.

Role, primary, window and backup drive move from wrolpi.yaml to sync.yaml:
they belong to this machine, and are never copied from a primary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Sync tab shows the configured backup drive

getSettings copies each setting by name and did not list the backup drive,
so the Sync tab showed "None" while one was configured.  It also dropped
`version`.  A test now fails for any setting the API sends that getSettings
drops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A backup can go to a folder on its drive, and remembers its last success

The Backup Drive panel takes a folder on the chosen drive.  The drive is
checked through the folder's mount point: the folder is written only when the
drive with the saved UUID is mounted there, so a folder left on the system
drive by an unplugged drive, or on another drive mounted inside the chosen
one, is refused.  A missing folder is created once the drive is checked.

When the last backup succeeded is saved in sync.yaml, so it is known after a
restart.  "Stop Backup" is disabled while the database snapshot is written (it
runs to the end), and a finished backup clears the request to stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The backup drive has its own window

The sync window was shown in the Sync panel but also scheduled the backup
drive.  The Backup Drive panel now has its own window (sync.yaml
backup_window_start/end), saved with the drive; the sync window is shown,
and used, only for a secondary's sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sync and the backup drive can each exclude folders

A secondary or a backup drive may be smaller than the primary's library.
Each Sync tab panel has its own excluded folders in sync.yaml, chosen in
a folder browser: a secondary browses its primary's folders (through
Caddy), the backup drive browses this WROLPi's media directory.  Excluded
folders are never copied; what was copied before is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The folder browser does not offer lost+found

Every ext4 drive has a root-only lost+found folder; a sync and a backup
never copy it, so it is not offered for exclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A sync never writes into config or tags, whatever their case

On a media drive that ignores case (exFAT, NTFS, macOS), a manifest path
like Config/wrolpi.db is the real config folder. The protected and
unlisted top-level folders are now compared case-insensitively in the
manifest, in resolve_destination, in excluded-folder validation, and in
both folder listings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A backup writes the database snapshot only when the database changed

Every backup rewrote the whole database snapshot (2.5 GB on a busy
WROLPi), even on the window's idle hourly runs. The snapshot is now
skipped when the live database (its size and mtime, and those of a
non-empty WAL) and the drive's copy are both unchanged since the last
snapshot. The record is in memory, so the first backup after a restart
writes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove the unused save_sync_config switch

Nothing activated it; SyncConfig saves directly.  The test server's
log_message override is called by http.server, so it goes in the
vulture baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:53:45 -07:00
lrnselfreliance
d88db90ce9
The Settings upgrade can switch to a chosen branch (#954)
* The Settings upgrade can switch to a chosen branch

The Upgrade panel gains a Branch input: a dropdown listing the current
branch, master, and release that also takes a typed branch name.
/api/upgrade/start accepts an optional {"branch": ...}; without it the
current branch is upgraded as before.  A branch is written into the systemd
env file and passed to `upgrade.sh -b`, so only plain git branch names are
accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The Controller upgrade refuses a branch that is not a plain branch name

The Controller's upgrade script wrote its branch straight into the systemd
env file, so a newline in it added a variable, and a leading dash became an
upgrade.sh option.  It now accepts the same branch names as the API, and no
script parameter may span lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:49:26 -07:00
lrnselfreliance
2e0e50c3aa
3D previews load ASCII STLs, stand upright, and fit the preview modal (#952)
* STL files preview with three's own STLLoader

react-stl-viewer inlines a copy of three-stdlib 2.17.2's STLLoader, which reads
ASCII STL text with LoaderUtils.decodeText.  three removed that in r175, so
every ASCII STL preview failed.  The three-stdlib override never reached that
copy, and react-stl-viewer has had no release since January 2024.

The 3MF preview's three.js viewer now previews STL files too, loading each
format with the loader that ships with three, so the loader always matches the
installed three.  react-stl-viewer and the three-stdlib override are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* An e2e spec previews ASCII and binary STL files in a real browser

The STL preview only fails in a browser with WebGL, so the spec opens both
encodings on the live stack and checks the dimensions line appears without a
load error.  fb:reset can now write binary files ({base64: ...}).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* STL previews stand Z-up models upright

STLs are Z-up (the 3D printing convention) and three is Y-up, so the
preview rotates STL geometry to stand the model upright, as react-stl-viewer
did.  Model Z reads as Height again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* 3MF previews stand Z-up models upright too

3MF is Z-up like STL, so the rotation moves from the STL loader into the
shared viewer and applies to every model.  Model Z reads as Height.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* File previews show their buttons without scrolling

.preview-fit reserved a fixed 18.75rem for the rest of the preview modal,
which was short: every framed preview overflowed a little, and a 3D model's
dimensions line pushed the action buttons below the fold.  The preview modal
now fills its height as a flex column, and the .preview-fit takes whatever
height is left.  Previews without a .preview-fit keep their content's height.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* 3D previews frame the model and light it brightly

- The camera fits the model's bounding sphere into the narrower field of view,
  from a raised three-quarter front view, instead of backing off 2.2x along Z.
- A hemisphere light and a headlight that follows the camera (from a little
  above and left) replace lights whose intensities were set before three's
  r155 lighting change and read dim.
- Geometry without normals (3MF) gets creased normals, so flat faces no longer
  shade into their neighbors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 15:33:58 -07:00
lrnselfreliance
aff5f1eb1f
A directory rename is finished by the file worker, not the request (#950)
* A directory rename is finished by the file worker, not the request

A directory rename queued a move, waited for it, then removed the old
directory and updated nested Collections itself.  A rename that outlasted
Sanic's RESPONSE_TIMEOUT (60 s) was cancelled while waiting: the move still
finished, but the old directory and its emptied subdirectories were left
behind, nested Collections kept their old directories, and the browser was
told the rename failed.

The move job now finishes the rename (nested Collections, then removing the
old directory, empty directories only), and the request returns once the
move is queued.  A failure is reported by the job's event.  An empty
directory can be renamed (it raised before).  The worker status carries
`rename_from`, so the Files page progress bar labels the rename, and the
file browser fetches again when the file worker is done.

A directory rename no longer queues a refresh of its parent: the file
worker runs the newest task first, so it scanned the directory before the
rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A queued rename returns its job, and the file browser waits for it

Review fixes:

- The cancellation test patched `wait_for_job`, which this path no longer
  calls, so it cancelled nothing.  It now asserts the invariant: the
  request only queues the rename (it never waits, and nothing has moved
  when it returns), and the file worker alone finishes it.

- The file browser inferred the end of a rename from the file worker's
  polled status.  A short job can start and finish between two polls, and
  the first idle poll (before the job started) spent the "fetch once"
  token, so the table could keep the old name.  POST /api/files/rename now
  returns `{"job_id": ...}` (null when the rename is already done), GET
  /api/files/jobs/<job_id> reports pending/complete/failed, and the file
  browser polls that job before fetching.

- useBrowseFiles ignores a fetch that settles after a newer one, so a
  listing requested before a rename cannot replace the one after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:09:33 -07:00
lrnselfreliance
e5987fdd7e
ASCII STL files preview again (#948)
react-stl-viewer pins three-stdlib 2.17.2, whose STLLoader decodes ASCII STL
files with three's LoaderUtils.decodeText.  three removed that function, so
every ASCII STL preview failed while binary STLs (which never decode text)
still worked.  Override three-stdlib to 2.36.1, which ships its own decodeText.

Jest can now load three: a small transform converts three/three-stdlib ES
modules to CommonJS without babel-preset-react-app, whose class-properties
plugin rejects three's static class blocks at our pinned Babel version.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:55:55 -07:00
Joel Shearon
fe85815c35
Format inventory expiration dates instead of epoch timestamps (#901)
Legacy inventory items stored date fields as Unix epoch seconds. The table showed the raw numbers, and sorting and expired highlighting did not treat them as dates.

- Frontend parses date values as YYYY-MM-DD or legacy epoch seconds; display, mobile view, export, search, sort and the expired check all use calendar dates with no timezone shift.
- The legacy migration converts epoch seconds to YYYY-MM-DD, and import repairs already-migrated files once.
- Jest runs with a pinned timezone; the expiry tests pass under any process timezone.

Closes #41

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 12:27:21 -06:00
lrnselfreliance
d75be8e1a4
Download forms: allow a frequency of "Once" (#804)
* app: a download frequency of "Once" no longer blocks submission

useForm treated any falsy value as a missing required field.  "Once" is
frequency 0, so the Channel/Playlist, RSS, and Zim download forms never
became ready and the Download button stayed disabled.  Only null,
undefined, '', false, and NaN count as missing now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* downloads: editing to "Once" stores a once-download, not frequency 0

Once-downloads have a NULL frequency.  A 0 frequency counted as recurring
(frequency != None) but could never be scheduled (not frequency), so an
edited download stayed on the recurring list and could break renewal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:00:24 -06:00
lrnselfreliance
3c362799b5
Resume EPUBs and comic books where the reader left off (#798)
* Resume EPUBs and comic books where the reader left off

The EPUB viewer (epub.html) accepts `?cfi=` and posts its location and
progress to the page that embeds it on every page turn.  Progress comes from
the chapter and the page within it, rather than book.locations.generate(),
which stalls a Raspberry Pi on a large book.  A search result's `loc` still
wins over the saved position.

The comic viewer opens at the saved page and saves each page shown.

Both resume on the doc page and in the file preview.  The resume point is
decided once per opened file, so saves made while reading do not change the
EPUB viewer's URL and reload it.  The doc page tracks only the FileGroup's
primary file; another format beside it is not tracked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* EPUB/comic progress: never save a place the server would discard

The server treats a position at the start (the cover, under 1%) as not
started, and replaces the saved place with progress 0.  Both viewers
reported the place they opened on, so opening a finished book un-finished
it, and peeking at the cover erased the reader's page.

- The comic viewer does not save the page it opens on, nor the cover; a
  peek at the cover keeps the last page saved.
- The EPUB viewer posts nothing at the start of the book.
- Progress before the end is kept between 1% and 94% (resumableProgress), so
  early pages of a long book and the last pages before the end keep their
  position.  Only the last page (comics) or the end (EPUB) is finished.
- Non-finite EPUB progress is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:19:51 -06:00
lrnselfreliance
1ad451cddf
Resume videos and audio where the user left off; show progress on cards (#791)
* Resume videos and audio where the user left off; show progress on cards

Videos and audio resume where the user left off, on the video page and in
the file preview, and keep playing.  `?t=` starts at an explicit time
instead.

useFileProgress saves how far the user is through a file: every 15 seconds
while it changes, and a final save (a beacon, which survives the page
closing) on pause, end, hiding the page, leaving it, or switching files.
Each save is also kept in sessionStorage; on open the newer of that and the
server's position wins, so coming straight back resumes exactly even if the
final save is still in flight.

Cards show a progress bar along the bottom of their poster.  The dashboard's
Recently Viewed Files gains All / Continue; Continue uses the new
`in_progress` file search filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Progress review fixes: in_progress totals, latest dashboard request, preview viewed

- The in_progress search filter skips the uncounted fast path and is part of
  the totals cache key, so Continue never reports All's (or the library's)
  count.
- The dashboard's Recently Viewed hook applies only its latest response, and
  clears the list when switching All/Continue until the new files arrive.
- POST /api/files/file returns `viewed`, so a preview can tell a file
  finished elsewhere from an older position this tab holds.
- The progress hook tests restore Blob, fetch and sendBeacon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:29:22 -06:00
lrnselfreliance
bcbc42632d
Remove CircleCI (#796)
GitHub Actions runs every job CircleCI did (#789), so CircleCI only
duplicated them.  The README badge now shows the backend tests on master.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:19:38 -06:00
lrnselfreliance
531721c157
Fix file view tracking: docs mark viewed, previews count one view (#786)
Opening a doc on its page never set FileGroup.viewed, so docs were missing
from Recently Viewed.  GET /api/docs/:id now marks the doc viewed, with the
same skip_viewed parameter as archives and videos.

The file preview marked a file viewed more than once: opening from a
?preview= URL sent two tracked requests, and every tag change re-fetched the
file with tracking on.  The fetches now skip tracking, and the preview marks
each opening viewed once.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:04:13 -06:00
lrnselfreliance
a1292005ce
Other search tab: search Channels by name, add Domains and Playlists, fix the estimate (#785)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Count Channels with any of the Tags in the Other search estimate

The Other tab lists Channels with any of the selected Tags, but its estimate
only counted Channels with the first Tag, so the two disagreed when several
Tags were selected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Search Channels by name in the Other search tab

The Other tab only listed Channels with the selected Tags, so typing a
Channel's name found nothing.  Channels now match the search text by name,
and the Tags when there are both.  The list and its estimate share one query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add Domains to the Other search tab

Domains match the search text by name and/or any of the Tags, like Channels.
The list and its estimate share one query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add Playlists to the Other search tab

Domains and Playlists are Collections, so the Other tab searches both with
the existing /api/collections/search (by kind) instead of a Domain-only
endpoint.  The search and the estimate share search_collections_query.  The
search now loads each Collection's Downloads up front instead of once per
Collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 18:22:12 -06:00
lrnselfreliance
93d878952c
Add a Repos module that mirrors git repositories (#783)
Repos keeps local mirrors of git repositories, cloned over https and kept up to date on a schedule.

- A Repo is a Collection (kind='repo') plus a `repository` row.  The new `git` downloader clones it, then updates it
  (weekly by default, or Never: the clone is kept as it is and only updated on request).
- Repos are saved in `repos/<tag>/<name>` (the Repos Directory, ignored by default) or a directory the user chooses.
  An existing clone can be imported where it is.  repos.yaml is the source of truth.
- A GitHub-like page shows each repo's README, files, and history, with a ZIP download.  READMEs are searchable
  under the global search's Other tab.
- git is hardened against hostile repos and configs: https only, pinned addresses with no private addresses (for the
  repo and its submodules), sanitized configs, no hooks or links, and only clones WROLPi made or imported are
  updated.
- Tagging a Repo moves its clone as one directory.  Repo changes wait for repos.yaml to be imported at startup.
- The file mover groups unindexed files with no primary file as one FileGroup, and never replaces a file at the
  destination.  A failed tag move restores the Collection's tag.  A Collection's directory cannot leave the media
  directory.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:16:58 -06:00
lrnselfreliance
04779b8557
Preview markdown files formatted, with a raw toggle (#780)
* Preview markdown files formatted, with a raw toggle

Markdown files are typed text/markdown (libmagic reports them as text/plain, or
as a programming language when they hold code) and keep the text indexer, so
they stay deep-searchable.  Existing files switch type on their next refresh.

The file preview renders markdown with markdown-it into the page, so it follows
the theme tokens and the night media filter.  A Formatted/Raw toggle shows the
source instead, and the choice is remembered.  Relative links and images
resolve against the file's directory; URLs that climb out of /media/ are
dropped.  Raw HTML in a file is escaped rather than parsed, and the output is
also sanitized with DOMPurify.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Do not load images from other sites in the markdown preview

An image from another site (http:, https:, or protocol-relative //) loaded as
soon as the preview opened, telling that site who was reading the file and
when.  Such an image now becomes a link labeled with its alt text, so the
reader chooses whether to follow it.  Inside a link (a README badge) it becomes
the outer link's text, since links cannot nest.  Media and embedded data:
images still load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 13:26:42 -06:00
lrnselfreliance
cc8afa5370
Add a Solar calculator (#778)
* Add a Solar calculator with offline NASA POWER data

Estimates a solar array's typical-year production from a location and the
panels; everything else has defaults under an optional "Fine-tune" section.

The model works from 12 monthly averages: Klein's average day per month, the
Erbs monthly diffuse fraction, Collares-Pereira & Rabl / Liu & Jordan hourly
split, HDKR transposition for fixed, single-axis and dual-axis arrays, then
PVWatts' ASHRAE incidence loss, SAPM cell temperature, DC power, system
losses and optional inverter.  "Best for the year" and "Best for winter" sweep
the tilt.

Sunlight, diffuse, albedo and temperature come from NASA POWER's 2001-2020
climatology on a 1 degree grid, packed into app/public/data (3.1 MB, ~1 MB
gzipped) by scripts/build_solar_climatology.py and fetched only when the
calculator opens.  Users can replace it with their own monthly values.

Every setting lives in the URL, with defaults left out, so the Share button
shares the whole configuration.

Against PVWatts v8 for a 1 kW fixed array: Phoenix +0.6%, Seattle +8.8%,
Anchorage +8.0% annually; these are recorded as tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Open the Solar calculator from the map, pins and current location

The map's right-click (or long-press) menu gains "Calculate Solar
Performance", and each saved pin a sun button; both open the calculator at
that location.  A button beside Latitude fills in the browser's current
location.  Locations are rounded to 4 decimals, and longitudes from a map
panned around the world are wrapped back to -180..180.

The map removed lat/lon/z from the URL when it closed, even after navigating
to another page, which stripped the calculator's location from its URL (and
from links shared with the Share button).  It now only cleans up while the
URL is still a map page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add off-grid sizing to the Solar calculator

A new "Off-grid sizing" tab sizes a battery system for the darkest month
from a list of devices (watts x hours per day):

- Battery bank for a number of days without sun, for LiFePO4 or lead-acid,
  at 12, 24 or 48 V.  Outdoor lead-acid is enlarged for the cold; outdoor
  lithium warns against charging below 0 C.
- Panels needed, from the worst month's DC production per kW at the chosen
  angle, with a button to apply the count.  A PWM controller's loss (battery
  voltage over the panel's Vmp) raises the count.
- Charge controller current, and the string voltage on the coldest morning
  against the controller's limit, since cold panels exceed their rated Voc.
  The coldest morning defaults to 25 C below the coldest monthly average.

All inputs live in the URL like the rest of the calculator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bump wrolpi-help for the Solar calculator help page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Credit NASA POWER as it asks, and say how the data was changed

- The calculator gains a "Data attribution" link under the location with
  NASA POWER's requested acknowledgment, the data reference (Climatology API
  v2.10.0, downloaded 2026/10/04), the CC BY 4.0 license, and how WROLPi
  modified the data (rounded to one byte per value, temperature averaged
  into 1 degree cells, interpolated between cells), which CC BY 4.0 requires.
- The build script writes the same text to solar-climatology.LICENSE.txt
  beside the data, with the API version taken from NASA's responses and the
  date the tiles were downloaded.  A test keeps the page and file in step.
- Bumps wrolpi-help for the updated attribution in the help page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: fix PWM sizing, Voc sign, unit switching and loading on review

- PWM usable power now uses the string's voltage (Vmp x panels in series),
  delivers nothing when the charging voltage is at or above the string's Voc,
  and falls off between Vmp and Voc.  The "too little voltage" warning covers
  PWM as well as MPPT.  Previously two panels in series counted double, and a
  panel that could not charge a 48 V bank counted as fully usable.
- A Voc temperature coefficient entered without its minus sign is treated as
  negative, in the math, the URL and the input, so the cold-morning check
  cannot be made too generous.
- Switching Fahrenheit on or off converts the entered temperatures instead
  of reinterpreting them.
- While the sunlight data downloads, the off-grid tab shows a loader rather
  than asking for a location, and holds the cold-morning verdict until the
  site's temperatures are known or one has been entered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: move the data attribution to the bottom of the page

The full NASA POWER acknowledgment, license and changes now sit in a small
"Data attribution" section at the end of the calculator, always visible,
instead of behind a link under the location.  The source line under the
location points to it.  Bumps wrolpi-help to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Solar: drop the data source line under the location

The attribution at the bottom of the page is enough.  The line under the
location now only appears to report the data's state: loading, failed to
load, or replaced by the user's own values.  Bumps wrolpi-help for the
refreshed screenshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 18:36:45 -06:00
lrnselfreliance
4fea50943b
map: clicking a POI shows its kind, detail, elevation and coordinates (#738)
The Protomaps basemap keeps a few attributes per POI beyond the name the
map already labels: kind, kind_detail and elevation.  Clicking a POI now
opens a popup with those, plus the feature's coordinates (or the click
location when the feature has no point).  The same POI appears at several
min_zoom levels within one tile, so features are deduped by name and kind.

Map popups (this one and the existing pin popup) now follow the active
theme's tokens instead of MapLibre's fixed white, covering light, dark,
night and amber.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-27 08:23:03 -06:00
lrnselfreliance
15d3792b0e
videos: transcode can remove the audio stream (silent video) (#737)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
A video's audio stream can be dropped the same way its video stream could
already be: a new REMOVE_AUDIO audio target maps only the video stream with
-an, needs a video container, and is refused for audio files.  Output
verification insists the result has no audio stream.

Removing both streams is refused by the validator, the API, resolve_output,
and the modal: choosing Remove for one stream resets the other to Keep and
withdraws the other's Remove option.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:49:14 -06:00
lrnselfreliance
63ddad69e8
app: api helpers report failure instead of returning an empty value (#736)
Six helpers turned a non-OK response into an empty value, so their
consumers rendered a server error as "nothing here":

* getConfigBackups -> {dates: []}: "No backups available for this
  config."  A thrown request also left "Loading backups..." forever.
* getInventoryBackups -> []: "No backups yet."
* getDownloaders -> {downloaders: []} on a throw, the error body on a
  non-OK response (which then crashed the .map): the RSS form's
  Downloader select sat empty.
* searchMap -> {results: []}: "No map locations found".
* getMapSearchStatus -> {indexed: [], missing: []}: "No maps are
  installed".
* flasherSearch -> [null, null], then `|| []`: "No .bin firmware
  found".

Invariants now enforced and tested:
* On a non-OK response each helper returns undefined; an empty value
  only ever comes from the server.
* Each consumer shows an error for a failed or thrown fetch and keeps
  its "none" message for a confirmed empty result.  The Downloader
  select is disabled and says it is loading or could not fetch.  A
  failed map status is unknown, never "no maps are installed".
* The map search tab loads from the first render, and a query shorter
  than the two characters the request needs now says so instead of
  "No map locations found".

Archive browsers and recent tags are left as they are: a failed fetch
there hides an optional picker or suggestion list without claiming
anything.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 17:47:32 -06:00
lrnselfreliance
20541d7184
app: number fields no longer throw "e.preventDefault is not a function" (#735)
`useForm`'s input handlers called `e.preventDefault()` unconditionally, but the
Mantine-backed `NumberField` (Sleep between requests, Depth, Maximum Pages)
hands them a bare `{target}` object because NumberInput emits a value, not an
event.  Every keystroke in those fields threw.

`handleInputEvent` now only calls `preventDefault` when the argument actually
has one, and `getInputProps`' wrapper no longer repeats the call.  Audited every
other `preventDefault` call site: all are bound to real DOM/React events or a
guarded no-argument call.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:41:52 -06:00
lrnselfreliance
eead78007d
app: status and settings contexts say whether their first poll landed (#734)
Both contexts start as {} and nothing distinguished that from a
server that reported nothing, so for up to one poll interval after a
page load consumers rendered the negative case as fact: the Settings
form appeared with blank values and an "undefined/config/wrolpi.yaml"
path (the ready check `settings ? true : undefined` was always true),
the Downloads toggle was clickable and read "Downloading Disabled",
the upgrade panel said "up to date, vundefined", and the five hardware
toggles on the Controller page claimed "not supported on this server".

Invariants now enforced and tested:
* StatusContext.loaded is true once a poll returned a real status;
  the API being down does not count.  SettingsContext.loaded is true
  once settings were fetched, and `failed` when a fetch returned
  nothing before that; a failed refresh after a load keeps the last
  known values and `loaded`.
* The Settings page shows its loader until loaded and its error only
  when failed.  The upgrade panel shows a loader until status loaded.
  The Downloads toggle is disabled with a neutral label until then.
* A subsystem toggle is disabled without claiming "unsupported" while
  status is unknown (undefined); "unsupported" (null) is reserved for
  a loaded status that lacks the field.  A call in flight is unknown,
  not unsupported.

The context fixtures declare loaded so existing specs render as
before.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:25:46 -06:00
lrnselfreliance
306386e801
app: Flasher saved configs and Statistics distinguish pending from empty (#733)
The Flasher's saved configurations started as [] with no loading
flag, and the Saved Firmwares tab is the default tab, so every load
painted "No saved configurations yet." until the fetch returned.  The
api helper returned [] on a non-OK response too, so a failed fetch
read as an empty list.

useStatistics used useState() with no initial value, so the
Statistics page's first render saw undefined and flashed "Failed to
fetch statistics" for one frame before the effect set {} and the
spinner appeared.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] / object = loaded.  The
  api helper returns undefined (and toasts) on a non-OK response.
* The Saved Firmwares tab shows a placeholder while pending, an error
  on failure, and "No saved configurations yet." only once the server
  confirmed there are none.  Only the newest configs request may write
  state, since save and delete refetch.
* The Statistics page shows its loader while pending and its error
  only for a failed fetch.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 17:05:02 -06:00
lrnselfreliance
6e251a009a
app: map pins and subscriptions tables distinguish pending from empty (#732)
* app: map pins and subscriptions tables distinguish pending from empty

Both lists were initialised to [], which SortableTable renders as
"loaded and empty".  The Pins tab therefore showed "No pins.
Right-click on the map to add one." until the pins arrived, and the
Manage tab's subscriptions table showed "No results" until the catalog
did.  A non-OK response (the api helpers return undefined) or a thrown
request left the same messages up permanently.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] = confirmed empty, so
  SortableTable shows its placeholder while pending and each empty
  message appears only once the server confirmed the list is empty.
* A failed fetch renders an error in place of the table.
* Only the newest request for each list may write state, since edits,
  deletes, subscribes and unsubscribes refetch while the mount request
  may still be in flight.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* app: the regions preview waits for the catalog

"View All Regions" stayed enabled while the catalog was pending or
failed, and the modal received a fresh `catalog || []` on every
render.  Its map effect depends on that array, and ManageMap
re-renders on every status poll, so an open preview tore its WebGL
map down and rebuilt it every three seconds, and a failed catalog was
previewed as zero regions over the error message.

The button is disabled until the catalog is an array, and the modal
is mounted only then, so it receives the state array itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 16:55:58 -06:00
lrnselfreliance
ae745f1a18
app: search dropdown offers "Channels: No results" only after a search (#730)
useSuggestions started with channels: [] and reset to that on every
keystroke, and the normalizer treats an empty list as "the server
found none".  So the global search dropdown showed a "Channels: No
results" row on focus before anything was typed, and on every
keystroke through the debounce and the request.  Because that row
counts as a result, the SearchBox's own "Searching…" state never
appeared.

The channel, domain, author and subject groups now start as null,
meaning not fetched; an empty list only ever comes from the server.
The count summary tolerates the null.  Tests render the hook and
check no Channels group exists before a search or while the request
is pending, and that the "No results" row and real rows still appear
after the response.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 15:07:00 -06:00
lrnselfreliance
86d4bbab98
Bookmarks: a nav menu of user links, stored in bookmarks.yaml (#729)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* bookmarks: user bookmarks in the navigation bar, stored in bookmarks.yaml

A "Bookmarks" tab between Calculators and Flasher opens on hover or click and
lists the user's bookmarks, with directories as flyout submenus (Menu.Sub).
The mobile hamburger shows directories as labelled groups instead.  The tab
always shows, with an "Edit bookmarks" item, so the feature can be found
before any bookmark exists.

Bookmarks live in a new config-only bookmarks.yaml: a nested tree where a
node with `children` is a directory and a node with `url` is a bookmark.
Every node carries an integer id so the API can rename, move and delete
without relying on names.  A URL may be a path on this WROLPi, a port on this
host (`:8096/` or `http://:8096/`, resolved against whatever hostname the
browser used, so it works over LAN IP, mDNS and the hotspot alike), or an
absolute URL.  Each bookmark chooses whether to open in a new tab.

The editor at /more/bookmarks follows the playlist items table: numbered rows,
up/down/edit/delete, and inline "Add a bookmark" / "Add a directory" forms
with a Location picker.  Deleting a directory asks first, since its contents
go with it.  The menu and the editor are in the component gallery.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* bookmarks: mark More current when the editor is folded inside it; rem indents in the mobile menu

Found on the QA Pi at a width where Bookmarks overflows into More: the More
button carried no mark on /more/bookmarks, because the bookmarks link has no
`to` for isLinkActive to match.  isLinkActive now knows the bookmarks link owns
its editor page, and the Bookmarks tab uses the same rule.

The mobile menu's directory labels sat a few pixels left of the labels around
them: the indent was in em, and a Menu.Label is set smaller than a Menu.Item.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* bookmarks: parse URLs instead of matching a prefix; validate the whole tree on every write

The URL contract was checked by prefix on both sides, and a prefix is not a
form: `//evil.com` starts like a path and `:8096@evil.com` starts like a port,
and a browser takes both off this host.  A path is now a single leading slash
that resolves to this origin, a port form is an optional scheme, `:`, digits
and then only a path, query or fragment (no `@`, no backslash, no control
characters), and anything else must parse as http(s).  The same rules live in
wrolpi/bookmarks.py and BookmarksContext.js, with the same bypass strings in
both test suites.

The config's dataclass validator only saw that `bookmarks` was a list, so a
hand-edited file or the generic config API could store a `javascript:` URL
that the bookmarks API would refuse, and the nav rendered it.  The config now
validates the tree itself on every write and import: URL forms, exactly one of
url/children, unique integer ids, boolean new_tab, and no cycles (a YAML
anchor can build one).  An invalid file is not imported and, since it was never
successfully imported, is never overwritten.

On the page, a stored URL that does not resolve is rendered as text, never as
an href.  A refused add keeps what was typed instead of clearing the form, and
an edit whose move is refused still refreshes the nav (the rename was
committed) and keeps the modal open.  An IPv6 host is bracketed before a port.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* help: document Bookmarks

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 13:11:36 -06:00
lrnselfreliance
9caa140fb6
app: search Other tab distinguishes pending channels from none (#728)
* app: search Other tab distinguishes pending channels from none

useSearchChannels initialised channels to [] and loading to false, and
only set loading inside its mount effect, so the Other tab's first
render painted "No Channels" with a count of 0 before the search had
run.  A failed search was rendered the same way, since the hook had no
catch and the API helper returns undefined on a non-OK response.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] = no channels; loading
  is true from the first render because the search starts on mount.
* Changing tags resets to pending, and only the newest request may
  write state, so a slow response for previous tags is dropped.
* The tab shows a loader while pending, an error on failure, and
  "No Channels" only once the server confirmed there are none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* app: useSearchChannels follows its caller's tags

The hook read its tag argument only as the initial state, so the
Other tab never refetched when the filter modal changed the URL's tags
while the tab stayed mounted; the list, including a stale "No
Channels", was whatever the mount fetched.

The internal tag state now follows the argument's contents, keyed on
the serialized list because the caller builds a new array every
render.  Tests change the hook argument after a loaded result and
expect a second search and a reset to pending, check that a new array
with the same tags does not refetch, and keep the slow-response drop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 12:01:37 -06:00
lrnselfreliance
83362a65c1
app: video comments and description distinguish pending from empty (#727)
* app: video comments and description distinguish pending from empty

useVideoExtras initialised both values to null and the page treated
null the same as "none", so every video open (and every prev/next)
showed "No comments have been downloaded. Refresh the video?" with a
live Refresh button until the comments arrived.  The Description tab
showed "No description available." the same way.  A failed fetch was
also rendered as "none", and the previous video's data lingered until
the next video's responses arrived.

Invariants now enforced and tested:
* null = pending, undefined = fetch failed, [] / '' = loaded and empty.
  The API serialises a missing info_json field as null, so the hook
  normalises that to empty; consumers never see null for "none".
* Switching video resets both to pending, and a slow response for the
  previous video is dropped.
* The page renders a placeholder while pending, an error message on
  failure, and offers Refresh only once the server confirmed there are
  no comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* app: useVideoExtras drops any request that is not the newest

Matching on the video id let a stale request land when the user left
and returned to the same video, or saved an edit while the page's
first description request was still in flight.  Each fetch now takes a
generation number and only the newest generation may write state,
per value, so a description refetch never invalidates a comments
request.

The edit dialog adopts a description that arrives while it is open
only if the user has not typed, so a draft written over the
placeholder is not wiped by the empty or failed value that follows.

Tests: 7 -> none -> 8 -> 7 with the first request rejecting last; a
save refetch beating the initial request; the draft surviving; the
pending placeholder asserted; the console spy restored in finally.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-26 11:22:37 -06:00
lrnselfreliance
5358c75a2b
Jobs queue, video Edit dropdown: transcode/remux, edit title & description (#711)
* Jobs queue and video transcode from the video page

Add an in-memory FIFO Job queue (wrolpi/jobs.py): functions register by
name, callers enqueue by name with JSON kwargs, and the perpetual-tasks
owner runs them one at a time.  Logging emitted inside a Job and the
output of its subprocesses are captured into a per-Job log; Jobs report
progress and can be cancelled (a running Job's subprocess is killed).
Jobs are not persisted; an API restart forgets them.

Add a transcode Job and POST /api/videos/<id>/transcode.  ffmpeg reports
progress via -progress pipe:1.  The existing FileGroup is pointed at the
new file before the refresh so tags and history survive a container
change.

Replace the video page Refresh button with an Edit dropdown (Refresh,
Transcode).  The Transcode modal pre-fills from the preferred codecs in
the Videos settings and shows the Job's progress, log tail and Cancel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: move Delete into the Edit dropdown

The Edit dropdown now holds Refresh, Transcode and Delete, so the action
row is Download / Edit / Add to Playlist.  Delete keeps its confirmation
(a Confirm dialog) and the tagged-files prompt, and is disabled in WROL
Mode like the other items.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: Edit > Edit to change a video's title

PUT /api/videos/<file_group_id> writes the title (and fulltitle) into the
Video's info json, creating the file when the video has none, then
validates the Video again so its title and search text follow.  Files
stay the source of truth: the edit survives a refresh, and a re-download
of the metadata replaces it (the modal says so).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: edit the description in Edit > Edit

PUT /api/videos/<file_group_id> now takes optional title and description
and changes only the fields given; an empty description clears it.  The
modal sends only what changed and refetches the description after saving.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: allow a container-only remux

Keeping both streams is no longer refused: ffmpeg copies them into the
chosen container with fast start.  The container select starts at the
file's own container, the hint explains what a remux does, and the button
reads Remux when nothing is re-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* WROL Mode: refuse video edits and transcodes past the API

The routes already returned 403, but a transcode Job queued before WROL
Mode was enabled still ran, update_video could be called from anywhere,
and the Edit modal's Save button stayed live.  The Job now fails before
ffmpeg starts, update_video carries wrol_mode_check, and the modal
disables Save with a notice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: verify the output before replacing the original

After ffmpeg exits, the output is probed and compared with the source
(container, presence of video/audio streams, codec per stream, duration
within 1s or 1%) and its last 30 seconds are decoded to a null output.
Any failure removes the output and keeps the original.

The tail decode replaces the download-time completeness check, which
grabbed a single frame near the end: one ffmpeg_video_complete (sync for
the downloader, async for Jobs) now serves both, and is stricter, since
every frame of the tail plus the audio must decode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: claim atomically, cap pending, own-session subprocesses; retarget before delete

Review fixes.  The worker read a Job's status and wrote RUNNING as two
steps, so a cancel landing between them was overwritten and the Job ran;
_claim_job now does both under the lock.  enqueue_job refuses past
JOB_PENDING_LIMIT pending Jobs.  JobContext.run_command starts the
subprocess in its own session and raises CancelledError when the command
was killed, so a handler cannot mistake it for a normal exit.

The transcode Job now retargets the FileGroup (committed) while the
original file still exists, and deletes the original afterwards
(transcode_video_file gained keep_original).  A crash in between leaves
both files with the database pointing at a real one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: the Edit menu owns the transcode Job; polling survives Close and errors

Review fixes.  The transcode Job's id and polling (useJob) move from the
modal to VideoEditMenu, so closing the modal no longer stops the page
from learning the Job finished (and refetching the video, whose path may
have changed).  While a Job runs the menu shows Transcoding..., reopening
shows its progress instead of the form, and Delete waits.  A failed poll
request is retried instead of ending the polling.  getJobs checks the
response like its siblings.  Design narration trimmed from comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: audio files, and Remove video to turn a video into an audio file

video_codec 'none' (REMOVE_VIDEO) drops the video stream (-vn, audio
required); an audio-only source is audio-only output whatever the video
target says, and a video target on it is refused.  Audio output gets an
audio container (m4a/ogg/mp3) checked against the codec it will hold, and
a video container asked for audio output is replaced by the codec's
natural one.  faststart only for mp4/m4a.  Verification learned the
audio-only shape and ffprobe's container names (mkv reports matroska).

The modal offers 'Remove video (audio only)', switches to audio
containers and follows the audio codec, reads Extract Audio, and opens
for audio files (no video codec select).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video page: yellow Edit button

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: remove the temp output on cancel; tell the user when a closed-modal Job fails or is lost

Review fixes.  The temp output was only removed on Exception, but a
cancelled Job's run_command raises CancelledError (a BaseException), so a
cancel left the partial file for a refresh to index.  Closing the modal
during a Job that then failed showed nothing; it now toasts the error
(and a cancellation).  Polling gives up on a 404 or five failures in a
row (the API restarted and forgot its Jobs), warning the user and
unlocking Transcode and Delete instead of reading Transcoding... forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Jobs: _claim_job docstring states the invariant, not the old race

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode: fragmented mp4 option for very long videos

A whole-file mp4 index grows with the frame count; for a many-hour video
it reaches tens of MB, which a phone must fetch and parse before the
first frame, and every seek walks it.  `fragmented` writes the file with
-movflags +frag_keyframe+empty_moov+default_base_moof+global_sidx and
10s fragments: a ~1 KB moov plus a global seek index, so playback starts
and seeks quickly.  mp4/m4a only; a remux, so it takes seconds per GB.

Measured on an 11-hour h264/aac file: moov 40 MB -> 1.4 KB, seek index
2 x 120 KB, 10k fragments, duration and codecs unchanged, remux in 25s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Transcode modal: the Fragmented label says audio for audio output

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Video edits keep the original: custom_title/custom_description in the info json's wrolpi section

Editing a title or description no longer overwrites yt-dlp's keys.  The
values go into the info json's `wrolpi` section (custom_title,
custom_description); the modeler prefers them when deriving the Video, so
a refresh keeps the edit while the original title and description stay in
the file.  An empty custom description means none.

Both re-download paths carry the wrolpi section over: the metadata-only
refresh merges it into the fresh info json, and a full re-download
snapshots it before yt-dlp overwrites the file and restores it (then
re-validates) in finalize_download.  FileGroup.get_wrolpi_json reads it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:25:23 -06:00
lrnselfreliance
d701846254
Hotspot captive portal: welcome page with the WROLPi's address (#709)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* Hotspot captive portal: welcome page with the WROLPi's address

Devices that join the hotspot probe a fixed plain-HTTP URL to check for
Internet access.  NetworkManager's shared-mode dnsmasq now resolves those
probe hostnames (and `wrolpi`) to the hotspot address via an include the
Controller writes when the hotspot starts, the probes land on the Controller
(port 80), and are redirected to /portal: "This is your WROLPi, its address is
10.42.0.1, open WROLPi / open the Controller".

Only the probe hostnames are hijacked so the hotspot keeps forwarding
everything else when the WROLPi has an uplink.  The include is corrected and
the connection re-activated once if NetworkManager picked a non-default
subnet.  Writing the include is best-effort; the hotspot starts regardless.

`hotspot.captive_portal` (controller.yaml, default true) toggles the probe
redirects and the include; like the other hotspot settings it applies on the
next hotspot start, so saving while the hotspot is up reports
restart_required and both Controller UIs say so.  The status endpoint gains
the hotspot address and portal URL, shown in both UIs.  repair.sh only
creates the include directory so a disabled setting survives upgrades.
Native only: Docker has no hotspot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: address review

* React: re-read the hotspot address whenever the HotspotToggle starts or
  stops the hotspot, matching the fallback UI's refresh-on-status.
* A WPA3 start that reverts to WPA2 leaves the hotspot up, so it now gets
  the captive portal address correction too.
* The corrective `nmcli connection up` logs a warning on a non-zero exit;
  dnsmasq would otherwise keep the old address silently.
* HotspotStatusResponse.captive_portal defaults to true like everywhere else.
* Trim design narrative from the module docstring and React comments.
* The include-before-activate test now observes the include at the moment
  of the `nmcli ... hotspot` call instead of inferring from call order.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: never forward any record type for probe hosts

`address=` only synthesizes A/AAAA answers.  iOS also asks for the HTTPS
record type; dnsmasq forwarded that upstream, Apple's answer aliased to
captive.g.aaplimg.com, and the phone fetched the genuine "Success" from
Apple instead of the portal.  Each probe host now also gets `local=`, so
dnsmasq answers every type itself, and the CDN alias is a probe host too.

Observed with dnsmasq query logging on a hotspot with an eth0 uplink.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: let acknowledged clients finish so the OS offers "Done"

A phone's sign-in sheet only shows "Done" once the OS re-probes and gets
the genuine success reply.  The Controller now remembers each client (by
hotspot IP) that has loaded the portal page and answers its later probes
with the reply each OS expects (Apple "Success" HTML, Android 204, Windows
"Microsoft Connect Test", ...).  The set is cleared when the hotspot
starts so a new session shows the page again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: acknowledge on Continue, not on view

iOS fetches the portal page in the background before opening the sheet,
then re-probes after the sheet's own navigation.  Acknowledging on view
made that re-probe return "Success" as the sheet's content.  Phones only
re-check the network (and offer "Done") after a full-page navigation, so
the welcome page now has a Continue button leading to /portal/continue,
which is the only place a client is acknowledged.  The Continue page says
to tap Done and links to WROLPi and the Controller.

Also hijack clients3.google.com (older Android probe host).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* help: bump wrolpi-help to the captive portal and Speed Test docs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* captive portal: comments describe the Continue design; clear acks on stop

Three comments still described acknowledging on view, which c72a7f7b
removed.  Acknowledged clients are now also cleared when the hotspot
stops, so a lease reused in a later session sees the portal page.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 20:10:02 -06:00
lrnselfreliance
91f92683ba
Speed Test calculator: ping, download, and upload against the API (#708)
* Speed Test calculator: ping, download, and upload against the API

Users asking whether the hotspot is fast enough, or how many people can
stream video at once, had no objective tool that works offline.  This
adds one under Calculators > Tools & Reference.

Backend (wrolpi/speedtest.py, /api/speedtest):
* ping: tiny no-store JSON the browser times for round trip latency.
* download: streams slices of a pre-built random buffer for a clamped
  number of seconds (0.1-15).  Random so no proxy or compression can
  flatter the result; slices so no bytes are generated per request.
* upload: streaming-body handler that reads and discards the body and
  reports bytes and elapsed time.  Requests over 8 MiB are refused.
* info: the client address Caddy forwarded plus how many tests are in
  flight.  Sanic now trusts one proxy hop (PROXIES_COUNT = 1) so
  request.remote_addr is the browser, not Caddy.  Nothing else read
  that field before.

There is deliberately no one-test-at-a-time lock.  Several people
testing at once is how a user learns how the link shares between
streamers, so the server only counts in-flight tests (a shared_ctx
Value created pre-fork so every worker sees the same number) and the
UI reports how many others ran alongside.

Both Caddyfiles route /api/speedtest/* unbuffered, as the SSE chat
route already is, so throughput is measured live rather than in bursts.

Frontend:
* speedtest.js holds the logic with no React in it (bucketed
  throughput, steady-state average that drops the first second, ping
  stats, stream-capacity tiers, hotspot address check) so Jest drives
  it with fake fetch and streams.
* SpeedTestCalculator runs 20 pings, a 10 s download, and a 10 s upload
  with a live figure and rolling trace, then shows Mbps and MB/s, ping
  verdict, the client address with a hotspot badge, and a table of how
  many 480p..4K streams the download speed supports.  Upload is
  sequential POSTs of one 4 MiB Blob because Safari cannot stream a
  request body.  Nothing is saved.
* Sparkline is a new inline-SVG ui component (no chart dependency),
  added to /theme-sample and checked in all four themes.  A missing
  sample is a gap, not a plunge to zero.

The page states that it measures the network path, not the drive that
actually serves video files.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* speedtest: bound uploads in time, close the Start race, address review

Review on #708 found three bugs and some polish:

* Start could be clicked again while the first fetchInfo() was still
  pending, because the phase only left `idle` after that await.  A
  second click made a second AbortController and a second concurrent
  run, and Cancel only reached the later one.  start() now refuses when
  a controller already exists and enters a `starting` phase
  synchronously, so the button is Cancel before the first await.

* The upload phase could overrun the advertised 10 s by however long
  the in-flight 4 MiB POST took: on a slow hotspot, the very link this
  tool exists for, that is tens of seconds.  runUpload() now aborts the
  in-flight request at the phase deadline (counting only completed
  requests) and uses 1 MiB bodies so a few-Mbps link still completes
  several inside the phase.

* The server upload sink was capped by size only, so a slow or paused
  body could hold a worker until 8 MiB eventually arrived.  It now also
  enforces MAX_UPLOAD_SECONDS (15 s, matching download) and answers
  408 when exceeded.  Both caps together are what "cannot hold a
  worker" actually requires.

Also: a NaN `seconds` now gets a NaN message instead of "must not be
negative"; the module, Sparkline, and gallery docstrings are trimmed
to purpose; and the upload counter is covered for the mid-stream 413,
the 408, and a broken body.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 18:47:18 -06:00
lrnselfreliance
e0fdb762fc
zim: recompute the outdated-Zims flag after startup and on Delete (#706)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* zim: recompute the outdated-Zims flag after startup and on Delete

The Dashboard shows the "Outdated Zim Files" banner from the persisted
`outdated_zims` flag, but the modal lists files from a live scan.  The two
disagreed: the flag was restored from the DB on startup and never
recomputed once the outdated file had been removed, so the banner showed
with nothing to delete, and the modal displayed an endless placeholder.

- The startup recompute now runs after the DB is up and configs are
  imported, so it overrides the value `init_flags` restored from the DB.
  Previously it ran before the DB was up, and the restored value won.
- DELETE /api/zim/outdated always recomputes the flag, even when nothing
  was deleted, so clicking Delete clears a stale banner.
- The modal re-scans when opened and says when nothing was found instead
  of showing a placeholder.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* zim: enforce restore-then-scan ordering; no Kiwix restart on empty delete

- Startup calls flags.init_flags() then flag_outdated_zim_files() right
  after db_up, before the DB config import, so the disk scan always
  overwrites the DB-restored value in one local sequence.
- DELETE /api/zim/outdated recomputes the flag in a finally.
- remove_outdated_zim_files skips refresh and the Kiwix restart when no
  file was deleted.
- Trim narrative comments to the constraint they enforce.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-12 15:19:36 -06:00
lrnselfreliance
0b61d3ab08
videos: codec download preferences with transcode and strict-fail options (#701)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
* videos: codec download preferences with transcode and strict-fail options

Add video_codecs/audio_codecs ordered preference lists to the video,
channel, and RSS downloaders, alongside the existing resolution
selector.  Codec preference is codec-major: a preferred codec at a
lower resolution beats another codec at a higher resolution.

When 'transcode' is enabled, a downloaded file whose streams do not
match the preferences is re-encoded with ffmpeg after download (only
the mismatching stream; targets are extensible maps, currently h264
and aac/opus/mp3).  When 'strict_codecs' is enabled and transcode is
not, a download with no matching format fails permanently instead of
falling back.

Empty codec lists produce a byte-identical yt-dlp command to before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* videos: honor strict_codecs when transcode cannot produce the preference

Review fixes:
- strict_codecs is now suppressed only when transcoding can produce a
  preferred codec for every non-empty preference list (new
  transcode_can_satisfy_codecs); av1-only preferences with transcode
  enabled fail under strict instead of silently keeping the file.
- No partial transcode: an unfixable video mismatch blocks an
  audio-only conversion (and vice versa); the mismatch falls through
  to strict/keep.
- fsync the transcode output before rename, and the directory after,
  so power loss on a Pi cannot leave a truncated final file.
- mp3 yt-dlp pattern matches mp4a.6B as well as mp4a.6b.
- ffmpeg maps use 0:V:0 (excludes attached pictures) and 0🅰️0?
  (tolerates missing audio).
- Strict toggle is no longer disabled by transcode in the UI; popup
  wording corrected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* videos: serialize transcodes machine-wide via shared_ctx lock

Sanic runs multiple worker processes, so an asyncio primitive cannot
serialize transcodes across download workers.  A multiprocessing.Lock
in shared_ctx now guarantees only one ffmpeg transcode runs at a time
on the machine; waiting workers poll the lock asynchronously so the
event loop stays responsive while a transcode holds it for hours.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* videos: transcode-lock review fixes

- Document that a crashed lock holder requires an API restart; this is
  accepted rather than adding owner-death recovery.
- Move the 'Transcoding' user notification to after the lock is
  acquired, so it fires when work begins rather than before a
  potentially hours-long wait.
- Soften the lock test docstring to what it measures (in-process
  wait/release); trim the acquire docstring to the polling rationale.
- Give transcode_lock its own section header in contexts.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-07 09:12:00 -06:00
lrnselfreliance
88d5d55283
files: left-align the preview toolbar (#698)
* files: left-align the preview toolbar, Open joins the button row

The modal actions row packs its children to the right, which suits a
confirming button but stranded the preview's toolbar. The toolbar now
fills the row so its buttons read from the left, with Open first in
the same row as Download. On phones the tags selector always takes its
own line instead of squeezing in beside the buttons.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* tags: the add-tag button is violet like every other tag button

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* share: a real button everywhere but the navbar corner

The share trigger was a bare anchor-wrapped glyph, which matched the
navbar's other corner icons but looked out of place beside real buttons
in the file preview toolbar. It is an IconButton by default now; the
navbar keeps the bare glyph via asIcon.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-03 16:03:17 -06:00
lrnselfreliance
9b8ebdc9b3
files: readable text previews in dark themes (#697)
* files: readable text previews in dark themes

A same-origin text document inherits the page's color-scheme, so dark
themes rendered the browser's default light text on the preview's
forced white background. Force a light color-scheme on the preview
iframe so text is always black on white.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* files: text previews follow the theme

Instead of pinning the preview iframe to a light scheme, follow the
theme: dark, night, and amber render the document light-on-dark against
the theme's panel color; light stays black-on-white.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-03 15:47:44 -06:00
lrnselfreliance
97a6566e3d
Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page (#678)
* Fix API stalls: non-blocking switch worker, threaded searches, indexed Videos page

Three response-time fixes found by benchmarking the read-only API endpoints
hot and cold on real libraries:

* `switch_worker` runs on the Sanic worker's event loop every 0.1s and called
  the blocking `multiprocessing.Event.wait(timeout=1)`.  With no switch pending
  (the normal state) this froze the loop for a second at a time; about a third
  of all requests, including trivial ones, took ~1,000 ms.  It now polls
  `is_set()` and returns immediately.

* The search handlers (videos, files, archives, docs, zims) and `/api/statistics`
  ran synchronous SQLite queries on the event loop, so one slow search stalled
  every other request on that worker.  They now run via `asyncio.to_thread`.

* The unfiltered Videos page (`POST /api/videos/search` with only an order)
  used `mimetype LIKE 'video/%' OR ...` plus `COUNT(*) OVER()`, which visits
  every video's index entry before LIMIT applies: seconds cold on a large
  library on a spinning disk, and ~10 s hot for a deep page.  It now seeks each
  exact mimetype on the (mimetype, effective_datetime) index for `limit + offset`
  rows and merges them, with the total from a separate count cached for 60 s.
  Results are identical to the generic query.

Also adds `scripts/api_perf.py` (times every read-only endpoint against a host,
optionally restarting the API over SSH for cold numbers) and
`scripts/api_perf_compare.py` (merges per-host results).  Host profiles live in a
gitignored JSON file; an example is included.

Measured on an 8-core docker host with a 10 GB library: median hot request
1,007 ms -> 22 ms; Videos page 1 hot 287 -> 108 ms, page 50 1,345 -> 120 ms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Serve the video description from its own endpoint instead of Video.__json__

`Video.__json__` computed `description` as `c_text or get_video_description()`,
where the fallback opens and parses the video's info json from disk.  The list
endpoints serialize every result with `__json__`, so any page containing videos
without an indexed description (common: many videos have none) read their info
json files on every request, cold, for a field the list never displays.  The
description is not part of search either: search runs against the FTS copy of
`c_text`, so the fallback never fed the index.

Drop `description` from `__json__` and serve it from
`GET /api/videos/<id>/description`, like comments and captions.  The endpoint
returns the indexed `c_text`, falling back to the info json only there, for a
row indexed before descriptions were stored.  The player page fetches it through
`useVideoExtras` alongside the comments.

Tests assert the list and detail responses never touch the info json, and cover
the endpoint's indexed, fallback, and not-found cases.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review: thread-local zim sessions, fast-path equivalence test, comments

* Zim search, single-zim search, and search estimates run libzim off the event
  loop, each opening its own session inside the worker thread rather than
  passing the request's SQLAlchemy Session across threads.
* The Videos fast-path test now runs the same requests through the generic
  query and asserts identical rows and paging for both date orders.  The one
  intentional difference is documented: past the end, the fast path still
  reports the true total.
* switch_worker returns before its try block so the empty path does not clear
  the event; its comment states the invariant only.  The copied "synchronous
  SQLite" comments are reduced to one line in one place.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 16:17:53 -06:00
lrnselfreliance
e66e1e9309
app: the navbar corner wrapped onto a second row on small phones (#672)
* app: the navbar corner wrapped onto a second row on small phones

Every status-icon slot carried a 1.5em separator margin -- ~80px across
three slots -- which wrapped the mobile bar's whole corner onto a second
row on any screen narrower than about 380px. The margin is now 0.25em on
both bars: the corner's gap and the slots' own padding still space the
icons, and on desktop the freed room just goes back to the links that
useOverflowNav folds into "More".

The mobile bar is extracted as MobileNav (mirroring DesktopNav) so a
Cypress suite can mount it and hold it to one row at phone widths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: prove 320px with a faithful fixture, trim the CSS narration

The suite claimed 360px covered narrower devices because the fixture was
wider than production, but the fixture also lacked the real theme
toggle's 0.8em right margin, so the safety margin was overstated. The
fixture now matches a phone -- 32px logo home link, the toggle's margin,
and a coarse pointer so the search link drops its keyboard hint, with a
test asserting the hint is really absent -- and 320px is measured
directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 18:37:23 -06:00
lrnselfreliance
b46faa005a
ui: a download form that emptied itself after every submit (#665)
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Controller Tests / controller-tests (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
* ui: a download form that emptied itself after every submit

A successful submit reset the whole form, so adding a second download meant
re-choosing the tags, the destination and the video settings that the first
one had already been given.

It now empties only the URLs.  Everything else stays for the next download,
and a Clear button beside Download puts the form back to its defaults when
the next one is unrelated.

`useForm` gains `clearPathsOnSuccess`; `clearOnSuccess` is untouched for the
forms that want the whole reset.

* review: the one download form that kept its URLs

Scrape took a `clearOnSuccess` prop that no caller passed, so mapping it to
the URL paths left create-mode scrape clearing nothing.  It follows the same
rule as Videos and Archive now: the edit form supplies `onSuccess`, and only
that keeps the form as it was.

`clearPaths` also wrote every path it was given, so a `urls` form gained a
`url: ''` it never declared -- which the comment beside it and the test named
for it both said it would not do.  It skips a path neither the defaults nor
the form has, and the test asserts the key is absent.
2026-08-13 17:14:30 -06:00
lrnselfreliance
4c71e84696
ui: two upload dialogs that had drifted apart (#664)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
* ui: two upload dialogs that had drifted apart

The dashboard and the file browser each had their own upload component: two
dropzones, two overwrite toggles, two copies of the progress-bar rendering,
against the same `useUploadFile` state.  The file browser's dialog was made
fullscreen with a drop target that grows; the dashboard's stayed a `large`
dialog with a strip to drop onto, because it was a different file.

They differ in one thing that is real: the file browser knows the destination
because the user selected it, and the dashboard has to ask.  That is now the
`destination` prop -- `undefined` asks, a string uploads there, and '' is the
media directory rather than a missing destination.

Everything else is one component, so the next change to either entry point
lands on both.

* review: an upload dialog that looped on the media directory

Pressing Upload in the file browser with nothing selected passes `''` -- the
media directory.  Three checks along that path read it as "no destination":

* The clearing effect ran, and `doClear` is a new function every render whose
  own setState scheduled the next one.  React ended it with "Maximum update
  depth exceeded", 16 warnings deep, on the default Upload path.  The effect
  now clears only while asking, and `doClear` is stable.
* The title read `Upload to: /media/wrolpi/`.
* `doUpload` bailed, so a drop on the media root uploaded nothing.

All three predate this branch; moving the dialog is what put them behind one
component.  Tests cover the title, what the hook is told, and that a fixed
destination is never the thing that empties the queue.
2026-08-13 15:15:04 -06:00
lrnselfreliance
82fdf9dd92
ui: an upload target smaller than the dialog around it (#663)
The upload dialog was 682x367 on a 1440x900 desktop -- 47% of the width, 41% of the
height -- and the drop target inside it was a strip about 90px tall.  A small thing to
drag a file onto, with the file browser it came from covering most of the screen behind.

Upload and Move ask for `fullScreen`, which is edge to edge on a phone and 90% above the
tablet breakpoint: filling a desktop reads as the app having been replaced rather than as
something opened over it.  Move holds a DirectorySearch above a row per path being moved,
which grows with the selection; 620px showed a handful and scrolled the rest.

Two things had to follow.  Mantine sizes a fullscreen dialog's content to the viewport but
leaves the body at its content height, so the footer landed two thirds up the screen with
a quarter of the viewport empty beneath it -- the body takes the leftover now and the
actions ride to the bottom of it.  And the drop target itself grows into the room, with a
floor so it stays large once progress bars claim their own space.

Rename and the confirmations keep their sizes: a fullscreen dialog for one text field or
one sentence reads as broken.  New Folder keeps `large` too -- it is a text field and a
path preview, the same shape as rename, not the directory picker its name suggests.

`size='fullscreen'` is a DIFFERENT thing -- full width, height from the content, which is
what a log or a preview wants.  Both are documented where the sizes are declared, and the
gallery grows a fullscreen example so the difference is visible in all four themes.

339 component tests pass, 69 suites / 1214 jest tests pass, npm run build is clean.
2026-08-13 13:06:18 -06:00