Commit graph

44 commits

Author SHA1 Message Date
Roland
f4303476cf ui: scale the whole interface by 10%, off the root font-size
The interface reads about 10% smaller than the pre-migration Semantic build, and the
drift is not one number.  Measured on the QA Pi at 1280px, Semantic's headings were
h1 28 / h2 24 / h3 18 / h4 15 / h5 14 against our 26 / 21 / 17 / 14 / 13, and its video
card a fixed 290px with an 18px title against our 238px with a 13px one -- while our
BODY text and chrome went the other way, a 16px base against Semantic's 14 and 42px
buttons against 36.  No per-component nudge reproduces what a user comparing the two
sees, which is the whole page at once.

Ten percent is not a fudge factor: our heading scale times 1.1 is 28.6 / 23.1 / 18.7 /
15.4 / 14.3, which is Semantic's scale to within a pixel at every step.  Verified in
the browser -- those are the numbers it now renders.

`--ui-scale: 1.1` and `html { font-size: calc(100% * var(--ui-scale)) }`, with
everything that should grow written in rem:

- 102 lengths in ui.css, App.css and tokens.css converted, restricted by property:
  font-size, padding, margin, gap, width/height, border-radius, offsets.
- Borders, outlines and shadows deliberately NOT converted.  A hairline is a hairline
  at any text size.
- 46 inline `fontSize` px values across 8 files.  `em` values were left alone: they
  already resolve against the parent.
- Ten inline-styled blocks in the library (Card's text block, Loading, Placeholder,
  ColoredInput, Confirm's action row, CardGroup's gap) moved into ui.css, because an
  inline px size cannot be reached by a scale or seen by a test.
- CardGroup converts its `minWidth` prop to rem, which is what returns a phone to one
  card per row: as px the track ignored the scale, so a 430px screen fitted two 200px
  cards where the old build fitted one 290px card.

Mantine needs nothing: its 644 size declarations are `calc(Xrem * var(--mantine-scale))`
and scale with the root.  `theme.scale` must stay at 1 -- it exists to CANCEL a
customised root font-size, so setting it to 1.1 as well would leave Mantine's half of
the interface at the old size beside our larger text.  A test pins that.

`zoom: 1.1` on the root was tried first and rejected.  It works and it is one line, but
it scales at paint: every 1px hairline becomes 1.1px and softens on a 1x display, and
viewport units do not shrink to compensate, so `100vh` paints 110% of the screen -- a
95vw modal overflowed sideways and the CBZ viewer's fullscreen overflowed down.  Rem
scales real lengths, keeps borders crisp, leaves viewport units alone, and preserves a
user's own browser font-size preference instead of overriding it.

A percentage rather than a px root size, for that last reason.

Five guards in themes/ui-scale.test.js, each planted against: the scale itself, that
every scalable declaration is rem, that borders are NOT, that no inline px font-size
survives anywhere including the gallery, and that Mantine's scale stays 1.

Also removes `.ui.modal.themed-confirm`, a dead Semantic-era rule and the source of the
one `95vw` that would have overflowed: `.ui.modal` no longer exists and nothing sets
that class.

Verified: 1141 tests / 65 suites; clean build; browser-checked at 1280px and at a 430px
phone; navbar single-row from 780 to 1800px; hairlines still 1px and no horizontal
overflow in all four themes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 19:46:42 -06:00
Roland
155b490adb review: name the icon-only controls, and make two guards mean what they say
Acting on the review of #626.  No bugs; one suggestion and three nits.

The map's icon-only controls had no accessible name.  IconButton requires a
label by construction and AddToPlaylistButton supplies one, but the plain Button
and APIButton used for edit, delete and the catalog preview reached assistive tech
as unnamed buttons -- and my own Cypress fixtures carefully passed `aria-label`
on the same pattern while production did not, which is its own kind of tell.  All
four are named now, and named with the row's subject rather than the verb alone:
"Delete Water tank", not "Delete", because a table of pins otherwise offers a
screen reader a column of identical buttons.

A source guard holds that: it scans for controls that carry an icon and no label
text, and reports any without an `aria-label`.  Paired with a premise that it
found some, since a scanner matching nothing reads exactly like a clean file.

The Group guard asserted only that `<Group` appeared somewhere in the cell, which
would pass on an empty Group left beside the bare siblings it was meant to wrap.
It now checks that every control in the cell is inside the Group -- and the first
version of THAT was wrong too: it sliced from `<Group` to the end of the cell
rather than to `</Group>`, so controls after the closing tag still counted as
inside, and the empty-Group plant stayed green.  Slicing between the tags makes
it fail.

Coverage the review asked for: the Semantic size names the call sites actually
use (`small`, `big`, `huge`) rather than only the Mantine ones, since the
translation step was untested on the exact spellings that were broken; icon-only
buttons stay square while loading, when the glyph is replaced by a spinner sized
from the button's height; and `iconAfter`-only squares as well as `icon`-only.

On the size passthrough, which was flagged as the highest-risk change outside the
map: the buttons that grow are Settings' Shutdown and Reboot (`huge`, to 60px),
Settings' Upgrade and Save and Channels' Save (`big`, to 50px), and VideoPlayer's
Refresh (`large`, to 50px).  All six read as deliberate emphasis -- the two
biggest are the two destructive ones -- so they are left honoured rather than
re-broken at the call site.  `size='small'` resolves to `sm`, which is what the
accidental default already was, so the detail pages do not move.

1134 jest / 64 suites, 246 in ui-layout.cy.js, clean npm run build.
2026-08-02 17:58:25 -06:00
Roland
300d21f0ac ui: the map pin buttons were three sizes for three reasons
Reported as "all different sizes", and the pin row's three actions measured
46x30, 30x30 and 54x36.  Three causes.

APIButton dropped `size`.  `useAPIButton` declared `size = 'medium'` as a
parameter and then never put it into the props it hands to Button, so every
APIButton in the app rendered at Mantine's default regardless of what its call
site asked for -- seventeen call sites pass one, from `xs` on the map pins to
`big` and `huge` on Settings.  The delete pin was 36px tall beside two 30px
siblings because of it.  Passing the parameter default through would have fixed
the bug by breaking everything else -- every APIButton naming no size would have
grown from `sm` to `md` -- so the default is undefined now and Button's own
default stands.  Eight buttons on three other pages do grow as a result, which is
their call sites finally being honoured: VideoPlayer's `large`, one of Channels'
`big`, and Settings' two `big` and two `huge`.

An icon-only Button was not square.  A Button is as wide as its content plus its
horizontal padding; an ActionIcon is square by construction; and the pin row
mixes the two, because AddToPlaylistButton renders an IconButton when it has no
label.  Their heights already agreed -- ActionIcon's scale was remapped onto
Button's some commits ago -- and this is the other axis of the same alignment,
keyed off Mantine's own `--button-height` so the two stay in step through every
size.

They were also misaligned and touching, which is a third thing: bare inline-block
siblings align on text BASELINES, and a Button's baseline is not an ActionIcon's,
so the middle control sat 4.3px above the other two.  A Group, as every other row
of buttons in the app uses.

Written test-first at Roland's request.  Seven cypress cases went red on the
sizes -- widths 46 against 30, 54 against 36, 62 against 42, 70 against 50, and
"xs is shorter than lg: expected 36 to be below 32" for the dropped size -- with
both inverses green throughout: a labelled button stays as wide as its label, and
an APIButton naming no size keeps the height it had.

The alignment claim is measured in ui-layout.cy.js, including the premise that
bare siblings really are misaligned, since three controls of identical height not
lining up is not believable otherwise.  The call sites are held by a source
guard, because MapPins fetches from the API and builds its rows through
SortableTable.

That guard earned itself twice over.  Written for the pin rows, it found the
map-files subscribe cell putting a preview button beside a subscribe button with
the same defect.  Then a first attempt at a general "any cell with two controls"
rule had to be abandoned: a cell whose buttons are the arms of a ternary renders
only one and needs no Group, and excluding ternaries to allow for that also
excluded the subscribe cell, whose `role={isSubscribed ? ... : ...}` says nothing
about how many controls it holds.  Named cells with a stated scope beat a general
rule that quietly stops covering things.

Two slips of mine in that file, both caught by running it: `expect(value, message)`
is chai and throws in jest, and `onSave(pin.id, ...)` was not unique to the cell
it named -- the edit row's TextInput calls it from onKeyDown in a different cell,
so the check passed against the wrong one.

1132 jest / 64 suites, 243 in ui-layout.cy.js, clean npm run build.  Confirmed
live: three 30x30 controls, vertical spread 0, 10px gaps.
2026-08-02 17:58:25 -06:00
Roland
752b3f8c8a ui: only the top modal answers Escape, and the audited widths
Two things, both reported from the running app.

Nested modals could not exist.  Mantine binds a window-level keydown listener
per open modal, gated only on whether that modal is open -- nothing checks which
one the user is looking at:

    useWindowEvent('keydown', (event) => {
      if (event.key === 'Escape' && closeOnEscape && ... && opened) onClose();
    }, {capture: true});

Two open modals meant two listeners, so one Escape closed both.  Reproduced on
the dashboard: the download modal plus the search modal over it, one press, both
gone, back to the page.  Both also sat at z-index 200, so which one drew on top
was DOM order rather than which the user opened.

Mantine's own answer is Modal.Stack with a stackId, which is the only thing that
gates closeOnEscape.  Not used, for two reasons: every call site would have to
be rewritten to hoist its modal into a stack, and it sets `__hidden` on
everything below the top -- the parent disappears while the child is open.  The
parent should be dimmed and still there, as it was before the migration.

So the wrapper keeps the register itself: a module-level list of open modals,
most recent on top, from which each modal takes closeOnEscape, trapFocus and a
z-index two above its parent.  Module-level rather than React context because
the two modals in the bug are not nested in the JSX at all -- the search modal
is rendered by KeyboardShortcutsProvider near the app root and the download
modal by the dashboard.  They are siblings that happen to be open together, and
no context relates them.  The increment is small on purpose: Mantine puts
popovers at 300, and a stack that climbed past it would paint over the select
dropdowns inside its own forms.

Widths: Roland audited the modal list and chose sizes against Mantine's scale.
Sixteen get wider -- twelve to `large` (620), four to `fullscreen` -- and eleven
keep the width they had, with the size now written down so the decision is in
the source rather than in a default.  Flasher was passing Mantine's raw `sm`,
which worked but bypassed the naming the audit used; it says `tiny` now.

The size table itself is guarded, because it is the single thing that halved
every modal during the migration and a quiet edit to it moves seventy at once.
Semantic's widths, measured on the pre-migration build still running on the QA
Pi: mini 360, tiny 540, small 720, large 1080, no-size 900.  Mantine's, read out
of its stylesheet: 320/380/440/620/780.  A `small` modal went 720 -> 440 and one
with no size went 900 -> 440.  Three cases now hold the scale, the mapping, and
the rule that no call site may name a size the table does not know -- an unknown
name is passed to Mantine, ignored, and silently rendered at the default.

Tested in a real browser: none of it is visible to jsdom -- the listener is on
window with capture, the stacking is z-index, and "still visible behind" is
geometry.  Note that `should('be.visible')` is the wrong assertion for the
parent: it is an occlusion check, so a child drawn correctly over its parent
trips it.  The test uses a wide viewport and asserts the parent is rendered and
shows to both sides of the child.  Planting the pre-fix behaviour reddens three.

1124 jest / 62 suites, 8 in modal-stack.cy.js, clean npm run build.  The 12
failures in Tags/Channels/Download/Inventory.cy.js are the known pre-existing
Semantic-era selector failures, unchanged.
2026-08-02 16:01:46 -06:00
Roland
906705e5b4 ui: thirty-five files off Semantic, leaving fifteen
Phase 2, part 4.  Baseline 50 -> 15.

Migrated: all eight calculators, the rest of inventory and collections, the
Map/Zim/Docs cluster, Videos, Files, Search, Archive, DashboardPage,
DatesSelector, HelpModal, Events, Tags, Flasher's neighbours, admin Settings
and Status, Download, and App.

Also the three shared modules everything imports: api.js and customHooks.js
swap the Semantic toast library for the library's shim (same call shape), and
useForm.js renders the library's Message.  useForm accepts both the old
`{content, header, positive|negative}` shape and the new `{kind, title}`,
because the pages building those objects migrate separately.

Two library gaps closed, both found by call sites needing them:

  ActionInput is a forwardRef now.  Its ref reaches the input rather than the
  wrapper, because callers use it to focus or select the field -- a ref to the
  surrounding div would do neither, which is why one call site had resorted to
  hand-copying the component's markup.

  Header takes a `color` token name.  Three separate call sites had reached for
  an inline style to colour a heading that carries meaning ("3 items to add"
  against "2 to remove"), which is how a hex ends up in markup.

Two files needed repair after being interrupted mid-edit: MapViewer had a
Segment left in its error branch, and VideoPlayer had lost the useContext
import its own code still used.  Neither jest nor tsc noticed; only
`npm run build` did, which is the whole reason that check exists.

Fifteen files remain: Archive, BulkTagModal, Channels, Docs, FileBrowser,
FilePreview, Flasher, Nav, Playlists, Theme.tsx, Upload, VideoPlayer, Zim,
ControllerPage, admin/Downloads.  Theme.tsx is deliberately last -- the
unmigrated files still import its Semantic-backed wrappers, so it can only go
once they have.
2026-07-30 12:54:24 -06:00
lrnselfreliance
9a5d1f1630
Add ordered collections (playlists): polymorphic items + item API (#475)
* Add Playlists: ordered, mixed-type collections

A playlist (Collection kind='playlist') is a manually-curated, ordered list that
mixes item types. Squash of the playlists feature branch (24 commits).

Data model & API:
- CollectionItem is polymorphic via item_kind: 'file' (FileGroup), 'zim'
  (zim_id + zim_entry), or 'url' (link to anything the WROLPi browser opens,
  e.g. /map?lat=&lon=&z=). CHECK constraint enforces exactly one kind; per-kind
  unique constraints; migration a7b1c2d3e4f5.
- Endpoints: create playlist; add/remove/reorder items (idempotent adds: 201 new,
  200 existing); GET returns the ordered items. All mutations wrol_mode_check'd.

Config (source of truth):
- playlists.yaml (PlaylistsConfig) stores each playlist's metadata AND ordered
  items (file by relative path, zim by path+entry, url inline); round-trips a
  DB wipe. Supports backup import (DB_CONFIG_FILE_NAMES).
- The redundant generic CollectionsConfig (collections.yaml) is deleted:
  channels/domains own their configs, authors/subjects are re-derived.

On-disk sync (sync_playlists_directory switch):
- Each playlist materializes into the configurable Playlists Directory
  (playlists_destination, Settings UI input, excluded from indexing) with
  position-prefixed entries: NNNN_<file> hard links (copy fallback across
  filesystems) and NNNN_<title>.html stubs for zim/url items.
- Tagged playlists are namespaced like channels: <playlists>/<tag>/<name>;
  a tag change moves the directory and prunes the empty old tag dir.
- Deletion safety: managed files are only removed when they are our marked
  stubs or still have another hard link (st_nlink > 1) -- never a last copy.
  Stray loose files at the playlists root are removed; stub URLs are
  scheme-allowlisted (no javascript:/data:).

Frontend:
- /playlists list page (CollectionTable, name filter, New Playlist modal, Tag
  column) between Files and Zim in the nav.
- View-only detail page (Name link) and edit page (Edit button): themed table
  with model icons/colors + poster thumbnails, items link to their model page
  (video/archive/doc) or preview, up/down reorder, remove, add-URL form,
  single-tag selector; all editing disabled in WROL Mode.
- Reusable green "Add to Playlist" button on Video/Archive/Doc detail pages,
  the FilePreview toolbar, Zim search results, and Map pins (added as a /map
  URL item with a map-marker icon).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Playlist edit page parity: rename, custom directory, tag modal move suggestion

Match the ChannelEditPage/DomainEditPage UX on the playlist edit page:
Back/View buttons, an edit form (name, description, directory), and the
shared CollectionTagModal with the violet "Tag" button.  The tag modal now
suggests moving the playlist to its managed tag location
(playlists/<tag>/<name>); accepting the suggestion stores directory=None so
the sync keeps auto-managing the location.  Tag buttons on the Channel and
Domain edit pages are now violet to match.

Backend: update_collection learns rename (with per-kind validation and
duplicate checks) and playlist custom directories (cleanup of the abandoned
directory, config round-trip, commit-before-switch to avoid the background
sync racing pre-commit state).  tag_collection's duplicated untag/tag
branches are merged, and the per-kind config-save dispatch is consolidated
into _activate_config_save().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Deduplicate Collection error handling and Tag creation

Delete the duplicate UnknownCollection in wrolpi/collections/errors.py and
re-export the canonical one from wrolpi/errors.py, which carries
status_code=NOT_FOUND (the local copy inherited 500).  Replace the repeated
query/raise pattern in reorganize.py with Collection.find_by_id.

Add Tag.get_or_create_tag and use it for collection tagging; flush the new
Tag before returning so callers can read .id immediately -- previously
tagging a collection with a brand-new tag name set tag_id=None.  Regression
test included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 19:46:34 -06:00
Roland
c726654de9
Map search added to global search. Preview cards are displayed for all results in new Map tab.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
Map viewer has simple search.

Map search databases can be downloaded from CDN, or built locally. Map searches can be built offline, but will not have enriched things like state names downloaded from Wikidata.
2026-04-13 12:17:08 -06:00
Roland
b8a532c8d9
Key changes:
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
  and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
  sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
  renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
8fc98741d0
Adding noreferrer to all external links. 2026-02-15 17:26:36 -07:00
Roland
56419cf5a2
Convert ManageMap page to use SortableTable. 2026-02-15 14:16:35 -07:00
Roland
98ea05b689
Reworking FileBrowser selection using context to reduce prop drilling. 2026-02-03 16:36:44 -07:00
Roland
8ea3ea8a67
Updating code to use new version of react-router. 2025-12-07 11:54:20 -07:00
Roland
8a7f0737f0 Reformatting code. 2025-12-06 15:11:37 -07:00
Roland
ee36b90e8e
Renaming HelpPopup to InfoPopup to avoid confusion with future Help features. 2025-01-24 14:28:04 -07:00
Roland
f7bcf29cce
Creating useMessageDismissal which can be used to dismiss hint messages to the user. 2025-01-22 18:58:18 -07:00
Roland
d8b3db9d63
Using iostat to detect busy disk issues, reporting them in the UI. Detecting under-voltage/over-current issues using dmesg and reporting them in the UI. 2025-01-10 11:33:21 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
2ad363281f
Delaying map startup because it prevents the API from starting. 2024-10-16 18:56:00 -06:00
Roland
f63a3cc0a6
Fixing map iframe help. 2024-06-18 11:35:42 -06:00
Roland
bd431165e5
Displaying helpful error when iframes do not load. 2024-06-17 10:40:30 -06:00
Roland
38de2f9b40
Moving HTTP kiwix/help services to 9085/9086 ports. Nginx will serve HTTPS proxy for kiwix/help services at the old ports. Modifying apache2 to use HTTPS to serve map.
Using new https_proxy docker container to serve HTTPS for zim, map and help.
2024-06-12 21:38:51 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
Python 3.11 is now required, old versions were causing unpredictability in tests.  (Sanic does not yet
support 3.12)

Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
 https://github.com/sanic-org/sanic/issues/2921

New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes.  The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation.  Now all multiprocessing tools are attached to the
app.shared_ctx.  This means `api_app` is imported in many, many places.

This forced a change in how the DownloadManager works.  Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue.  Now, a single worker checks
for new downloads and sends a Sanic signal.

Flags have been reworked to use the `api_app`.  I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
62c4afce0b
Modifying height of iframes for Map/Zim viewers to show full page of iframe. 2024-04-16 10:34:05 -06:00
Roland
619a88c429
Reworking default/empty/error states for all pages. Null is pending, undefined is error. 2023-09-07 20:13:36 -06:00
Roland
81e661d14c
Combining information messages into <InfoMessage/>. 2023-08-23 20:14:23 -06:00
Roland
b1f19c8903
Using <APIButton/> for map import button. 2023-08-15 15:12:45 -06:00
Roland
59c4ab2502
Replacing icon property when icon has been requested.
Creating <APIButton/> which handles success/failure/pending during API calls.
2023-07-21 16:52:15 -06:00
Roland
03a2df217d
Zim/Kiwix (Wikipedia) feature
* A new zim module has been added, this module allows several things to interact with Zim files.
  * Allow searching of Zim files using libzim.
  * Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
  * Creation of a new zim docker container which allows viewing of the Zim file contents.

* This commit also adds move/rename/mkdir features in the File Browser.
  * File tags are preserved when moving/renaming.
2023-07-10 21:30:40 -06:00
Roland
4d5eba59f7
Changing primary button colors to violet. 2023-04-05 10:18:52 -06:00
Roland
1812a1690f
Matching return style in all js. 2023-02-17 16:09:24 -07:00
Roland
0621cd184d
Reworking empty-states for all pages and tables. Removing custom isEmpty in favor of lodash. 2023-01-04 09:14:33 -07:00
Roland
a1c74a3443
Adding osmium note in docker import warning. 2022-11-05 14:38:51 -06:00
Roland
d15972ddbe
Increasing slow map import warning temperature. 2022-11-04 10:01:11 -06:00
Roland
dad276d6ec
Creating isEmpty function to check if dict/array have some contents. 2022-11-04 09:58:38 -06:00
Roland
2cd458930a
Moving map import messages into their own functions. 2022-11-04 09:03:13 -06:00
Roland
6854fa3692
Handling merging multiple pbf files in map importing. 2022-10-27 20:22:11 -06:00
Roland
5bf648001b
Adding useTitle to every function page. 2022-10-25 15:57:44 -06:00
Roland
7fa89dca58
Using Themed Loader when it has a label. 2022-10-25 15:19:05 -06:00
Roland
b928b37b8d
Using unthemed Loader in ManageMap. 2022-10-25 15:13:18 -06:00
dependabot[bot]
d4ed5fa5ce Fixing JS warnings.
Handling all combinations of toggle in light and dark.

Changing CPUUsageProgress colors to be more readable in dark mode.

Moving theme handling into Theme.js.  Importing element wrappers from Theme.js to handle element theming.

Changing video menu to dark mode.

Creating ThemeContext to share Dark Mode setting.

Removing duplicate code for search functions.

Displaying X when downloads are disabled.

Using react context to share settings across components.

Refusing to start workers when WROL Mode is enabled.

Changing warning to debug.

Queuing recurring downloads first.

Adding Video.url index.

Adding special handling for youtube shorts to videos.

Reworking Downloads to have one worker per process.

Testing creating downloads and getting downloaders.

Testing video validation and poster generation.  Testing video statistics.

Increasing inventory test coverage.

Adding pytest.ini to automatically detect async test.

Adding more archive test coverage.

Testing Archive CRUD.

Catching and logging when encountering error during indexing.

Using FileIcon when Archive has no screenshot.

A Video will be assigned to a Channel by its info json first, then to its directory.

Including suffix in File json.

Adding index for File.suffix.

Gathering suffix during indexing.

Passing entire file down to FileIcon.

Moving logging up.

Returning File.associated in file json.

Associated files are last in file search.

Removing duplicate refreshing code by creating new _refresh_directory function.

Changing File.idempotency to a datetime to avoid having to clear idempotency.

Truncating indexes to avoid error with large files.

Don't overwrite video data when info json is empty.

Using already_downloaded for channel videos.

Awaiting refresh in Channel.refresh.

Testing that videos already downloaded aren't downloaded again.

Adding indexes for video/archive referenced paths.

Changing grid width for mobile or computer.

Adding button to refresh specific directories.

Restoring Channel edit link.

Fixing caption path in video_modeler.

Adding ZIP files to mimetype colors.

Using model instead of first file's model.

Fetching File models in batches to avoid issuing a SELECT for every File with a model.

Flushing generated poster file.

Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id.

Attaching Video to it's Channel during download.

Using ExternalCardLink for ArchivePage singlefile link.

Validating and indexing videos after download.

Handling glob when file name contains matching characters.

Reworking and testing split_path_stem_and_suffix and glob_shared_stem.

Adding associated video files to DB during upsert.

Adding pending downloads to dashboard.

Hiding status when download is open.

Handling when load is empty.

Adding load statistics to Dashboard.

Limiting concurrent after_refresh functions.

Handling when no onSelect is declared.

Adding ImageRow for FileTable.

Moving image date into Card.Meta.

Using ExternalLink for FileCard.

Using ExternalLink for ImageCard.

Handling when error occurs during refresh of a directory.  Committing during refresh to avoid losing all files if an error occurs.

Removing api health check to avoid spamming logs.

Creating ImageCard to display image posters when they aren't too large.

Removing archive.domain_id not null.

Reworking file refresh to process all files in a single directory.

Bumping version for major file rework.

No longer checking for video corruption.

Adding duration by default to speed up tests.

Using register_indexer for all indexers.

Adding CPU usage progress bar in Welcome.

Modifying TextIndexer to ignore large text files.

Counting all files during refresh.

Adding refresh_logger to clarify logging.  Checking for wrol_mode before refresh functions.

Removing unused idempotency from video/channel tables.

Using magic module to speed up mimetype identification.

All tests pass.

Archive tests pass.

Changing video get endpoint to return 'file' to avoid ambiguity.

Video tests pass.

Simplifying regex filtering.

Using regex to ignore case when filtering.

Converting all channel classes to functions.

Restoring channel refresh and download buttons.

Matching format of Archive and Video pages.

Adding sorting to Archives search.

Reworking video orders for file search.  Using SearchInput for videos search.

Archives can be searched on the Archives page.  Domains can be filtered.

Reworking Archive search to use Files.  Videos, Archives and Global search use the FilesView.

Videos are claimed by their Channel based on the Channel's directory.  Prev and Next video cards are displayed on the VideoPage.

Creating useVideo for reworked VideoWrapper and VideoPage.

Adding Paginator to FilesView.

Moving file refresh button out of manage tab.  Removing Files page tabs.  Limiting refresh to one at a time.

Adding class for card icon to match poster height.

Video and Archives always re-claim their files on refresh.

Archive files are indexed and claimed.

FileView table is compact and striped.

Videos page uses FileView.

Updated navlinks to router v6.

Removing ! character from file names.

Disabling select button when cards are visible.

Adding show/hide button for FileTable selection.

The Archive Page is displayed.

Adding mimetype dropdown.

Upgrading to React Router v6 for useSearchParams.

Validating video during modeling.  Removing screenshot during corruption check because it was very slow.  Updating app to use video files as search results.

Updating video search to return Files.

Video and WROLPi tests pass.

Creating function to refresh files within a channel's directory.

Removing troubleshooting print.  Updating result order in test.

Reworking video_factory to handle a specific video file location.

Returning video model on file search.

Getting session using video_file.

Screenshotting when checking for video corruption.

Reducing ffmpeg calls by gathering duration during poster generation.

Moving video file matching into model.

VideoIndexer is moved into video module.  Video files are validated.

Moving files module into wrolpi in preparation to use it in modules.

Title and contents are indexed for text file.

Files are indexed on refresh by using their indexer.

Zip files are indexed using their contents.

Testing multiple zip files.

Adding unique to file.path.  Checking that the correct indexer is used.

Adding index to file.modification_datetime and file.size.

Handling file info gathering in File.do_index.  Only reading contents of file if file.indexed is False.

Video model is found during file refresh.  A Video record is created that has the associated files.

Videos without associated files are found.

Videos are found using registered video_modeler() rather than in refresh code.

Removing old video columns.

Handling video info json files.

Trying to import each module individually.  Will continue to import modules even if one fails.

Reworking refresh so it performs less DB calls.

Converting file.path to primary key.

file.path is always a pathlib.Path.

Updating video_factory to use new file system, testing it.

Bump sanic from 21.9.1 to 21.12.2 in /docker/archive

Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2)

---
updated-dependencies:
- dependency-name: sanic
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-09-23 11:25:55 -06:00
Roland
6a654cd3af
Wrapping most API requests in trys/catches. 2022-05-01 21:03:30 -06:00
Roland
5178f72b66
Reworking map import to handle postgres data dumps. 2022-03-31 07:31:15 -06:00
Roland
c815aa68c6
Calculating time-to-import for map pbf files in the backend. These times were created using a test RPi4. Encoding URL of archive files to prevent error which blocked displaying of archives. 2022-03-12 09:50:16 -07:00
Roland
888352c984
Squashed commit of the following:
commit 26bde0fb48d59a86c2fb9f53a7218d49f22d4860
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 8 13:25:56 2022 -0700

    Using osm2pgsql append to preserve other imported maps.

commit c6c52623eaf02e691b1e75dcf39c3a8334ead55a
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 8 08:20:38 2022 -0700

    Using placeholder instead of loader when table is loading.

commit 44fec6490e26603c88a98f6bc70fc18686e17f09
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:58:34 2022 -0700

    Simplifying tableRow function.

commit 11b8b57f1fa59556397620f3cad9a33a0237501d
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:53:51 2022 -0700

    Removing scientific humanFileSize.

commit 0a09d0a6963afebed7f388aad71bbdd3151d2e9e
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:46:22 2022 -0700

    Fixing active loader.

commit 07f4deeac861351d7483ef57278eb2e0941e31fb
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:40:44 2022 -0700

    Increasing recommended map file size.

commit af3ac492c81f4c8f84563a448bc272ae9334d32d
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:35:13 2022 -0700

    Reworking logic order for simplicity.  Fetching downloads on interval.

commit 3d9d31fcec87ed02ced7f67c34907faa5a3b54e5
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:24:39 2022 -0700

    Handling when pbf import status are loading.

commit 3798961a48e7c9eb165ca4e4051f239dbb088d0c
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:53:50 2022 -0700

    Fetching import status every 30 seconds.

commit f7a8193b0955d14ead8090aa74ce3c5a681b57ec
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:33:17 2022 -0700

    Stopping renderd during installation.

commit 5bf050b59e0a6e391184596f5e6f0737dea1221d
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:26:59 2022 -0700

    Removing duplciate serve.

commit 892e3315eeeb5d2516996b901a32a73a66dae078
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:22:14 2022 -0700

    Setting bash flags in sub-install script.

commit d5d2acd2cd85e089c78aa85a27c5d966222a1347
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:20:19 2022 -0700

    Moving bulk of installation script into separate script so the latest code will be used during initial installation.

commit ab030d8af834a4a8772edd06df360c51e72fbb02
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 18:40:34 2022 -0700

    Removing mod_tile/ajt directory instead of entire contents of mod_tile.  Showing import as relative to media directory.

commit c05199db4ecf56f4d096051b58689457882af12a
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 18:39:59 2022 -0700

    Remvoing marker from map.

commit ba5df4cf62d61b55281277a6254836e973c766ae
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 11:57:41 2022 -0700

    Creating import_map.sh script.

commit 1a16e46130e775efe375adb22917ba12e44855cd
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 11:11:01 2022 -0700

    Increasing ram usage during import to avoid hanging import.

commit 0f5ce1a9f806f1c0878dd804b04587b6340ffd16
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 19:19:47 2022 -0700

    Increasing map_file.size size.

commit eb6e01512af59fb4a18ab15643c7bc61f3d48e53
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:55:55 2022 -0700

    Copying leaflet js/css files into docker container and to Apache directory so they will be available offline.

commit 5b687324e119f021673f42e8d079b83034b70d53
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:36:06 2022 -0700

    Matching tab height on Map page.

commit 8031786d672e18366e9a2eb2d7bdfcd8611d27f3
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:34:20 2022 -0700

    Shrinking HelpPopup Icon.

commit 5ea94488e43bbc5c414e61c9173a3cfa908c7104
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:31:30 2022 -0700

    Testing if no PBFs are requested to be imported.

commit c4e86be1faed78bd3af1f3487471c01a1958a433
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:18:48 2022 -0700

    Using Icon instead of Button for HelpPopup.

commit 51d94f6d85885803813544fea16a56372babaa82
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:07:03 2022 -0700

    PBF files are sorted by the api.

commit 40f3a7ff04a83db0153f64a88a37f1d56ce043aa
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Mar 4 18:35:51 2022 -0700

    Adding help popup.

commit 1da8d9fa403ec2a3b3c16c11a3937de37c895a35
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Mar 4 09:41:01 2022 -0700

    Moving message up.  Clearing cached tiles after import.

commit 3192adf95192554c194c9e0a8aeb149822462682
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Mar 3 21:37:13 2022 -0700

    Removing osm2pgsql from api docker because it seems to be unsupported.

commit 0e10522f3cdc7edd374c14f230bb2dbd68ab1a94
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Mar 3 21:36:39 2022 -0700

    Disabling import button for docker.  Adding download instructions.

commit e1d8f55d159fb4d4ef2ece859ef7cc1059885791
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Mar 3 21:14:55 2022 -0700

    Map files can be imported on RPi.

commit 3128530e09968430a71b27cd2e75a567f85a708b
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Mar 3 15:31:54 2022 -0700

    Adding shm_size to increase size to render large map tiles.

commit f3cf80fae0ce46f44ea9ac5033f68a32d0197ae3
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Mar 2 10:39:18 2022 -0700

    Can get status of map imports.

commit 3b7bbce004cd55a5d89980c2c09124b72c450bc6
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 1 20:51:13 2022 -0700

    Adding map link to 50x.html.  Swapping links using hostname to make it easier on the user.

commit a8b6b60fe182233f40c2f1ecd825636c3d37fd23
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 1 12:01:50 2022 -0700

    Moving Inventory link to end.

commit c5346b8dc2ff83262d6fd153130912809dfd7574
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 1 12:00:47 2022 -0700

    Squashed commit of the following:

    commit 39f0b8f3fc049fffd140d551cd91d7d9dba6d416
    Author: Roland <roland@learningselfreliance.com>
    Date:   Tue Mar 1 12:00:13 2022 -0700

        Starting map refresh function.

    commit 2fac9ddf07e378389ddd85ca972f33b3323d30ac
    Author: Roland <roland@learningselfreliance.com>
    Date:   Mon Feb 28 16:43:47 2022 -0700

        Updating readme with map requirements.

    commit 8325e1ea89537128329e313ab4b5588f8809d1c9
    Author: Roland <roland@learningselfreliance.com>
    Date:   Mon Feb 28 16:05:35 2022 -0700

        Fixing order of map db init scripts.  Creating index.html to display map.  Using :8084 index file to display the map.

    commit 8f81539dd9d25fc26607f472c156d41ddcfc11f5
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 19:17:14 2022 -0700

        Installing apache2.

    commit c7aaef07f8291070aeaf5e7df553e34aca0719ec
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 18:41:07 2022 -0700

        Always configure postgres extensions.  Initialize gis with dc.

    commit aa725ed4b8ec46d17accebd96dd41508ea2d9caf
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 15:48:20 2022 -0700

        Map is displayed as iframe.

    commit de9bb92dcaf44996e7727adce7774f01895ddf52
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 12:16:56 2022 -0700

        Running script in openstreetmap directory.

    commit f277388095d2221aa6407643ab655f26dc349eb7
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 12:06:41 2022 -0700

        Bumping version for map.

    commit afd4aaeffb2aced2dc5c1a8f94038885c8287265
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 11:59:31 2022 -0700

        Checking that the installation script is on an rpi.

    commit a799382ceac3c201fc01d41fd63e37607282831a
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 11:49:21 2022 -0700

        Only running mod_tile install once.

    commit 0b98bb19cd4d86c9630a863cab4eb46410d1c9d1
    Author: Roland <roland@learningselfreliance.com>
    Date:   Fri Feb 25 11:31:21 2022 -0700

        Building out map install script.

    commit 7c799775c11df573fa6cd65451adea28c0832cb0
    Author: Roland <roland@learningselfreliance.com>
    Date:   Thu Feb 24 12:16:53 2022 -0700

        Creating OSM installation script.

    commit d3b93a3ee97d1ae4a00b6a0423ed59ea8b897ccd
    Author: Roland <roland@learningselfreliance.com>
    Date:   Tue Feb 22 15:39:04 2022 -0700

        Removing unused map module.
2022-03-08 13:26:03 -07:00