Roland
b8a532c8d9
Key changes:
...
- Add MapLibre GL JS viewer with light/dark themes, terrain layer,
and embedded static map page for coordinate sharing
- Add map pin support with persistent map_pins.yaml config
- Add map region subscriptions and downloader with progress tracking,
sourcing pre-extracted PMTiles from a cold CDN
- Add map export to PNG and GPS coordinate copy via context menu
- Add MapViewer React component alongside reworked Map page
- Add map catalog and region-building script (build_map_regions.py)
- Remove OSM import/render scripts, leaflet files, Apache configs,
renderd service, and osm-map-dumper Docker container
- Update Caddyfile to serve PMTiles and map static assets
- Update build/install/repair scripts for Protomaps dependencies
- Drop map_file table, add/drop map_subscription table (alembic)
- Adding button to skip primary drive mounting during onboarding.
2026-04-04 13:48:19 -06:00
Roland
0ceac364ec
Hiding traceback when zim file does not match kiwix naming scheme.
2026-03-29 10:19:28 -06:00
Roland
4918ea5ef6
Creating Zim search endpoint that searches all auto-search zim files.
2026-03-20 14:16:56 -06:00
Roland
2e1cb4bfaf
Fixing bug in file refreshing where old file locations were not removed from the DB because they shared a their original stem.
2026-03-11 15:03:30 -06:00
Roland
71e3b307c3
Restarting wrolpi-kiwix when zim files are modeled.
2026-02-14 17:37:09 -07:00
Roland
9005c0deff
Adding toggle to disable the tags directory feature.
2026-01-31 18:08:42 -07:00
Roland
173284699d
FileWorker: Async queue-based file synchronization system
...
Introduces FileWorker for background file operations with dual-queue
architecture (public/private). Handles refresh, move, and rename tasks
with progress tracking, job completion monitoring, and batch processing.
Compares DB FileGroups to filesystem state using FileGroupDiff to detect
new/modified/deleted files. Designed for memory efficiency on Raspberry Pi.
2026-01-23 15:29:15 -07:00
Roland
2a27d354a5
Creating AutoAwaitTestClient for tests which automatically runs background tasks and expires db objects.
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2026-01-05 18:26:49 -07:00
Roland
7183cf5e85
Automatically restarting kiwix container when new zim is downloaded using new Controller.
2025-12-19 17:55:27 -07:00
Roland
3e3694869b
Fixing bug where not all archives were modeled.
2025-12-18 17:19:57 -07:00
Roland
10d34d7390
Fixing download config import issue.
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run
2025-12-16 19:59:28 -07:00
Roland
344fe9035f
Removing optional_session, passing session down from request middleware. Moving session param to the front of functions.
CodeQL / Analyze (push) Has been cancelled
Jest Tests / jest-tests (push) Has been cancelled
2025-12-11 16:03:18 -07:00
Roland
8a7f0737f0
Reformatting code.
2025-12-06 15:11:37 -07:00
Roland
5b29144d70
Testing that nearly-empty configs are ignored on import. Adding setting to disable automatic upgrade checking.
2025-12-06 15:11:37 -07:00
Roland
1d8738ace2
Running alembic check in cicd.
2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92
Creating a new Collection model for use in organizing FileGroups.
...
Replacing Channels and Domains with Collection.
Moving downloading relationships from Channel to Collection.
Created domains.yaml config to manage tagged domain collections.
Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
c635267eda
Fixing bug where Archive screenshots would not be tracked when a user uploads a screenshot.
CodeQL / Analyze (push) Has been cancelled
2025-10-12 17:19:54 -06:00
Roland
0ac1e559bf
Updating Zim downloading so that Zim files are indexed and modeled when downloaded with new style of modeling like Video or Archive.
2025-03-21 22:12:49 -06:00
Roland
f72ecf2682
Adding Khan Academy to zim subscriptions. Updating some kiwix catalog sizes.
2025-03-03 16:32:35 -07:00
Roland
581d5eaf6e
Adding toggle to Channel page to stop downloading comments for certain channels.
2025-03-01 18:23:29 -07:00
Roland
c82baa1ad8
Using upsert_file to handle Zim file download.
2025-02-18 12:19:16 -07:00
Roland
3d6ccad0c0
Adding feature which adds toggles to Zim files to enable/disable auto search.
2025-02-02 12:43:04 -07:00
Roland
531ec3d65e
Creating new /api/archive/upload which accepts a file from the SingleFile browser extension and converts it to an Archive.
2025-01-30 18:41:32 -07:00
Roland
220cb8010f
Using aria2c to download files. Removing custom file download functions.
2025-01-10 11:32:37 -07:00
Roland
7d6b7114a7
Adding TRACE level for logging beyond debug. Reworking async_client for tests, not all tests need the App to handle signals, so any test that needs signals can use the new await_switches fixture which primes the app.
2024-12-11 11:52:33 -07:00
Roland
fe521c99f3
Adding edit form for Scrape downloader.
2024-12-03 12:44:38 -07:00
Roland
b7d163e7ff
Config files are initialized on startup, if they are missing.
2024-11-28 18:13:43 -07:00
Roland
8909d28a6b
Updating zim downloader with better error.
2024-10-20 15:37:24 -06:00
Roland
82acca00b6
Changing aiohttp function to be context managers to simplify calls.
2024-10-16 18:56:49 -06:00
Roland
d34301a085
Configs are imported manually in main. Configs now have dump/save/import methods. Configs will refuse to save if they have not been imported.
2024-10-16 18:56:44 -06:00
Roland
d3e5b56520
Bumping python requirements versions.
2024-10-16 18:56:41 -06:00
Roland
d54735c4bb
Configs can be saved in background using switches.
2024-10-16 18:56:38 -06:00
Roland
0f232fd533
Moving FileGroups instead of refreshing files after they have been moved.
2024-10-16 18:56:25 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config.
2024-10-16 18:56:19 -06:00
Roland
c839acb270
Converting Tag methods to use tag id or name.
2024-10-16 18:56:01 -06:00
Roland
2ad363281f
Delaying map startup because it prevents the API from starting.
2024-10-16 18:56:00 -06:00
Roland
53d2402d32
Only files that have been tagged will be returned when new 'Any' tag button is used to filter file search results.
2024-10-16 18:55:56 -06:00
Roland
cdc73d9d16
Trimming video file name to avoid error when file name is too long.
2024-10-16 18:55:52 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads.
2024-10-16 18:55:46 -06:00
Roland
9ee73af61d
Converting APIError to use SanicException as base.
2024-05-01 18:04:25 -06:00
Roland
58de8c7c5b
Upgrading Sanic and yt-dlp. Major changes occurred because Sanic changed how multiprocessing works.
...
Python 3.11 is now required, old versions were causing unpredictability in tests. (Sanic does not yet
support 3.12)
Sanic has been upgraded to 23.6.0, which is the latest version that avoids this bug:
https://github.com/sanic-org/sanic/issues/2921
New strategy for multiprocessing is to create all multiprocessing tools in one process, then fork
to other processes. The previous strategy was to declare multiprocessing tools at the top of every
file, or wherever they were needed at import/creation. Now all multiprocessing tools are attached to the
app.shared_ctx. This means `api_app` is imported in many, many places.
This forced a change in how the DownloadManager works. Previously, it would continually run
download workers which would pull downloads from a multiprocessing.Queue. Now, a single worker checks
for new downloads and sends a Sanic signal.
Flags have been reworked to use the `api_app`. I removed the `which` flag functionality because the
`which` are called at import and needed their own multiprocessing.Event.
2024-04-30 10:55:25 -06:00
Roland
b05b2e7a6d
Allowing maintainer to change special directories.
2024-03-09 16:51:26 -07:00
Roland
758ff8e810
Creating background process that fetches comments for Videos that do not yet have comments.
...
Displaying star for Uploader's comments, heart for favorited comments.
Adding wrolpi help files desktop shortcut to rpi image.
Using zim file names as FileGroup.title.
Removing accidental output commit.
2024-03-01 17:15:56 -07:00
Roland
32346dedc8
Fixing download url for Electronics and Unix zim files.
2024-02-19 16:00:51 -07:00
Roland
4f4b2bdc72
Fixing Arduino download URL.
2024-02-13 14:22:25 -07:00
Roland
afecb62114
Adding medical wiki projects.
2024-02-13 14:16:16 -07:00
Roland
acaec9eacc
Testing special english stack overflow url.
2024-02-12 13:05:14 -07:00
Roland
80d9b74a03
Fixing english stackoverflow download url.
2024-02-12 12:51:12 -07:00
Roland
971eb4f99a
Adding many more Stack Exchange Zim subscriptions.
2024-02-12 12:40:56 -07:00
Roland
8b67dc87ff
Checking that tests are happening in tmp directory.
2024-02-05 14:52:38 -07:00
