Commit graph

80 commits

Author SHA1 Message Date
Roland
63a77b87e6 Removing collection metadata. 2025-12-06 15:11:37 -07:00
Roland
2a1ed5ab2b Fixing alembic revisions to not use external models. 2025-12-06 15:11:37 -07:00
Roland
1d8738ace2 Running alembic check in cicd. 2025-12-06 15:11:37 -07:00
Roland
a1e0fb4e92 Creating a new Collection model for use in organizing FileGroups.
Replacing Channels and Domains with Collection.

Moving downloading relationships from Channel to Collection.

Created domains.yaml config to manage tagged domain collections.

Adding turn-key upgrade solution in the UI.
2025-12-06 15:11:37 -07:00
Roland
64660d721f
Fixing channel summary trigger.
Some checks failed
CodeQL / Analyze (push) Has been cancelled
2025-08-22 11:03:33 -06:00
Roland
75c05f6e25 Adding db trigger for file_group.effective_datetime. 2025-08-17 11:19:25 -06:00
Roland
25f9568154 Adding summary columns to Channel table. Adding triggers to keep columns updated. 2025-08-17 10:50:55 -06:00
Roland
338d961284 Adding file_group.effective_datetime to speed up video and archive browsing.
Some checks are pending
CodeQL / Analyze (push) Waiting to run
2025-08-16 11:38:22 -06:00
Roland
581d5eaf6e
Adding toggle to Channel page to stop downloading comments for certain channels. 2025-03-01 18:23:29 -07:00
Roland
3d6ccad0c0
Adding feature which adds toggles to Zim files to enable/disable auto search. 2025-02-02 12:43:04 -07:00
Roland
5621d47309
Video downloads have many more options. Downloads are migrated.
Storing video settings in session storage to help user start downloads consistently.

Applying channel tag name to new channels when download videos or channels.

Initializing map tiles when importing and resetting map.
2024-11-28 09:48:57 -07:00
Roland
ca51f3a268
Making Channel.tag_id migration lazier to avoid rare errors. 2024-10-16 18:57:04 -06:00
Roland
d34301a085
Configs are imported manually in main. Configs now have dump/save/import methods. Configs will refuse to save if they have not been imported. 2024-10-16 18:56:44 -06:00
Roland
d54735c4bb
Configs can be saved in background using switches. 2024-10-16 18:56:38 -06:00
Roland
317f3f3dfb
Top processes are displayed on Status page. Nice of renderd is lowered to avoid clobbering system at startup. 2024-10-16 18:56:27 -06:00
Roland
45167e4f32
A Channel can be tagged with a single Tag. The Channel can be moved into a Tag Directory according to the config. 2024-10-16 18:56:19 -06:00
Roland
218fb3e760
Replacing Tag names with escaped tag names to avoid errors in Tags Directory. 2024-10-16 18:56:10 -06:00
Roland
63738bd70f
Removing ChannelDownload in favor of Download.channel_id. 2024-10-16 18:55:54 -06:00
Roland
50d95dad19
Channels now have Channel.channel_downloads. A Channel can have many Downloads. 2024-10-16 18:55:46 -06:00
Roland
0fd545d2e8
Setting FileGroup.censored when refreshing channel files. 2024-06-17 14:48:41 -06:00
Roland
00fe799b8f
User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
Roland
758ff8e810
Creating background process that fetches comments for Videos that do not yet have comments.
Displaying star for Uploader's comments, heart for favorited comments.

Adding wrolpi help files desktop shortcut to rpi image.

Using zim file names as FileGroup.title.

Removing accidental output commit.
2024-03-01 17:15:56 -07:00
Roland
0629c8b464
Converting VideoPlayer to use FileGroup.url. Fixing downgrade to use FileGroup.url. 2023-10-29 21:13:38 -06:00
Roland
15687e534a
Moving data from models to FileGroup.
* Ebook.creator -> FileGroup.author
* Video.viewed -> FileGroup.viewed
* Video.duration -> FileGroup.length
* Video.url -> FileGroup.url
* Archive.url -> FileGroup.url
2023-10-29 17:31:54 -06:00
Roland
ae9d188391
Adding FileGroup.published_date/author/etc. Creating script which converts a Singlefile to an Archive FileGroup files.
Parsing SingleFile datetime using dates.strpdate.

Adding timezone when no timezone info is available.

Installing python-dateutil to assist in handling diverse datetime strings.

Reworking refresh progress to use idempotency to display more accurate progress.  Only logging that files have been invalidated if they used to be indexed.

Clearing archives to force re-index.

Ignoring 'Unknown' author from PDF metadata. Using file_title when pdf metadata title is empty.
2023-10-23 18:09:08 -06:00
Roland
6e0f9a999c
Combining install scripts into a new repair script. Removing create_db script in favor of repair script. 2023-10-03 12:15:29 -06:00
Roland
2147bf90e3
Removing Video.validated because it was only used in the script. 2023-08-15 13:17:30 -06:00
Roland
e8e5b5727f
Creating Video.validated which is used by find_corrupt_videos.py to search. 2023-08-08 13:42:55 -06:00
Roland
36083200e6
Creating Video.ffprobe_json column which stores ffprobe information about the video file. Creating find_corrupt_videos.py script which automatically re-downloads videos which do not have both video and audio streams. 2023-08-03 21:40:25 -06:00
Roland
03a2df217d
Zim/Kiwix (Wikipedia) feature
* A new zim module has been added, this module allows several things to interact with Zim files.
  * Allow searching of Zim files using libzim.
  * Allow downloading of Kiwix projects (wikipedia, wikibooks, etc) on a schedule which replaces the outdated files.
  * Creation of a new zim docker container which allows viewing of the Zim file contents.

* This commit also adds move/rename/mkdir features in the File Browser.
  * File tags are preserved when moving/renaming.
2023-07-10 21:30:40 -06:00
Roland
fedbbe31d7
Changing default Survival tag to Primitive. 2023-05-18 17:25:33 -06:00
Roland
7c1cc07365
Only joining FileGroup table when searching Archives when necessary. 2023-05-16 14:56:53 -06:00
Roland
0c0115dcd2
Adding Pests to default Tags. 2023-05-10 15:25:06 -06:00
Roland
387282b445
Adding my public gpg key.
Moving upload destination input out of drop Segment.

Disabling uploads when downloads are disabled.

Inverting labels for checkboxes in inverted modal.

Deleting file_groups which no longer contain the exclusive primary path.

Adding debug logging for all API errors.

Checking that upload output is not a directory.

Expanding nginx request size.

Adding docs to upload endpoint.

Using dot-circle icon for more image mimetypes.

Using Form.Field to make upload destination required.

Adding border to inverted Segments.

Many files can be uploaded in chunks.

Files can be uploaded, progress is displayed.

Showing white border around share QR code.

Displaying WROLPi colors for download icons.

Creating button to clear DirectorySearch.

Excluding channel/domain directories so other directories can be returned.

Ordering search results.

Ignoring when no channel is found only if we have a destination.

Limiting directory search results.

Displaying categories when searching destination directories.

Handling when live_chat.json file exists.

Allowing refresh of directory directly.

Logging which download was successful.

Removing timeout.

Creating index for directory names.

Sleeping to catch cancel.

Arbitrary files can be downloaded using the new FileDownloader.
2023-04-28 13:56:32 -06:00
Roland
d9a737d46b
Adding default to TagFile.created_at. 2023-04-10 08:41:35 -06:00
Roland
189f25399a
Adding WROL to default tags. 2023-04-09 17:14:22 -06:00
Roland
18273286dc
Removing FileGroup.full_stem in favor of primary_path. This is to handle when two unrelated files share a name ('some text.txt' and 'some text.jpg'). 2023-04-09 16:59:20 -06:00
Roland
92d0795000
Adding file_group_textsearch_idx index. 2023-04-04 16:01:31 -06:00
Roland
ae2b6daa33
FileBrowser files can be tagged. A Tag icon is displayed on each file row in the browser. 2023-04-01 11:52:41 -06:00
Roland
900f605d27
Replacing default tag colors. 2023-03-29 17:37:13 -06:00
Roland
a7c4fd1119
Displaying progress bar during refresh. 2023-03-27 20:46:31 -06:00
Roland
1dd19a8f5e
Reworking tag endpoints in a more restful way. 2023-03-27 16:17:33 -06:00
Roland
9b5c82219d
Adding indexes after file_groups. 2023-03-25 20:55:44 -06:00
Roland
5615ce542d
Making example label a tag.
Tags can be created and deleted in UI.

Adding tag_file using id to avoid session error.

Using UTC in test.

Matching tag_names style in test.

Tag icon is displayed on file rows.

Archives, Videos and Files can be filtered by their tags.

Tags can be added and removed from UI.

Tags are displayed in VideoPlayer.

The tags of a file are returned in it's JSON.

Video favorites are converted to Favorite tag on channel config import.

Updating tests to match frontend usage.

PDF creator and modification time are displayed.

Reworking file search to use Video.__json__ for complex video relations.

Archives can be viewed.

Removing ebook.title.

Reworking alembic migration to be an upgrade rather than a replace.

Skipping test on circleci.

Clearing out video caption index only if extract_subtitles if off.

Channel tests pass.

Some video tests pass.

Missing videos can be found.  A video can be downloaded.

Orphaned files can be found.

Videos can be tagged.

Videos can be searched.

Videos can be modeled.

Converting some tests to use FileGroup.primary_path.

Adding archive and domains to alembic migration.

Testing cascading Archive titles.

Archives now uses FileGroup.

archive.lib tests pass.

Archive is partially converted to FileGroup.

Calculating FileGroup.primary file on refresh, rather than on select.

wrolpi tests pass.

Using FilesType to convert between str/pathlib.Path for FileGroup.files.

Using list of files in FileGroup.files instead of dictionary.

Storing files in dedicated FileGroup.files to avoid overwriting them.

Adding comment explaining CASE.

Not overwriting FileGroup.data on refresh if no change has been detected.

Removing File model in favor of FileGroup.data.

Adding typing to Files methods.

Files are indexed using the primary mimetype of the FileGroup.

files test_lib tests pass.

Removing file_group_file in favor of simple one-to-many relationship.

PDF files are indexed during refresh.

Adding Tag and TagFile to tag files with specific tags.

Only iterating when full_stem_ids has contents.

Testing that FileGroups are related to their Files.

Files and file_groups are created and linked.

Bump webpack from 5.75.0 to 5.76.1 in /app (#160)

Bumps [webpack](https://github.com/webpack/webpack) from 5.75.0 to 5.76.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](https://github.com/webpack/webpack/compare/v5.75.0...v5.76.1)

---
updated-dependencies:
- dependency-name: webpack
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-25 17:54:41 -06:00
Roland
93f0e7494f
Creating Event and Flag systems. Events are used to send and display a toast to the App. Flags are used to signal if some action has been performed, or if a package is installed. 2022-12-23 16:04:09 -07:00
Roland
d8c166379b
Removing unused File columns and indexes. 2022-12-16 10:00:46 -07:00
Roland
6fcb992540
Storing example ebooks for testing.
Special handling of json mimetype for json files.

Moving types up to avoid overwriting mime.types.

Adding custom types to nginx to server ebooks better.

Displaying EBookCard correctly when ebook was not indexed correctly.

Global search can search for all ebooks (mobi/epub).

Testing when mimetype suffix is searched.

Adding eBook to global search filter list.  Opening mimetype search to match mimetypes that may not end with expected suffix.

Adding log warning to explain refresh speed.

Generating ebook covers after refresh.

Testing that another nearby book removes the cover.

Calibre covers can be discovered and associated with an ebook.

Handling if ebooklib is not yet installed.

Adding ebook table and modeler to allow searching the contents of epub files.

Marking all PDFs as un-indexed, so they will be indexed.

Replacing null characters in PDF text.

Testing PDFIndexer against example PDF file.

Installing PyPDF2 to extract text for PDFIndexer.
2022-12-11 12:36:53 -07:00
Roland
c106fdaaf7
Invalidating video files for orphaned video meta files. 2022-12-07 18:27:45 -07:00
Roland
d0d69519cd
Bumping version for new feature.
Skipping video epoch test on circleci.

Created script which finds and cleans up (renames/deletes) orphaned video meta files.

Reworking orphaned video query, some video files were leaking through.

Modifying File table to allow finding of orphaned video files.

Extracting epoch from video info_json for more accurate upload_date.

Extracting epoch from video info_json for more accurate upload_date.
2022-12-07 18:15:28 -07:00
Roland
d689f3b5e7
Creating index for archive.url. 2022-11-02 15:21:34 -06:00