rt/wrolpi
1
0
Fork 0
mirror of https://github.com/lrnselfreliance/wrolpi.git synced 2026-10-26 19:01:39 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
wrolpi/alembic/versions
History
Roland 73fafcd645
Search: fast ABC mode by default, opt-in deep search 
The file_group.textsearch tsvector indexes all four text columns;
d_text (captions, document text, zip contents) is ~90% of the data,
making ts_rank searches take seconds on large libraries.  Add a
textsearch_abc generated column (title/author/description only, ~8x
smaller) and default all searches to it, with deep=true opting into the
full column.

Database:
- Migration adds the generated column and a GIN index (rewrites the
  table; a few minutes on large libraries).

Backend:
- FilesSearchRequest, VideoSearchRequest, ArchiveSearchRequest, and
  DocSearchRequest accept `deep`; each search function picks the column.
- /api/search_file_estimates returns both counts in one index-bound
  query (COUNT FILTER on the abc subset): `file_groups` (fast) and
  `file_groups_deep`.

UI (one coherent story across the count surfaces):
- The typing dropdown always suggests the deep count (what is findable).
- The tab badge shows the active mode's count so it matches the result
  list.
- The gap is reconciled by a hint offering deep search: count-aware on
  the global search ("No results in fast search. N results available in
  deep search." / "N more results available..."), generic on the module
  pages (no deep estimate there).  The hint resets pagination.
- "Search Depth" toggle in the search filter modals, persisted as
  ?deep=true.

Tests that relied on d_text matches now pass deep=True, with fast-mode
assertions alongside.  Also skip flaky test_download_channel on
CircleCI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 10:32:16 -06:00
..
00f11f309c53_filegroup_censored.py Setting FileGroup.censored when refreshing channel files. 2024-06-17 14:48:41 -06:00
1a70196c05bb_download_location.py DownloadResult.try_again is not necessary. 2022-05-10 21:20:36 -06:00
1c041aeee0fc_doc_indexer.py User can select from/to years when searching files. Indexing doc and docx files. 2024-03-07 12:12:48 -07:00
1f2325523525_download_destination_migrate.py Video downloads have many more options. Downloads are migrated. 2024-11-28 09:48:57 -07:00
2bfb3b50b178_tag_names_migration.py Configs can be saved in background using switches. 2024-10-16 18:56:38 -06:00
2fb39c1861c7_create_mapfile_table.py Squashed commit of the following: 2022-03-08 13:26:03 -07:00
3b6918aeca4b_file_group.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
3c33b1dae069_video_validated.py Feature/video validate (#74) 2022-01-14 15:54:51 -07:00
4b54ad052e2e_reindex_archives.py Adding FileGroup.published_date/author/etc. Creating script which converts a Singlefile to an Archive FileGroup files. 2023-10-23 18:09:08 -06:00
4c8da5502c8a_drop_channel_skip_download_videos.py Combining install scripts into a new repair script. Removing create_db script in favor of repair script. 2023-10-03 12:15:29 -06:00
4f03b9548f6e_file_group_effective_date_trigger.py Adding db trigger for file_group.effective_datetime. 2025-08-17 11:19:25 -06:00
5ad9efbf3570_filegroup_length.py Converting VideoPlayer to use FileGroup.url. Fixing downgrade to use FileGroup.url. 2023-10-29 21:13:38 -06:00
6b0b249a4f81_archive_published_datetime.py Moving data from models to FileGroup. 2023-10-29 17:31:54 -06:00
8bdb48a486b9_video_ffprobe_json.py Creating Video.ffprobe_json column which stores ffprobe information about the video file. Creating find_corrupt_videos.py script which automatically re-downloads videos which do not have both video and audio streams. 2023-08-03 21:40:25 -06:00
8d0d81bc9c34_channel_channel_downloads.py Fixing alembic migration that imported non-existent code. 2026-05-29 11:15:32 -06:00
8df411232c17_video_indexes.py Creating indexes on video columns to speedup video search. 2021-12-24 14:53:47 -07:00
9ec4c765ef8d_remove_inventories_version.py Configs are imported manually in main. Configs now have dump/save/import methods. Configs will refuse to save if they have not been imported. 2024-10-16 18:56:44 -06:00
26c5a62ec683_make_filegroup_directory_not_null.py Creating FileGroup.directory, converting paths to relative to avoid updating all the paths on move. 2025-12-24 11:26:45 -07:00
32d96eb9122d_video_censored.py Creating indexes on video columns to speedup video search. 2021-12-24 14:53:47 -07:00
36cae041ad31_files.py Refresh files removed files that were deleted. 2022-02-02 08:50:03 -07:00
053f17f07c4e_init_inventory_tables.py A new start 2021-11-01 16:43:09 -06:00
53f6780aa599_drop_urls.py Fixing alembic revisions to not use external models. 2025-12-06 15:11:37 -07:00
54c19aa847da_channel_summary_indexes.py Fixing channel summary trigger. 2025-08-22 11:03:33 -06:00
62acddc55091_archive_textsearch_coalesce_fix.py Applying textsearch coalesce fix to archive.textsearch. 2022-01-04 11:14:15 -07:00
69e9754d1a63_filegroup_effective_datetime.py Adding file_group.effective_datetime to speed up video and archive browsing. 2025-08-16 11:38:22 -06:00
70db5ba8763d_files_rework.py Creating video_channel_id_idx index. 2022-09-23 12:31:11 -06:00
75d34a2b442d_archive_datetime_index.py Only joining FileGroup table when searching Archives when necessary. 2023-05-16 14:56:53 -06:00
76add63ed225_json_to_jsonb.py Converting JSON columns to JSONB for performance. 2026-01-06 06:23:32 -07:00
84b6993a6091_drop_channel_next_download.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
90ac16f9140a_remove_file_directory_and_stem.py Removing unused File columns and indexes. 2022-12-16 10:00:46 -07:00
170d1be52bc7_download_destination.py Video downloads have many more options. Downloads are migrated. 2024-11-28 09:48:57 -07:00
365c985893b8_ebook_creator.py Moving data from models to FileGroup. 2023-10-29 17:31:54 -06:00
577e414f1bc4_add_archive_collection_id_index.py Batch processing in get_collections to speed up collection pages. 2025-12-07 17:42:04 -07:00
717f36a4da94_ebook.py Storing example ebooks for testing. 2022-12-11 12:36:53 -07:00
751aecf46b88_zim.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
930b9b7adb79_wrolpi_flags.py Creating Event and Flag systems. Events are used to send and display a toast to the App. Flags are used to signal if some action has been performed, or if a package is installed. 2022-12-23 16:04:09 -07:00
1870bec7c81e_initialize_archive_tables.py A new start 2021-11-01 16:43:09 -06:00
6489cd50c889_file_stem_and_directory.py Bumping version for new feature. 2022-12-07 18:15:28 -07:00
13542a230c78_archive_url_index.py Creating index for archive.url. 2022-11-02 15:21:34 -06:00
31896cc29b71_archive_contents_search.py Archives can be searched by their contents. 2021-12-31 20:14:14 -07:00
66407d145b76_.py Creating a new Collection model for use in organizing FileGroups. 2025-12-06 15:11:37 -07:00
158973fda20f_lower_filegroup_primary_path.py Adding file_group.effective_datetime to speed up video and archive browsing. 2025-08-16 11:38:22 -06:00
184329ed5137_downloader.py Arbitrary videos can be downloaded. (#63) 2021-12-20 16:09:11 -07:00
0400005fe2dc_video_textsearch_coalesce_fix.py Applying textsearch coalesce fix to archive.textsearch. 2022-01-04 11:14:15 -07:00
467235201c70_download_sub_downloader.py RSSDownloader can schedule downloads for other Downloaders using sub_downloader. 2022-06-08 11:56:42 -06:00
a1b2c3d4e5f6_add_collection_file_format.py FileGroups in Collections can be reorganized in the Collection's directory. 2026-01-31 18:08:41 -07:00
a2b3c4d5e6f7_drop_map_file_table.py Key changes: 2026-04-04 13:48:19 -06:00
a7b1c2d3e4f5_collection_item_polymorphic.py Add ordered collections (playlists): polymorphic items + item API (#475) 2026-06-11 19:46:34 -06:00
aac864072193_increase_file_size_column.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
add_unique_collection_name_kind.py Creating a new Collection model for use in organizing FileGroups. 2025-12-06 15:11:37 -07:00
af4527503bdf_add_filegroup_directory_column.py Creating FileGroup.directory, converting paths to relative to avoid updating all the paths on move. 2025-12-24 11:26:45 -07:00
b2b9700f5048_fix_corrupted_media_paths.py FileGroups in Collections can be reorganized in the Collection's directory. 2026-01-31 18:08:41 -07:00
b2e4f6a8c1d3_drop_inventory_tables.py Rebuild Inventory as a config-only system with food catalog and calculators 2026-06-14 17:14:54 -06:00
b3c4d5e6f7a8_create_map_subscription_table.py Key changes: 2026-04-04 13:48:19 -06:00
b43f70f369d0_remove_duplicate_channel_fields_.py Creating a new Collection model for use in organizing FileGroups. 2025-12-06 15:11:37 -07:00
b97c1313f986_video_modification_datetime.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
ba98bd360b7a_add_collection_id_to_channel_model.py Removing model dependency in migration. 2026-01-31 19:29:02 -07:00
bddb5e95a91e_video_textsearch_index.py Changing weight to lowest. 2021-12-31 22:36:16 -07:00
bf704baa185d_zim_flag.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
c3d5f7a9b1e2_download_last_download_attempt.py Add per-domain and global daily download limits 2026-06-28 17:17:42 -06:00
c4d5e6f7a8b9_drop_map_subscription_table.py Key changes: 2026-04-04 13:48:19 -06:00
c72d305cf11e_download_settings.py Adding Download.settings column in alembic migration. 2022-10-14 13:08:39 -06:00
c9561bda3cf0_file_group_indexes.py Adding default to TagFile.created_at. 2023-04-10 08:41:35 -06:00
d0bbe45e56c3_reindex_pdfs.py Storing example ebooks for testing. 2022-12-11 12:36:53 -07:00
d2eace02f5b9_file_path_on_cascade.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
d4e6f8a0c2b5_file_group_suffix.py Add ESP32 firmware flasher 2026-07-01 18:34:04 -06:00
d4e446637f18_channel_tag.py Making Channel.tag_id migration lazier to avoid rare errors. 2024-10-16 18:57:04 -06:00
d5e6f7a8b9c0_doc_section.py Adding simple search to epub viewer. 2026-04-13 17:23:20 -06:00
d8fcdb7773c0_drop_channel_link.py Moving non-video functions to wrolpi.common. Handling Channel/Downlaod relationship in Channel.update. 2022-04-29 19:09:25 -06:00
d96a04eb2785_video_have_comments.py Creating background process that fetches comments for Videos that do not yet have comments. 2024-03-01 17:15:56 -07:00
d371333159ab_video_view_count_bigint.py Changing video view_count to bigint for popular videos. 2026-02-08 12:33:37 -07:00
db0a1560d923_downloader_info.py Squashed commit of the following: 2021-12-30 20:45:11 -07:00
dbf05c8e8dcc_invalidate_videos_for_orphaned_video_.py Invalidating video files for orphaned video meta files. 2022-12-07 18:27:45 -07:00
dc6637ced17b_zim_auto_search.py Adding feature which adds toggles to Zim files to enable/disable auto search. 2025-02-02 12:43:04 -07:00
e5f7a9c1d3b6_file_group_textsearch_abc.py Search: fast ABC mode by default, opt-in deep search 2026-07-03 10:32:16 -06:00
e6f7a8b9c0d1_reindex_epubs_for_spine_order.py Adding simple search to epub viewer. 2026-04-13 17:23:20 -06:00
e7f8a9b0c1d2_rename_ebook_to_doc.py Add Docs module for searching and organizing PDFs and EPUBs 2026-03-20 14:17:06 -06:00
f0d0086a73d1_init_videos_tables.py A new start 2021-11-01 16:43:09 -06:00
f2c6326cad29_video_source_id_index.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
f8c65e7e6802_channel_refresh.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
f38cfe4b5cb2_collections_feature.py Running alembic check in cicd. 2025-12-06 15:11:37 -07:00
f7229a67e333_channel_download_missing_data.py Adding toggle to Channel page to stop downloading comments for certain channels. 2025-03-01 18:23:29 -07:00
f9928a3b6bc5_file_directories.py Zim/Kiwix (Wikipedia) feature 2023-07-10 21:30:40 -06:00
migrate_domains_to_collections.py Creating domain collection for archives that have no domain. 2026-02-01 16:19:40 -07:00
migrate_download_channel_to_collection.py Removing collection metadata. 2025-12-06 15:11:37 -07:00