diff --git a/docs/controller/controller-admin-tab.png b/docs/controller/controller-admin-tab.png index 00c5e4e..74541a5 100644 Binary files a/docs/controller/controller-admin-tab.png and b/docs/controller/controller-admin-tab.png differ diff --git a/docs/controller/dashboard-system-actions.png b/docs/controller/dashboard-system-actions.png new file mode 100644 index 0000000..57d14d6 Binary files /dev/null and b/docs/controller/dashboard-system-actions.png differ diff --git a/docs/controller/dashboard.md b/docs/controller/dashboard.md index 5be78a3..e3b7907 100644 --- a/docs/controller/dashboard.md +++ b/docs/controller/dashboard.md @@ -45,6 +45,8 @@ Quick action buttons for system-wide operations: - **Reboot** - Reboots the entire system - **Shutdown** - Shuts down the system (native mode only) +![Dashboard system actions](dashboard-system-actions.png) + ### Services Table Lists all WROLPi services with their current status: diff --git a/docs/controller/system-actions-overview.png b/docs/controller/system-actions-overview.png index 0a7e17a..931bf6a 100644 Binary files a/docs/controller/system-actions-overview.png and b/docs/controller/system-actions-overview.png differ diff --git a/docs/index.md b/docs/index.md index c2c6132..d0a69b7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,6 +8,9 @@ Information about a WROLPi system. * [Getting Started](system/getting-started.md) * [Getting Help](system/getting-help.md) +* [Controller](controller/index.md) +* [Collections](system/collections/index.md) + * [Reorganize Collections](system/collections/reorganization.md) * [Logs](system/logs.md) * [Systems](system/index.md) * [Recommended Hardware](system/recommended-hardware.md) diff --git a/docs/system/collections/batch-reorganize.png b/docs/system/collections/batch-reorganize.png new file mode 100644 index 0000000..78e4f26 Binary files /dev/null and b/docs/system/collections/batch-reorganize.png differ diff --git a/docs/system/collections/conflict-recommended.png b/docs/system/collections/conflict-recommended.png new file mode 100644 index 0000000..bd5db2d Binary files /dev/null and b/docs/system/collections/conflict-recommended.png differ diff --git a/docs/system/collections/conflict-resolution.png b/docs/system/collections/conflict-resolution.png new file mode 100644 index 0000000..c2771d3 Binary files /dev/null and b/docs/system/collections/conflict-resolution.png differ diff --git a/docs/system/collections/index.md b/docs/system/collections/index.md new file mode 100644 index 0000000..0717ba5 --- /dev/null +++ b/docs/system/collections/index.md @@ -0,0 +1,109 @@ +# Collections + +Collections are logical groupings of files in WROLPi. They organize related content together, +such as all videos from a channel or all archives from a domain. + +## How Collections Work + +Collections are tied to specific directories. All files within a collection's directory +automatically belong to that collection. When you download new content to that directory, +it's automatically included. + +Examples: + +- Video channels stored in `/media/wrolpi/videos/My Favorite Channel` +- Archives from a domain stored in `/media/wrolpi/archives/example.com` + +> **A Collection cannot contain another Collection!** + +## Collection Kinds + +Collections are categorized by kind: + +| Kind | Purpose | Default Location | +|-----------|------------------------------------|-----------------------------| +| `channel` | Video channels (YouTube, etc.) | `/media/wrolpi/videos/` | +| `domain` | Website archives (SingleFile, etc.)| `/media/wrolpi/archives/` | + +## Automatic Collection Creation + +WROLPi automatically creates collections in several scenarios: + +- **Video downloads**: When downloading from a channel, WROLPi creates a channel collection +- **Archive downloads**: When archiving a website, WROLPi creates a domain collection + +## Collection Properties + +Each collection tracks: + +- **Name**: The collection identifier (channel name, domain name, etc.) +- **Directory**: The directory containing the collection's files (must be unique) +- **Tag**: Optional tag for categorization and filtering +- **File Format**: The naming format used at last reorganization +- **Item Count**: Number of files in the collection +- **Total Size**: Combined size of all files + +## Working with Collections + +### Viewing Collections + +Collections appear in several places: + +- **Videos page**: Channel collections show in the Channels view +- **Archives page**: Domain collections show in the Domains view + +### Collection Actions + +From a collection page you can: + +- **Download new content**: Add more videos/archives to the collection +- **Tag files**: Apply or remove tags from all items +- **Reorganize**: Move files to match a new naming format (see [Reorganization](reorganization.md)) +- **Refresh**: Rescan the directory for new files + +## Download Destinations + +Download destinations are determined by two template settings: + +### New Collections + +When downloading from a new channel or domain, WROLPi creates a collection using the +**special directories template** from Settings. This determines where the collection's +directory is created: + +- `videos_destination` - Template for new video channels +- `archive_destination` - Template for new archive domains + +Example: If `videos_destination` is `%(name)s`, downloading from "Tech Channel" creates +the collection at `/media/wrolpi/videos/Tech Channel/`. + +### Existing Collections + +When downloading to an existing collection, files are placed within the collection's +directory using the **file format template**. This template controls how files are +named and organized within the collection: + +``` +%(upload_year)s/%(title)s.%(ext)s +``` + +Example: A video uploaded in 2024 titled "Tutorial" goes to +`/media/wrolpi/videos/Tech Channel/2024/Tutorial.mp4`. + +### Template Variables + +| Variable | Description | +|-------------------|------------------------------------| +| `%(name)s` | Collection name | +| `%(tag)s` | Tag name (if tagged) | +| `%(kind)s` | Collection kind (channel/domain) | +| `%(upload_year)s` | Year of upload (videos) | +| `%(upload_month)s`| Month of upload (videos) | +| `%(title)s` | Video/archive title | +| `%(ext)s` | File extension | + +## Related Documentation + +- [Reorganization](reorganization.md) - Moving files to match new naming formats +- [Videos](../../modules/videos/index.md) - Video channel management +- [Archives](../../modules/archives/index.md) - Website archive management diff --git a/docs/system/collections/reorganization.md b/docs/system/collections/reorganization.md new file mode 100644 index 0000000..80e3078 --- /dev/null +++ b/docs/system/collections/reorganization.md @@ -0,0 +1,185 @@ +# Collection Reorganization + +When you change your file naming format, existing files remain in their old locations. +Reorganization moves files to match the new format. + +## When to Reorganize + +Reorganization is needed when: + +- You change your video or archive `file_name_format` in settings +- You want files organized by year/month when they weren't before +- You want to restructure how files are named within a collection + +## Checking Reorganization Status + +Collections that need reorganization show a notification in the UI. You can also check +the status from the collection page. + +A collection needs reorganization when its stored `file_format` differs from the +current format in your settings. + +## Reorganization Preview + +Before moving any files, WROLPi shows a preview of what will change: + +- Total files in the collection +- Number of files that will move +- Sample moves showing before/after paths +- Any conflicts that need resolution + +![Reorganization preview](reorganize-preview.png) + +## Conflict Resolution + +A conflict occurs when multiple source files would move to the same destination. +For example, two videos with the same title would conflict if the format doesn't +include unique identifiers. + +### Detecting Conflicts + +Conflicts are detected during the preview phase, before any files move. WROLPi +shows which files conflict and provides information to help you decide which to keep. + +![Conflict resolution showing quality rankings](conflict-resolution.png) + +### Quality Ranking + +WROLPi assigns a quality rank to help identify which duplicate to keep. Higher +ranked files have more complete metadata and are recommended to keep. + +**Video quality factors:** + +- Has URL and source ID +- Has info JSON with complete metadata +- Has comments downloaded +- Has captions/subtitles +- Has audio streams +- Has poster image +- Has precise publish date + +**Archive quality factors:** + +- Has readability content +- Has screenshot +- Has title and author +- Has extracted text + +![Recommended file to keep highlighted](conflict-recommended.png) + +### Resolving Conflicts + +For each conflict group: + +1. Review the conflicting files +2. The highest-ranked file shows as "recommended to keep" +3. Delete the lower-quality duplicates +4. After deletion, refresh the preview +5. Repeat until no conflicts remain + +Only after all conflicts are resolved can reorganization proceed. + +## Executing Reorganization + +Once conflicts are resolved, click **Reorganize Files** to begin moving files. + +![Reorganize button](reorganize-button.png) + +The process: + +1. Each file group moves to its new location +2. All related files move together (video + subtitles + poster, etc.) +3. Empty source directories are cleaned up +4. Database records update to reflect new paths +5. Collection's file format updates to match current settings + +### Progress Tracking + +During reorganization, progress displays showing: + +- Files completed vs total +- Current file being moved +- Percentage complete + +### Stopping and Resuming + +If reorganization is interrupted (system restart, error, etc.): + +- Files already moved stay in their new locations +- The collection's format is not updated +- You can retry reorganization to continue from the partial state +- WROLPi detects which files still need moving + +## Batch Reorganization + +When multiple collections need reorganization, you can reorganize them all at once. + +Navigate to: + +- **Videos > Settings > Reorganize All Channels** for video channels +- **Archive > Settings > Reorganize All Domains** for archive domains + +Batch reorganization: + +- Processes collections one at a time +- Shows overall progress across all collections +- Stops on first failure with error details +- Can be retried after resolving issues + +![Batch reorganization page](batch-reorganize.png) + +## Common Issues + +### File Not Found + +If a source file doesn't exist: + +- The file may have been manually moved or deleted +- Run a file refresh to resync the database +- Retry reorganization + +### Destination Already Exists + +If the destination path already has a file: + +- This indicates a conflict that wasn't detected +- The file is skipped +- Check for duplicate files with similar names + +### Permission Errors + +If files can't be moved: + +- Check that WROLPi has write access to both source and destination +- Verify disk space is available +- Check for mounted/unmounted drives + +## Format Variables Reference + +Available variables for file naming formats: + +### Video Formats + +| Variable | Description | Example | +|--------------------|----------------------------|-------------| +| `%(title)s` | Video title | My Video | +| `%(uploader)s` | Channel/uploader name | WROLPi | +| `%(channel)s` | Alias for uploader | WROLPi | +| `%(upload_date)s` | Upload date (YYYYMMDD) | 20240315 | +| `%(upload_year)s` | Upload year | 2024 | +| `%(upload_month)s` | Upload month (zero-padded) | 03 | +| `%(id)s` | Platform video ID | dQw4w9WgXcQ | +| `%(ext)s` | File extension | mp4 | + +### Archive Formats + +| Variable | Description | Example | +|-------------------------|------------------------------|---------------------| +| `%(title)s` | Page title | Article Title | +| `%(domain)s` | Source domain | example.com | +| `%(download_datetime)s` | Download datetime | 2024-03-15-14-30-00 | +| `%(download_date)s` | Download date | 2024-03-15 | +| `%(download_year)s` | Download year | 2024 | +| `%(download_month)s` | Download month (zero-padded) | 03 | +| `%(download_day)s` | Download day (zero-padded) | 15 | +| `%(ext)s` | File extension | html | diff --git a/docs/system/collections/reorganize-button.png b/docs/system/collections/reorganize-button.png new file mode 100644 index 0000000..9dd63a1 Binary files /dev/null and b/docs/system/collections/reorganize-button.png differ diff --git a/docs/system/collections/reorganize-preview.png b/docs/system/collections/reorganize-preview.png new file mode 100644 index 0000000..540bfbf Binary files /dev/null and b/docs/system/collections/reorganize-preview.png differ diff --git a/docs/system/databases.md b/docs/system/databases.md index 0b66aaa..453f3c9 100644 --- a/docs/system/databases.md +++ b/docs/system/databases.md @@ -17,6 +17,8 @@ This contains the tables for the API. | alembic_version | Tracks Alembic migration. | | `alembic` | | archive | Data about a FileGroup Archive. | file_group_id, domain_id | `modules/archive/models.py` | | channel | Data about a video Channel. | | `modules/videos/models.py` | +| collection | A logical grouping of FileGroups. | tag_id | `wrolpi/collections/models.py`| +| collection_item | Links a FileGroup to a Collection. | collection_id, file_group_id | `wrolpi/collections/models.py`| | directory | Directories in the media directory. | | `wrolpi/files/models.py` | | domains | Groups Archives by shared URL domain. | | `modules/archive/models.py` | | download | Data about what should be downloaded and when. | | `wrolpi/downloader.py` |