Add ordered collections (playlists): polymorphic items + item API (#475)

* Add Playlists: ordered, mixed-type collections

A playlist (Collection kind='playlist') is a manually-curated, ordered list that
mixes item types. Squash of the playlists feature branch (24 commits).

Data model & API:
- CollectionItem is polymorphic via item_kind: 'file' (FileGroup), 'zim'
  (zim_id + zim_entry), or 'url' (link to anything the WROLPi browser opens,
  e.g. /map?lat=&lon=&z=). CHECK constraint enforces exactly one kind; per-kind
  unique constraints; migration a7b1c2d3e4f5.
- Endpoints: create playlist; add/remove/reorder items (idempotent adds: 201 new,
  200 existing); GET returns the ordered items. All mutations wrol_mode_check'd.

Config (source of truth):
- playlists.yaml (PlaylistsConfig) stores each playlist's metadata AND ordered
  items (file by relative path, zim by path+entry, url inline); round-trips a
  DB wipe. Supports backup import (DB_CONFIG_FILE_NAMES).
- The redundant generic CollectionsConfig (collections.yaml) is deleted:
  channels/domains own their configs, authors/subjects are re-derived.

On-disk sync (sync_playlists_directory switch):
- Each playlist materializes into the configurable Playlists Directory
  (playlists_destination, Settings UI input, excluded from indexing) with
  position-prefixed entries: NNNN_<file> hard links (copy fallback across
  filesystems) and NNNN_<title>.html stubs for zim/url items.
- Tagged playlists are namespaced like channels: <playlists>/<tag>/<name>;
  a tag change moves the directory and prunes the empty old tag dir.
- Deletion safety: managed files are only removed when they are our marked
  stubs or still have another hard link (st_nlink > 1) -- never a last copy.
  Stray loose files at the playlists root are removed; stub URLs are
  scheme-allowlisted (no javascript:/data:).

Frontend:
- /playlists list page (CollectionTable, name filter, New Playlist modal, Tag
  column) between Files and Zim in the nav.
- View-only detail page (Name link) and edit page (Edit button): themed table
  with model icons/colors + poster thumbnails, items link to their model page
  (video/archive/doc) or preview, up/down reorder, remove, add-URL form,
  single-tag selector; all editing disabled in WROL Mode.
- Reusable green "Add to Playlist" button on Video/Archive/Doc detail pages,
  the FilePreview toolbar, Zim search results, and Map pins (added as a /map
  URL item with a map-marker icon).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Playlist edit page parity: rename, custom directory, tag modal move suggestion

Match the ChannelEditPage/DomainEditPage UX on the playlist edit page:
Back/View buttons, an edit form (name, description, directory), and the
shared CollectionTagModal with the violet "Tag" button.  The tag modal now
suggests moving the playlist to its managed tag location
(playlists/<tag>/<name>); accepting the suggestion stores directory=None so
the sync keeps auto-managing the location.  Tag buttons on the Channel and
Domain edit pages are now violet to match.

Backend: update_collection learns rename (with per-kind validation and
duplicate checks) and playlist custom directories (cleanup of the abandoned
directory, config round-trip, commit-before-switch to avoid the background
sync racing pre-commit state).  tag_collection's duplicated untag/tag
branches are merged, and the per-kind config-save dispatch is consolidated
into _activate_config_save().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Deduplicate Collection error handling and Tag creation

Delete the duplicate UnknownCollection in wrolpi/collections/errors.py and
re-export the canonical one from wrolpi/errors.py, which carries
status_code=NOT_FOUND (the local copy inherited 500).  Replace the repeated
query/raise pattern in reorganize.py with Collection.find_by_id.

Add Tag.get_or_create_tag and use it for collection tagging; flush the new
Tag before returning so callers can read .id immediately -- previously
tagging a collection with a brand-new tag name set tag_id=None.  Regression
test included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
lrnselfreliance 2026-06-09 19:37:34 -06:00 • committed by GitHub
parent f365bb5411
commit 20f65e1063
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
40 changed files with 3188 additions and 359 deletions

View file

@ -0,0 +1,58 @@
"""Make collection_item polymorphic: file | zim | url items.
CollectionItem previously only referenced a FileGroup. Playlists (kind='playlist') also need to
order Zim articles (zim_id + zim_entry, mirroring TagZimEntry) and arbitrary URLs (a link the WROLPi
browser can open, e.g. a map location). This adds item_kind plus the zim/url/title columns, relaxes
file_group_id to nullable, and adds per-kind unique constraints and a CHECK that exactly one kind's
columns are populated. Existing rows default to item_kind='file'.
Revision ID: a7b1c2d3e4f5
Revises: e6f7a8b9c0d1
Create Date: 2026-06-08
"""
from alembic import op
import sqlalchemy as sa
revision = 'a7b1c2d3e4f5'
down_revision = 'e6f7a8b9c0d1'
branch_labels = None
depends_on = None
CHECK_SQL = (
"(item_kind = 'file' AND file_group_id IS NOT NULL AND zim_id IS NULL AND url IS NULL)"
" OR (item_kind = 'zim' AND zim_id IS NOT NULL AND zim_entry IS NOT NULL"
" AND file_group_id IS NULL AND url IS NULL)"
" OR (item_kind = 'url' AND url IS NOT NULL AND file_group_id IS NULL AND zim_id IS NULL)"
)
def upgrade():
op.add_column('collection_item',
sa.Column('item_kind', sa.String(), nullable=False, server_default='file'))
op.add_column('collection_item', sa.Column('zim_id', sa.Integer(), nullable=True))
op.add_column('collection_item', sa.Column('zim_entry', sa.Text(), nullable=True))
op.add_column('collection_item', sa.Column('url', sa.Text(), nullable=True))
op.add_column('collection_item', sa.Column('title', sa.Text(), nullable=True))
# `file` is no longer the only kind, so a row need not have a FileGroup.
op.alter_column('collection_item', 'file_group_id', existing_type=sa.Integer(), nullable=True)
op.create_foreign_key('collection_item_zim_id_fkey', 'collection_item', 'zim',
['zim_id'], ['id'], ondelete='CASCADE')
op.create_unique_constraint('uq_collection_zim_entry', 'collection_item',
['collection_id', 'zim_id', 'zim_entry'])
op.create_unique_constraint('uq_collection_url', 'collection_item', ['collection_id', 'url'])
op.create_check_constraint('ck_collection_item_kind', 'collection_item', CHECK_SQL)
def downgrade():
op.drop_constraint('ck_collection_item_kind', 'collection_item', type_='check')
op.drop_constraint('uq_collection_url', 'collection_item', type_='unique')
op.drop_constraint('uq_collection_zim_entry', 'collection_item', type_='unique')
op.drop_constraint('collection_item_zim_id_fkey', 'collection_item', type_='foreignkey')
# Restore NOT NULL (valid only if no non-file rows remain).
op.alter_column('collection_item', 'file_group_id', existing_type=sa.Integer(), nullable=False)
op.drop_column('collection_item', 'title')
op.drop_column('collection_item', 'url')
op.drop_column('collection_item', 'zim_entry')
op.drop_column('collection_item', 'zim_id')
op.drop_column('collection_item', 'item_kind')

View file

@ -23,6 +23,7 @@ import {SemanticToastContainer} from "react-semantic-toasts-2";
import {FilePreviewProvider} from "./components/FilePreview";
import {TagsProvider} from "./Tags";
import {ZimRoute} from "./components/Zim";
import {PlaylistsRoute} from "./components/Playlists";
import {DocsRoute} from "./components/Docs";
import ErrorBoundary from "./components/ErrorBoundary";
import {KeyboardShortcutsProvider} from "./components/KeyboardShortcutsProvider";
@ -104,6 +105,7 @@ const router = createBrowserRouter(createRoutesFromElements(<Route
<Route path='docs/*' element={<ErrorBoundary><DocsRoute/></ErrorBoundary>}/>
<Route path='map/*' element={<ErrorBoundary><MapRoute/></ErrorBoundary>}/>
<Route path='zim/*' element={<ErrorBoundary><ZimRoute/></ErrorBoundary>}/>
<Route path='playlists/*' element={<ErrorBoundary><PlaylistsRoute/></ErrorBoundary>}/>
<Route path='files/*' element={<ErrorBoundary><FilesRoute/></ErrorBoundary>}/>
</Route>));

View file

@ -2264,3 +2264,101 @@ export async function searchDocs(offset, limit, searchStr, order, tagNames, auth
throw Error(message);
}
}
export async function fetchPlaylists() {
const response = await apiGet(`${COLLECTIONS_API}?kind=playlist`);
if (response.ok) {
return (await response.json())['collections'];
}
const message = await getErrorMessage(response, 'Failed to fetch playlists.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}
export async function getPlaylist(playlistId) {
const response = await apiGet(`${COLLECTIONS_API}/${playlistId}`);
if (response.ok) {
return (await response.json())['collection'];
}
const message = await getErrorMessage(response, 'Failed to fetch playlist.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}
export async function createPlaylist(name, description) {
// Trailing slash avoids Sanic's 302 redirect on the root POST route (which would drop the body).
const response = await apiPost(`${COLLECTIONS_API}/`, {name, description, kind: 'playlist'});
if (response.ok) {
return (await response.json())['collection'];
}
const message = await getErrorMessage(response, 'Failed to create playlist.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}
export async function deletePlaylist(playlistId) {
const response = await apiDelete(`${COLLECTIONS_API}/${playlistId}`);
if (!response.ok) {
const message = await getErrorMessage(response, 'Failed to delete playlist.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}
}
export async function addPlaylistItem(playlistId, item) {
// item: {item_kind, url?, title?, file_group_id?, zim_id?, zim_entry?, position?}
const response = await apiPost(`${COLLECTIONS_API}/${playlistId}/items`, item);
if (response.ok) {
return (await response.json())['item'];
}
const message = await getErrorMessage(response, 'Failed to add item.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}
export async function removePlaylistItem(playlistId, itemId) {
const response = await apiDelete(`${COLLECTIONS_API}/${playlistId}/items/${itemId}`);
if (!response.ok) {
const message = await getErrorMessage(response, 'Failed to remove item.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}
}
export async function reorderPlaylistItems(playlistId, itemIds) {
const response = await apiPut(`${COLLECTIONS_API}/${playlistId}/items/order`, {item_ids: itemIds});
if (response.ok) {
return (await response.json())['collection'];
}
const message = await getErrorMessage(response, 'Failed to reorder items.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}
// Update a playlist's name, description, and/or custom directory ('' clears the directory).
export async function updatePlaylist(playlistId, {name, description, directory}) {
const response = await apiPut(`${COLLECTIONS_API}/${playlistId}`, {name, description, directory});
if (response.ok) {
return (await response.json())['collection'];
}
const message = await getErrorMessage(response, 'Failed to update playlist.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}
// Set (tagName) or clear (tagName='') a playlist's single tag. When `directory` is given, the
// playlist also moves there (the managed tag location is normalized back to auto-managed).
export async function setPlaylistTag(playlistId, tagName, directory) {
const body = {tag_name: tagName || ''};
if (directory !== null && directory !== undefined) {
body.directory = directory;
}
const response = await apiPut(`${COLLECTIONS_API}/${playlistId}`, body);
if (response.ok) {
return (await response.json())['collection'];
}
const message = await getErrorMessage(response, 'Failed to update playlist tag.');
toast({type: 'error', title: 'Error', description: message, time: 5000});
throw Error(message);
}

View file

@ -0,0 +1,132 @@
import React, {useEffect, useState} from "react";
import {Input, Message} from "semantic-ui-react";
import {toast} from "react-semantic-toasts-2";
import {Button, Form, List, Loader, Modal} from "./Theme";
import {addPlaylistItem, createPlaylist, fetchPlaylists} from "../api";
/**
* Modal that adds one or more items to a playlist — an existing one or a newly-created one.
*
* `items` is a list of item payloads, each ready to POST to .../items, e.g.
* {item_kind: 'file', file_group_id: 123}
* {item_kind: 'zim', zim_id: 1, zim_entry: 'A/Fire', title: 'Fire'}
* {item_kind: 'url', url: '/map?...', title: '...'}
*/
export function AddToPlaylistModal({open, onClose, items = [], onComplete}) {
const [playlists, setPlaylists] = useState(null); // null=loading, []/[...]=ok, undefined=error
const [selected, setSelected] = useState(null);
const [newName, setNewName] = useState('');
const [adding, setAdding] = useState(false);
useEffect(() => {
if (!open) return;
setSelected(null);
setNewName('');
setPlaylists(null);
fetchPlaylists().then(setPlaylists).catch(() => setPlaylists(undefined));
}, [open]);
const addAll = async (playlistId, playlistName) => {
setAdding(true);
try {
for (const payload of items) {
await addPlaylistItem(playlistId, payload);
}
const n = items.length;
toast({
type: 'success', title: 'Added to playlist',
description: `${n} item${n === 1 ? '' : 's'} added to "${playlistName}"`, time: 3000,
});
if (onComplete) onComplete();
onClose();
} catch (e) {
// Error toast already shown by the API client.
} finally {
setAdding(false);
}
};
const handleAddExisting = async () => {
const playlist = (playlists || []).find(p => p.id === selected);
if (playlist) await addAll(playlist.id, playlist.name);
};
const handleCreateAndAdd = async () => {
const name = newName.trim();
if (!name) return;
try {
const playlist = await createPlaylist(name);
if (playlist && playlist.id) await addAll(playlist.id, playlist.name);
} catch (e) {
}
};
const creating = !!newName.trim();
return <Modal open={open} onClose={onClose} size='tiny'>
<Modal.Header>Add to Playlist</Modal.Header>
<Modal.Content>
{playlists === undefined && <Message error><Message.Header>Could not load playlists</Message.Header></Message>}
{playlists === null && <Loader active inline='centered'/>}
{Array.isArray(playlists) && playlists.length > 0 &&
<List divided selection>
{playlists.map(p => <List.Item key={p.id} active={selected === p.id}
onClick={() => setSelected(p.id)}>
<List.Icon name={selected === p.id ? 'check circle' : 'circle outline'}
verticalAlign='middle'/>
<List.Content><List.Header>{p.name}</List.Header></List.Content>
</List.Item>)}
</List>}
{Array.isArray(playlists) && playlists.length === 0 &&
<p>No playlists yet — create one below.</p>}
<Form onSubmit={handleCreateAndAdd} style={{marginTop: '1em'}}>
<Form.Field>
<label>Or create a new playlist</label>
<Input placeholder='New playlist name...' value={newName}
onChange={(e, {value}) => setNewName(value)}/>
</Form.Field>
</Form>
</Modal.Content>
<Modal.Actions>
<Button onClick={onClose}>Cancel</Button>
{creating
? <Button primary loading={adding} onClick={handleCreateAndAdd}>Create &amp; Add</Button>
: <Button primary disabled={!selected} loading={adding} onClick={handleAddExisting}>Add</Button>}
</Modal.Actions>
</Modal>;
}
/**
* A button that opens AddToPlaylistModal for one of:
* - one FileGroup (fileGroupId) or many (fileGroupIds),
* - a Zim article (zim={zimId, entry, title}), or
* - a URL (url={url, title}) — link to anything the WROLPi browser can open (e.g. a map location).
* Extra props are forwarded to the Button (color, size, disabled, icon, content, ...).
*/
export function AddToPlaylistButton({fileGroupId, fileGroupIds, zim, url, content = 'Add to Playlist', onComplete, ...buttonProps}) {
const [open, setOpen] = useState(false);
let items;
if (url && url.url) {
items = [{item_kind: 'url', url: url.url, title: url.title || null}];
} else if (zim && zim.zimId != null && zim.entry) {
items = [{item_kind: 'zim', zim_id: zim.zimId, zim_entry: zim.entry, title: zim.title || null}];
} else {
const ids = fileGroupIds || (fileGroupId != null ? [fileGroupId] : []);
items = ids.map(id => ({item_kind: 'file', file_group_id: id}));
}
// An empty `content` makes a compact icon-only button (omit content entirely so Semantic does
// not reserve label padding); otherwise render the labeled "Add to Playlist" button.
const contentProps = content ? {content} : {};
return <>
<Button color='green' icon='list' onClick={() => setOpen(true)}
disabled={items.length === 0} {...contentProps} {...buttonProps}/>
<AddToPlaylistModal open={open} onClose={() => setOpen(false)} items={items}
onComplete={onComplete}/>
</>;
}

View file

@ -85,6 +85,7 @@ import {Media, ThemeContext} from "../contexts/contexts";
import {Button, Card, darkTheme, Header, Loader, Placeholder, Popup, Segment, Statistic, Tab} from "./Theme";
import {BulkTagModal} from "./BulkTagModal";
import {taggedImageLabel, TagsSelector} from "../Tags";
import {AddToPlaylistButton} from "./AddToPlaylist";
import {toast} from "react-semantic-toasts-2";
import {API_ARCHIVE_UPLOAD_URI, Downloaders} from "./Vars";
import {CollectionTable} from "./collections/CollectionTable";
@ -343,6 +344,7 @@ function ArchivePage() {
{updateButton}
{deleteButton}
{generateScreenshotButton}
<AddToPlaylistButton fileGroupId={archiveFile.id}/>
</Segment>
<Segment>
@ -669,7 +671,7 @@ export function DomainEditPage() {
type="button"
size='small'
onClick={() => setTagEditModalOpen(true)}
color='green'
color='violet'
style={{marginTop: '1em'}}
>Tag</Button>;

View file

@ -191,7 +191,7 @@ export function ChannelEditPage() {
type="button"
size='small'
onClick={() => setTagEditModalOpen(true)}
color='green'
color='violet'
style={{marginTop: '1em'}}
>Tag</Button>;

View file

@ -27,6 +27,7 @@ import {useAuthors, useDoc, useOneQuery, useSearchDocs, useSubjects} from "../ho
import {TagsSelector} from "../Tags";
import {toast} from "react-semantic-toasts-2";
import {CollectionTable} from "./collections/CollectionTable";
import {AddToPlaylistButton} from "./AddToPlaylist";
import {CbzViewer} from "./CbzViewer";
import _ from "lodash";
@ -438,6 +439,7 @@ function DocPage() {
onClick={() => handleDelete(false)}
obeyWROLMode={true}
>Delete</APIButton>
<AddToPlaylistButton fileGroupId={docFile.id}/>
</>;
return <>

View file

@ -11,6 +11,7 @@ import {Button, Modal} from "./Theme";
import {toast} from "react-semantic-toasts-2";
import {useOneQuery} from "../hooks/customHooks";
import {ShareButton} from "./Share";
import {AddToPlaylistButton} from "./AddToPlaylist";
import {pathDirectory} from "./FileBrowser";
import {InlineErrorBoundary} from "./ErrorBoundary";
import {useLocation} from "react-router";
@ -496,6 +497,9 @@ export function FilePreviewProvider({children}) {
{downloadButton}
{directoryButton}
<ShareButton/>
{previewFile && previewFile['id'] &&
<AddToPlaylistButton fileGroupId={previewFile['id']} content={null} size='small'
title='Add to Playlist'/>}
</div>
<div className='preview-toolbar-tags'>{tagsDisplay}</div>
<div className='preview-toolbar-group'>

View file

@ -29,6 +29,7 @@ import {Media, StatusContext} from "../contexts/contexts";
import {MAP_VIEWER_URI} from "./Vars";
import {Modal} from "./Theme";
import MapViewer from "./MapViewer";
import {AddToPlaylistButton} from "./AddToPlaylist";
import maplibregl from "maplibre-gl";
import {Protocol} from "pmtiles";
import layers from "protomaps-themes-base";
@ -532,6 +533,12 @@ function MapPins() {
<TableCell>{pin.created}</TableCell>
<TableCell collapsing>
<Button size='tiny' icon='edit' onClick={() => setEditingId(pin.id)}/>
<AddToPlaylistButton
size='tiny'
icon='list'
content=''
url={{url: `/map?lat=${pin.lat}&lon=${pin.lon}&z=14`, title: pin.label}}
/>
<APIButton
size='tiny'
color='red'

View file

@ -53,6 +53,7 @@ const allLinks = [
{text: 'Docs', to: '/docs', key: 'docs'},
{text: 'Map', to: '/map', key: 'map'},
{text: 'Files', to: '/files', key: 'files'},
{text: 'Playlists', to: '/playlists', key: 'playlists'},
{text: 'Zim', to: '/zim', key: 'zim'},
{text: 'Inventory', to: '/inventory', key: 'inventory'},
{to: '/more/otp', text: 'One Time Pad', key: 'otp', end: true},

View file

@ -0,0 +1,585 @@
import React, {useEffect, useState} from "react";
import {Link, Route, Routes, useNavigate, useParams} from "react-router";
import {Grid, Image, Input, Message} from "semantic-ui-react";
import {toast} from "react-semantic-toasts-2";
import {Button, Form, Header, Icon, Loader, Modal, Segment, Table} from "./Theme";
import {
APIButton,
BackButton,
DirectorySearch,
encodeMediaPath,
ErrorMessage,
findPosterPath,
InfoPopup,
mimetypeColor,
mimetypeIconName,
PageContainer,
PreviewLink,
SearchInput,
useTitle,
} from "./Common";
import {useOneQuery, useWROLMode} from "../hooks/customHooks";
import {CollectionTable} from "./collections/CollectionTable";
import {CollectionEditForm} from "./collections/CollectionEditForm";
import {CollectionTagModal} from "./collections/CollectionTagModal";
import {
addPlaylistItem,
createPlaylist,
deletePlaylist,
fetchPlaylists,
getCollectionTagInfo,
getPlaylist,
removePlaylistItem,
reorderPlaylistItems,
setPlaylistTag,
updatePlaylist,
} from "../api";
import {ThemeContext} from "../contexts/contexts";
const PLAYLIST_COLUMNS = [
{key: 'name', label: 'Name', sortable: true},
{key: 'item_count', label: 'Items', sortable: true, width: 2},
{key: 'tag_name', label: 'Tag', width: 3},
{key: 'actions', type: 'actions', label: 'Manage', width: 2, align: 'right'},
];
// The "Name" column links to the view-only page; the "Edit" button links to the edit page.
const PLAYLIST_ROUTES = {search: '/playlists/:id', edit: '/playlists/:id/edit', id_field: 'id'};
function usePlaylists() {
const [playlists, setPlaylists] = useState(null); // null=loading, []/[...]=ok, undefined=error
const refetch = async () => {
try {
setPlaylists(await fetchPlaylists());
} catch (e) {
console.error(e);
setPlaylists(undefined);
}
};
useEffect(() => {
refetch();
}, []);
return {playlists, refetch};
}
function usePlaylist(playlistId) {
const [playlist, setPlaylist] = useState(null);
const refetch = async () => {
try {
setPlaylist(await getPlaylist(playlistId));
} catch (e) {
console.error(e);
setPlaylist(undefined);
}
};
useEffect(() => {
refetch();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [playlistId]);
return {playlist, refetch};
}
export function PlaylistsPage() {
useTitle('Playlists');
const {playlists, refetch} = usePlaylists();
const [searchStr, setSearchStr] = useOneQuery('name');
const [modalOpen, setModalOpen] = useState(false);
const [name, setName] = useState('');
const searchInputRef = React.useRef();
const navigate = useNavigate();
const handleCreate = async () => {
const trimmed = name.trim();
if (!trimmed) return;
try {
const playlist = await createPlaylist(trimmed);
toast({type: 'success', title: 'Playlist created', description: trimmed, time: 3000});
setName('');
setModalOpen(false);
if (playlist && playlist.id) {
navigate(`/playlists/${playlist.id}`);
} else {
await refetch();
}
} catch (e) {
// Error toast already shown by the API client.
}
};
const header = <div style={{marginBottom: '1em'}}>
<Grid stackable columns={2}>
<Grid.Row>
<Grid.Column>
<SearchInput
placeholder='Name filter...'
size='large'
searchStr={searchStr}
disabled={!Array.isArray(playlists) || playlists.length === 0}
onClear={() => setSearchStr('')}
onChange={setSearchStr}
onSubmit={null}
inputRef={searchInputRef}
/>
</Grid.Column>
<Grid.Column textAlign='right'>
<Button secondary onClick={() => setModalOpen(true)}>New Playlist</Button>
</Grid.Column>
</Grid.Row>
</Grid>
</div>;
const createModal = <Modal open={modalOpen} onClose={() => setModalOpen(false)} size='tiny'>
<Modal.Header>New Playlist</Modal.Header>
<Modal.Content>
<Form onSubmit={handleCreate}>
<Form.Field>
<label>Name</label>
<Input autoFocus placeholder='Playlist name...' value={name}
onChange={(e, {value}) => setName(value)}/>
</Form.Field>
</Form>
</Modal.Content>
<Modal.Actions>
<Button onClick={() => setModalOpen(false)}>Cancel</Button>
<Button primary disabled={!name.trim()} onClick={handleCreate}>Create</Button>
</Modal.Actions>
</Modal>;
return <>
{header}
{createModal}
<CollectionTable
collections={playlists}
columns={PLAYLIST_COLUMNS}
routes={PLAYLIST_ROUTES}
searchStr={searchStr}
emptyMessage='No playlists yet'
/>
</>;
}
// Icon (name + color) for a playlist item. File items use the FileGroup's mimetype/model icon
// and color (video, pdf, ebook, image, ...) so they match the rest of the UI; zim/url are fixed.
function itemIcon(item) {
if (item.item_kind === 'file' && item.file_group) {
const fg = item.file_group;
const lowerPath = (fg.primary_path || '').toLowerCase();
return {name: mimetypeIconName(fg.mimetype, lowerPath), color: mimetypeColor(fg.mimetype, lowerPath)};
}
if (item.item_kind === 'zim') return {name: 'book'};
if (item.item_kind === 'url') {
// A map-location URL (e.g. /map?lat=&lon=) gets a map marker; other URLs get the link icon.
const u = (item.url || '').toLowerCase();
if (u.startsWith('/map')) return {name: 'map marker alternate'};
return {name: 'linkify'};
}
return {name: 'file'};
}
function itemLabel(item) {
if (item.item_kind === 'file' && item.file_group) {
return item.title || item.file_group.title || item.file_group.primary_path || 'File';
}
if (item.item_kind === 'zim' && item.zim) {
return item.title || item.zim.entry;
}
if (item.item_kind === 'url') {
return item.title || item.url;
}
return item.title || 'Item';
}
// Only http(s) and relative WROLPi paths are safe for an href; reject javascript:/data:/etc. so a
// shared playlist's url item cannot execute script when opened. Exported for tests.
export function safeHref(url) {
if (!url) return null;
try {
const u = new URL(url, window.location.origin);
if (u.protocol === 'http:' || u.protocol === 'https:') {
return url; // Keep the original (may be a relative WROLPi path).
}
} catch {
return null;
}
return null;
}
function itemLink(item) {
if (item.item_kind === 'url') {
return safeHref(item.url);
}
if (item.item_kind === 'zim' && item.zim) {
// Built from a numeric id + encoded entry — no user-supplied scheme.
return `/api/zim/${item.zim.id}/entry/${encodeURIComponent(item.zim.entry)}`;
}
return null;
}
// The in-app model page for a FileGroup, or null when there is none (image, audio, 3D, ...).
function fileModelUrl(fg) {
if (!fg) return null;
if (fg.model === 'video') return `/videos/${fg.id}`;
if (fg.model === 'archive') return `/archives/${fg.id}`;
if (fg.model === 'doc') return `/docs/${fg.id}`;
return null;
}
// Render a playlist item's title as the appropriate link:
// - file with a model page (video/archive/doc) -> in-app Link to that page
// - file without a model page (image, etc.) -> PreviewLink (opens the preview modal)
// - zim/url -> the entry/url in a new tab
function ItemTitle({item, label}) {
if (item.item_kind === 'file' && item.file_group) {
const url = fileModelUrl(item.file_group);
if (url) {
return <Link to={url}>{label}</Link>;
}
return <PreviewLink file={item.file_group}>{label}</PreviewLink>;
}
const link = itemLink(item);
return link
? <a href={link} target='_blank' rel='noopener noreferrer'>{label}</a>
: <span>{label}</span>;
}
// The ordered items table, shared by the view and edit pages. When `editable`, the reorder/remove
// action buttons (and their column) are shown; otherwise the table is read-only.
function PlaylistItemsTable({items, editable, onMove, onRemove}) {
return <Table unstackable celled>
<Table.Header>
<Table.Row>
<Table.HeaderCell collapsing>#</Table.HeaderCell>
<Table.HeaderCell collapsing>Item</Table.HeaderCell>
<Table.HeaderCell>Title</Table.HeaderCell>
{editable && <Table.HeaderCell collapsing/>}
</Table.Row>
</Table.Header>
<Table.Body>
{items.map((item, index) => {
const label = itemLabel(item);
const poster = item.item_kind === 'file' && item.file_group
? findPosterPath(item.file_group) : null;
return <Table.Row key={item.id} verticalAlign='middle'>
<Table.Cell collapsing>{String(index + 1).padStart(2, '0')}</Table.Cell>
<Table.Cell collapsing textAlign='center' style={{width: '90px'}}>
{poster
? <Image src={`/media/${encodeMediaPath(poster)}`} alt=''
centered style={{maxHeight: '45px', maxWidth: '80px', width: 'auto'}}/>
: <Icon {...itemIcon(item)} size='large'/>}
</Table.Cell>
<Table.Cell>
<ItemTitle item={item} label={label}/>
</Table.Cell>
{editable && <Table.Cell collapsing textAlign='right'>
<Button icon='arrow up' size='mini' disabled={index === 0}
onClick={() => onMove(index, -1)}/>
<Button icon='arrow down' size='mini' disabled={index === items.length - 1}
onClick={() => onMove(index, 1)}/>
<Button icon='trash' color='red' size='mini'
onClick={() => onRemove(item.id)}/>
</Table.Cell>}
</Table.Row>;
})}
</Table.Body>
</Table>;
}
// View-only playlist page (opened from the "Name" column). No reorder/remove/add/delete controls;
// just an "Edit" button to switch to the edit page.
export function PlaylistViewPage() {
const {playlistId} = useParams();
const {playlist} = usePlaylist(playlistId);
const {t} = React.useContext(ThemeContext);
useTitle(playlist && playlist.name ? `${playlist.name} Playlist` : 'Playlist');
if (playlist === undefined) {
return <ErrorMessage>Could not fetch playlist</ErrorMessage>;
}
if (playlist === null) {
return <Loader active inline='centered'/>;
}
const items = playlist.items || [];
return <>
<BackButton/>
<Header as='h1'>
{playlist.name}
<Link to={`/playlists/${playlistId}/edit`}>
<Icon name='edit' style={{marginLeft: '0.5em'}}/>
</Link>
</Header>
{playlist.description && <p {...t}>{playlist.description}</p>}
{items.length === 0
? <Message><Message.Header>This playlist is empty</Message.Header></Message>
: <PlaylistItemsTable items={items} editable={false}/>}
</>;
}
// Edit playlist page (opened from the edit icon), mirroring ChannelEditPage: Back/View buttons at
// the top, a form segment for name/description/tag, then a segment with the ordered items.
export function PlaylistEditPage() {
const {playlistId} = useParams();
const {playlist, refetch} = usePlaylist(playlistId);
const navigate = useNavigate();
const wrolMode = useWROLMode();
const [url, setUrl] = useState('');
const [urlTitle, setUrlTitle] = useState('');
const [name, setName] = useState('');
const [description, setDescription] = useState('');
const [directory, setDirectory] = useState('');
const [tagModalOpen, setTagModalOpen] = useState(false);
useTitle(playlist && playlist.name ? `Edit ${playlist.name} Playlist` : 'Edit Playlist');
// Seed the form fields once the playlist loads (keyed on id so a refetch doesn't clobber edits).
const playlistDbId = playlist && playlist.id;
useEffect(() => {
if (playlistDbId) {
setName(playlist.name || '');
setDescription(playlist.description || '');
setDirectory(playlist.directory || '');
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [playlistDbId]);
if (playlist === undefined) {
return <ErrorMessage>Could not fetch playlist</ErrorMessage>;
}
if (playlist === null) {
return <Loader active inline='centered'/>;
}
const items = playlist.items || [];
// Editing is blocked while in WROL Mode (the API enforces this too); fall back to a read-only view.
const editable = !wrolMode;
const handleSave = async () => {
try {
await updatePlaylist(playlistId, {
name: name.trim(),
description: description.trim(),
directory: directory.trim(),
});
toast({
type: 'success', title: 'Playlist Updated',
description: 'Playlist was successfully updated', time: 3000,
});
await refetch();
} catch (e) {
// Error toast already shown by the API client.
}
};
// Set or clear (tagName=null) the playlist's tag from the tag modal. `directory` is the
// modal's move-to suggestion (null when the move toggle is off).
const handleTagSave = async (tagName, directory) => {
try {
await setPlaylistTag(playlistId, tagName || '', directory);
await refetch();
} catch (e) {
}
};
const handleAddUrl = async () => {
const trimmed = url.trim();
if (!trimmed) return;
try {
await addPlaylistItem(playlistId, {
item_kind: 'url', url: trimmed, title: urlTitle.trim() || null,
});
setUrl('');
setUrlTitle('');
await refetch();
} catch (e) {
// Error toast already shown.
}
};
const handleRemove = async (itemId) => {
try {
await removePlaylistItem(playlistId, itemId);
await refetch();
} catch (e) {
}
};
const move = async (index, delta) => {
const ids = items.map(i => i.id);
const target = index + delta;
if (target < 0 || target >= ids.length) return;
[ids[index], ids[target]] = [ids[target], ids[index]];
try {
await reorderPlaylistItems(playlistId, ids);
await refetch();
} catch (e) {
}
};
const handleDelete = async () => {
try {
await deletePlaylist(playlistId);
navigate('/playlists');
} catch (e) {
}
};
const deleteButton = <APIButton
color='red'
size='small'
confirmContent='Are you sure you want to delete this playlist? Its directory will be removed.'
confirmButton='Delete'
confirmHeader='Delete Playlist?'
onClick={handleDelete}
obeyWROLMode={true}
style={{marginTop: '1em'}}
>Delete</APIButton>;
const tagButton = <Button
type="button"
size='small'
onClick={() => setTagModalOpen(true)}
color='violet'
disabled={!editable}
style={{marginTop: '1em'}}
>Tag</Button>;
const actionButtons = <>
{deleteButton}
{tagButton}
</>;
// Minimal form object for CollectionEditForm (mirrors the useForm interface it consumes).
const form = {error: null, loading: false, disabled: !editable || !name.trim(), ready: true};
return <>
<BackButton/>
<Link to={`/playlists/${playlistId}`}>
<Button>View</Button>
</Link>
<CollectionEditForm
form={form}
title='Edit Playlist'
wrolModeContent='Playlist editing is disabled while in WROL Mode.'
actionButtons={actionButtons}
appliedTagName={playlist.tag_name}
onSubmit={handleSave}
>
<Grid.Row>
<Grid.Column width={8}>
<Form.Field>
<label>Playlist Name</label>
<Input
placeholder='Playlist name...'
value={name}
disabled={!editable}
onChange={(e, {value}) => setName(value)}
/>
</Form.Field>
</Grid.Column>
<Grid.Column width={8}>
<Form.Field>
<label>
Directory
<InfoPopup content='Optional custom directory. When empty, the playlist
lives in the Playlists Directory (under its tag, if tagged).'/>
</label>
<DirectorySearch
value={directory}
disabled={!editable}
onSelect={value => setDirectory(value || '')}
/>
</Form.Field>
</Grid.Column>
</Grid.Row>
<Grid.Row>
<Grid.Column>
<Form.Field>
<label>Description</label>
<Input
placeholder='Optional description...'
value={description}
disabled={!editable}
onChange={(e, {value}) => setDescription(value)}
/>
</Form.Field>
</Grid.Column>
</Grid.Row>
</CollectionEditForm>
{/* Tag Modal */}
<CollectionTagModal
open={tagModalOpen}
onClose={() => setTagModalOpen(false)}
currentTagName={playlist.tag_name}
originalDirectory={playlist.directory || ''}
getTagInfo={(tagName) => getCollectionTagInfo(playlistId, tagName)}
onSave={handleTagSave}
collectionName="Playlist"
/>
{/* Items Segment */}
<Segment>
<Header as='h1'>Items</Header>
{items.length === 0
? <Message>
<Message.Header>This playlist is empty</Message.Header>
{editable && <Message.Content>Add a link below.</Message.Content>}
</Message>
: <PlaylistItemsTable items={items} editable={editable} onMove={move} onRemove={handleRemove}/>}
<Header as='h4'>Add a link</Header>
<Form onSubmit={handleAddUrl}>
<Form.Group>
<Form.Field width={8}>
<Input placeholder='URL (e.g. /map?lat=40.76&lon=-111.89&z=10)'
value={url} disabled={!editable}
onChange={(e, {value}) => setUrl(value)}/>
</Form.Field>
<Form.Field width={6}>
<Input placeholder='Title (optional)' value={urlTitle} disabled={!editable}
onChange={(e, {value}) => setUrlTitle(value)}/>
</Form.Field>
<Form.Field>
<Button primary type='submit' disabled={!editable || !url.trim()}>Add</Button>
</Form.Field>
</Form.Group>
</Form>
</Segment>
</>;
}
export function PlaylistsRoute() {
return <PageContainer>
<Routes>
<Route path='/' exact element={<PlaylistsPage/>}/>
<Route path=':playlistId' exact element={<PlaylistViewPage/>}/>
<Route path=':playlistId/edit' exact element={<PlaylistEditPage/>}/>
</Routes>
</PageContainer>;
}

View file

@ -0,0 +1,43 @@
import {safeHref} from './Playlists';
// safeHref is the only thing standing between a stored playlist url item and an executable href,
// so every dangerous-scheme vector must return null.
describe('safeHref', () => {
it('allows http(s) and relative WROLPi paths', () => {
expect(safeHref('/map?lat=40.76&lon=-111.89&z=12')).toBe('/map?lat=40.76&lon=-111.89&z=12');
expect(safeHref('/api/zim/1/entry/home')).toBe('/api/zim/1/entry/home');
expect(safeHref('relative/path')).toBe('relative/path');
expect(safeHref('http://example.com/page')).toBe('http://example.com/page');
expect(safeHref('https://example.com/page')).toBe('https://example.com/page');
});
it('rejects empty values', () => {
expect(safeHref('')).toBeNull();
expect(safeHref(null)).toBeNull();
expect(safeHref(undefined)).toBeNull();
});
it('rejects dangerous schemes', () => {
expect(safeHref('javascript:alert(1)')).toBeNull();
expect(safeHref('data:text/html,<script>alert(1)</script>')).toBeNull();
expect(safeHref('vbscript:msgbox(1)')).toBeNull();
expect(safeHref('blob:https://example.com/uuid')).toBeNull();
expect(safeHref('file:///etc/passwd')).toBeNull();
});
it('rejects case and whitespace smuggling tricks', () => {
expect(safeHref('JaVaScRiPt:alert(1)')).toBeNull();
expect(safeHref(' javascript:alert(1)')).toBeNull();
expect(safeHref('\tjavascript:alert(1)')).toBeNull();
// The URL parser strips tab/newline anywhere, exactly as the browser does when resolving
// the href -- so this must be detected as a javascript: URL and rejected.
expect(safeHref('java\tscript:alert(1)')).toBeNull();
expect(safeHref('java\nscript:alert(1)')).toBeNull();
});
it('treats percent-encoded schemes as harmless relative paths', () => {
// Browsers do not percent-decode before scheme detection; this resolves as a relative
// path, which is safe to keep.
expect(safeHref('javascript%3Aalert(1)')).toBe('javascript%3Aalert(1)');
});
});

View file

@ -15,6 +15,12 @@ import {
FormInput as SFormInput,
Header as SHeader,
Icon as SIcon,
List as SList,
ListContent as SListContent,
ListDescription as SListDescription,
ListHeader as SListHeader,
ListIcon as SListIcon,
ListItem as SListItem,
Loader as SLoader,
Menu as SMenu,
Modal as SModal,
@ -52,6 +58,12 @@ import {
FormInputProps,
HeaderProps,
IconProps,
ListProps,
ListItemProps,
ListIconProps,
ListContentProps,
ListHeaderProps,
ListDescriptionProps,
LoaderProps,
MenuProps,
ModalProps,
@ -398,6 +410,39 @@ export const Modal: ModalComponent = Object.assign(ModalBase, {
Header: ModalHeader,
});
// ----------------------------------------------------------------------------
// List Compound Component
// ----------------------------------------------------------------------------
// Semantic's List supports `inverted`; the inverted class on the parent handles the item text,
// but List.Icon must also receive `inverted` so its color flips in dark mode (the parent's
// inverted class alone does not restyle the icon).
interface ListComponent extends React.FC<ListProps> {
Item: React.FC<ListItemProps>;
Icon: React.FC<ListIconProps>;
Content: React.FC<ListContentProps>;
Header: React.FC<ListHeaderProps>;
Description: React.FC<ListDescriptionProps>;
}
const ListBase: React.FC<ListProps> = (props) => {
const {i} = useContext(ThemeContext);
return <SList {...i} {...props}/>
};
const ListIcon: React.FC<ListIconProps> = (props) => {
const {i} = useContext(ThemeContext);
return <SListIcon {...i} {...props}/>
};
export const List: ListComponent = Object.assign(ListBase, {
Item: SListItem,
Icon: ListIcon,
Content: SListContent,
Header: SListHeader,
Description: SListDescription,
});
// ----------------------------------------------------------------------------
// Confirm Component
// ----------------------------------------------------------------------------

View file

@ -25,6 +25,7 @@ import {ThemeContext} from "../contexts/contexts";
import {Button, darkTheme, Header, Icon, Loader, Segment, Tab} from "./Theme";
import {VideoCard} from "./Videos";
import {TagsSelector} from "../Tags";
import {AddToPlaylistButton} from "./AddToPlaylist";
import {Comment, CommentGroup, Input, Label, Transition} from "semantic-ui-react";
import {TaggedDeleteConfirmModal} from "./TaggedDeleteConfirmModal";
@ -568,6 +569,7 @@ function VideoPage({videoFile, prevFile, nextFile, fetchVideo, ...props}) {
obeyWROLMode={true}
disabled={!videoFile.url}
>Refresh</APIButton>
<AddToPlaylistButton fileGroupId={videoFile.id}/>
</p>
</Segment>

View file

@ -45,6 +45,7 @@ import {
} from "./Common";
import Grid from "semantic-ui-react/dist/commonjs/collections/Grid";
import {TagsSelector} from "../Tags";
import {AddToPlaylistButton} from "./AddToPlaylist";
import {
deleteOutdatedZims,
fetchZims,
@ -201,10 +202,13 @@ const ZimSearchEntry = ({zimId, onTag, onUntag, entry}) => {
</Modal.Content>
<Modal.Actions>
<Grid>
<Grid.Column mobile={10} tablet={14}>
<Grid.Column mobile={10} tablet={12}>
<TagsSelector selectedTagNames={tag_names} onAdd={localAddTag} onRemove={localUntag}/>
</Grid.Column>
<Grid.Column width={2}>
<Grid.Column width={4} textAlign='right'>
<AddToPlaylistButton
zim={{zimId, entry: path, title: (title || path).replace(/<[^>]*>/g, '')}}
content={null} title='Add to Playlist'/>
<Button color='blue' as='a' href={url} target='_blank' rel='noopener noreferrer'>Open</Button>
</Grid.Column>
</Grid>

View file

@ -24,6 +24,7 @@ const TYPE_LABELS = {
domain: 'Domains',
download: 'Downloads',
inventory: 'Inventories',
playlist: 'Playlists',
};
const TYPE_COLUMNS = {
@ -34,6 +35,7 @@ const TYPE_COLUMNS = {
domain: [{key: 'name', header: 'Name'}, {key: 'directory', header: 'Directory'}],
download: [{key: 'url', header: 'URL'}, {key: 'downloader', header: 'Downloader'}],
inventory: [{key: 'name', header: 'Name'}],
playlist: [{key: 'name', header: 'Name'}, {key: 'items', header: 'Items'}],
};
function groupItemsByType(items) {

View file

@ -473,6 +473,7 @@ export function SettingsPage() {
throttle_on_startup: settings.throttle_on_startup,
videos_destination: settings.videos_destination,
zims_destination: settings.zims_destination,
playlists_destination: settings.playlists_destination,
save_ffprobe_json: settings.save_ffprobe_json,
});
}, [JSON.stringify(settings)]);
@ -862,6 +863,19 @@ export function SettingsPage() {
</Form.Field>
</GridColumn>
</GridRow>
<GridRow columns={2}>
<GridColumn>
<Form.Field>
<label>Playlists Directory</label>
<Input
label={mediaDirectoryLabel}
value={state.playlists_destination}
disabled={!editSpecialDirectories}
onChange={(e, d) => handleInputChange(e, 'playlists_destination', d.value)}
/>
</Form.Field>
</GridColumn>
</GridRow>
<GridRow columns={1}>
<GridColumn>
<Toggle

View file

@ -1,5 +1,6 @@
from . import lib
from .config import (collections_config, CollectionsConfig, save_collections_config)
from . import sync # noqa: F401 (registers the sync_playlists_directory switch handler)
from .config import (playlists_config, PlaylistsConfig, save_playlists_config)
from .errors import UnknownCollection
from .models import Collection, CollectionItem
from .types import collection_type_registry
@ -7,10 +8,10 @@ from .types import collection_type_registry
__all__ = [
'Collection',
'CollectionItem',
'CollectionsConfig',
'PlaylistsConfig',
'UnknownCollection',
'collection_type_registry',
'collections_config',
'lib',
'save_collections_config',
'playlists_config',
'save_playlists_config',
]

View file

@ -10,6 +10,7 @@ from sanic_ext.extensions.openapi import openapi
from wrolpi.api_utils import json_response
from wrolpi.common import wrol_mode_check
from wrolpi.errors import ValidationError
from wrolpi.schema import JSONErrorResponse
from . import lib, schema
from .errors import UnknownCollection
@ -53,6 +54,100 @@ async def get_collections_endpoint(request: Request):
return json_response(response_data)
@collection_bp.post('/')
@openapi.definition(
summary='Create a collection (defaults to a playlist)',
body=schema.CollectionCreateRequest,
validate=True,
)
@openapi.response(HTTPStatus.CREATED, description="Collection created")
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@wrol_mode_check
async def create_collection_endpoint(request: Request, body: schema.CollectionCreateRequest):
"""Create a new Collection. Defaults to kind='playlist' (a manually-curated, ordered list)."""
session = request.ctx.session
try:
collection = lib.create_collection(
session, name=body.name, description=body.description, kind=body.kind or 'playlist')
except ValidationError as e:
return json_response({'error': str(e)}, status=HTTPStatus.BAD_REQUEST)
data = lib.get_collection_with_stats(session, collection.id)
return json_response({'collection': data}, status=HTTPStatus.CREATED)
@collection_bp.post('/<collection_id:int>/items')
@openapi.definition(
summary='Add an item (file, zim, or url) to a collection',
body=schema.AddItemRequest,
validate=True,
)
@openapi.response(HTTPStatus.CREATED, description="Item added")
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@wrol_mode_check
async def add_collection_item_endpoint(request: Request, collection_id: int, body: schema.AddItemRequest):
"""Append (or insert at ``position``) a file/zim/url item into the collection."""
session = request.ctx.session
try:
item, created = lib.add_collection_item(
session, collection_id, item_kind=body.item_kind,
file_group_id=body.file_group_id, zim_id=body.zim_id, zim_entry=body.zim_entry,
url=body.url, title=body.title, position=body.position)
except UnknownCollection as e:
return json_response({'error': str(e)}, status=HTTPStatus.NOT_FOUND)
except ValidationError as e:
return json_response({'error': str(e)}, status=HTTPStatus.BAD_REQUEST)
# 201 when a new item was created; 200 when the item already existed (idempotent add).
status = HTTPStatus.CREATED if created else HTTPStatus.OK
return json_response({'item': item.dict()}, status=status)
@collection_bp.delete('/<collection_id:int>/items/<item_id:int>')
@openapi.summary('Remove an item from a collection')
@openapi.response(HTTPStatus.NO_CONTENT, description="Item removed")
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@wrol_mode_check
async def remove_collection_item_endpoint(request: Request, collection_id: int, item_id: int):
"""Remove a single item by its id and close the ordering gap."""
session = request.ctx.session
try:
removed = lib.remove_collection_item(session, collection_id, item_id)
except UnknownCollection as e:
return json_response({'error': str(e)}, status=HTTPStatus.NOT_FOUND)
if not removed:
return json_response({'error': f'Item {item_id} not found in collection {collection_id}'},
status=HTTPStatus.NOT_FOUND)
return response.raw('', HTTPStatus.NO_CONTENT)
@collection_bp.put('/<collection_id:int>/items/order')
@openapi.definition(
summary='Reorder a collection\'s items',
body=schema.ReorderItemsRequest,
validate=True,
)
@openapi.response(HTTPStatus.OK, description="Items reordered")
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@wrol_mode_check
async def reorder_collection_items_endpoint(request: Request, collection_id: int,
body: schema.ReorderItemsRequest):
"""Set the item order from a full list of item ids (used by drag-and-drop)."""
session = request.ctx.session
try:
lib.reorder_collection_items(session, collection_id, body.item_ids)
except UnknownCollection as e:
return json_response({'error': str(e)}, status=HTTPStatus.NOT_FOUND)
except ValueError as e:
return json_response({'error': str(e)}, status=HTTPStatus.BAD_REQUEST)
data = lib.get_collection_with_stats(session, collection_id)
return json_response({'collection': data})
@collection_bp.get('/<collection_id:int>')
@openapi.summary('Get a single collection by ID')
@openapi.response(HTTPStatus.OK, description="Collection details with statistics")
@ -102,7 +197,8 @@ async def put_collection_endpoint(request: Request, collection_id: int, body: sc
collection_id=collection_id,
directory=body.directory,
tag_name=body.tag_name,
description=body.description
description=body.description,
name=body.name,
)
# Return updated collection data

View file

@ -1,126 +1,153 @@
import pathlib
import shutil
from dataclasses import dataclass, field
from typing import List
from wrolpi.common import ConfigFile, logger
from wrolpi.common import ConfigFile, get_media_directory, get_relative_to_media_directory, logger
from wrolpi.db import get_db_session
from wrolpi.events import Events
from wrolpi.switches import register_switch_handler, ActivateSwitchMethod
from wrolpi.switches import ActivateSwitchMethod, register_switch_handler
logger = logger.getChild(__name__)
__all__ = ['CollectionsConfig', 'collections_config', 'save_collections_config']
__all__ = ['PlaylistsConfig', 'playlists_config', 'save_playlists_config', 'get_playlists_config']
@dataclass
class CollectionsConfigValidator:
"""Validator for collections config file."""
class PlaylistsConfigValidator:
"""Validator for the playlists config file."""
version: int = 0
collections: List[dict] = field(default_factory=list)
playlists: List[dict] = field(default_factory=list)
class CollectionsConfig(ConfigFile):
"""
Config file for Collections.
class PlaylistsConfig(ConfigFile):
"""Config file for playlists (Collections with kind='playlist').
This config stores Collection metadata (name, description, directory, tag)
but NOT the individual items. Items are stored in the database.
For directory-restricted collections, items are auto-populated from the directory.
For unrestricted collections, items are managed manually through the API.
Unlike other collection kinds, playlists are user-curated and ordered, with no other config
that owns them — so this config stores each playlist's metadata AND its ordered item list
(files/zim-articles/urls), making the config the source of truth that survives a DB rebuild.
Format:
collections:
- name: "My Videos"
description: "Favorite videos"
directory: "videos/favorites" # optional, enables auto-population
tag_name: "favorites" # optional
- name: "Reading List"
description: "Books to read"
# No directory means manual item management
playlists:
- name: "Fire Making"
description: "..."
tag_name: "..." # optional
items:
- file: "videos/foo.mp4"
- zim: "zims/wikipedia.zim"
entry: "A/Fire"
title: "Making Fire"
- url: "/map?lat=1&lon=2&z=10"
title: "The spot"
"""
file_name = 'collections.yaml'
validator = CollectionsConfigValidator
default_config = dict(
version=0,
collections=[],
)
# Use wider width to accommodate longer paths
file_name = 'playlists.yaml'
validator = PlaylistsConfigValidator
default_config = dict(version=0, playlists=[])
width = 120
def __getitem__(self, item):
return self._config[item]
def __setitem__(self, key, value):
self._config[key] = value
@property
def collections(self) -> List[dict]:
"""Get list of collection configs."""
return self._config.get('collections', [])
def playlists(self) -> List[dict]:
return self._config.get('playlists', [])
def import_config(self, file: pathlib.Path = None, send_events=False):
"""Import collections from config file into database."""
from .models import Collection
"""Import playlists (and their ordered items) from the config into the database."""
from modules.zim.models import Zim
from wrolpi.files.models import FileGroup
from wrolpi.tags import Tag
super().import_config(file, send_events)
from .models import Collection, CollectionItem
file = file or self.get_file()
file_str = str(self.get_relative_file())
collections_data = self._config.get('collections', [])
if not collections_data:
logger.info(f'No collections to import from {file_str}')
# A missing config is NOT an empty config (mirrors ChannelsConfig). Deleting here would
# cascade: the sync prunes the on-disk playlist dirs and the next dump writes an empty
# config -- a single lost file would permanently destroy every playlist.
if not file.is_file():
logger.info(f'No playlists config file, skipping import: {file_str}')
self.successful_import = True
return
logger.info(f'Importing {len(collections_data)} collections from {file_str}')
super().import_config(file, send_events)
playlists_data = self._config.get('playlists', [])
# An empty playlists list never deletes DB records (mirrors ChannelsConfig). Deleting all
# playlists is done through the API, which dumps the resulting (empty) config itself.
if not playlists_data:
logger.info(f'No playlists in config, preserving existing DB playlists: {file_str}')
self.successful_import = True
return
try:
with get_db_session(commit=True) as session:
# Track which collections were imported
imported_dirs = set() # absolute paths for directory-restricted collections
imported_pairs = set() # (name, kind) for unrestricted collections
# Import each collection using from_config
for idx, collection_data in enumerate(collections_data):
try:
name = collection_data.get('name')
if not name:
logger.error(f'Collection at index {idx} has no name, skipping')
continue
# Use Collection.from_config to create/update
# This will auto-populate items if directory exists
collection = Collection.from_config(collection_data, session)
if collection.directory:
# Normalize to absolute string for comparison
imported_dirs.add(str(collection.directory))
else:
imported_pairs.add((collection.name, collection.kind))
except Exception as e:
logger.error(f'Failed to import collection at index {idx}', exc_info=e)
imported_names = set()
for data in playlists_data:
name = data.get('name')
if not name:
logger.error(f'Playlist config entry has no name, skipping: {data}')
continue
# Delete collections that are no longer in config
all_collections = session.query(Collection).all()
for collection in all_collections:
if collection.directory:
if str(collection.directory) not in imported_dirs:
logger.info(
f'Deleting collection {repr(collection.name)} at {collection.directory} (no longer in config)')
session.delete(collection)
else:
if (collection.name, collection.kind) not in imported_pairs:
logger.info(
f"Deleting unrestricted collection {repr(collection.name)} kind={collection.kind} (no longer in config)")
session.delete(collection)
collection = session.query(Collection).filter_by(
name=name, kind='playlist').one_or_none()
if not collection:
collection = Collection(name=name, kind='playlist')
session.add(collection)
collection.description = data.get('description')
total_imported = len(imported_dirs) + len(imported_pairs)
logger.info(f'Successfully imported {total_imported} collections from {file_str}')
# Optional custom directory (media-relative); None when not customized.
directory = data.get('directory')
collection.directory = (get_media_directory() / directory) if directory else None
tag_name = data.get('tag_name')
collection.tag = Tag.get_by_name(session, tag_name) if tag_name else None
session.flush([collection])
# Rebuild items to match the config's order exactly.
for item in list(collection.items):
session.delete(item)
session.flush()
position = 1
for item_data in (data.get('items') or []):
item = None
if 'file' in item_data:
path = get_media_directory() / item_data['file']
fg = session.query(FileGroup).filter_by(primary_path=str(path)).one_or_none()
if fg:
item = CollectionItem(collection_id=collection.id, item_kind='file',
file_group_id=fg.id)
else:
logger.warning(f'Playlist {name!r}: file not indexed, skipping '
f'{item_data["file"]!r}')
elif 'zim' in item_data:
zpath = get_media_directory() / item_data['zim']
zim = session.query(Zim).filter_by(path=str(zpath)).one_or_none()
if zim:
item = CollectionItem(collection_id=collection.id, item_kind='zim',
zim_id=zim.id, zim_entry=item_data.get('entry'))
else:
logger.warning(f'Playlist {name!r}: zim not found, skipping '
f'{item_data["zim"]!r}')
elif 'url' in item_data:
item = CollectionItem(collection_id=collection.id, item_kind='url',
url=item_data['url'])
if item is not None:
item.title = item_data.get('title')
item.position = position
position += 1
session.add(item)
session.flush()
imported_names.add(name)
# Delete playlists that are no longer present in the config.
for collection in session.query(Collection).filter_by(kind='playlist').all():
if collection.name not in imported_names:
logger.info(f'Deleting playlist {collection.name!r} (no longer in config)')
session.delete(collection)
logger.info(f'Imported {len(imported_names)} playlists from {file_str}')
self.successful_import = True
except Exception as e:
self.successful_import = False
message = f'Failed to import {file_str} config!'
@ -129,35 +156,134 @@ class CollectionsConfig(ConfigFile):
Events.send_config_import_failed(message)
raise
def preview_backup_import(self, backup_date: str, mode: str) -> dict:
"""Preview which playlists a backup import would add/remove (keyed by playlist name)."""
backup_file = self._get_backup_file(backup_date)
backup_data = self.read_config_file(backup_file)
current_data = self.read_config_file() if self.get_file().is_file() else dict(playlists=[], version=0)
backup_playlists = backup_data.get('playlists', [])
current_playlists = current_data.get('playlists', [])
current_names = {p.get('name') for p in current_playlists if p.get('name')}
backup_names = {p.get('name') for p in backup_playlists if p.get('name')}
add = []
remove = []
unchanged = 0
for playlist in backup_playlists:
name = playlist.get('name')
if name and name not in current_names:
add.append(dict(type='playlist', name=name, items=len(playlist.get('items') or [])))
elif name:
unchanged += 1
if mode == 'overwrite':
for name in sorted(current_names - backup_names):
remove.append(dict(type='playlist', name=name))
return dict(mode=mode, add=add, remove=remove, unchanged=unchanged)
def import_backup(self, backup_date: str, mode: str, send_events: bool = False):
"""Restore playlists from a dated backup of playlists.yaml.
'overwrite' replaces the config with the backup; 'merge' adds backup playlists whose names
are not already in the current config. Either way the result is imported into the DB.
"""
self._preserve_current_config()
backup_file = self._get_backup_file(backup_date)
config_file = self.get_file()
if mode == 'overwrite':
config_file.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(backup_file, config_file)
elif mode == 'merge':
backup_data = self.read_config_file(backup_file)
current_data = self.read_config_file() if config_file.is_file() else dict(playlists=[], version=0)
current_names = {p.get('name') for p in current_data.get('playlists', []) if p.get('name')}
merged_playlists = list(current_data.get('playlists', []))
for playlist in backup_data.get('playlists', []):
if playlist.get('name') and playlist['name'] not in current_names:
merged_playlists.append(playlist)
merged_data = dict(
playlists=merged_playlists,
version=current_data.get('version', 0) + 1,
)
config_file.parent.mkdir(parents=True, exist_ok=True)
self.write_config_data(merged_data, config_file)
self.import_config(send_events=send_events)
# Materialize the restored playlists on disk.
from .sync import sync_playlists_directory
sync_playlists_directory.activate_switch()
def dump_config(self, file: pathlib.Path = None, send_events=False, overwrite=False):
"""Dump all collections from database to config file."""
"""Dump all playlists (with their ordered items) to the config file."""
from .models import Collection
logger.info('Dumping collections to config')
# playlists.yaml is derived from the database (the source of truth here), so the dump should
# always win. Sync our in-memory version up to the on-disk version first, otherwise a Sanic
# worker (or freshly-restarted process) that hasn't re-imported would refuse to overwrite a
# newer on-disk config.
target = file or self.get_file()
try:
if target.exists():
disk_version = (self.read_config_file(target) or {}).get('version') or 0
if disk_version > (self._config.get('version') or 0):
self._config['version'] = disk_version
except Exception as e:
logger.warning(f'Could not read existing playlists config version: {e}')
with get_db_session() as session:
# Order by name for consistency
collections = session.query(Collection).order_by(Collection.name).all()
playlists = session.query(Collection).filter_by(kind='playlist') \
.order_by(Collection.name).all()
data = []
for collection in playlists:
entry = {'name': collection.name}
if collection.description:
entry['description'] = collection.description
if collection.tag_name:
entry['tag_name'] = collection.tag_name
if collection.directory:
# Custom playlist directory; stored relative to the media directory.
entry['directory'] = str(get_relative_to_media_directory(collection.directory))
entry['items'] = [c for c in (i.to_config() for i in collection.items) if c]
data.append(entry)
self._config['playlists'] = data
# Use to_config to export each collection
collections_data = [collection.to_config() for collection in collections]
self._config['collections'] = collections_data
logger.info(f'Dumping {len(collections_data)} collections to config')
logger.info(f'Dumping {len(data)} playlists to config')
self.save(file, send_events, overwrite)
# Global instance
collections_config = CollectionsConfig()
playlists_config = PlaylistsConfig()
# Switch handler for saving collections config
@register_switch_handler('save_collections_config')
def save_collections_config():
"""Save the collections config when the switch is activated."""
collections_config.background_dump.activate_switch()
@register_switch_handler('save_playlists_config')
def save_playlists_config():
"""Save the playlists config when the switch is activated."""
playlists_config.background_dump.activate_switch()
# Explicit type for activate_switch helper
save_collections_config: ActivateSwitchMethod
def get_playlists_config() -> PlaylistsConfig:
return playlists_config
@register_switch_handler('import_playlists_config')
def import_playlists_config():
"""Re-import playlists.yaml into the database.
Activated when a global refresh completes: after a DB rebuild, playlists.yaml is imported at
startup before any files are indexed, so its file/zim items are skipped. The config still
holds them; once the refresh has re-indexed the files, this re-import restores those items.
"""
playlists_config.import_config()
# Restored items need their hardlinks/stubs re-created.
from .sync import sync_playlists_directory
sync_playlists_directory.activate_switch()
save_playlists_config: ActivateSwitchMethod
import_playlists_config: ActivateSwitchMethod

View file

@ -1,14 +1,8 @@
from wrolpi.errors import APIError
from wrolpi.errors import APIError, UnknownCollection
__all__ = ['UnknownCollection', 'ReorganizationConflict']
class UnknownCollection(APIError):
"""Cannot find Collection"""
code = 'UNKNOWN_COLLECTION'
summary = 'Cannot find Collection'
class ReorganizationConflict(APIError):
"""Reorganization would cause filename conflicts"""
code = 'REORGANIZATION_CONFLICT'

View file

@ -17,8 +17,8 @@ from wrolpi.errors import FileWorkerConflict
from wrolpi.errors import ValidationError
from wrolpi.events import Events
from wrolpi.tags import Tag
from .errors import UnknownCollection
from .models import Collection, validate_collection_directory
from .types import collection_type_registry
logger = logger.getChild(__name__)
@ -26,6 +26,10 @@ __all__ = [
'get_collections',
'get_collection_with_stats',
'get_domain_statistics',
'create_collection',
'add_collection_item',
'remove_collection_item',
'reorder_collection_items',
'update_collection',
'refresh_collection',
'tag_collection',
@ -35,6 +39,23 @@ __all__ = [
]
def _activate_config_save(kind: str):
"""Activate the config-save switch for the config that owns this kind of Collection.
Author/subject collections are re-derived on refresh and have no config."""
if kind == 'domain':
# Local import to avoid circular import: collections -> archive -> collections
from modules.archive.lib import save_domains_config
save_domains_config.activate_switch()
elif kind == 'channel':
# Local import to avoid circular import: collections -> videos -> collections
from modules.videos.lib import save_channels_config
save_channels_config.activate_switch()
elif kind == 'playlist':
from .config import save_playlists_config
save_playlists_config.activate_switch()
def get_collections(session: Session, kind: Optional[str] = None) -> List[dict]:
"""
Get all collections, optionally filtered by kind.
@ -169,10 +190,7 @@ def get_collection_with_stats(session: Session, collection_id: int) -> dict:
Raises:
UnknownCollection: If collection not found
"""
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
collection = Collection.find_by_id(session, collection_id)
# Get base collection data
data = collection.__json__()
@ -216,13 +234,17 @@ def get_collection_with_stats(session: Session, collection_id: int) -> dict:
data['total_size'] = 0
else:
# Generic collections (author, subject, etc.): count actual CollectionItem rows.
# Generic collections (author, subject, playlist, etc.): count actual CollectionItem rows.
from .models import CollectionItem
item_count = session.query(func.count(CollectionItem.id)).filter(
CollectionItem.collection_id == collection_id
).scalar()
data['item_count'] = item_count or 0
# Playlists are manually curated and ordered, so return the full ordered item list.
if collection.kind == 'playlist':
data['items'] = [item.dict() for item in collection.items]
return data
@ -232,6 +254,7 @@ def update_collection(
directory: Optional[str] = None,
tag_name: Optional[str] = None,
description: Optional[str] = None,
name: Optional[str] = None,
) -> Collection:
"""
Update a collection's properties.
@ -242,6 +265,7 @@ def update_collection(
directory: New directory (relative or absolute path), or None to clear
tag_name: New tag name, empty string to clear, or None to leave unchanged
description: New description, or None to leave unchanged
name: New name (rename), or None to leave unchanged
Returns:
Updated Collection object
@ -250,12 +274,27 @@ def update_collection(
UnknownCollection: If collection not found
ValidationError: If validation fails
"""
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
collection = Collection.find_by_id(session, collection_id)
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
# Rename if provided.
if name is not None:
name = name.strip()
if not name:
raise ValidationError('Collection name cannot be empty')
if name != collection.name:
if not collection_type_registry.validate(collection.kind, name):
raise ValidationError(f'Invalid {collection.kind} name {name!r}')
duplicate = session.query(Collection).filter(
Collection.name == name,
Collection.kind == collection.kind,
Collection.id != collection.id,
).first()
if duplicate:
raise ValidationError(f'A {collection.kind} named {name!r} already exists')
collection.name = name
# Update directory if provided
old_directory = collection.directory
if directory is not None:
if directory:
# Convert relative path to absolute with validation
@ -271,33 +310,157 @@ def update_collection(
# Update tag if provided
if tag_name is not None:
if tag_name:
# Set or create tag
if not collection.directory:
# Set or create tag. Playlists manage their own on-disk layout (the sync namespaces them
# under the tag), so they may be tagged without a `directory`; other kinds require one.
if collection.kind != 'playlist' and not collection.directory:
raise ValidationError(
f"Collection '{collection.name}' has no directory. "
f"Set a directory before tagging."
)
tag = session.query(Tag).filter_by(name=tag_name).one_or_none()
if not tag:
tag = Tag(name=tag_name)
session.add(tag)
session.flush()
tag = Tag.get_or_create_tag(session, name=tag_name)
# Assign the relationship too so `collection.tag_name` is fresh below.
collection.tag = tag
collection.tag_id = tag.id
elif tag_name == '':
# Clear tag (empty string explicitly clears)
collection.tag = None
collection.tag_id = None
session.flush()
# Trigger domain config save if this is a domain collection
if collection.kind == 'domain':
# Local import to avoid circular import: collections -> archive -> collections
from modules.archive.lib import save_domains_config
save_domains_config.activate_switch()
if collection.kind == 'playlist':
from .sync import (cleanup_playlist_directory, get_playlists_directory,
_default_playlist_subdir, sync_playlists_directory)
# A directory equal to the managed tag location is not "custom" -- store None so the sync
# keeps auto-managing it (future tag changes keep moving the playlist automatically).
if collection.directory and \
pathlib.Path(collection.directory) == _default_playlist_subdir(get_playlists_directory(), collection):
collection.directory = None
session.flush()
# Commit before the cleanup and switch activations so the background sync reads the NEW
# state -- otherwise it can run against the pre-commit row, re-create the old directory,
# and undo the cleanup (same pattern as tag_collection).
session.commit()
# A custom directory lives outside the playlists root, so the sync's orphan pruning never
# sees it -- clean up the abandoned directory here when the playlist moves away from it.
if old_directory and str(old_directory) != str(collection.directory or ''):
cleanup_playlist_directory(old_directory)
# Re-sync the on-disk playlist (a tag/directory change moves it).
sync_playlists_directory.activate_switch()
_activate_config_save(collection.kind)
return collection
def create_collection(session: Session, name: str, description: Optional[str] = None,
kind: str = 'playlist') -> Collection:
"""Create a new Collection (defaults to a 'playlist').
Raises:
ValidationError: if the name is empty/invalid for the kind, or a duplicate exists.
"""
name = (name or '').strip()
if not name:
raise ValidationError('Collection name is required')
if not collection_type_registry.is_registered(kind):
raise ValidationError(f'Unknown collection kind: {kind!r}')
if not collection_type_registry.validate(kind, name):
description_msg = collection_type_registry.get_description(kind) or 'Invalid name format'
raise ValidationError(f'Invalid {kind} name {name!r}. {description_msg}')
existing = session.query(Collection).filter_by(name=name, kind=kind).first()
if existing:
raise ValidationError(f'A {kind} named {name!r} already exists')
collection = Collection(name=name, description=description, kind=kind)
session.add(collection)
session.flush([collection])
_activate_config_save(kind)
return collection
def add_collection_item(session: Session, collection_id: int, item_kind: str,
file_group_id: Optional[int] = None, zim_id: Optional[int] = None,
zim_entry: Optional[str] = None, url: Optional[str] = None,
title: Optional[str] = None, position: Optional[int] = None):
"""Add a file/zim/url item to a collection.
Adding an item that already exists is idempotent: the existing CollectionItem is returned.
Returns:
(CollectionItem, created): ``created`` is False when the item already existed.
Raises:
UnknownCollection: if the collection does not exist.
ValidationError: if the item is invalid for its type.
"""
from .models import CollectionItem
collection = Collection.find_by_id(session, collection_id)
if item_kind == 'file':
from wrolpi.files.models import FileGroup
if not file_group_id:
raise ValidationError('file_group_id is required for a file item')
file_group = session.query(FileGroup).filter_by(id=file_group_id).one_or_none()
if not file_group:
raise ValidationError(f'No FileGroup with id {file_group_id}')
created = session.query(CollectionItem).filter_by(
collection_id=collection_id, file_group_id=file_group_id).first() is None
try:
item = collection.add_file_group(file_group, position=position, session=session)
except ValueError as e:
raise ValidationError(str(e))
if title:
item.title = title
elif item_kind == 'zim':
created = session.query(CollectionItem).filter_by(
collection_id=collection_id, zim_id=zim_id, zim_entry=zim_entry).first() is None
try:
item = collection.add_zim_entry(session, zim_id, zim_entry, title=title, position=position)
except ValueError as e:
raise ValidationError(str(e))
elif item_kind == 'url':
created = session.query(CollectionItem).filter_by(
collection_id=collection_id, url=url).first() is None
try:
item = collection.add_url(session, url, title=title, position=position)
except ValueError as e:
raise ValidationError(str(e))
else:
raise ValidationError(f'Unknown item_kind: {item_kind!r}')
session.flush()
from .config import save_playlists_config
save_playlists_config.activate_switch()
from .sync import sync_playlists_directory
sync_playlists_directory.activate_switch()
return item, created
def remove_collection_item(session: Session, collection_id: int, item_id: int) -> bool:
"""Remove an item from a collection by CollectionItem id. Returns True if removed."""
collection = Collection.find_by_id(session, collection_id)
removed = collection.remove_item(session, item_id)
if removed:
from .config import save_playlists_config
save_playlists_config.activate_switch()
from .sync import sync_playlists_directory
sync_playlists_directory.activate_switch()
return removed
def reorder_collection_items(session: Session, collection_id: int, item_ids: List[int]):
"""Reorder a collection's items from a full list of CollectionItem ids."""
collection = Collection.find_by_id(session, collection_id)
collection.reorder_items(session, item_ids)
from .config import save_playlists_config
save_playlists_config.activate_switch()
from .sync import sync_playlists_directory
sync_playlists_directory.activate_switch()
def _get_external_collection_file_paths(session, collection_id: int, directory: pathlib.Path) -> list[pathlib.Path]:
"""Return primary_path for FileGroups associated with a collection but outside its directory.
@ -351,10 +514,7 @@ def refresh_collection(collection_id: int, send_events: bool = True) -> None:
from wrolpi.files.worker import file_worker
with get_db_session() as session:
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
collection = Collection.find_by_id(session, collection_id)
if not collection.directory:
raise ValidationError(
@ -416,85 +576,17 @@ async def tag_collection(
ValidationError: If tagging requirements not met
FileWorkerConflict: If a file operation is in progress and directory change is requested
"""
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
collection = Collection.find_by_id(session, collection_id)
# Track old directory before any changes for potential file moving
old_directory = collection.directory
# If tag_name is None, remove the tag from the collection
if tag_name is None:
collection.tag_id = None
# Update directory if provided (user may want to move files when removing tag)
if directory:
target_directory = validate_collection_directory(directory)
else:
target_directory = collection.directory
# Check if we need to move files
need_to_move = (
target_directory is not None and
old_directory is not None and
target_directory != old_directory
)
if __debug__ and logger.isEnabledFor(TRACE_LEVEL):
logger.trace(f'tag_collection: {repr(collection.name)} need_to_move={need_to_move}, '
f'{old_directory} -> {target_directory}')
if need_to_move and flags.file_worker_busy.is_set():
raise FileWorkerConflict('Refusing to move collection while FileWorker is busy')
session.flush()
# Trigger config save based on collection kind
if collection.kind == 'domain':
# Local import to avoid circular import: collections -> archive -> collections
from modules.archive.lib import save_domains_config
save_domains_config.activate_switch()
elif collection.kind == 'channel':
# Local import to avoid circular import: collections -> videos -> collections
from modules.videos.lib import save_channels_config
save_channels_config.activate_switch()
# Move files if directory changed - run in background so closing tab won't cancel it
if need_to_move:
target_directory.mkdir(parents=True, exist_ok=True)
# Commit tag changes before starting background move
session.commit()
background_task(_background_move_collection(collection.id, target_directory))
# Return info about the un-tagging operation
relative_dir = get_relative_to_media_directory(target_directory) if target_directory else None
return {
'collection_id': collection.id,
'collection_name': collection.name,
'tag_name': None,
'directory': str(relative_dir) if relative_dir else None,
'will_move_files': need_to_move,
}
# Get or create the tag
tag = session.query(Tag).filter_by(name=tag_name).one_or_none()
if not tag:
tag = Tag(name=tag_name)
session.add(tag)
session.flush()
# Determine target directory
# Determine target directory. Collections can be tagged without a directory for UI
# search/filtering, so a missing directory is left missing.
if directory:
# User specified a directory - validate it
target_directory = validate_collection_directory(directory)
elif collection.directory:
# Use existing directory
target_directory = collection.directory
else:
# No directory provided and collection has none - keep it that way
# Collections can be tagged without a directory for UI search/filtering
target_directory = None
target_directory = collection.directory
# Check if we need to move files
need_to_move = (
@ -510,24 +602,19 @@ async def tag_collection(
if need_to_move and flags.file_worker_busy.is_set():
raise FileWorkerConflict('Refusing to move collection while FileWorker is busy')
# Apply the tag
collection.tag_id = tag.id
# Only update directory if we have a target (don't auto-generate for directory-less collections)
# Note: directory update is handled by move_collection if need_to_move, otherwise set directly
if target_directory is not None and not need_to_move:
collection.directory = target_directory
# Apply (or remove) the tag
if tag_name is None:
collection.tag_id = None
else:
collection.tag_id = Tag.get_or_create_tag(session, name=tag_name).id
# Only update directory if we have a target (don't auto-generate for directory-less
# collections). Directory update is handled by move_collection if need_to_move.
if target_directory is not None and not need_to_move:
collection.directory = target_directory
session.flush()
# Trigger config save based on collection kind
if collection.kind == 'domain':
# Local import to avoid circular import: collections -> archive -> collections
from modules.archive.lib import save_domains_config
save_domains_config.activate_switch()
elif collection.kind == 'channel':
# Local import to avoid circular import: collections -> videos -> collections
from modules.videos.lib import save_channels_config
save_channels_config.activate_switch()
_activate_config_save(collection.kind)
# Move files if directory changed - run in background so closing tab won't cancel it
if need_to_move:
@ -536,7 +623,6 @@ async def tag_collection(
session.commit()
background_task(_background_move_collection(collection.id, target_directory))
# Return info about the tagging operation
relative_dir = get_relative_to_media_directory(target_directory) if target_directory else None
return {
'collection_id': collection.id,
@ -569,10 +655,22 @@ def get_tag_info(
Raises:
UnknownCollection: If collection not found
"""
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
collection = Collection.find_by_id(session, collection_id)
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
# Playlists: suggest the managed tag location (<playlists>/<tag>/<name>), even when the
# playlist has no explicit directory (most don't -- the sync manages their location).
if collection.kind == 'playlist':
from wrolpi.common import escape_file_name
from .sync import get_playlists_directory
base = get_playlists_directory()
if tag_name:
base = base / escape_file_name(tag_name)
suggested = base / escape_file_name(collection.name)
return {
'suggested_directory': str(get_relative_to_media_directory(suggested)),
'conflict': False,
'conflict_message': None,
}
# For collections without a directory, return None - no directory suggestions needed
if collection.directory is None:
@ -643,10 +741,7 @@ def delete_collection(
Raises:
UnknownCollection: If collection not found
"""
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
collection = Collection.find_by_id(session, collection_id)
collection_dict = {
'id': collection.id,
@ -656,22 +751,17 @@ def delete_collection(
# Orphan child Archives if this is a domain collection
if collection.kind == 'domain':
# Local imports to avoid circular import: collections -> archive -> collections
# Local import to avoid circular import: collections -> archive -> collections
from modules.archive.models import Archive
from modules.archive.lib import save_domains_config
archives = session.query(Archive).filter_by(collection_id=collection_id).all()
for archive in archives:
archive.collection_id = None
session.flush()
# Trigger domain config save
save_domains_config.activate_switch()
# Orphan child Videos and delete Channel if this is a channel collection
elif collection.kind == 'channel':
# Local imports to avoid circular import: collections -> videos -> collections
from modules.videos.models import Channel, Video
from modules.videos.lib import save_channels_config
# Find the Channel that references this Collection
channel = session.query(Channel).filter_by(collection_id=collection_id).one_or_none()
@ -686,13 +776,27 @@ def delete_collection(
session.delete(channel)
session.flush()
# Trigger channel config save
save_channels_config.activate_switch()
# A playlist's custom directory lives outside the playlists root; remember it so its managed
# files can be cleaned up after the delete (the sync's orphan pruning only covers the root).
playlist_custom_directory = collection.directory if collection.kind == 'playlist' else None
# Delete the collection
session.delete(collection)
session.flush()
# Clean up the playlist's on-disk directory and re-sync.
if collection_dict['kind'] == 'playlist':
from .sync import cleanup_playlist_directory, sync_playlists_directory
# Commit before the cleanup and switch activations so the background sync reads the NEW
# state (the deleted row); a sync of pre-commit state would re-create the directory.
session.commit()
if playlist_custom_directory:
cleanup_playlist_directory(playlist_custom_directory)
sync_playlists_directory.activate_switch()
# Persist the deletion to the owning config.
_activate_config_save(collection_dict['kind'])
return collection_dict

View file

@ -2,7 +2,7 @@ import pathlib
from typing import Optional, List
from sqlalchemy import Column, Integer, String, ForeignKey, Text, DateTime, Index, UniqueConstraint, func, BigInteger, \
or_
or_, CheckConstraint
from sqlalchemy.orm import relationship, Session
from sqlalchemy.orm.collections import InstrumentedList
@ -761,6 +761,105 @@ class Collection(ModelHelper, Base):
).update({'position': CollectionItem.position + shift_by})
session.flush()
def _next_position(self, session: Session) -> int:
"""Return the position to use when appending a new item to the end."""
max_position = session.query(func.max(CollectionItem.position)).filter_by(
collection_id=self.id
).scalar()
return (max_position or 0) + 1
def _place_item(self, session: Session, item: 'CollectionItem', position: Optional[int]):
"""Assign ``position`` to ``item`` (appending or inserting with a shift) and persist it."""
if position is None:
item.position = self._next_position(session)
else:
self._shift_positions(session, position, shift_by=1)
item.position = position
session.add(item)
session.flush([item])
def add_url(self, session: Session, url: str, title: Optional[str] = None,
position: Optional[int] = None) -> 'CollectionItem':
"""Add a `url` item (a link the WROLPi browser can open). Idempotent per URL."""
if not url:
raise ValueError('A url item requires a url')
existing = session.query(CollectionItem).filter_by(collection_id=self.id, url=url).first()
if existing:
logger.warning(f'URL {url!r} already in Collection {self.id}')
return existing
item = CollectionItem(collection_id=self.id, item_kind='url', url=url, title=title)
self._place_item(session, item, position)
return item
def add_zim_entry(self, session: Session, zim_id: int, zim_entry: str, title: Optional[str] = None,
position: Optional[int] = None) -> 'CollectionItem':
"""Add a `zim` item (a single article within a Zim file). Idempotent per (zim, entry)."""
# Local import to avoid a circular import (zim -> tags -> collections).
from modules.zim.models import Zim
zim = session.query(Zim).filter_by(id=zim_id).one_or_none()
if not zim:
raise ValueError(f'No Zim with id {zim_id}')
# Validate the entry exists within the Zim.
try:
zim.find_entry(zim_entry)
except Exception as e:
raise ValueError(f'Zim entry not found: {zim_entry!r}') from e
existing = session.query(CollectionItem).filter_by(
collection_id=self.id, zim_id=zim_id, zim_entry=zim_entry).first()
if existing:
logger.warning(f'Zim entry {zim_entry!r} already in Collection {self.id}')
return existing
item = CollectionItem(collection_id=self.id, item_kind='zim', zim_id=zim_id,
zim_entry=zim_entry, title=title)
self._place_item(session, item, position)
return item
def remove_item(self, session: Session, item_id: int) -> bool:
"""Remove an item by its CollectionItem id (any item_kind). Returns True if removed."""
item = session.query(CollectionItem).filter_by(collection_id=self.id, id=item_id).first()
if not item:
return False
position = item.position
session.delete(item)
session.flush()
# Close the gap left behind.
self._shift_positions(session, position + 1, shift_by=-1)
return True
def reorder_items(self, session: Session, ordered_item_ids: List[int]):
"""Set item order from a full list of CollectionItem ids (for drag-and-drop reordering).
Any items not present in ``ordered_item_ids`` are appended after, keeping their prior order.
@raise ValueError: if an id does not belong to this collection.
"""
items = session.query(CollectionItem).filter_by(collection_id=self.id).all()
by_id = {i.id: i for i in items}
for item_id in ordered_item_ids:
if item_id not in by_id:
raise ValueError(f'Item {item_id} is not in Collection {self.id}')
position = 1
seen = set()
for item_id in ordered_item_ids:
if item_id in seen:
# Ignore duplicate ids so positions stay contiguous (no gaps).
continue
by_id[item_id].position = position
seen.add(item_id)
position += 1
# Preserve any unlisted items after the explicitly-ordered ones.
for item in sorted(items, key=lambda i: i.position):
if item.id not in seen:
item.position = position
position += 1
session.flush()
def get_items(self, session: Session, limit: Optional[int] = None, offset: int = 0) -> List['CollectionItem']:
"""Get items in this collection, ordered by position."""
@ -1041,14 +1140,35 @@ class Collection(ModelHelper, Base):
class CollectionItem(ModelHelper, Base):
"""
Junction table between Collection and FileGroup.
Maintains ordering and metadata about the relationship.
An ordered item within a Collection.
An item is one of three kinds (``item_kind``):
- ``file``: references a FileGroup (video, archive, ebook, image, doc, ...).
- ``zim``: references a single Zim article (``zim_id`` + ``zim_entry`` path), like TagZimEntry.
- ``url``: an arbitrary URL the WROLPi browser can open (e.g. a map location ``/map?lat=&lon=``).
Exactly one kind's columns are populated; enforced by a CHECK constraint.
"""
__tablename__ = 'collection_item'
ITEM_KINDS = ('file', 'zim', 'url')
id = Column(Integer, primary_key=True)
collection_id = Column(Integer, ForeignKey('collection.id', ondelete='CASCADE'), nullable=False)
file_group_id = Column(Integer, ForeignKey('file_group.id', ondelete='CASCADE'), nullable=False)
# Which kind of item this is.
item_kind = Column(String, nullable=False, server_default='file', default='file')
# `file` items: a FileGroup.
file_group_id = Column(Integer, ForeignKey('file_group.id', ondelete='CASCADE'), nullable=True)
# `zim` items: a single article within a Zim file.
zim_id = Column(Integer, ForeignKey('zim.id', ondelete='CASCADE'), nullable=True)
zim_entry = Column(Text, nullable=True)
# `url` items: any URL.
url = Column(Text, nullable=True)
# Display title (used for `url`/`zim` items, and as the on-disk stem; optional for `file`).
title = Column(Text, nullable=True)
# Position in the collection (for ordering)
position = Column(Integer, nullable=False, default=0)
@ -1059,6 +1179,7 @@ class CollectionItem(ModelHelper, Base):
# Relationships
collection = relationship('Collection', back_populates='items')
file_group: FileGroup = relationship('FileGroup')
zim = relationship('Zim')
# Indexes for performance
__table_args__ = (
@ -1066,15 +1187,56 @@ class CollectionItem(ModelHelper, Base):
Index('idx_collection_item_collection_position', 'collection_id', 'position'),
Index('idx_collection_item_file_group_id', 'file_group_id'),
Index('idx_collection_item_position', 'position'),
# A FileGroup/Zim-article/URL appears at most once per collection. NULLs are distinct in
# Postgres, so each unique constraint only applies to items of that kind.
UniqueConstraint('collection_id', 'file_group_id', name='uq_collection_file_group'),
UniqueConstraint('collection_id', 'zim_id', 'zim_entry', name='uq_collection_zim_entry'),
UniqueConstraint('collection_id', 'url', name='uq_collection_url'),
# Exactly one kind's columns are populated.
CheckConstraint(
"(item_kind = 'file' AND file_group_id IS NOT NULL AND zim_id IS NULL AND url IS NULL)"
" OR (item_kind = 'zim' AND zim_id IS NOT NULL AND zim_entry IS NOT NULL"
" AND file_group_id IS NULL AND url IS NULL)"
" OR (item_kind = 'url' AND url IS NOT NULL AND file_group_id IS NULL AND zim_id IS NULL)",
name='ck_collection_item_kind',
),
)
def __repr__(self):
return f'<CollectionItem collection={self.collection_id} file_group={self.file_group_id} position={self.position}>'
return (f'<CollectionItem collection={self.collection_id} kind={self.item_kind} '
f'position={self.position}>')
def dict(self) -> dict:
"""Return dictionary representation."""
d = super(CollectionItem, self).dict()
if self.file_group:
if self.item_kind == 'file' and self.file_group:
d['file_group'] = self.file_group.__json__()
elif self.item_kind == 'zim' and self.zim:
d['zim'] = {'id': self.zim_id, 'entry': self.zim_entry, 'path': str(self.zim.path)}
return d
def to_config(self) -> Optional[dict]:
"""Serialize this item for playlists.yaml.
Files/Zims are referenced by their media-relative path so they survive a DB rebuild.
Returns None if the item cannot currently be represented (e.g. missing relationship).
"""
if self.item_kind == 'file' and self.file_group:
try:
ref = str(get_relative_to_media_directory(self.file_group.primary_path))
except ValueError:
ref = str(self.file_group.primary_path)
data = {'file': ref}
elif self.item_kind == 'zim' and self.zim:
try:
ref = str(get_relative_to_media_directory(self.zim.path))
except ValueError:
ref = str(self.zim.path)
data = {'zim': ref, 'entry': self.zim_entry}
elif self.item_kind == 'url':
data = {'url': self.url}
else:
return None
if self.title:
data['title'] = self.title
return data

View file

@ -17,7 +17,6 @@ from sqlalchemy.orm import Session, joinedload
from wrolpi.common import logger, get_media_directory
from wrolpi.db import get_db_session
from .errors import UnknownCollection
from .models import Collection
logger = logger.getChild(__name__)
@ -236,9 +235,7 @@ def get_reorganization_preview(
with get_db_session() as session:
return get_reorganization_preview(collection_id, session, sample_size)
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
collection = Collection.find_by_id(session, collection_id)
if not collection.directory:
raise ValueError(f"Collection '{collection.name}' has no directory. Cannot reorganize.")
@ -362,9 +359,7 @@ def get_batch_reorganization_preview(
"""
from sqlalchemy.orm import joinedload
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
collection = Collection.find_by_id(session, collection_id)
# Get the current file format from config
current_config_format = collection._get_current_file_format()
@ -891,9 +886,7 @@ def execute_reorganization(
with get_db_session(commit=True) as session:
return execute_reorganization(collection_id, session)
collection = session.query(Collection).filter_by(id=collection_id).one_or_none()
if not collection:
raise UnknownCollection(f"Collection with ID {collection_id} not found")
collection = Collection.find_by_id(session, collection_id)
if not collection.directory:
raise ValueError(f"Collection '{collection.name}' has no directory. Cannot reorganize.")

View file

@ -7,12 +7,45 @@ from dataclasses import dataclass, field
from typing import List, Optional
@dataclass
class CollectionCreateRequest:
"""Request body for creating a collection (default kind 'playlist')."""
name: str
description: Optional[str] = None
kind: str = 'playlist'
@dataclass
class AddItemRequest:
"""Request body for adding an item to a collection.
``item_kind`` selects which fields are required:
- 'file': file_group_id
- 'zim': zim_id + zim_entry
- 'url': url
"""
item_kind: str = 'file'
file_group_id: Optional[int] = None
zim_id: Optional[int] = None
zim_entry: Optional[str] = None
url: Optional[str] = None
title: Optional[str] = None
position: Optional[int] = None
@dataclass
class ReorderItemsRequest:
"""Request body for reordering a collection's items by their ids."""
item_ids: List[int] = field(default_factory=list)
@dataclass
class CollectionUpdateRequest:
"""Request body for updating a collection."""
directory: Optional[str] = None
tag_name: Optional[str] = None
description: Optional[str] = None
name: Optional[str] = None
@dataclass

355
wrolpi/collections/sync.py Normal file
View file

@ -0,0 +1,355 @@
"""On-disk synchronization of playlist collections.
Each playlist (kind='playlist') is materialized into a per-playlist subdirectory of the configured
playlists directory. A tagged playlist is namespaced under its tag (like channel tags), an untagged
one lives directly under the playlists directory:
<media>/playlists/<tag>/<name>/ tagged playlist
<media>/playlists/<name>/ untagged playlist
Items are ordered by a position prefix within the subdirectory:
0001_<file> hard links of a FileGroup's files (file item)
0002_<title>.html a stub redirecting to a Zim article (zim item)
0003_<title>.html a stub redirecting to a URL (url item)
This gives an offline-browsable, ordered view of the playlist. The playlists directory is excluded
from indexing (see ``WROLPiConfig.import_config``), so these managed files are not re-indexed.
Only files matching the ``NNNN_`` position prefix are managed here; anything else a user puts in the
directory is left untouched.
Deletion safety: a managed file is only removed if it is one of our generated HTML stubs, or it still
has another hard link outside the playlists directory (``st_nlink > 1``). We never delete a file
that would lose its last copy -- mirroring the Tags Directory behaviour.
"""
import html
import pathlib
import re
import shutil
import urllib.parse
from wrolpi.common import get_media_directory, get_wrolpi_config, escape_file_name, logger
from wrolpi.db import get_db_session
from wrolpi.switches import register_switch_handler, ActivateSwitchMethod
logger = logger.getChild(__name__)
__all__ = ['sync_playlists_directory', 'get_playlists_directory', 'validate_playlists_destination',
'cleanup_playlist_directory']
# Files we create are prefixed with a zero-padded 1-based position, e.g. "0001_".
MANAGED_NAME = re.compile(r'^\d{4}_')
README_NAME = 'README.txt'
# Embedded in generated stub HTML so we can recognize (and therefore safely delete) our own stubs,
# even though a stub is not a hard link.
STUB_MARKER = '<!--wrolpi-playlist-stub-->'
def get_playlists_directory() -> pathlib.Path:
return get_media_directory() / get_wrolpi_config().playlists_destination
def validate_playlists_destination(destination: str, config=None) -> str:
"""Validate a playlists_destination value, returning an error message (empty string if valid).
The sync deletes loose files at the destination's root and prunes empty directories beneath it,
so the destination must be a plain relative directory inside the media directory which does not
overlap (equal/ancestor/descendant) any other content destination.
"""
config = config or get_wrolpi_config()
parts = pathlib.PurePosixPath(destination or '').parts
if not parts:
return 'Playlists directory cannot be empty'
if pathlib.PurePosixPath(destination).is_absolute():
return 'Playlists directory must be relative to media directory'
if '..' in parts or '.' in parts:
return 'Playlists directory cannot contain ".." or "."'
others = {
'videos_destination': config.videos_destination,
'archive_destination': config.archive_destination,
'map_destination': config.map_destination,
'zims_destination': config.zims_destination,
'tags directory': 'tags', # Fixed path, see wrolpi.tags.get_tags_directory.
}
for name, other in others.items():
if not other:
continue
# Destinations like 'videos/%(channel_tag)s/...' are templates; any directory can match a
# template segment, so only the static prefix is comparable.
other_parts = []
for part in pathlib.PurePosixPath(other).parts:
if '%(' in part:
break
other_parts.append(part)
if not other_parts:
continue
other_parts = tuple(other_parts)
# Reject equality or any ancestor/descendant relationship: the sync would delete or prune
# files belonging to the other destination.
overlap = min(len(parts), len(other_parts))
if parts[:overlap] == other_parts[:overlap]:
return f'Playlists directory cannot overlap {name} ({other!r})'
return ''
def _position_prefix(position: int) -> str:
return f'{position:04d}_'
def _safe_redirect_url(url: str) -> str:
"""Neutralize dangerous URL schemes for use in a stub's href/meta-refresh.
Allows http(s) and scheme-less (relative) URLs like ``/api/...`` or ``/map?...``; anything else
(``javascript:``, ``data:``, ...) becomes ``about:blank`` so opening a stub cannot run script.
"""
url = (url or '').strip()
# Strip control characters that could smuggle a scheme past the check (e.g. "java\tscript:").
url = ''.join(c for c in url if ord(c) >= 0x20 and c != '\x7f')
scheme = urllib.parse.urlparse(url).scheme.lower()
if scheme and scheme not in ('http', 'https'):
return 'about:blank'
return url
def _stub_html(title: str, url: str) -> str:
"""A tiny HTML file that redirects to (and links to) a local URL."""
safe_url = html.escape(_safe_redirect_url(url), quote=True)
safe_title = html.escape(title or url or 'link')
return (
'<!DOCTYPE html>\n<html lang="en"><head><meta charset="utf-8">'
f'{STUB_MARKER}'
f'<meta http-equiv="refresh" content="0; url={safe_url}">'
f'<title>{safe_title}</title></head>\n'
f'<body><p>Opening <a href="{safe_url}">{safe_title}</a> &hellip;</p></body></html>\n'
)
def _zim_entry_url(zim_id: int, zim_entry: str) -> str:
return f'/api/zim/{zim_id}/entry/{urllib.parse.quote(zim_entry or "")}'
def _item_title(item) -> str:
if item.title:
return item.title
if item.item_kind == 'url':
return item.url or 'link'
if item.item_kind == 'zim':
return (item.zim_entry or 'article').split('/')[-1]
return 'item'
def _hardlink_or_copy(source: pathlib.Path, link: pathlib.Path):
"""Hard-link source to link (replacing a stale link); fall back to copy across filesystems."""
if link.exists():
try:
if link.stat().st_ino == source.stat().st_ino:
return # Already the correct hardlink.
except FileNotFoundError:
pass
link.unlink()
link.parent.mkdir(parents=True, exist_ok=True)
try:
link.hardlink_to(source)
except OSError:
# Cross-filesystem (EXDEV) or other link failure -> copy instead.
shutil.copy2(source, link)
def _default_playlist_subdir(playlists_directory: pathlib.Path, collection) -> pathlib.Path:
"""The managed location for a playlist: tagged playlists are namespaced under their tag
(``<playlists>/<tag>/<name>``), untagged playlists live directly under the playlists
directory (``<playlists>/<name>``). Ignores any explicit custom directory."""
base = playlists_directory
if collection.tag_name:
base = base / escape_file_name(collection.tag_name)
return base / escape_file_name(collection.name)
def _playlist_subdir(playlists_directory: pathlib.Path, collection) -> pathlib.Path:
"""The directory a playlist's files live in.
A playlist with an explicit custom ``directory`` uses it as-is (it may live anywhere in the
media directory); otherwise the managed location is used.
"""
if collection.directory:
return pathlib.Path(collection.directory)
return _default_playlist_subdir(playlists_directory, collection)
def _is_stub(path: pathlib.Path) -> bool:
"""True if this file is one of our generated HTML stubs (carries STUB_MARKER)."""
try:
with path.open('rb') as fh:
return STUB_MARKER.encode() in fh.read(512)
except OSError:
return False
def _safe_unlink_managed(path: pathlib.Path) -> bool:
"""Delete a managed playlist file only when it is safe to do so.
Safe means: it is one of our generated stubs, or it still has another hard link outside the
playlists directory (``st_nlink > 1``). A file that would lose its last copy is never deleted
(we warn instead). Returns True if the file is gone afterward.
"""
try:
st = path.stat()
except FileNotFoundError:
return True
if _is_stub(path) or st.st_nlink > 1:
logger.debug(f'Removing playlist file: {path}')
path.unlink()
return True
logger.warning(f'Refusing to delete playlist file without another hardlink: {path}')
return False
def cleanup_playlist_directory(directory: pathlib.Path):
"""Clean up a directory a playlist no longer occupies (deleted, moved, or re-pointed).
Custom playlist directories live outside the playlists root, so the orphan pruning in
``sync_playlists_directory`` never sees them -- the mutation that abandons the directory calls
this instead. Managed (``NNNN_``) files are removed with the usual stub/hardlink safety, then
the directory is deleted only if it is empty; user files are never touched.
"""
directory = pathlib.Path(directory)
if not directory.is_dir():
return
for child in list(directory.iterdir()):
if child.is_file() and MANAGED_NAME.match(child.name):
_safe_unlink_managed(child)
if next(directory.iterdir(), None) is None:
logger.debug(f'Removing abandoned playlist directory: {directory}')
directory.rmdir()
else:
logger.warning(f'Keeping abandoned playlist directory with remaining files: {directory}')
def _sync_one_playlist(playlists_directory: pathlib.Path, collection) -> pathlib.Path:
"""Materialize a single playlist's items into its subdirectory. Returns the subdir."""
directory = _playlist_subdir(playlists_directory, collection)
directory.mkdir(parents=True, exist_ok=True)
desired = set() # Names (relative to `directory`) we should have created.
for item in collection.items: # Ordered by position.
prefix = _position_prefix(item.position)
if item.item_kind == 'file' and item.file_group and item.file_group.files:
for source in item.file_group.my_paths():
if not source.is_file():
continue
name = f'{prefix}{source.name}'
_hardlink_or_copy(source, directory / name)
desired.add(name)
elif item.item_kind == 'zim':
name = f'{prefix}{escape_file_name(_item_title(item))}.html'
(directory / name).write_text(_stub_html(_item_title(item),
_zim_entry_url(item.zim_id, item.zim_entry)))
desired.add(name)
elif item.item_kind == 'url':
name = f'{prefix}{escape_file_name(_item_title(item))}.html'
(directory / name).write_text(_stub_html(_item_title(item), item.url))
desired.add(name)
# Remove our previously-managed files that are no longer wanted (safely; only NNNN_ files).
for path in directory.iterdir():
if path.is_file() and MANAGED_NAME.match(path.name) and path.name not in desired:
_safe_unlink_managed(path)
return directory
def _write_readme(playlists_directory: pathlib.Path):
playlists_directory.mkdir(parents=True, exist_ok=True)
readme = playlists_directory / README_NAME
if not readme.is_file():
readme.write_text(
'This directory contains your playlists as ordered files.\n\n'
'WARNING: This directory is controlled by WROLPi. Managed files (named like 0001_*)\n'
'are created and deleted automatically, and any loose file placed directly in this\n'
'directory will be removed.\n'
)
def _delete_stray_root_files(playlists_directory: pathlib.Path):
"""Remove loose files at the playlists root.
Only the README and playlist (or tag) subdirectories belong at the root of this
WROLPi-managed directory, so any other file a user drops in (e.g. ``playlists/foo``) is deleted.
"""
for path in playlists_directory.iterdir():
if path.is_file() and path.name != README_NAME:
logger.debug(f'Removing stray file from playlists root: {path}')
path.unlink()
def _prune_orphan_dirs(playlists_directory: pathlib.Path, wanted_subdirs):
"""Remove playlist directories (and now-empty tag directories) that are no longer wanted.
Managed files inside an orphaned directory are removed with the same safety as item removal
(stub or another-hardlink only). A directory with surviving files -- because it holds
non-managed files, or files we refused to delete -- is kept.
"""
wanted = set(wanted_subdirs)
# Keep wanted playlist dirs and the tag directories that contain them.
keep = set(wanted)
for directory in wanted:
for parent in directory.parents:
if parent == playlists_directory:
break
keep.add(parent)
# Deepest first, so a playlist leaf dir is emptied before we try to remove its tag dir.
all_dirs = sorted((p for p in playlists_directory.rglob('*') if p.is_dir()),
key=lambda p: len(p.parts), reverse=True)
for path in all_dirs:
if path in keep:
continue
for child in list(path.iterdir()):
if child.is_file() and MANAGED_NAME.match(child.name):
_safe_unlink_managed(child)
if next(path.iterdir(), None) is None:
logger.debug(f'Removing orphaned playlist directory: {path}')
path.rmdir()
else:
logger.warning(f'Refusing to delete playlist directory with remaining files: {path}')
@register_switch_handler('sync_playlists_directory')
def sync_playlists_directory():
"""Synchronize all playlist collections to the playlists directory."""
from .models import Collection
# The sync deletes files; refuse to run against a destination that escapes the media directory
# or overlaps another content destination (e.g. a hand-edited wrolpi.yaml).
if error := validate_playlists_destination(get_wrolpi_config().playlists_destination):
message = f'Refusing to sync playlists directory: {error}'
logger.error(message)
raise RuntimeError(message)
playlists_directory = get_playlists_directory()
media_directory = get_media_directory().resolve()
resolved = playlists_directory.resolve()
if resolved == media_directory or not resolved.is_relative_to(media_directory):
message = f'Refusing to sync playlists directory outside media directory: {resolved}'
logger.error(message)
raise RuntimeError(message)
with get_db_session() as session:
try:
collections = session.query(Collection).filter_by(kind='playlist').all()
if not collections and not playlists_directory.is_dir():
# No playlists and no directory: nothing to materialize or clean up. Don't create
# an empty managed directory (with its README) on systems that use no playlists.
return
_write_readme(playlists_directory)
wanted_subdirs = {_sync_one_playlist(playlists_directory, c) for c in collections}
_prune_orphan_dirs(playlists_directory, wanted_subdirs)
_delete_stray_root_files(playlists_directory)
except Exception as e:
logger.error('Failed to sync playlists directory', exc_info=e)
raise
sync_playlists_directory: ActivateSwitchMethod

View file

@ -1,10 +1,8 @@
import pathlib
import pytest
import yaml
from sqlalchemy.orm import Session
from wrolpi.collections import collections_config
from wrolpi.collections.models import Collection
from wrolpi.common import get_media_directory
@ -16,11 +14,10 @@ async def test_collection_lifecycle_end_to_end(async_client, test_session: Sessi
Lifecycle:
- Start with no collections
- Create a directory-restricted collection with a unique directory containing some videos
- Dump to config (collections.yaml)
- Add a new video to that directory; ensure collection fetches show it
- Remove one video from the collection only (files remain)
- Delete another video entirely (files removed and item gone from collection)
- Delete the collection; dump config; files remain (except the deleted video)
- Delete the collection; files remain (except the deleted video)
"""
# 1) Start clean
assert test_directory.is_dir()
@ -57,19 +54,7 @@ async def test_collection_lifecycle_end_to_end(async_client, test_session: Sessi
fg_ids = {i.file_group_id for i in items}
assert {v1.file_group_id, v2.file_group_id, v3.file_group_id}.issubset(fg_ids)
# 4) Dump collections config; verify entry present
collections_config.dump_config()
config_file = collections_config.get_file()
assert config_file.is_file(), f"Expected config file at {config_file}"
data = yaml.safe_load(config_file.read_text())
assert isinstance(data, dict)
dumped = data.get('collections', [])
# Config stores relative paths for portability
assert any(c.get('name') == 'Lifecycle Test Collection' and c.get('directory') == str(rel_dir) and c.get(
'kind') == 'channel' for c in dumped)
# 5) Add a new video to the same directory; ensure fetch methods include it after population
# 4) Add a new video to the same directory; ensure fetch methods include it after population
v4 = video_factory(with_video_file=coll_dir / 'v4.mp4')
test_session.commit()
@ -107,12 +92,7 @@ async def test_collection_lifecycle_end_to_end(async_client, test_session: Sessi
# 8) Finally, delete the collection
test_session.delete(coll)
test_session.commit()
# Dump config again; collection should be removed from config
collections_config.dump_config()
data = yaml.safe_load(config_file.read_text())
dumped = data.get('collections', [])
assert not any(c.get('name') == 'Lifecycle Test Collection' and c.get('directory') == str(rel_dir) for c in dumped)
assert test_session.query(Collection).count() == 0
# All remaining files should still exist except the one for the deleted video
assert v1.file_group.primary_path.exists()

View file

@ -1,19 +1,14 @@
import pathlib
import pytest
import yaml
from sqlalchemy.orm import Session
from wrolpi.collections import collections_config
from wrolpi.collections.models import Collection
from wrolpi.common import get_media_directory
# from wrolpi.switches import await_switches
@pytest.mark.asyncio
async def test_tagging_directory_collection_moves_files_and_updates_config(
async def test_tagging_directory_collection_moves_files(
test_session: Session,
test_directory: pathlib.Path,
video_factory,
@ -44,15 +39,6 @@ async def test_tagging_directory_collection_moves_files_and_updates_config(
# Verify it populated
assert {i.file_group_id for i in coll.get_items(test_session)} == {v1.file_group_id, v2.file_group_id}
# Dump config and verify (config stores relative paths for portability)
collections_config.dump_config()
cfg_file = collections_config.get_file()
data = yaml.safe_load(cfg_file.read_text())
dumped = data.get('collections', [])
assert any(
c.get('name') == 'Funny Clips' and c.get('directory') == str(rel_dir) and c.get('kind') == 'channel' for c in
dumped)
# Tag the collection -> should move to videos/<Tag>/<Collection Name>
tag = await tag_factory('Funny')
coll.set_tag(tag.id)
@ -79,18 +65,12 @@ async def test_tagging_directory_collection_moves_files_and_updates_config(
assert str(dest_dir) in str(v1.file_group.primary_path)
assert str(dest_dir) in str(v2.file_group.primary_path)
# Dump config again and verify updated directory and tag (config stores relative paths)
collections_config.dump_config()
data = yaml.safe_load(cfg_file.read_text())
dumped = data.get('collections', [])
rel_dest_dir = dest_dir.relative_to(get_media_directory())
assert any(
c.get('name') == 'Funny Clips' and c.get('directory') == str(rel_dest_dir) and c.get('tag_name') == 'Funny' for
c in dumped)
# The tag is recorded on the collection.
assert coll.tag_name == 'Funny'
@pytest.mark.asyncio
async def test_tagging_unrestricted_collection_updates_config_only(
async def test_tagging_unrestricted_collection_does_not_move_files(
test_session: Session,
test_directory: pathlib.Path,
make_files_structure,
@ -105,44 +85,25 @@ async def test_tagging_unrestricted_collection_updates_config_only(
fg1, fg2 = make_files_structure(files, file_groups=True, session=test_session)
test_session.commit()
# Create an unrestricted collection and add both files
# Create an unrestricted collection (no directory) and add both files manually.
coll = Collection.from_config(test_session, {'name': 'Loose Stuff', 'kind': 'channel'})
# Unrestricted, so add manually
coll.add_file_groups(test_session, [fg1, fg2])
test_session.commit()
assert coll.directory is None
# Dump config and verify directory is absent/None
collections_config.dump_config()
cfg_file = collections_config.get_file()
data = yaml.safe_load(cfg_file.read_text())
dumped = data.get('collections', [])
entry = next(c for c in dumped if c.get('name') == 'Loose Stuff')
assert 'directory' not in entry
# Tag the collection; files should not move
# Tag the collection; files should not move (no directory to move into).
before_paths = (fg1.primary_path, fg2.primary_path)
# Create a Tag synchronously and assign by name
# from wrolpi.tags import Tag
# tag = Tag(name='Favorites', color='#00ff00')
# test_session.add(tag)
# test_session.commit()
await tag_factory('Favorites')
coll.set_tag('Favorites')
test_session.commit()
# Assert paths unchanged
# Assert paths unchanged and the tag was applied, directory still unset.
test_session.refresh(fg1)
test_session.refresh(fg2)
after_paths = (fg1.primary_path, fg2.primary_path)
assert before_paths == after_paths
# Dump config and verify tag written, directory still not set
collections_config.dump_config()
data = yaml.safe_load(cfg_file.read_text())
dumped = data.get('collections', [])
entry = next(c for c in dumped if c.get('name') == 'Loose Stuff')
assert entry.get('tag_name') == 'Favorites'
assert 'directory' not in entry
assert coll.tag_name == 'Favorites'
assert coll.directory is None
def test_set_tag_on_directory_less_domain_collection(test_session):
@ -221,3 +182,27 @@ async def test_tag_collection_removes_tag_and_updates_directory(test_session, te
# Verify result contains correct directory
assert result['directory'] == str(untagged_dir.relative_to(test_directory))
@pytest.mark.asyncio
async def test_tag_collection_creates_new_tag(test_session, test_directory, await_switches):
"""Tagging a collection with a Tag that does not exist yet creates the Tag and links it."""
from wrolpi.collections.lib import tag_collection
from wrolpi.tags import Tag
directory = test_directory / 'archive' / 'example.com'
directory.mkdir(parents=True, exist_ok=True)
collection = Collection(name='example.com', kind='domain', directory=directory)
test_session.add(collection)
test_session.commit()
assert test_session.query(Tag).filter_by(name='Brand New').one_or_none() is None
result = await tag_collection(test_session, collection_id=collection.id, tag_name='Brand New')
test_session.commit()
await await_switches(timeout=2)
tag = test_session.query(Tag).filter_by(name='Brand New').one()
assert collection.tag_id == tag.id
assert collection.tag_name == 'Brand New'
assert result['tag_name'] == 'Brand New'

View file

@ -0,0 +1,390 @@
"""Tests for playlist CollectionItem operations: create, add file/url items, reorder, remove."""
from http import HTTPStatus
import pytest
from wrolpi.collections.models import Collection
@pytest.mark.asyncio
async def test_create_playlist(async_client, test_session):
"""A playlist collection can be created via the API."""
_, response = await async_client.post('/api/collections', json={'name': 'Fire Making'})
assert response.status_code == HTTPStatus.CREATED, response.json
data = response.json['collection']
assert data['name'] == 'Fire Making'
assert data['kind'] == 'playlist'
assert data['item_count'] == 0
assert data['items'] == []
# Persisted to the database.
assert test_session.query(Collection).filter_by(name='Fire Making', kind='playlist').one()
@pytest.mark.asyncio
async def test_create_playlist_rejects_blank_and_duplicate(async_client, test_session):
_, response = await async_client.post('/api/collections', json={'name': ' '})
assert response.status_code == HTTPStatus.BAD_REQUEST
_, response = await async_client.post('/api/collections', json={'name': 'Dup'})
assert response.status_code == HTTPStatus.CREATED
_, response = await async_client.post('/api/collections', json={'name': 'Dup'})
assert response.status_code == HTTPStatus.BAD_REQUEST
@pytest.mark.asyncio
async def test_add_url_item(async_client, test_session):
"""A url item (e.g. a map location) can be added and is returned in order."""
_, response = await async_client.post('/api/collections', json={'name': 'Links'})
cid = response.json['collection']['id']
_, response = await async_client.post(f'/api/collections/{cid}/items', json={
'item_kind': 'url', 'url': '/map?lat=40.76&lon=-111.89&z=12', 'title': 'Salt Lake City'})
assert response.status_code == HTTPStatus.CREATED, response.json
item = response.json['item']
assert item['item_kind'] == 'url'
assert item['url'] == '/map?lat=40.76&lon=-111.89&z=12'
assert item['title'] == 'Salt Lake City'
assert item['position'] == 1
_, response = await async_client.get(f'/api/collections/{cid}')
data = response.json['collection']
assert data['item_count'] == 1
assert [i['url'] for i in data['items']] == ['/map?lat=40.76&lon=-111.89&z=12']
@pytest.mark.asyncio
async def test_add_url_item_requires_url(async_client, test_session):
_, response = await async_client.post('/api/collections', json={'name': 'Links'})
cid = response.json['collection']['id']
_, response = await async_client.post(f'/api/collections/{cid}/items', json={'item_kind': 'url'})
assert response.status_code == HTTPStatus.BAD_REQUEST
@pytest.mark.asyncio
async def test_add_item_to_unknown_collection(async_client, test_session):
_, response = await async_client.post('/api/collections/99999/items',
json={'item_kind': 'url', 'url': '/x'})
assert response.status_code == HTTPStatus.NOT_FOUND
@pytest.mark.asyncio
async def test_add_file_item(async_client, test_session, test_directory):
"""An existing FileGroup can be added as a file item."""
from wrolpi.files.models import FileGroup
_, response = await async_client.post('/api/collections', json={'name': 'Mixed'})
cid = response.json['collection']['id']
path = test_directory / 'guide.pdf'
path.write_bytes(b'%PDF-1.4 test')
fg = FileGroup.from_paths(test_session, path)
test_session.commit()
_, response = await async_client.post(f'/api/collections/{cid}/items', json={
'item_kind': 'file', 'file_group_id': fg.id})
assert response.status_code == HTTPStatus.CREATED, response.json
assert response.json['item']['item_kind'] == 'file'
_, response = await async_client.get(f'/api/collections/{cid}')
items = response.json['collection']['items']
assert len(items) == 1
assert items[0]['file_group']['id'] == fg.id
@pytest.mark.asyncio
async def test_reorder_and_remove_items(async_client, test_session):
"""Items can be reordered by id and removed, with positions kept contiguous."""
_, response = await async_client.post('/api/collections', json={'name': 'Order'})
cid = response.json['collection']['id']
ids = []
for name in ('a', 'b', 'c'):
_, response = await async_client.post(f'/api/collections/{cid}/items', json={
'item_kind': 'url', 'url': f'/u/{name}', 'title': name})
ids.append(response.json['item']['id'])
# Reverse the order.
_, response = await async_client.put(f'/api/collections/{cid}/items/order',
json={'item_ids': list(reversed(ids))})
assert response.status_code == HTTPStatus.OK, response.json
items = response.json['collection']['items']
assert [i['url'] for i in items] == ['/u/c', '/u/b', '/u/a']
assert [i['position'] for i in items] == [1, 2, 3]
# Remove the middle item (originally 'b').
_, response = await async_client.delete(f'/api/collections/{cid}/items/{ids[1]}')
assert response.status_code == HTTPStatus.NO_CONTENT
_, response = await async_client.get(f'/api/collections/{cid}')
items = response.json['collection']['items']
assert [i['url'] for i in items] == ['/u/c', '/u/a']
assert [i['position'] for i in items] == [1, 2]
@pytest.mark.asyncio
async def test_tag_playlist_without_directory(async_client, test_session):
"""A playlist can be tagged (and untagged) even though it has no directory."""
_, response = await async_client.post('/api/collections', json={'name': 'Tagged'})
cid = response.json['collection']['id']
# Set a tag -- unlike domains/channels, no directory is required.
_, response = await async_client.put(f'/api/collections/{cid}', json={'tag_name': 'Cooking'})
assert response.status_code == HTTPStatus.OK, response.json
assert response.json['collection']['tag_name'] == 'Cooking'
# Clearing the tag works too.
_, response = await async_client.put(f'/api/collections/{cid}', json={'tag_name': ''})
assert response.status_code == HTTPStatus.OK, response.json
assert response.json['collection']['tag_name'] is None
@pytest.mark.asyncio
async def test_playlist_tag_info_suggests_managed_location(async_client, test_session):
"""tag_info for a playlist suggests the managed tag location, like channels/domains."""
_, response = await async_client.post('/api/collections', json={'name': 'Fire Making'})
cid = response.json['collection']['id']
# With a tag: <playlists>/<tag>/<name>.
_, response = await async_client.post(f'/api/collections/{cid}/tag_info',
json={'tag_name': 'Survival'})
assert response.status_code == HTTPStatus.OK, response.json
assert response.json['suggested_directory'] == 'playlists/Survival/Fire Making'
assert response.json['conflict'] is False
# Without a tag (untagging): <playlists>/<name>.
_, response = await async_client.post(f'/api/collections/{cid}/tag_info', json={'tag_name': None})
assert response.status_code == HTTPStatus.OK, response.json
assert response.json['suggested_directory'] == 'playlists/Fire Making'
@pytest.mark.asyncio
async def test_playlist_directory_matching_managed_location_is_cleared(async_client, test_session):
"""Saving a directory equal to the managed tag location stores None (sync keeps auto-managing)."""
from wrolpi.collections.models import Collection
_, response = await async_client.post('/api/collections', json={'name': 'Managed'})
cid = response.json['collection']['id']
# Tag + move to the suggested (managed) location in one update, like the tag modal does.
_, response = await async_client.put(f'/api/collections/{cid}', json={
'tag_name': 'Survival', 'directory': 'playlists/Survival/Managed'})
assert response.status_code == HTTPStatus.OK, response.json
assert response.json['collection']['tag_name'] == 'Survival'
# The directory equals the managed location, so it is normalized to None (auto-managed).
collection = test_session.query(Collection).filter_by(id=cid).one()
test_session.refresh(collection)
assert collection.directory is None
# A genuinely custom directory is kept.
_, response = await async_client.put(f'/api/collections/{cid}', json={
'directory': 'custom/spot'})
assert response.status_code == HTTPStatus.OK, response.json
assert response.json['collection']['directory'] == 'custom/spot'
@pytest.mark.asyncio
async def test_rename_playlist(async_client, test_session):
"""A playlist can be renamed; blank and duplicate names are rejected."""
_, response = await async_client.post('/api/collections', json={'name': 'Old Name'})
cid = response.json['collection']['id']
_, response = await async_client.put(f'/api/collections/{cid}', json={'name': 'New Name'})
assert response.status_code == HTTPStatus.OK, response.json
assert response.json['collection']['name'] == 'New Name'
# Renaming to another playlist's name is rejected.
_, response = await async_client.post('/api/collections', json={'name': 'Other'})
assert response.status_code == HTTPStatus.CREATED
_, response = await async_client.put(f'/api/collections/{cid}', json={'name': 'Other'})
assert response.status_code == HTTPStatus.BAD_REQUEST
# A blank name is rejected.
_, response = await async_client.put(f'/api/collections/{cid}', json={'name': ' '})
assert response.status_code == HTTPStatus.BAD_REQUEST
@pytest.mark.asyncio
async def test_adding_duplicate_item_is_idempotent(async_client, test_session):
"""Re-adding the same url returns the existing item with 200 (not a new 201)."""
_, response = await async_client.post('/api/collections', json={'name': 'Dups'})
cid = response.json['collection']['id']
_, r1 = await async_client.post(f'/api/collections/{cid}/items',
json={'item_kind': 'url', 'url': '/u/x'})
assert r1.status_code == HTTPStatus.CREATED
first_id = r1.json['item']['id']
_, r2 = await async_client.post(f'/api/collections/{cid}/items',
json={'item_kind': 'url', 'url': '/u/x'})
assert r2.status_code == HTTPStatus.OK # already existed -> not created
assert r2.json['item']['id'] == first_id
_, rg = await async_client.get(f'/api/collections/{cid}')
assert rg.json['collection']['item_count'] == 1
@pytest.mark.asyncio
async def test_reorder_ignores_duplicate_ids(async_client, test_session):
"""Duplicate ids in the reorder list must not leave a position gap."""
_, response = await async_client.post('/api/collections', json={'name': 'DupOrder'})
cid = response.json['collection']['id']
ids = []
for name in ('a', 'b', 'c'):
_, r = await async_client.post(f'/api/collections/{cid}/items',
json={'item_kind': 'url', 'url': f'/u/{name}'})
ids.append(r.json['item']['id'])
# Pass the first id twice; positions must stay contiguous (1, 2, 3).
_, response = await async_client.put(f'/api/collections/{cid}/items/order',
json={'item_ids': [ids[0], ids[1], ids[0]]})
assert response.status_code == HTTPStatus.OK, response.json
positions = sorted(i['position'] for i in response.json['collection']['items'])
assert positions == [1, 2, 3], positions
@pytest.mark.asyncio
async def test_playlist_directory_sync(async_client, test_session, test_directory):
"""Adding items materializes ordered files into the playlists directory; reorder re-prefixes."""
from wrolpi.files.models import FileGroup
_, response = await async_client.post('/api/collections', json={'name': 'Sync Test'})
cid = response.json['collection']['id']
# A real file becomes a FileGroup, added as a file item.
path = test_directory / 'lesson.pdf'
path.write_bytes(b'%PDF-1.4 test')
fg = FileGroup.from_paths(test_session, path)
test_session.commit()
_, rf = await async_client.post(f'/api/collections/{cid}/items',
json={'item_kind': 'file', 'file_group_id': fg.id})
file_item_id = rf.json['item']['id']
# A url item.
_, ru = await async_client.post(f'/api/collections/{cid}/items',
json={'item_kind': 'url', 'url': '/map?lat=1&lon=2&z=3', 'title': 'Map Spot'})
url_item_id = ru.json['item']['id']
playlist_dir = test_directory / 'playlists' / 'Sync Test'
names = sorted(p.name for p in playlist_dir.iterdir())
assert any(n.startswith('0001_') and n.endswith('lesson.pdf') for n in names), names
assert any(n.startswith('0002_') and n.endswith('.html') for n in names), names
# The file is a hard link to the source (same inode).
link = next(p for p in playlist_dir.iterdir() if p.name.startswith('0001_'))
assert link.stat().st_ino == path.stat().st_ino
# Reorder so the url is first; prefixes must follow the new positions.
await async_client.put(f'/api/collections/{cid}/items/order',
json={'item_ids': [url_item_id, file_item_id]})
names = sorted(p.name for p in playlist_dir.iterdir())
assert any(n.startswith('0001_') and n.endswith('.html') for n in names), names
assert any(n.startswith('0002_') and n.endswith('lesson.pdf') for n in names), names
# Removing the url item drops its stub and re-sequences the file to 0001_.
await async_client.delete(f'/api/collections/{cid}/items/{url_item_id}')
names = sorted(p.name for p in playlist_dir.iterdir())
assert not any(n.endswith('.html') for n in names), names
assert any(n.startswith('0001_') and n.endswith('lesson.pdf') for n in names), names
@pytest.mark.asyncio
async def test_url_stub_neutralizes_dangerous_scheme(async_client, test_session, test_directory):
"""A javascript: url item must not produce an executable href in its on-disk stub."""
_, response = await async_client.post('/api/collections', json={'name': 'XSS'})
cid = response.json['collection']['id']
await async_client.post(f'/api/collections/{cid}/items',
json={'item_kind': 'url', 'url': 'javascript:alert(1)', 'title': 'evil'})
stub = next(p for p in (test_directory / 'playlists' / 'XSS').iterdir() if p.name.endswith('.html'))
content = stub.read_text()
assert 'javascript:' not in content, content
assert 'about:blank' in content
@pytest.mark.asyncio
async def test_playlists_config_round_trip(async_client, test_session, test_directory):
"""A playlist's items survive a dump -> wipe DB -> import cycle (config is source of truth)."""
from wrolpi.collections.config import playlists_config
from wrolpi.files.models import FileGroup
_, r = await async_client.post('/api/collections', json={'name': 'Roundtrip'})
cid = r.json['collection']['id']
path = test_directory / 'a.pdf'
path.write_bytes(b'%PDF-1.4 x')
fg = FileGroup.from_paths(test_session, path)
test_session.commit()
await async_client.post(f'/api/collections/{cid}/items',
json={'item_kind': 'file', 'file_group_id': fg.id})
await async_client.post(f'/api/collections/{cid}/items',
json={'item_kind': 'url', 'url': '/map?x=1', 'title': 'Spot'})
# Dump to playlists.yaml, wipe playlists from the DB, then re-import from config.
playlists_config.dump_config()
test_session.query(Collection).filter_by(kind='playlist').delete()
test_session.commit()
assert test_session.query(Collection).filter_by(kind='playlist').count() == 0
playlists_config.import_config()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(name='Roundtrip', kind='playlist').one()
assert [i.item_kind for i in collection.items] == ['file', 'url']
assert collection.items[0].file_group_id == fg.id
assert collection.items[1].url == '/map?x=1'
assert collection.items[1].title == 'Spot'
assert [i.position for i in collection.items] == [1, 2]
@pytest.mark.asyncio
async def test_add_zim_item_end_to_end(async_client, test_session, test_directory):
"""A Zim article can be added, renders an on-disk stub, and survives a config round trip."""
from modules.zim.models import Zim
from wrolpi.collections.config import playlists_config
from wrolpi.vars import PROJECT_DIR
zim_path = test_directory / 'wikipedia.zim'
zim_path.write_bytes((PROJECT_DIR / 'test/zim.zim').read_bytes())
zim = Zim.from_paths(test_session, zim_path)
test_session.commit()
_, response = await async_client.post('/api/collections', json={'name': 'Articles'})
cid = response.json['collection']['id']
# An entry that does not exist in the Zim is rejected.
_, response = await async_client.post(f'/api/collections/{cid}/items', json={
'item_kind': 'zim', 'zim_id': zim.id, 'zim_entry': 'nonexistent-entry'})
assert response.status_code == HTTPStatus.BAD_REQUEST, response.json
# A real entry is added.
_, response = await async_client.post(f'/api/collections/{cid}/items', json={
'item_kind': 'zim', 'zim_id': zim.id, 'zim_entry': 'home', 'title': 'Home Page'})
assert response.status_code == HTTPStatus.CREATED, response.json
item = response.json['item']
assert item['item_kind'] == 'zim'
assert item['zim'] == {'id': zim.id, 'entry': 'home', 'path': str(zim_path)}
# The on-disk stub redirects to the Zim entry API.
stub = test_directory / 'playlists' / 'Articles' / '0001_Home Page.html'
assert stub.is_file()
assert f'/api/zim/{zim.id}/entry/home' in stub.read_text()
# Config round trip: the zim item is stored by media-relative path and restored after a wipe.
playlists_config.dump_config()
test_session.query(Collection).filter_by(kind='playlist').delete()
test_session.commit()
playlists_config.import_config()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(name='Articles', kind='playlist').one()
assert [i.item_kind for i in collection.items] == ['zim']
assert collection.items[0].zim_id == zim.id
assert collection.items[0].zim_entry == 'home'
assert collection.items[0].title == 'Home Page'
@pytest.mark.asyncio
async def test_remove_unknown_item(async_client, test_session):
_, response = await async_client.post('/api/collections', json={'name': 'Empty'})
cid = response.json['collection']['id']
_, response = await async_client.delete(f'/api/collections/{cid}/items/99999')
assert response.status_code == HTTPStatus.NOT_FOUND

View file

@ -0,0 +1,251 @@
"""Tests for the on-disk playlist sync: tag namespacing, safe deletion, and directory cleanup."""
import pytest
from wrolpi.collections.models import Collection
from wrolpi.collections.sync import sync_playlists_directory, get_playlists_directory, \
validate_playlists_destination
from wrolpi.common import get_wrolpi_config
from wrolpi.tags import Tag
def _make_playlist(session, name, tag=None):
collection = Collection(name=name, kind='playlist')
session.add(collection)
session.flush([collection])
if tag is not None:
collection.tag_id = tag.id
session.flush()
return collection
def test_untagged_playlist_lives_at_top_level(test_session, test_directory):
"""An untagged playlist syncs to <playlists>/<name>."""
_make_playlist(test_session, 'Fire Making')
test_session.commit()
sync_playlists_directory()
assert (get_playlists_directory() / 'Fire Making').is_dir()
def test_tagged_playlist_namespaced_under_tag(test_session, test_directory):
"""A tagged playlist syncs to <playlists>/<tag>/<name>."""
tag = Tag(name='Survival')
test_session.add(tag)
test_session.flush()
_make_playlist(test_session, 'Fire Making', tag=tag)
test_session.commit()
sync_playlists_directory()
assert (get_playlists_directory() / 'Survival' / 'Fire Making').is_dir()
assert not (get_playlists_directory() / 'Fire Making').exists()
def test_remove_item_deletes_hardlink_keeps_original(test_session, test_directory):
"""Removing a file item deletes its hard link from the playlist dir, but keeps the original."""
from wrolpi.files.models import FileGroup
source = test_directory / 'guide.pdf'
source.write_bytes(b'%PDF-1.4 test')
file_group = FileGroup.from_paths(test_session, source)
test_session.commit()
collection = _make_playlist(test_session, 'Docs')
item = collection.add_file_group(file_group, session=test_session)
test_session.commit()
sync_playlists_directory()
link = get_playlists_directory() / 'Docs' / '0001_guide.pdf'
assert link.is_file(), 'hard link should be created'
assert source.is_file()
collection.remove_item(test_session, item.id)
test_session.commit()
sync_playlists_directory()
assert not link.exists(), 'hard link should be removed when the item is removed'
assert source.is_file(), 'the original file must survive'
def test_delete_playlist_removes_its_directory(test_session, test_directory):
"""Deleting a playlist removes its on-disk directory."""
collection = _make_playlist(test_session, 'Temp')
collection.add_url(test_session, '/map?lat=1&lon=2&z=3', title='spot')
test_session.commit()
sync_playlists_directory()
directory = get_playlists_directory() / 'Temp'
assert directory.is_dir()
# Delete the playlist; the next sync should prune its now-orphaned directory.
test_session.delete(collection)
test_session.commit()
sync_playlists_directory()
assert not directory.exists()
def test_sync_refuses_to_delete_non_hardlinked_file(test_session, test_directory):
"""A managed-looking file with no other hard link (and not a stub) is preserved, not deleted."""
_make_playlist(test_session, 'Safe')
test_session.commit()
sync_playlists_directory()
directory = get_playlists_directory() / 'Safe'
orphan = directory / '0001_precious.bin'
orphan.write_bytes(b'\x00\x01 irreplaceable')
# The playlist has no items, so this managed-looking file is "stale" -- but it has no other
# hard link and is not one of our stubs, so it must NOT be deleted.
sync_playlists_directory()
assert orphan.is_file()
def test_sync_deletes_stray_root_file(test_session, test_directory):
"""A loose file dropped into the playlists root (not README, not a playlist dir) is removed.
Only README.txt and playlist subdirectories belong at the root of this WROLPi-managed directory.
"""
playlist = _make_playlist(test_session, 'Keeper')
playlist.add_url(test_session, '/u/x', title='x')
test_session.commit()
sync_playlists_directory()
playlists_directory = get_playlists_directory()
stray = playlists_directory / 'foo'
stray.write_text('junk a user dropped here')
sync_playlists_directory()
assert not stray.exists(), 'a stray root file should be deleted during sync'
# The README and real playlist directories are preserved.
assert (playlists_directory / 'README.txt').is_file()
assert (playlists_directory / 'Keeper').is_dir()
def test_sync_refuses_dangerous_destination(test_session, test_directory):
"""The sync deletes files, so it must refuse a destination that escapes the media directory
or overlaps another content destination (e.g. from a hand-edited wrolpi.yaml)."""
_make_playlist(test_session, 'Any')
test_session.commit()
config = get_wrolpi_config()
original = config._config['playlists_destination']
try:
# Traversal outside the media directory.
config._config['playlists_destination'] = '..'
with pytest.raises(RuntimeError):
sync_playlists_directory()
# The media directory itself.
config._config['playlists_destination'] = '.'
with pytest.raises(RuntimeError):
sync_playlists_directory()
# Another content destination: syncing here would delete loose files in videos/.
config._config['playlists_destination'] = 'videos'
with pytest.raises(RuntimeError):
sync_playlists_directory()
finally:
config._config['playlists_destination'] = original
# The default destination still works.
sync_playlists_directory()
assert (get_playlists_directory() / 'Any').is_dir()
def test_validate_playlists_destination(test_directory):
"""Destination validation rejects traversal and overlap with other content destinations."""
config = get_wrolpi_config()
assert validate_playlists_destination('playlists', config) == ''
assert validate_playlists_destination('my/playlists', config) == ''
assert validate_playlists_destination('', config)
assert validate_playlists_destination('/absolute', config)
assert validate_playlists_destination('../escape', config)
assert validate_playlists_destination('a/../../b', config)
# Equal to, child of, and parent of videos_destination are all rejected.
assert validate_playlists_destination('videos', config)
assert validate_playlists_destination('videos/playlists', config)
assert validate_playlists_destination('zims', config)
assert validate_playlists_destination('map/pins', config)
assert validate_playlists_destination('tags', config)
def test_retag_moves_directory_and_prunes_empty_tag_dir(test_session, test_directory):
"""Changing a playlist's tag moves its directory and prunes the now-empty old tag directory."""
tag = Tag(name='Survival')
test_session.add(tag)
test_session.flush()
collection = _make_playlist(test_session, 'Fire Making', tag=tag)
collection.add_url(test_session, '/u/x', title='x')
test_session.commit()
sync_playlists_directory()
assert (get_playlists_directory() / 'Survival' / 'Fire Making').is_dir()
# Remove the tag.
collection.tag_id = None
test_session.commit()
sync_playlists_directory()
assert (get_playlists_directory() / 'Fire Making').is_dir(), 'moved to top level'
assert not (get_playlists_directory() / 'Survival').exists(), 'empty tag dir pruned'
def test_custom_directory_playlist(test_session, test_directory):
"""A playlist with a custom `directory` syncs there instead of under the playlists root."""
custom = test_directory / 'survival' / 'fire'
collection = _make_playlist(test_session, 'Fire Making')
collection.directory = custom
collection.add_url(test_session, '/u/x', title='x')
test_session.commit()
sync_playlists_directory()
assert (custom / '0001_x.html').is_file()
# No directory is created under the playlists root for this playlist.
assert not (get_playlists_directory() / 'Fire Making').exists()
def test_custom_directory_ignores_tag(test_session, test_directory):
"""An explicit custom directory wins over tag namespacing."""
tag = Tag(name='Survival')
test_session.add(tag)
test_session.flush()
custom = test_directory / 'my' / 'spot'
collection = _make_playlist(test_session, 'Fire Making', tag=tag)
collection.directory = custom
collection.add_url(test_session, '/u/x', title='x')
test_session.commit()
sync_playlists_directory()
assert (custom / '0001_x.html').is_file()
assert not (get_playlists_directory() / 'Survival').exists()
def test_cleanup_playlist_directory(test_session, test_directory):
"""cleanup_playlist_directory removes managed files (safely) and the directory if empty."""
from wrolpi.collections.sync import cleanup_playlist_directory
custom = test_directory / 'old' / 'spot'
collection = _make_playlist(test_session, 'Mover')
collection.directory = custom
collection.add_url(test_session, '/u/x', title='x')
test_session.commit()
sync_playlists_directory()
assert (custom / '0001_x.html').is_file()
cleanup_playlist_directory(custom)
assert not custom.exists(), 'managed stub removed and empty dir deleted'
# A directory holding a non-managed file is kept (only managed files are removed).
custom2 = test_directory / 'old' / 'spot2'
custom2.mkdir(parents=True)
(custom2 / 'keep.txt').write_text('user file')
cleanup_playlist_directory(custom2)
assert (custom2 / 'keep.txt').is_file()
assert custom2.is_dir()

View file

@ -0,0 +1,264 @@
"""Tests for PlaylistsConfig disaster recovery: a missing or empty playlists.yaml must never
delete playlists from the database (mirroring ChannelsConfig's guards)."""
from copy import deepcopy
import pytest
from wrolpi.collections.config import get_playlists_config
from wrolpi.collections.models import Collection
@pytest.fixture
def playlists_config(async_client):
"""The global playlists config, reset to defaults (it is not reset by
initialize_configs_contexts like other configs)."""
config = get_playlists_config()
config._config = deepcopy(config.default_config)
return config
def _make_playlist(session, name='Survives'):
collection = Collection(name=name, kind='playlist')
session.add(collection)
collection.add_url(session, '/map?lat=1&lon=2&z=3', title='spot')
session.commit()
return collection
@pytest.mark.asyncio
async def test_import_missing_config_preserves_db(test_session, test_directory, playlists_config):
"""If playlists.yaml does not exist, importing must NOT delete playlists from the DB.
A missing config (partial restore, fresh config directory, disk problem) is not the same as
an explicit "no playlists"; deleting here would cascade into the on-disk sync pruning the
playlist directories and the next dump writing an empty config -- total, permanent loss.
"""
_make_playlist(test_session)
assert not playlists_config.get_file().is_file()
playlists_config.import_config()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(kind='playlist').one()
assert collection.name == 'Survives'
assert len(collection.items) == 1
# Nothing to import is a successful import (matches ChannelsConfig).
assert playlists_config.successful_import is True
@pytest.mark.asyncio
async def test_import_empty_config_preserves_db(test_session, test_directory, playlists_config):
"""A config file with an empty (or absent) playlists list must NOT delete DB playlists.
Matches ChannelsConfig: an empty list never deletes DB records. Deleting all playlists is
done through the UI/API, which dumps the (empty) config itself.
"""
_make_playlist(test_session)
file = playlists_config.get_file()
file.parent.mkdir(parents=True, exist_ok=True)
file.write_text('version: 1\nplaylists: []\n')
playlists_config.import_config()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(kind='playlist').one()
assert collection.name == 'Survives'
assert playlists_config.successful_import is True
@pytest.mark.asyncio
async def test_db_rebuild_round_trip(test_session, test_directory, playlists_config):
"""After a full DB rebuild, playlists are recovered from playlists.yaml.
File items can only be re-linked after the file refresh has re-indexed their FileGroups; until
then they are skipped with a warning, and restored by a later re-import (the config file still
holds them as media-relative paths).
"""
from wrolpi.files.models import FileGroup
pdf = test_directory / 'guide.pdf'
pdf.write_bytes(b'%PDF-1.4 x')
file_group = FileGroup.from_paths(test_session, pdf)
collection = Collection(name='Rebuild', kind='playlist')
test_session.add(collection)
test_session.flush([collection])
collection.add_file_group(file_group, session=test_session)
collection.add_url(test_session, '/map?x=1', title='Spot')
test_session.commit()
playlists_config.dump_config()
# Simulate a DB rebuild: collections AND file groups are gone.
test_session.query(Collection).filter_by(kind='playlist').delete()
test_session.query(FileGroup).delete()
test_session.commit()
playlists_config.import_config()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(name='Rebuild', kind='playlist').one()
# The file is not indexed yet, so only the url item could be restored.
assert [i.item_kind for i in collection.items] == ['url']
# The file refresh re-indexes the file; a re-import restores the file item in its position.
FileGroup.from_paths(test_session, pdf)
test_session.commit()
playlists_config.import_config()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(name='Rebuild', kind='playlist').one()
assert [i.item_kind for i in collection.items] == ['file', 'url']
assert [i.position for i in collection.items] == [1, 2]
@pytest.mark.asyncio
async def test_global_refresh_restores_skipped_playlist_items(
test_session, test_directory, playlists_config, refresh_files, await_switches):
"""A global refresh re-imports playlists.yaml, restoring items skipped at startup.
After a DB rebuild, playlists.yaml is imported before any files are indexed, so file items are
skipped. When the global refresh completes, the worker re-imports the config and the skipped
items are restored.
"""
pdf = test_directory / 'guide.pdf'
pdf.write_bytes(b'%PDF-1.4 x')
file = playlists_config.get_file()
file.parent.mkdir(parents=True, exist_ok=True)
file.write_text(
'version: 1\n'
'playlists:\n'
' - name: Rebuilt\n'
' items:\n'
' - file: guide.pdf\n'
' - url: /map?x=1\n'
)
# Startup import: the file is not indexed yet, so only the url item is restored.
playlists_config.import_config()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(name='Rebuilt', kind='playlist').one()
assert [i.item_kind for i in collection.items] == ['url']
# A global refresh indexes the file and triggers the re-import.
await refresh_files()
await await_switches()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(name='Rebuilt', kind='playlist').one()
assert [i.item_kind for i in collection.items] == ['file', 'url']
assert [i.position for i in collection.items] == [1, 2]
@pytest.mark.asyncio
async def test_backup_preview_and_import_merge(test_session, test_directory, playlists_config):
"""A backup can be previewed and merged: backup-only playlists are added, current ones win."""
file = playlists_config.get_file()
file.parent.mkdir(parents=True, exist_ok=True)
file.write_text(
'version: 3\n'
'playlists:\n'
' - name: Current\n'
' items:\n'
' - url: /u/current\n'
)
backup = playlists_config._get_backup_file('20260101')
backup.parent.mkdir(parents=True, exist_ok=True)
backup.write_text(
'version: 1\n'
'playlists:\n'
' - name: Current\n'
' items:\n'
' - url: /u/old\n'
' - name: Restored\n'
' items:\n'
' - url: /u/restored\n'
' - url: /u/restored2\n'
)
preview = playlists_config.preview_backup_import('20260101', 'merge')
assert preview['add'] == [dict(type='playlist', name='Restored', items=2)]
assert preview['remove'] == []
assert preview['unchanged'] == 1
playlists_config.import_backup('20260101', 'merge')
test_session.expire_all()
by_name = {c.name: c for c in test_session.query(Collection).filter_by(kind='playlist').all()}
assert sorted(by_name) == ['Current', 'Restored']
# Merge keeps the current config's version of 'Current'.
assert [i.url for i in by_name['Current'].items] == ['/u/current']
assert [i.url for i in by_name['Restored'].items] == ['/u/restored', '/u/restored2']
@pytest.mark.asyncio
async def test_backup_preview_and_import_overwrite(test_session, test_directory, playlists_config):
"""Overwrite mode replaces the config (and DB) with the backup's playlists."""
_make_playlist(test_session, name='CurrentOnly')
playlists_config.dump_config()
backup = playlists_config._get_backup_file('20260101')
backup.parent.mkdir(parents=True, exist_ok=True)
backup.write_text(
'version: 1\n'
'playlists:\n'
' - name: FromBackup\n'
' items:\n'
' - url: /u/backup\n'
)
preview = playlists_config.preview_backup_import('20260101', 'overwrite')
assert preview['add'] == [dict(type='playlist', name='FromBackup', items=1)]
assert preview['remove'] == [dict(type='playlist', name='CurrentOnly')]
playlists_config.import_backup('20260101', 'overwrite')
test_session.expire_all()
names = [c.name for c in test_session.query(Collection).filter_by(kind='playlist').all()]
assert names == ['FromBackup']
@pytest.mark.asyncio
async def test_import_still_deletes_playlists_removed_from_config(
test_session, test_directory, playlists_config):
"""A non-empty config remains authoritative: playlists absent from it are deleted."""
_make_playlist(test_session, name='Kept')
_make_playlist(test_session, name='Removed')
file = playlists_config.get_file()
file.parent.mkdir(parents=True, exist_ok=True)
file.write_text(
'version: 1\n'
'playlists:\n'
' - name: Kept\n'
' items:\n'
' - url: /map?lat=1&lon=2&z=3\n'
)
playlists_config.import_config()
test_session.expire_all()
names = [c.name for c in test_session.query(Collection).filter_by(kind='playlist').all()]
assert names == ['Kept']
@pytest.mark.asyncio
async def test_custom_directory_round_trips(test_session, test_directory, playlists_config):
"""A playlist's custom directory survives a dump -> wipe -> import cycle."""
custom = test_directory / 'survival' / 'fire'
collection = _make_playlist(test_session, name='Custom Dir')
collection.directory = custom
test_session.commit()
playlists_config.dump_config()
# The config stores the directory relative to the media directory.
data = playlists_config.read_config_file()
entry = next(p for p in data['playlists'] if p['name'] == 'Custom Dir')
assert entry['directory'] == 'survival/fire'
# Wipe and re-import; the directory is restored.
test_session.query(Collection).filter_by(kind='playlist').delete()
test_session.commit()
playlists_config.import_config()
test_session.expire_all()
collection = test_session.query(Collection).filter_by(name='Custom Dir').one()
assert str(collection.directory) == str(custom)

View file

@ -135,3 +135,9 @@ collection_type_registry.register(
validate_channel_name,
'Channel name must be a non-empty string'
)
collection_type_registry.register(
'playlist',
validate_channel_name, # Permissive: any non-empty name.
'Playlist name must be a non-empty string'
)

View file

@ -456,7 +456,7 @@ def get_media_directory() -> Path:
DB_CONFIG_FILE_NAMES = {'tags.yaml', 'channels.yaml', 'domains.yaml', 'download_manager.yaml', 'inventories.yaml',
'collections.yaml'}
'playlists.yaml'}
class ConfigFile:
@ -809,6 +809,7 @@ class WROLPiConfigValidator:
map_default_location: dict = None
map_destination: str = None
nav_color: str = None
playlists_destination: str = None
require_cookies_unlocked: bool = None
require_media_mounted: bool = None
save_ffprobe_json: bool = None
@ -858,14 +859,14 @@ def get_all_configs() -> Dict[str, ConfigFile]:
if download_cache_config := get_download_cache_config():
all_configs[download_cache_config.file_name] = download_cache_config
from wrolpi.collections.config import collections_config as _collections_config
if _collections_config:
all_configs[_collections_config.file_name] = _collections_config
from modules.map.pins import get_map_pins_config
if map_pins_config := get_map_pins_config():
all_configs[map_pins_config.file_name] = map_pins_config
from wrolpi.collections.config import get_playlists_config
if playlists_config := get_playlists_config():
all_configs[playlists_config.file_name] = playlists_config
return all_configs
@ -938,15 +939,18 @@ async def import_all_db_configs() -> dict[str, bool]:
logger.warning(f'Failed to import inventories config: {e}')
results['inventories'] = False
# Collections (uses tags for tag_name)
# Playlists (kind='playlist' collections; uses tags for tag_name).
# NOTE: there is no generic collections config — channels/domains are owned by their own configs
# (channels.yaml/domains.yaml), and authors/subjects are re-derived on refresh. Playlists are
# user-curated with no other config home, so they own playlists.yaml.
try:
from wrolpi.collections.config import collections_config as _collections_config
_collections_config.import_config()
results['collections'] = True
logger.debug('collections config imported')
from wrolpi.collections.config import playlists_config
playlists_config.import_config()
results['playlists'] = True
logger.debug('playlists config imported')
except Exception as e:
logger.warning(f'Failed to import collections config: {e}')
results['collections'] = False
logger.warning(f'Failed to import playlists config: {e}')
results['playlists'] = False
# Map pins (YAML-only, no DB)
try:
@ -989,6 +993,7 @@ class WROLPiConfig(ConfigFile):
map_default_location=None,
map_destination='map',
nav_color='violet',
playlists_destination='playlists',
require_cookies_unlocked=True,
require_media_mounted=True,
save_ffprobe_json=True,
@ -1033,6 +1038,14 @@ class WROLPiConfig(ConfigFile):
self._config['map_destination'] = self.map_destination or self.default_config['map_destination']
self._config['videos_destination'] = self.videos_destination or self.default_config['videos_destination']
self._config['zims_destination'] = self.zims_destination or self.default_config['zims_destination']
self._config['playlists_destination'] = \
self.playlists_destination or self.default_config['playlists_destination']
# The playlists directory holds WROLPi-managed hardlinks/stubs; never index it.
ignored = list(self._config.get('ignored_directories') or [])
if self._config['playlists_destination'] not in ignored:
ignored.append(self._config['playlists_destination'])
self._config['ignored_directories'] = ignored
self.successful_import = True
except Exception as e:
@ -1214,6 +1227,14 @@ class WROLPiConfig(ConfigFile):
def archive_destination(self, value: str):
self.update({'archive_destination': value})
@property
def playlists_destination(self) -> str:
return self._config['playlists_destination']
@playlists_destination.setter
def playlists_destination(self, value: str):
self.update({'playlists_destination': value})
@property
def map_default_location(self) -> dict:
return self._config.get('map_default_location')

View file

@ -32,7 +32,6 @@ def attach_shared_contexts(app: Sanic):
app.shared_ctx.domains_config = manager.dict()
app.shared_ctx.archive_downloader_config = manager.dict()
app.shared_ctx.download_cache_config = manager.dict()
app.shared_ctx.collections_config = manager.dict()
app.shared_ctx.map_pins_config = manager.dict()
# Shared dicts.
app.shared_ctx.refresh = manager.dict()
@ -105,7 +104,6 @@ def reset_shared_contexts(app: Sanic):
app.shared_ctx.domains_config.clear()
app.shared_ctx.archive_downloader_config.clear()
app.shared_ctx.download_cache_config.clear()
app.shared_ctx.collections_config.clear()
# Shared dicts.
app.shared_ctx.refresh.clear()
app.shared_ctx.uploaded_files.clear()
@ -265,12 +263,6 @@ def initialize_configs_contexts(app: Sanic):
except Exception as e:
logger.error(f'Failed to initialize in-memory download cache config: {e}')
try:
from wrolpi.collections.config import collections_config as _collections_config
_collections_config.initialize(app.shared_ctx.collections_config)
except Exception as e:
logger.error(f'Failed to initialize in-memory collections config: {e}')
try:
from modules.map.pins import MAP_PINS_CONFIG
MAP_PINS_CONFIG.initialize(app.shared_ctx.map_pins_config)

View file

@ -376,16 +376,14 @@ class Download(ModelHelper, Base): # noqa
session.delete(self)
session.commit()
# Save appropriate config based on collection kind
# Save appropriate config based on collection kind. Channels/domains own their configs;
# other kinds have no generic collections config to save.
if collection_kind == 'channel':
from modules.videos.lib import save_channels_config
save_channels_config.activate_switch()
elif collection_kind == 'domain':
from modules.archive.lib import save_domains_config
save_domains_config.activate_switch()
elif collection_kind:
from wrolpi.collections.config import save_collections_config
save_collections_config.activate_switch()
# Save download config again because this download is now removed from the download lists.
save_downloads_config.activate_switch()

View file

@ -1241,6 +1241,10 @@ class FileWorker:
if is_global_refresh:
flags.refresh_complete.set()
flags.global_refresh_active.clear()
# After a DB rebuild, playlists.yaml was imported before any files were indexed, so its
# file/zim items were skipped. They are indexed now; re-import to restore them.
from wrolpi.collections.config import import_playlists_config
import_playlists_config.activate_switch()
# Mark job as complete if tracking
self._complete_job(task.job_id)

View file

@ -176,6 +176,7 @@ def get_settings(_: Request):
'map_destination': wrolpi_config.map_destination,
'nav_color': wrolpi_config.nav_color,
'media_directory': str(get_media_directory()), # Convert to string to avoid conversion to relative.
'playlists_destination': wrolpi_config.playlists_destination,
'tags_directory': wrolpi_config.tags_directory,
'throttle_on_startup': wrolpi_config.throttle_on_startup,
'version': __version__,
@ -240,6 +241,23 @@ async def update_settings(_: Request, body: schema.SettingsRequest):
elif not body.zims_destination:
new_config['zims_destination'] = wrolpi_config.default_config['zims_destination']
if body.playlists_destination:
# The playlists sync deletes files in its directory, so this destination gets stricter
# validation: no traversal, and no overlap with the other content destinations. Compare
# against the destinations in this same request when provided.
from types import SimpleNamespace
from wrolpi.collections.sync import validate_playlists_destination
effective = SimpleNamespace(
videos_destination=body.videos_destination or wrolpi_config.videos_destination,
archive_destination=body.archive_destination or wrolpi_config.archive_destination,
map_destination=body.map_destination or wrolpi_config.map_destination,
zims_destination=body.zims_destination or wrolpi_config.zims_destination,
)
if error := validate_playlists_destination(body.playlists_destination, effective):
raise InvalidConfig(error)
else:
new_config['playlists_destination'] = wrolpi_config.default_config['playlists_destination']
# Validate timezone if provided; empty string clears it.
if body.timezone is not None:
if body.timezone == '':

View file

@ -40,6 +40,7 @@ class SettingsResponse:
wrol_mode: bool
archive_destination: str = 'archive/%(domain_tag)s/%(domain)s'
map_destination: str = 'map'
playlists_destination: str = 'playlists'
videos_destination: str = 'videos/%(channel_tag)s/%(channel_name)s'
zims_destination: str = 'zims'
@ -78,6 +79,7 @@ class SettingsRequest:
map_destination: Optional[str] = None
nav_color: Optional[str] = None
media_directory: Optional[str] = None
playlists_destination: Optional[str] = None
tags_directory: Optional[bool] = None
throttle_on: Optional[bool] = None
throttle_on_startup: Optional[bool] = None

View file

@ -252,6 +252,28 @@ class Tag(ModelHelper, Base):
save_tags_config.activate_switch()
sync_tags_directory.activate_switch()
@staticmethod
def get_or_create_tag(session: Session, name: str = None, id_: int = None) -> 'Tag':
"""Gets or creates Tag object.
@warning: does not commit"""
if name and id_:
raise RuntimeError('Tag name and id cannot be used together')
if tag := session.query(Tag).filter(Tag.id == id_).one_or_none():
return tag
if tag := session.query(Tag).filter(Tag.name == name).one_or_none():
return tag
if not name:
raise InvalidTag('Tag name cannot be empty')
tag = Tag(name=name)
session.add(tag)
# Flush so the new Tag has an id (callers use `.id` immediately).
tag.flush(session)
return tag
@event.listens_for(Tag, 'after_insert')
@event.listens_for(Tag, 'after_update')

View file

@ -798,6 +798,37 @@ async def test_settings_special_directories(async_client, test_wrolpi_config):
assert get_wrolpi_config().archive_destination == 'archive/%(domain_tag)s/%(domain)s'
@pytest.mark.asyncio
async def test_settings_playlists_destination_validation(async_client, test_wrolpi_config):
"""The playlists destination is strictly validated: the sync deletes files there, so traversal
and overlap with other content destinations are rejected."""
# A valid change works.
data = {'playlists_destination': 'my_playlists'}
request, response = await async_client.patch('/api/settings', content=json.dumps(data))
assert response.status_code == HTTPStatus.NO_CONTENT
assert get_wrolpi_config().playlists_destination == 'my_playlists'
for invalid in (
'/absolute/not/allowed',
'../outside-media',
'a/../../b',
'videos', # equals/overlaps videos_destination's base
'videos/playlists', # inside videos_destination
'zims',
'tags',
):
data = {'playlists_destination': invalid}
request, response = await async_client.patch('/api/settings', content=json.dumps(data))
assert response.status_code == HTTPStatus.BAD_REQUEST, \
f'{invalid!r} should have been rejected'
# Empty restores the default.
data = {'playlists_destination': ''}
request, response = await async_client.patch('/api/settings', content=json.dumps(data))
assert response.status_code == HTTPStatus.NO_CONTENT
assert get_wrolpi_config().playlists_destination == 'playlists'
@pytest.mark.asyncio
async def test_settings_save_ffprobe_json(async_client, test_wrolpi_config):
"""Maintainer can toggle save_ffprobe_json setting."""