Arbitrary videos can be downloaded. (#63)

* Squashed commit of the following:

commit c527475fcb4d4906457be5c10f0e9c6dee7fedc1
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:16:36 2021 -0700

    Only saving a favorite video to the config if it has a source_id.

commit ae8b712ebeaef467b8d5a5b87ca0f4f276373a0d
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:14:26 2021 -0700

    Ading description and changing Save button to Download.

commit 03a72be6dfdfa383915950a80e3f1ee9febf6705
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:09:44 2021 -0700

    Removing uploader for now.

commit 5d0231284bf84e459687f6ab903b06bf4e3762bd
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:07:17 2021 -0700

    Removing unique data from example info.

commit 9cc721a32d4715d9d6433e51db3b35e039c78d68
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:02:33 2021 -0700

    Updating downloader tests.

commit 8e193b03349ae4613f105418cd3e8176c7101304
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:53:31 2021 -0700

    Toasting error when WROL Mode is enabled.  Disabling new downloads during WROL Mode.

commit 48c4311c31db8f2d2c8dd0a65f979a5f0a49a911
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:45:06 2021 -0700

    Raising error if download exits without a 0.  Clearing kill event when starting a new download.

commit 54ba3a890d59facc837410d34bf7afb40db3b685
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:16:46 2021 -0700

    Downloads can be restarted and killed.

commit bd60751262247259c3e9ceb5116e017d6314c219
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Dec 13 16:28:18 2021 -0700

    Downloads can be started again.  Downloads are sorted on Downloads page.

commit ea7f39231b92982ae0b263b76c5bd1902cc8169d
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 21:24:16 2021 -0700

    Building Downloads tab.

commit d3321872ac3743ff6df781b07ef0fb537f1d4c98
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 20:41:46 2021 -0700

    Not scheduling channels withtout a URL/frequency.

commit 84f268a0d4f58c85f86b6377900492e0766f6619
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 19:56:08 2021 -0700

    Channel downloads will be spread out when each channel is created.

commit 26fa6beea38ef3f8d6b7bf218f18222a51a87a60
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 13:08:11 2021 -0700

    Recurring downloads are renewed on startup.

commit fc2ab850533f1c3ad073f4692c7dea2be43e85d6
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 17:03:57 2021 -0700

    Fetching download again after do_download.

commit fdadad327fc09c17a49eb63b11e632fb72cc3195
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 16:20:51 2021 -0700

    Fixing error with test.  Mock was not being reset.

commit 070dee238e53377acd22097c63cf3794e68ed423
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 15:36:39 2021 -0700

    Fixed bug in date handling (big surprise).

commit e53a8992268d68620c679455adaec3a590f20602
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 16:48:36 2021 -0700

    Using chunks for regular channel refresh.

commit 3bd60c58d1bef0226325f321978c0434b36b3be5
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 15:22:02 2021 -0700

    Reformatting.

commit 4fdc325f25a32c311ca6543fe321624b2bd564b2
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 15:13:34 2021 -0700

    Creating function that refreshes NO CHANNEL videos.

commit d67feafb6c2994f5a4a1295920a70db369cc6816
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 14:35:54 2021 -0700

    Removing unused functions.

commit 6b3918e4e78fbac64920f85c4de0df873ec366a1
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 14:31:48 2021 -0700

    Removing reporter and events.  This will be replaced in the future with some sort of SSE.

commit 54fa1b0f5e93ca25c80b780dcdc3876b700273b1
Author: Roland <roland@learningselfreliance.com>
Date:   Sun Nov 28 17:55:43 2021 -0700

    Better error handling for archive button.

commit c31d124e511b27f626ed2df71486d47302b396f3
Author: Roland <roland@learningselfreliance.com>
Date:   Sun Nov 28 17:29:43 2021 -0700

    Displaying View Count in a human readable format.

commit 0ee2bd65588fd93b475cad6144e76f1b45191b39
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Nov 27 09:29:41 2021 -0700

    Fixing tests.

commit e64aff32bab95c311ea3747e1a21b58b99750d5e
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Nov 27 08:59:51 2021 -0700

    Handling when a Video has no Channel in UI.

commit b9549ecd5aac03c002adca7b7667f0a7189b662a
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Nov 26 08:35:36 2021 -0700

    Downloaders can raise UnrecoverableDownloadError to signal a downlaod should not be retried.

commit da88798e2584d3e84900a70e34166c8c208c64fb
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 20:40:29 2021 -0700

    Downloaders can refuse a download, which will then be deleted.

commit b865d856121db5ec8a3660c5d3c13b9846fde688
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 19:35:43 2021 -0700

    Removing relative path handling.

commit 6d7757870b02ba030d78b3ef7e43f28b906eea61
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 19:28:27 2021 -0700

    Using __json__ for Video and Channel models.

commit 649e4d6a61608272759d1e7ee02b490bb5c20b3e
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 18:23:38 2021 -0700

    Fixing video links in UI.

commit 9d081af12742070b8519bd752812130cb5dbe393
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:42:38 2021 -0700

    Fixing docs.

commit 1202cbe0f9ae744d7801b53a5071cded8ea39924
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:40:36 2021 -0700

    Fixing wrolpi tests.

commit 3f6af8cc1123f2f63a3fc0f21ec4d994c642cf13
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:13:35 2021 -0700

    Fixing tests to use MediaPath.

commit faec04f2a5e4c267a502c03884940a9c5bcfe387
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Nov 19 10:21:20 2021 -0700

    Testing when a video has no channel.

commit cbf22aad7a0b72e61685e0236e843554b57c4af1
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Nov 18 16:24:03 2021 -0700

    Using MediaPath for Video and Channel models.  Testing VideoDownloader.do_download.

commit 7916ed61e43439e6cd60f23ce3fdf6c1a75bf02d
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Nov 18 12:43:44 2021 -0700

    Channels/Videos downloaders can get gotten.

commit 808ef796cec01db6e7009065f9e26d25d9500ce2
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 21:45:19 2021 -0700

    Upgrading to Sanic 21.9.1.

commit ea32813be28e78393ac456ee782a7245c098188f
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 20:29:01 2021 -0700

    Creating Downloader and DownloaderManager classes which handle downloading URLs.

commit e859a77a77277ddf8b6e221ec9c9383c8ab91d66
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 12:41:39 2021 -0700

    Creating UI segement which allows the download/upload of media.

* Displaying all recurring downloads.

* Not downloading channels with empty url.

* Showing positive row for recurring downloads.

* Removing default last_successful_download.

* Only displaying Last Successful Download when there is one.

* Refreshing videos before finding missing videos.

* Using short video path name for log.

* Removing download history feature in favor of new Downloads page.

* Only showing info date if it exists.

* Making get_or_create_channel more reliable.

* Allowing null for channel download frequency.

* Logging when searching for channel.

* Storing channel's source_id in config.

* Importing using link instead of name.

* Getting channel id when importing channels.

* Removing redunant parentheses from repr.

* Preserving database channel values if there are not values in the config.

* Only highlighting pending downloads.

* Deleting old downloads periodically.

* Sorting recurring by next_download/frequency.

* Creating fixtures for channels, media directory and db session.

* Removing unused function.  Wrapping config for tests.

* Deleting channel download when deleting a channel.

* Testing that a channel's download is deleted when the channel is deleted.

* Updating youtube-dl

* Switching from youtube-dl in favor of yt-dlp.

* Sorting once-downlaods by last_successful_download then id.

* Adding special handling to video_url_resolver for full urls.

* Consolidating downloader methods and functions.  Reversing once-download sort order so the most recently downloaded are at the top.

* Getting session if one is not provided.

* Adding test for create_downloads.

* Not passing format option when getting filename.

* Linting

Signed-off-by: Roland <roland@learningselfreliance.com>
This commit is contained in:
lrnselfreliance 2021-12-20 16:08:13 -07:00 • committed by Roland
parent 2fff678aee
commit 46796f783a
No known key found for this signature in database
GPG key ID: 9D6DBD3ACF11AB93
61 changed files with 3347 additions and 1525 deletions

View file

@ -0,0 +1,51 @@
"""Downloader
Revision ID: 184329ed5137
Revises: 1870bec7c81e
Create Date: 2021-11-17 19:13:58.305140
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '184329ed5137'
down_revision = '1870bec7c81e'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
CREATE TABLE download (
id SERIAL PRIMARY KEY,
url TEXT NOT NULL,
status TEXT DEFAULT 'new',
next_download TIMESTAMPTZ,
last_successful_download TIMESTAMPTZ,
frequency INT,
attempts INT DEFAULT 0
)
''')
session.execute('ALTER TABLE video ADD COLUMN url TEXT')
session.execute('ALTER TABLE channel ADD COLUMN source_id TEXT')
session.execute('CREATE INDEX channel_source_id ON channel (source_id)')
if not DOCKERIZED:
session.execute('ALTER TABLE public.download OWNER TO wrolpi')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('DROP TABLE IF EXISTS download')
session.execute('DROP INDEX IF EXISTS channel_source_id')
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS source_id')
session.execute('ALTER TABLE video DROP COLUMN IF EXISTS url')

View file

@ -9,6 +9,7 @@
"query-string": "^6.13.0",
"react": "^16.13.1",
"react-dom": "^16.13.1",
"react-dropzone": "^11.4.2",
"react-router-dom": "^5.2.0",
"react-scripts": "^3.4.1",
"react-semantic-toasts": "^0.6.5",

View file

@ -4,21 +4,23 @@ import {NavBar} from "./components/Nav";
import {Route, Switch} from "react-router-dom";
import {FavoriteVideosPreview, VideosRoute, VideoWrapper, ViewedVideosPreview} from "./components/Videos";
import Admin from "./components/Admin";
import {Container, Header, Segment} from "semantic-ui-react";
import {Container, Divider, Header, Segment} from "semantic-ui-react";
import 'semantic-ui-offline/semantic.min.css';
import {SemanticToastContainer} from 'react-semantic-toasts';
import 'react-semantic-toasts/styles/react-semantic-alert.css';
import {AppsRoute} from "./components/Apps";
import {InventoryRoute} from "./components/Inventory";
import {ArchiveRoute} from "./components/Archive";
import {Saver} from "./components/Upload";
function Welcome() {
return (
<Container style={{marginTop: '2em'}}>
<Header as="h1">Welcome to WROLPi!</Header>
<h3>
Take your internet, off-grid.
</h3>
<Header as='h2'>Save your media</Header>
<Saver/>
<Divider/>
<Segment>
<FavoriteVideosPreview/>
</Segment>
@ -45,6 +47,7 @@ export default function App() {
<NavBar/>
</header>
<Switch>
<Route path='/videos/video/:video_id' exact component={VideoWrapper}/>
<Route path='/videos/channel/:channel_link/video/:video_id' exact component={VideoWrapper}/>
<Route path="/" exact={true} component={Welcome}/>
<Route path="/videos" component={VideosRoute}/>

View file

@ -32,7 +32,16 @@ async function apiCall(url, method, body, ms = 10000) {
try {
// await the response or error.
return await timeoutPromise(ms, promise);
let response = await timeoutPromise(ms, promise);
if (response.status === 403 && (await response.json())['code'] === 17) {
toast({
type: 'error',
title: 'WROL Mode is enabled!',
description: 'This functionality is not enabled while WROL Mode is enabled!',
time: 5000,
});
}
return response;
} catch (e) {
// Timeout, or some other exception.
if (e.message === 'promise timeout') {
@ -165,6 +174,13 @@ export async function saveConfig(config) {
return await apiPatch(`${API_URI}/settings`, config);
}
export async function getDownloads() {
let response = await apiGet(`${API_URI}/download`);
let data = await response.json();
return data;
}
export async function validateRegex(regex) {
let response = await apiPost(`${API_URI}/valid_regex`, {regex});
return (await response.json())['valid'];
@ -172,7 +188,7 @@ export async function validateRegex(regex) {
export async function favoriteVideo(video_id, favorite) {
let body = {favorite: favorite, video_id};
let response = await apiPost(`${VIDEOS_API}:favorite`, body);
let response = await apiPost(`${VIDEOS_API}/favorite`, body);
return (await response.json())['favorite'];
}
@ -182,28 +198,22 @@ export async function getStatistics() {
}
export async function refresh() {
let response = await apiPost(`${VIDEOS_API}:refresh`);
let response = await apiPost(`${VIDEOS_API}/refresh`);
return await response.json();
}
export async function download() {
let response = await apiPost(`${VIDEOS_API}:download`);
let response = await apiPost(`${VIDEOS_API}/download`);
return await response.json();
}
export async function downloadChannel(link) {
let url = `${VIDEOS_API}:download/${link}`;
let url = `${VIDEOS_API}/download/${link}`;
await fetch(url, {method: 'POST'});
}
export async function downloadHistory() {
let url = `${VIDEOS_API}/download_history`;
let resp = await apiGet(url);
return await resp.json();
}
export async function refreshChannel(link) {
let url = `${VIDEOS_API}:refresh/${link}`;
let url = `${VIDEOS_API}/refresh/${link}`;
await fetch(url, {method: 'POST'});
}
@ -316,7 +326,7 @@ export async function searchURLs(offset, limit, domain = null) {
}
export async function refreshArchives() {
return await apiPost(`${ARCHIVES_API}:refresh`);
return await apiPost(`${ARCHIVES_API}/refresh`);
}
export async function fetchDomains() {
@ -328,3 +338,15 @@ export async function fetchDomains() {
throw Error(`Unable to fetch domains`);
}
}
export async function postDownload(urls) {
let body = {'urls': urls};
let response = await apiPost(`${API_URI}/download`, body);
return response;
}
export async function killDownload(download_id) {
let response = await apiPost(`${API_URI}/download/${download_id}/kill`);
return response;
}

View file

@ -1,7 +1,20 @@
import React from 'react';
import {Button, Checkbox, Container, Divider, Form, Header, Loader, Tab} from "semantic-ui-react";
import {getConfig, saveConfig} from "../api";
import {
Button,
Checkbox,
Confirm,
Container,
Divider,
Form,
Header,
Loader,
Placeholder,
Tab,
Table
} from "semantic-ui-react";
import {getConfig, getDownloads, killDownload, postDownload, saveConfig} from "../api";
import TimezoneSelect from 'react-timezone-select';
import {secondsToDate, secondsToFrequency} from "./Common";
class Settings extends React.Component {
@ -152,11 +165,217 @@ class WROLMode extends React.Component {
}
}
class DownloadRow extends React.Component {
render() {
let {url, frequency, last_successful_download, status, next_download} = this.props;
let positive = false;
if (status === 'pending') {
positive = true;
}
return (
<Table.Row positive={positive}>
<Table.Cell>{url}</Table.Cell>
<Table.Cell>{secondsToFrequency(frequency)}</Table.Cell>
<Table.Cell>{last_successful_download ? secondsToDate(last_successful_download) : null}</Table.Cell>
<Table.Cell>{secondsToDate(next_download)}</Table.Cell>
<Table.Cell>{status}</Table.Cell>
</Table.Row>
);
}
}
class StoppableRow extends React.Component {
constructor(props) {
super(props);
this.state = {
stopOpen: false,
startOpen: false,
};
}
openStop = () => {
this.setState({stopOpen: true});
}
closeStop = () => {
this.setState({stopOpen: false});
}
openStart = () => {
this.setState({startOpen: true});
}
closeStart = () => {
this.setState({startOpen: false});
}
handleStop = async (e) => {
e.preventDefault();
await killDownload(this.props.id);
this.closeStop();
await this.props.fetchDownloads();
};
handleStart = async (e) => {
e.preventDefault();
await postDownload(`${this.props.url}`);
this.closeStart();
await this.props.fetchDownloads();
};
render() {
let {url, last_successful_download, status} = this.props;
let {stopOpen, startOpen} = this.state;
let buttonCell = <Table.Cell/>;
let positive = false;
let negative = false;
let warning = false;
if (status === 'pending' || status === 'new') {
positive = status === 'pending';
buttonCell = (
<Table.Cell>
<Button
onClick={this.openStop}
color='red'
>Stop</Button>
<Confirm
open={stopOpen}
content='Are you sure you want to stop this download? It will not be retried.'
confirmButton='Stop'
onCancel={this.closeStop}
onConfirm={this.handleStop}
/>
</Table.Cell>
);
} else if (status === 'failed' || status === 'deferred') {
negative = status === 'failed';
warning = status === 'deferred';
buttonCell = (
<Table.Cell>
<Button
onClick={this.openStart}
color='green'
>Start</Button>
<Confirm
open={startOpen}
content='Are you sure you want to restart this download?'
confirmButton='Start'
onCancel={this.closeStart}
onConfirm={this.handleStart}
/>
</Table.Cell>
);
}
return (
<Table.Row positive={positive} negative={negative} warning={warning}>
<Table.Cell>{url}</Table.Cell>
<Table.Cell>{last_successful_download ? secondsToDate(last_successful_download) : null}</Table.Cell>
<Table.Cell>{status}</Table.Cell>
{buttonCell}
</Table.Row>
);
}
}
class Downloads extends React.Component {
constructor(props) {
super(props);
this.state = {
once_downloads: null,
recurring_downloads: null,
pending_downloads: null,
};
}
async componentDidMount() {
await this.fetchDownloads();
}
fetchDownloads = async () => {
let data = await getDownloads();
this.setState({
once_downloads: data.once_downloads,
recurring_downloads: data.recurring_downloads,
pending_downloads: data.pending_downloads,
});
}
render() {
let tablePlaceholder = (
<Placeholder>
<Placeholder.Line/>
<Placeholder.Line/>
</Placeholder>
);
let stoppableHeader = (
<Table.Header>
<Table.Row>
<Table.HeaderCell>URL</Table.HeaderCell>
<Table.HeaderCell>Completed At</Table.HeaderCell>
<Table.HeaderCell>Status</Table.HeaderCell>
<Table.HeaderCell>Control</Table.HeaderCell>
</Table.Row>
</Table.Header>
);
let nonStoppableHeader = (
<Table.Header>
<Table.Row>
<Table.HeaderCell>URL</Table.HeaderCell>
<Table.HeaderCell>Download Frequency</Table.HeaderCell>
<Table.HeaderCell>Last Successful Download</Table.HeaderCell>
<Table.HeaderCell>Next Download</Table.HeaderCell>
<Table.HeaderCell>Status</Table.HeaderCell>
</Table.Row>
</Table.Header>
);
let onceTable = tablePlaceholder;
if (this.state.once_downloads !== null && this.state.once_downloads.length === 0) {
onceTable = <p>No downloads are scheduled to be downloaded.</p>
} else if (this.state.once_downloads !== null) {
onceTable = (<Table>
{stoppableHeader}
<Table.Body>
{this.state.once_downloads.map((i) => <StoppableRow {...i} fetchDownloads={this.fetchDownloads}/>)}
</Table.Body>
</Table>);
}
let recurringTable = tablePlaceholder;
if (this.state.recurring_downloads !== null && this.state.recurring_downloads.length === 0) {
recurringTable = <p>No downloads are scheduled to be downloaded.</p>
} else if (this.state.recurring_downloads !== null) {
recurringTable = (<Table>
{nonStoppableHeader}
<Table.Body>
{this.state.recurring_downloads.map((i) => <DownloadRow {...i}/>)}
</Table.Body>
</Table>);
}
return (
<div>
<Header as='h1'>Downloads</Header>
{onceTable}
<Header as='h1'>Recurring Downloads</Header>
{recurringTable}
</div>
);
}
}
class Admin extends React.Component {
render() {
const panes = [
{menuItem: 'Downloads', render: () => <Tab.Pane><Downloads/></Tab.Pane>},
{menuItem: 'Settings', render: () => <Tab.Pane><Settings/></Tab.Pane>},
{menuItem: 'WROL Mode', render: () => <Tab.Pane><WROLMode/></Tab.Pane>},
];

View file

@ -348,8 +348,17 @@ class ArchiveAddForm extends APIForm {
handleSubmit = async (e) => {
e.preventDefault();
await postArchive(this.state.inputs.url);
this.setState({inputs: {url: ''}}, this.fetchURLs);
this.setLoading();
try {
let response = await postArchive(this.state.inputs.url);
if (response.status === 204) {
this.setState({inputs: {url: ''}, success: true}, this.fetchURLs);
} else {
this.setState({loading: false, success: undefined, error: true});
}
} finally {
this.clearLoading();
}
}
fetchURLs = async (e) => {
@ -365,10 +374,13 @@ class ArchiveAddForm extends APIForm {
<Form.Input
name='url'
placeholder='https://wrolpi.org'
onChange={this.handleInputChange}
value={this.state.inputs.url}
disabled={this.state.disabled}
error={this.state.error}
onChange={this.handleInputChange}
success={this.state.success}
/>
<Form.Button primary>Archive</Form.Button>
<Form.Button primary disabled={this.state.disabled}>Archive</Form.Button>
</Form.Group>
</Form>
</>

View file

@ -101,13 +101,11 @@ class ChannelPage extends APIForm {
match_regex: '',
generate_posters: true,
calculate_duration: true,
download_frequency: 604800,
download_frequency: null,
},
errors: {},
};
this.calculateDuration = React.createRef();
this.generatePosters = React.createRef();
this.mkdir = React.createRef();
}
@ -304,7 +302,7 @@ class ChannelPage extends APIForm {
<Form.Group>
<Form.Field>
<label>Download Frequency</label>
<Dropdown selection
<Dropdown selection clearable
name='download_frequency'
disabled={this.state.disabled}
placeholder='Frequency'
@ -344,29 +342,6 @@ class ChannelPage extends APIForm {
onChange={this.checkRegex}
/>
</Form.Field>
<Form.Field>
<Checkbox
toggle
label="Generate posters, if not found"
name="generate_posters"
disabled={this.state.disabled}
checked={this.state.inputs.generate_posters}
ref={this.generatePosters}
onClick={() => this.handleCheckbox(this.generatePosters)}
/>
</Form.Field>
<Form.Field>
<Checkbox
toggle
label="Calculate video duration"
name="calculate_duration"
disabled={this.state.disabled}
checked={this.state.inputs.calculate_duration}
ref={this.calculateDuration}
onClick={() => this.handleCheckbox(this.calculateDuration)}
/>
</Form.Field>
</Segment>
</Accordion.Content>
</Accordion>
@ -399,24 +374,24 @@ class ChannelPage extends APIForm {
</Button>
{!this.props.create &&
<>
<Button color='red' onClick={this.handleDeleteButton}>
Delete
</Button>
<Confirm
open={this.state.deleteOpen}
content='Are you sure you want to delete this channel? No video files will be deleted.'
confirmButton='Delete'
onCancel={() => this.setState({deleteOpen: false})}
onConfirm={this.handleConfirm}
/>
<Button color='blue' onClick={this.downloadChannel}>
Download
</Button>
<Button color='inverted blue' onClick={this.refreshChannel}>
Refresh
</Button>
</>
<>
<Button color='red' onClick={this.handleDeleteButton}>
Delete
</Button>
<Confirm
open={this.state.deleteOpen}
content='Are you sure you want to delete this channel? No video files will be deleted.'
confirmButton='Delete'
onCancel={() => this.setState({deleteOpen: false})}
onConfirm={this.handleConfirm}
/>
<Button color='blue' onClick={this.downloadChannel}>
Download
</Button>
<Button color='inverted blue' onClick={this.refreshChannel}>
Refresh
</Button>
</>
}
</Container>
</Form>
@ -455,13 +430,13 @@ class ChannelRow extends React.Component {
{channel.video_count}
</Table.Cell>
<Table.Cell>
{channel.url ? secondsToDate(channel.info_date) : null}
{channel.url && channel.info_date ? secondsToDate(channel.info_date) : null}
</Table.Cell>
<Table.Cell>
{channel.url ? secondsToDate(channel.next_download) : null}
{channel.url && channel.next_download ? secondsToDate(channel.next_download) : null}
</Table.Cell>
<Table.Cell>
{channel.url ? secondsToFrequency(channel.download_frequency) : null}
{channel.url && channel.download_frequency ? secondsToFrequency(channel.download_frequency) : null}
</Table.Cell>
<Table.Cell textAlign='right'>
<Link className="ui button secondary" to={this.editTo}>Edit</Link>
@ -486,16 +461,17 @@ class MobileChannelRow extends ChannelRow {
</Link>
</p>
<p>
Last Update: {channel.url ? secondsToDate(channel.info_date) : null}
Last Update: {channel.url && channel.info_date ? secondsToDate(channel.info_date) : null}
</p>
<p>
Next Update: {channel.url ? secondsToDate(channel.next_download) : null}
Next Update: {channel.url && channel.next_download ? secondsToDate(channel.next_download) : null}
</p>
<p>
Videos: {channel.video_count}
</p>
<p>
Frequency: {channel.url ? secondsToFrequency(channel.download_frequency) : null}
Frequency: {channel.url && channel.download_frequency ?
secondsToFrequency(channel.download_frequency) : null}
</p>
</Table.Cell>
<Table.Cell width={6} colSpan={2} textAlign='right'>
@ -621,7 +597,7 @@ export class Channels extends React.Component {
</Table>
</Responsive>
<Responsive maxWidth={769}>
<Table striped basic unstackable size='medium'>
<Table striped basic unstackable size='small'>
<Table.Body>
{this.state.results.map((channel) =>
<MobileChannelRow key={channel.link} channel={channel}/>)}

View file

@ -87,12 +87,16 @@ export function uploadDate(d) {
}
export function VideoCard({video}) {
let channel = video.channel;
let channel_url = `/videos/channel/${channel.link}/video`;
let video_url = `/videos/video/${video.id}`;
let upload_date = uploadDate(video.upload_date);
let video_url = `/videos/channel/${channel.link}/video/${video.id}`;
let poster_url = video.poster_path ? `/media/${channel.directory}/${encodeURIComponent(video.poster_path)}` : null;
// A video may not have a channel.
let channel = video.channel ? video.channel : null;
let channel_url = null;
if (channel) {
channel_url = `/videos/channel/${channel.link}/video`;
video_url = `/videos/channel/${channel.link}/video/${video.id}`;
}
let poster_url = video.poster_path ? `/media/${encodeURIComponent(video.poster_path)}` : null;
return (
<Card style={{'width': '18em', 'margin': '1em'}}>
@ -110,9 +114,10 @@ export function VideoCard({video}) {
</Card.Header>
<Card.Description>
<Container textAlign='left'>
<Link to={channel_url} className="no-link-underscore video-card-link">
<b>{channel.name}</b>
</Link>
{channel &&
<Link to={channel_url} className="no-link-underscore video-card-link">
<b>{channel.name}</b>
</Link>}
<p>{upload_date}</p>
</Container>
</Card.Description>
@ -229,6 +234,27 @@ export function humanFileSize(bytes, si = false, dp = 1) {
return bytes.toFixed(dp) + ' ' + units[u];
}
export function humanNumber(num, dp = 1) {
// Convert large numbers to a more human readable format.
// >> humanNumber(1000)
// 1.0k
// >> humanNumber(1500000)
// 1.5m
const divisor = 1000;
if (Math.abs(num) < divisor) {
return num;
}
const units = ['k', 'm', 'b'];
let i = -1;
const r = 10 ** dp;
do {
num /= divisor;
++i;
} while (Math.round(Math.abs(num) * r) / r >= divisor && i < units.length - 1);
return num.toFixed(dp) + units[i];
}
export class APIForm extends React.Component {
constructor(props) {
@ -317,7 +343,7 @@ export class APIForm extends React.Component {
this.setState({
disabled: true,
error: false,
errors: this.getEmptyErrors,
errors: this.getEmptyErrors(),
loading: true,
success: false,
})

View file

@ -0,0 +1,74 @@
import React from "react";
import {Button, Form, Header, Segment, TextArea} from "semantic-ui-react";
import {postDownload} from "../api";
const validUrl = /^(http|https):\/\/[^ "]+$/;
class Downloader extends React.Component {
constructor(props) {
super(props);
this.state = {
urls: '',
valid: true,
};
}
submitDownload = async () => {
if (this.state.urls) {
let response = await postDownload(this.state.urls);
if (response.status === 204) {
this.setState({urls: ''});
}
}
}
handleInputChange = async (event, {name, value}) => {
this.setState({[name]: value}, this.validateUrls);
}
validateUrls = async () => {
// Validate that all URLs in state are valid.
if (this.state.urls) {
let urls = this.state.urls.split(/\r?\n/);
let valid = true;
urls.forEach((url) => {
if (valid && url && !validUrl.test(url)) {
valid = false;
}
})
this.setState({valid});
}
}
render() {
let textareaPlaceholder = 'Enter one URL per line';
let disabled = !this.state.urls || !this.state.valid;
return (
<Form onSubmit={this.submitDownload}>
<Header as='h4'>Enter the URLs you wish to bring offline.</Header>
<TextArea
placeholder={textareaPlaceholder}
name='urls'
onChange={this.handleInputChange}
value={this.state.urls}
error={!this.state.valid}
/>
<Button primary style={{marginTop: '1em'}} disabled={disabled}>Download</Button>
</Form>
)
}
}
export class Saver extends React.Component {
render() {
return (
<>
<Segment>
<Downloader/>
</Segment>
</>
)
}
}

View file

@ -3,7 +3,7 @@ import Icon from "semantic-ui-react/dist/commonjs/elements/Icon";
import {deleteVideo, favoriteVideo} from "../api";
import Button from "semantic-ui-react/dist/commonjs/elements/Button";
import {Link} from "react-router-dom";
import {humanFileSize, uploadDate, VideoCard} from "./Common";
import {humanFileSize, humanNumber, uploadDate, VideoCard} from "./Common";
import {Confirm, Container, Segment, Tab} from "semantic-ui-react";
import Grid from "semantic-ui-react/dist/commonjs/collections/Grid";
@ -24,15 +24,15 @@ function VideoPage(props) {
props.history.goBack();
}
let videoUrl = `${MEDIA_PATH}/${channel.directory}/${encodeURIComponent(video.video_path)}`;
let videoUrl = `${MEDIA_PATH}/${encodeURIComponent(video.video_path)}`;
let posterUrl = null;
if (video.poster_path) {
posterUrl = `${MEDIA_PATH}/${channel.directory}/${encodeURIComponent(video.poster_path)}`;
posterUrl = `${MEDIA_PATH}/${encodeURIComponent(video.poster_path)}`;
}
let captionsUrl = null;
if (video.caption_path) {
captionsUrl = `${MEDIA_PATH}/${channel.directory}/${encodeURIComponent(video.caption_path)}`;
captionsUrl = `${MEDIA_PATH}/${encodeURIComponent(video.caption_path)}`;
}
let description = 'No description available.';
@ -78,16 +78,19 @@ function VideoPage(props) {
let statisticsPane = {
menuItem: 'Statistics', render: () => <Tab.Pane>
<h3>Size</h3>
<p>{humanFileSize(video.size)}</p>
<p>{video.size ? humanFileSize(video.size) : '???'}</p>
<h3>Source URL</h3>
<p>{video.url || 'N/A'}</p>
<h3>Source ID</h3>
<p>{video.source_id}</p>
<p>{video.source_id || 'N/A'}</p>
<h3>View Count</h3>
<p>{viewCount}</p>
<p>{viewCount ? humanNumber(viewCount) : 'N/A'}</p>
<h3>File Name</h3>
<pre style={{backgroundColor: '#ccc', padding: '0.4em'}}>{video.channel.directory}/{video.video_path}</pre>
<pre style={{backgroundColor: '#ccc', padding: '0.4em'}}>{video.video_path}</pre>
</Tab.Pane>
}
@ -117,9 +120,9 @@ function VideoPage(props) {
<h2>{video.title}</h2>
{video.upload_date && <h3>{uploadDate(video.upload_date)}</h3>}
<h3>
<Link to={`/videos/channel/${channel.link}/video`}>
{channel && <Link to={`/videos/channel/${channel.link}/video`}>
{channel.name}
</Link>
</Link>}
</h3>
<p>

View file

@ -10,12 +10,11 @@ import Paginator, {
scrollToTop,
searchOrders,
secondsToString,
uploadDate,
VideoCards,
videoOrders
} from "./Common"
import VideoPage from "./VideoPlayer";
import {download, downloadHistory, getChannel, getStatistics, getVideo, refresh, searchVideos} from "../api";
import {download, getChannel, getStatistics, getVideo, refresh, searchVideos} from "../api";
import {Button, Dropdown, Form, Grid, Header, Icon, Input, Loader, Segment, Statistic, Tab} from "semantic-ui-react";
import * as QueryString from 'query-string';
import Container from "semantic-ui-react/dist/commonjs/elements/Container";
@ -28,15 +27,10 @@ class ManageVideos extends React.Component {
super(props);
this.state = {
streamUrl: null,
downloadHistory: null,
mostRecentDownload: '',
}
}
async componentDidMount() {
await this.downloadHistory();
}
download = async (e) => {
e.preventDefault();
let response = await download();
@ -53,15 +47,6 @@ class ManageVideos extends React.Component {
}
}
async downloadHistory() {
let response = await downloadHistory();
if (response && response.history.length > 0) {
let history = response.history;
let mostRecentDownload = uploadDate(history[0]['info_date']);
this.setState({downloadHistory: history, mostRecentDownload: mostRecentDownload});
}
}
render() {
return (
<Container>
@ -75,7 +60,6 @@ class ManageVideos extends React.Component {
</Button>
<label>Download any missing videos</label>
<br/>
Most recent download: {this.state.mostRecentDownload}
</p>
<p>
@ -102,6 +86,7 @@ export class VideoWrapper extends React.Component {
prev: null,
next: null,
channel: null,
no_channel: null,
}
}
@ -112,8 +97,13 @@ export class VideoWrapper extends React.Component {
async fetchVideo() {
// Get and display the Video specified in the Router match
let [video, prev, next] = await getVideo(this.props.match.params.video_id);
let channel = await getChannel(this.props.match.params.channel_link);
this.setState({video, prev, next, channel}, scrollToTop);
let channel_link = this.props.match.params.channel_link;
let channel = channel_link ? await getChannel(channel_link) : null;
let no_channel = false;
if (!channel) {
no_channel = true;
}
this.setState({video, prev, next, channel, no_channel}, scrollToTop);
}
async componentDidUpdate(prevProps, prevState) {
@ -125,7 +115,7 @@ export class VideoWrapper extends React.Component {
}
render() {
if (this.state.video && this.state.channel) {
if (this.state.video && (this.state.no_channel || this.state.channel)) {
return <VideoPage {...this.state} history={this.props.history} autoplay={true}/>
} else {
return <VideoPlaceholder/>

View file

@ -1,6 +1,6 @@
# Import conftest for test fixtures
import os
from wrolpi.common import import_modules
from wrolpi.conftest import * # noqa
os.environ['DB_PORT'] = '54321'
# Import modules and their conftest.py.
import_modules()

View file

@ -8,7 +8,6 @@ services:
- POSTGRES_PASSWORD=wrolpi
ports:
- '54321:5432'
- '5432:5432'
healthcheck:
test: [ 'CMD-SHELL', 'pg_isready -U postgres' ]

56
main.py
View file

@ -6,11 +6,13 @@ import logging
import sys
import pytz
from sanic import Sanic
from wrolpi import root_api, BEFORE_STARTUP_FUNCTIONS
from wrolpi.common import logger, get_config
from wrolpi import root_api, BEFORE_STARTUP_FUNCTIONS, after_startup, limit_concurrent
from wrolpi.common import logger, get_config, wrol_mode_enabled, import_modules
from wrolpi.dates import set_timezone
from wrolpi.vars import PROJECT_DIR, MODULES_DIR
from wrolpi.downloader import download_manager
from wrolpi.vars import PROJECT_DIR
logger = logger.getChild('wrolpi-main')
@ -40,7 +42,7 @@ def db_main(args):
return 0
async def main():
async def main(loop):
parser = argparse.ArgumentParser()
parser.add_argument('-v', '--verbose', action='count')
@ -70,18 +72,7 @@ async def main():
set_timezone(tz)
# Import the API in every module. Each API should attach itself to `root_api`.
try:
modules = [i.name for i in MODULES_DIR.iterdir() if i.is_dir() and not i.name.startswith('_')]
for module in modules:
module = f'modules.{module}.api'
logger.debug(f'Importing {module}')
__import__(module, globals(), locals(), [], 0)
except ImportError as e:
logger.fatal('No modules could be found!', exc_info=e)
raise
if not modules:
raise Exception('No modules could be found!')
import_modules()
# Run the startup functions
for func in BEFORE_STARTUP_FUNCTIONS:
@ -93,8 +84,8 @@ async def main():
except Exception as e:
logger.warning(f'Startup {func} failed!', exc_info=e)
# Run the API
return root_api.main(args)
# Run the API.
return root_api.main(loop, args)
async def set_log_level(args):
@ -113,7 +104,32 @@ async def set_log_level(args):
logger.warning(f'Logging level: {level_name}')
@after_startup
@limit_concurrent(1)
def periodic_downloads(app: Sanic, loop):
"""
A simple function that perpetually calls downloader_manager.do_downloads() after sleeping.
"""
if wrol_mode_enabled():
logger.warning(f'Not starting download worker because WROL Mode is enabled.')
return
# Set all downloads to new.
download_manager.reset_downloads()
async def _periodic_download():
download_manager.renew_recurring_downloads()
download_manager.delete_old_once_downloads()
await download_manager.do_downloads()
seconds = 60
logger.debug(f'Waiting {seconds} seconds to start next download')
await asyncio.sleep(seconds)
app.add_task(_periodic_download())
app.add_task(_periodic_download())
if __name__ == '__main__':
loop = asyncio.get_event_loop()
result = loop.run_until_complete(main())
loop_ = asyncio.get_event_loop()
result = loop_.run_until_complete(main(loop_))
sys.exit(result)

View file

@ -1,6 +1,31 @@
from abc import ABC
from wrolpi.common import logger
from wrolpi.downloader import Downloader, Download
from wrolpi.errors import UnrecoverableDownloadError
from . import lib
from .api import bp # noqa
PRETTY_NAME = 'Archive'
logger = logger.getChild(__name__)
class ArchiveDownloader(Downloader, ABC):
@staticmethod
def valid_url(url: str) -> bool:
"""
Archiver will attempt to archive anything, so it should be last!
"""
return True
def do_download(self, download: Download):
if download.attempts > 3:
raise UnrecoverableDownloadError(f'Max download attempts reached for {download.url}')
lib.new_archive(download.url, sync=True)
return True
# Archive downloader is the last downloader which should be used.
archive_downloader = ArchiveDownloader(priority=100)

View file

@ -64,7 +64,7 @@ async def search_archives(_: Request, data: dict):
return json_response(ret)
@bp.post(':refresh')
@bp.post('/refresh')
@wrol_mode_check
async def refresh_archives(_: Request):
asyncio.ensure_future(lib.refresh_archives())

View file

@ -6,17 +6,17 @@ import pathlib
import re
from itertools import groupby
from typing import Iterator
from urllib.parse import urlparse
import requests
from bs4 import BeautifulSoup
from sqlalchemy.orm import Session
from modules.archive.models import URL, Domain, Archive
from wrolpi.common import get_media_directory, logger, chunks
from wrolpi.common import get_media_directory, logger, chunks, extract_domain
from wrolpi.dates import now, strptime_ms, strftime_ms
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.errors import InvalidDomain, UnknownURL, PendingArchive, InvalidArchive
from wrolpi.vars import PYTEST
logger = logger.getChild(__name__)
@ -27,21 +27,6 @@ def get_archive_directory() -> pathlib.Path:
return get_media_directory() / 'archive'
def extract_domain(url):
"""
Extract the domain from a URL. Remove leading www.
>>> extract_domain('https://www.example.com/foo')
'example.com'
"""
parsed = urlparse(url)
domain = parsed.netloc
if domain.startswith('www.'):
# Remove leading www.
domain = domain[4:]
return domain
def get_domain_directory(url: str) -> pathlib.Path:
"""
Get the archive directory for a particular domain.
@ -161,7 +146,7 @@ def new_archive(url: str, sync: bool = False):
url_.latest_id = archive_id
url_.latest_datetime = now()
if sync:
if PYTEST or sync:
return _do_archive(url, archive_id)
else:
# Run the real archive process in the future.

View file

@ -312,7 +312,7 @@ class TestArchive(TestAPI):
archive: Archive = session.query(Archive).one()
self.assertEqual(url.latest, archive)
# latest_datetime is set, the timestamp in the file name is assumed to be UTC.
self.assertEqual(str(url.latest_datetime), '2021-10-05 22:20:10.346823-06:00')
self.assertEqual(str(url.latest_datetime), '2021-10-05 16:20:10.346823-06:00')
# Running the refresh does not result in a new archive.
_refresh_archives()

View file

@ -27,8 +27,8 @@ class Item(Base, ModelHelper):
inventory = relationship('Inventory', primaryjoin="Item.inventory_id==Inventory.id", back_populates='items')
def __repr__(self):
return f'<Item(id={self.id}, name={self.name}, brand={self.brand}, ' \
f'count={self.count}, item_size={self.item_size}, unit={self.unit}, inventory={self.inventory_id})>'
return f'<Item id={self.id}, name={self.name}, brand={self.brand}, ' \
f'count={self.count}, item_size={self.item_size}, unit={self.unit}, inventory={self.inventory_id} >'
class Inventory(Base, ModelHelper):
@ -43,7 +43,7 @@ class Inventory(Base, ModelHelper):
items: InstrumentedList = relationship('Item', foreign_keys='Item.inventory_id')
def __repr__(self):
return f'<Inventory(id={self.id}, name={self.name!r}, deleted={self.deleted_at})>'
return f'<Inventory id={self.id}, name={self.name!r}, deleted={self.deleted_at} >'
def dict(self):
d = super().dict()

View file

@ -0,0 +1 @@
from modules.videos import downloader

View file

@ -1,28 +1,25 @@
import asyncio
import random
from functools import wraps
from http import HTTPStatus
from multiprocessing import Event
import requests
from sanic import Blueprint, response
from sanic.request import Request
from wrolpi import before_startup
from wrolpi.common import create_websocket_feed, get_sanic_url, \
wrol_mode_check, ProgressReporter, get_config
wrol_mode_check
from wrolpi.common import logger
from wrolpi.root_api import add_blueprint, json_response
from wrolpi.schema import validate_doc, JSONErrorResponse
from .channel.api import channel_bp
from .downloader import update_channels, download_all_missing_videos
from .lib import process_video_meta_data, _refresh_videos, get_statistics, get_download_history
from .channel.lib import download_channel, spread_channel_downloads
from .lib import _refresh_videos, get_statistics
from .schema import StreamResponse, \
FavoriteRequest, FavoriteResponse, VideosStatisticsResponse, DownloadHistoryResponse
FavoriteRequest, FavoriteResponse, VideosStatisticsResponse
from .video.api import video_bp
from .video.lib import set_video_favorite
content_bp = Blueprint('Video Content', '/api/videos')
content_bp = Blueprint('VideoContent', '/api/videos')
bp = Blueprint('Videos', '/api/videos').group(
content_bp, # view and manage video content and settings
channel_bp, # view and manage channels
@ -35,8 +32,8 @@ logger = logger.getChild(__name__)
refresh_queue, refresh_event = create_websocket_feed('refresh', '/feeds/refresh', content_bp)
@content_bp.post(':refresh')
@content_bp.post(':refresh/<link:string>')
@content_bp.post('/refresh')
@content_bp.post('/refresh/<link:str>')
@validate_doc(
summary='Search for videos that have previously been downloaded and stored.',
produces=StreamResponse,
@ -63,8 +60,7 @@ async def refresh(_, link: str = None):
await refresh_videos(channel_links)
refresh_logger.info('refresh complete')
except Exception as e:
refresh_queue.put({'error': 'Refresh failed. See server logs.', 'message': str(e)})
except Exception:
raise
finally:
refresh_event.clear()
@ -78,8 +74,8 @@ async def refresh(_, link: str = None):
download_queue, download_event = create_websocket_feed('download', '/feeds/download', content_bp)
@content_bp.post(':download')
@content_bp.post(':download/<link:string>')
@content_bp.post('/download')
@content_bp.post('/download/<link:str>')
@validate_doc(
summary='Update channel catalogs, download any missing videos',
produces=StreamResponse,
@ -88,7 +84,7 @@ download_queue, download_event = create_websocket_feed('download', '/feeds/downl
],
)
@wrol_mode_check
async def download(_, link: str = None):
def download(_, link: str = None):
download_logger = logger.getChild('download')
stream_url = get_sanic_url(scheme='ws', path='/api/videos/feeds/download')
@ -98,44 +94,18 @@ async def download(_, link: str = None):
if refresh_event.is_set():
return response.json({'error': 'Refresh is running. Cannot download.'})
download_event.set()
download_channel(link)
async def do_download():
reporter = ProgressReporter(download_queue, 2)
reporter.set_progress_total(0, 1)
reporter.send_progress(0, 1, 'Download started')
try:
update_channels(reporter, link)
download_logger.info('Updated all channel catalogs')
download_all_missing_videos(reporter, link)
reporter.finish(1, 'All videos have been downloaded')
# Fill in any missing data for all videos.
reporter.message(0, 'Processing and cleaning files')
process_video_meta_data()
reporter.finish(0, 'Processing and cleaning complete')
download_logger.info('download complete')
except Exception as e:
logger.fatal(f'Download failed: {e}')
reporter.error(0, 'Download failed. See server logs.')
raise
finally:
download_event.clear()
coro = do_download()
asyncio.ensure_future(coro)
download_logger.debug('do_download scheduled')
return response.json({'code': 'stream-started', 'stream_url': stream_url})
@wraps(_refresh_videos)
async def refresh_videos(channel_links: list = None):
return _refresh_videos(refresh_queue, channel_links=channel_links)
return _refresh_videos(channel_links=channel_links)
@content_bp.post(':favorite')
@content_bp.post('/favorite')
@validate_doc(
summary='Toggle the favorite flag on a video',
consumes=FavoriteRequest,
@ -160,56 +130,6 @@ async def statistics(_: Request):
return json_response(ret, HTTPStatus.OK)
@content_bp.get('/download_history')
@validate_doc(
summary='Retrieve video download history',
produces=DownloadHistoryResponse,
)
async def download_history(_: Request):
history = await get_download_history()
ret = dict(history=history)
return json_response(ret, HTTPStatus.OK)
MIN_DOWNLOAD_FREQUENCY = 60 * 60 * 4 # 4 hours
MAX_DOWNLOAD_FREQUENCY = 60 * 60 * 12 # 12 hours
PERIODIC_DOWNLOAD_EVENT = Event()
async def _periodic_download(min_download_frequency: int, max_download_frequency: int):
"""
Wait some amount of time, download, then schedule the next download.
"""
sleep_seconds = random.randint(min_download_frequency, max_download_frequency)
logger.debug(f'Waiting {sleep_seconds} seconds before next periodic download')
await asyncio.sleep(sleep_seconds)
url = get_sanic_url(path='/api/videos:download')
resp = requests.post(url)
if resp.status_code != HTTPStatus.OK and resp.status_code != HTTPStatus.CONFLICT:
# Download failed and wasn't already running.
logger.warning(f'Periodic download failed with status_code={resp.status_code}')
logger.warning(f'Periodic download response={resp}')
# Schedule the next download
asyncio.ensure_future(_periodic_download(min_download_frequency, max_download_frequency))
@before_startup
def start_periodic_download():
"""
Periodically download the videos.
"""
if not PERIODIC_DOWNLOAD_EVENT.is_set():
PERIODIC_DOWNLOAD_EVENT.set()
logger.info(f'Starting periodic download')
config = get_config()
try:
min_download_frequency = int(config.get('min_download_frequency', MIN_DOWNLOAD_FREQUENCY))
max_download_frequency = int(config.get('max_download_frequency', MAX_DOWNLOAD_FREQUENCY))
except Exception:
logger.fatal(f'Failed to get min/max download frequencies from config!', exc_info=True)
raise
asyncio.ensure_future(_periodic_download(min_download_frequency, max_download_frequency))
def startup_spread_channel_downloads():
spread_channel_downloads()

View file

@ -1,4 +1,5 @@
#! /usr/bin/env python3
import pathlib
from pathlib import Path
from typing import Generator, List, Union
@ -7,8 +8,7 @@ import webvtt
from wrolpi.common import logger, chunks
from wrolpi.db import get_db_session
from wrolpi.errors import UnknownCaptionFile
from .common import get_absolute_video_caption
from wrolpi.media_path import MediaPath
from .models import Video
@ -43,9 +43,11 @@ def process_captions(video: Video):
"""
Parse and insert captions for a video record.
"""
caption_path = get_absolute_video_caption(video)
if not caption_path.exists():
raise UnknownCaptionFile()
if isinstance(video.caption_path, MediaPath):
caption_path = video.caption_path.path
else:
caption_path = pathlib.Path(video.caption_path)
try:
lines = get_unique_caption_lines(str(caption_path))
block = '\n'.join(lines)

View file

@ -5,8 +5,8 @@ from sanic import response, Blueprint
from sanic.request import Request
from wrolpi.common import logger, wrol_mode_check, get_relative_to_media_directory, make_media_directory
from wrolpi.root_api import json_response
from wrolpi.errors import UnknownDirectory
from wrolpi.root_api import json_response
from wrolpi.schema import validate_doc, JSONErrorResponse
from .lib import get_minimal_channels, delete_channel, update_channel, get_channel, create_channel
from ..schema import ChannelsResponse, ChannelResponse, ChannelPostRequest, \
@ -27,7 +27,7 @@ async def get_channels(_: Request):
return json_response({'channels': channels})
@channel_bp.route('/<link:string>', methods=['GET', 'OPTIONS'])
@channel_bp.route('/<link:str>', methods=['GET', 'OPTIONS'])
@validate_doc(
summary='Get a Channel',
produces=ChannelResponse,
@ -36,7 +36,7 @@ async def get_channels(_: Request):
),
)
def channel_get(_: Request, link: str):
channel = get_channel(link)
channel = get_channel(link=link)
channel.pop('info_json')
return json_response({'channel': channel})
@ -61,6 +61,15 @@ def channel_post(_: Request, data: dict):
else:
raise
if download_frequency := data.get('download_frequency'):
try:
download_frequency = int(download_frequency)
except ValueError:
if download_frequency in ('null', 'None'):
download_frequency = None
data['download_frequency'] = download_frequency
channel = create_channel(data)
# Refresh the videos asynchronously
@ -72,12 +81,11 @@ def channel_post(_: Request, data: dict):
{'Location': f'/api/videos/channels/{channel["link"]}'})
@channel_bp.put('/<link:string>')
@channel_bp.patch('/<link:string>')
@channel_bp.put('/<link:str>')
@channel_bp.patch('/<link:str>')
@validate_doc(
summary='Update a Channel',
consumes=ChannelPutRequest,
produces=SuccessResponse,
responses=(
(HTTPStatus.NOT_FOUND, JSONErrorResponse),
(HTTPStatus.BAD_REQUEST, JSONErrorResponse),
@ -90,7 +98,7 @@ def channel_update(_: Request, link: str, data: dict):
headers={'Location': f'/api/videos/channels/{channel.link}'})
@channel_bp.delete('/<link:string>')
@channel_bp.delete('/<link:str>')
@validate_doc(
summary='Delete a Channel',
produces=SuccessResponse,

View file

@ -1,16 +1,19 @@
from typing import List, Dict
from collections import defaultdict
from datetime import timedelta
from typing import List, Dict, Union
from sqlalchemy.orm.exc import NoResultFound
from wrolpi.common import sanitize_link, run_after, get_relative_to_media_directory, make_media_directory
from wrolpi.common import sanitize_link, run_after, get_relative_to_media_directory, make_media_directory, logger
from wrolpi.dates import today
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.downloader import download_manager, Download
from wrolpi.errors import UnknownChannel, UnknownDirectory, APIError, ValidationError
from ..common import check_for_channel_conflicts
from ..lib import save_channels_config
from ..models import Channel
DEFAULT_DOWNLOAD_FREQUENCY = 60 * 60 * 24 * 7 # weekly
DEFAULT_DOWNLOAD_TIMEOUT = 60.0 * 10.0 # Ten minutes
logger = logger.getChild(__name__)
async def get_minimal_channels() -> List[dict]:
@ -21,7 +24,7 @@ async def get_minimal_channels() -> List[dict]:
# Get all channels, even if they don't have videos.
stmt = '''
SELECT
c.id, name, link, directory, url, download_frequency, info_date, next_download
c.id, name, link, directory, url, download_frequency, info_date
FROM
channel AS c
ORDER BY LOWER(name)
@ -64,15 +67,69 @@ async def get_channels_video_count() -> Dict[int, int]:
return video_counts
@run_after(save_channels_config)
def delete_channel(link):
with get_db_session(commit=True) as session:
try:
channel = session.query(Channel).filter_by(link=link).one()
except NoResultFound:
raise UnknownChannel()
def get_channel(source_id: str = None, link: str = None, url: str = None, return_dict: bool = True
) -> Union[dict, Channel]:
"""
Attempt to find a Channel using the provided params. The params are in order of reliability.
channel.delete_with_videos()
Raises UnknownChannel if no channel is found.
"""
with get_db_session() as session:
# Try by source_id first.
channel = None
if source_id:
channel = session.query(Channel).filter_by(source_id=source_id).one_or_none()
if not channel and link:
channel = session.query(Channel).filter_by(link=link).one_or_none()
if not channel and url:
channel = session.query(Channel).filter_by(url=url).one_or_none()
if not channel:
raise UnknownChannel(f'No channel matches {link=} {source_id=} {url=}')
logger.debug(f'Found {channel=} using {source_id=} {link=} {url=}')
return channel.dict() if return_dict else channel
def _spread_by_frequency(channels: List[Channel]):
channels_by_frequency = defaultdict(lambda: [])
for channel in channels:
channels_by_frequency[channel.download_frequency].append(channel)
for frequency, channels in channels_by_frequency.items():
# The seconds between each download.
chunk = frequency // len(channels)
for channel in channels:
# My position in the channel group.
index = channels.index(channel)
# My next download will be distributed by my frequency and my position.
position = chunk * index
next_download = today() + timedelta(seconds=frequency + position)
yield dict(url=channel.url, frequency=frequency, next_download=next_download)
def spread_channel_downloads():
"""
Channels should be downloaded in a manner that is spread out over their frequency. For example, three channels
with a frequency of a week should be downloaded on different days that week.
"""
with get_db_session(commit=True) as session:
channels = list(session.query(Channel).filter(
Channel.url != None,
Channel.url != '',
Channel.download_frequency != None
).order_by(Channel.link).all()) # noqa
url_next_download = _spread_by_frequency(channels)
for info in url_next_download:
download = download_manager.get_download(session, url=info['url'])
if not download:
download = Download(url=info['url'])
session.add(download)
download.frequency = info['frequency']
download.next_download = info['next_download']
@run_after(save_channels_config)
@ -101,7 +158,10 @@ def update_channel(data, link):
try:
data['download_frequency'] = int(data['download_frequency'])
except ValueError:
raise APIError('Invalid download frequency')
if data['download_frequency'] in ('null', 'None', ''):
data['download_frequency'] = None
else:
raise APIError(f'Invalid download frequency {data["download_frequency"]}')
if data.get('match_regex') in ('None', 'null'):
data['match_regex'] = None
@ -119,35 +179,28 @@ def update_channel(data, link):
# Apply the changes now that we've OK'd them
channel.update(data)
spread_channel_downloads()
return channel
def get_channel(link) -> dict:
"""
Get a Channel by it's `link`. Raise UnknownChannel if it does not exist.
"""
with get_db_session() as session:
try:
channel = session.query(Channel).filter_by(link=link).one()
except NoResultFound:
raise UnknownChannel()
return channel.dict()
@run_after(save_channels_config)
def create_channel(data: dict) -> dict:
@run_after(spread_channel_downloads)
def create_channel(data: dict, return_dict: bool = True) -> Union[Channel, dict]:
"""
Create a new Channel. Check for conflicts with existing Channels.
"""
with get_db_session(commit=True) as session:
link = sanitize_link(data['name'])
try:
# Verify that the URL/Name/Link aren't taken
check_for_channel_conflicts(
session,
url=data.get('url'),
name=data['name'],
link=sanitize_link(data['name']),
link=link,
directory=str(data['directory']),
source_id=data.get('source_id'),
)
except APIError as e:
raise ValidationError() from e
@ -156,13 +209,34 @@ def create_channel(data: dict) -> dict:
name=data['name'],
url=data.get('url'),
match_regex=data.get('match_regex'),
link=sanitize_link(data['name']),
link=link,
directory=str(data['directory']),
download_frequency=data.get('download_frequency', DEFAULT_DOWNLOAD_FREQUENCY),
download_frequency=data.get('download_frequency'),
source_id=data.get('source_id'),
)
session.add(channel)
session.commit()
session.flush()
session.refresh(channel)
return channel.dict()
if return_dict:
return channel.dict()
else:
return channel
@run_after(save_channels_config)
def delete_channel(link):
with get_db_session(commit=True) as session:
try:
channel = session.query(Channel).filter_by(link=link).one()
except NoResultFound:
raise UnknownChannel()
channel.delete_with_videos()
def download_channel(link: str):
channel = get_channel(link=link, return_dict=False)
with get_db_session(commit=True) as session:
download_manager.create_download(channel.url, session)

View file

@ -1,4 +1,5 @@
import json
import os
import pathlib
import tempfile
from datetime import timedelta
@ -9,14 +10,16 @@ import mock
import pytest
from modules.videos.api import refresh_queue
from modules.videos.channel.lib import spread_channel_downloads, delete_channel
from modules.videos.lib import upsert_video
from modules.videos.models import Channel, Video
from modules.videos.test.common import create_channel_structure
from wrolpi.dates import now
from wrolpi.db import get_db_session
from wrolpi.downloader import download_manager, Download
from wrolpi.errors import UnknownFile
from wrolpi.root_api import api_app
from wrolpi.test.common import wrap_test_db, get_all_messages_in_queue, TestAPI
from wrolpi.test.common import wrap_test_db, TestAPI
class TestVideoAPI(TestAPI):
@ -39,83 +42,83 @@ class TestVideoAPI(TestAPI):
@wrap_test_db
def test_channel(self):
with tempfile.TemporaryDirectory() as media_dir, \
mock.patch('wrolpi.common.get_media_directory', lambda: pathlib.Path(media_dir)):
channel_directory = tempfile.TemporaryDirectory(dir=media_dir).name
pathlib.Path(channel_directory).mkdir()
new_channel = dict(
directory=channel_directory,
match_regex='asdf',
name='Example Channel 1',
url='https://example.com/channel1',
)
channel_directory = tempfile.TemporaryDirectory(dir=self.tmp_dir.name).name
channel_directory_name = channel_directory.split('/')[-1]
pathlib.Path(channel_directory).mkdir()
# Channel doesn't exist
request, response = api_app.test_client.get('/api/videos/channels/examplechannel1')
assert response.status_code == HTTPStatus.NOT_FOUND, f'Channel exists: {response.json}'
new_channel = dict(
directory=channel_directory,
match_regex='asdf',
name='Example Channel 1',
url='https://example.com/channel1',
)
# Create it
request, response = api_app.test_client.post('/api/videos/channels', data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.CREATED
location = response.headers['Location']
request, response = api_app.test_client.get(location)
created = response.json['channel']
self.assertIsNotNone(created)
self.assertIsNotNone(created['id'])
# Channel doesn't exist
request, response = api_app.test_client.get('/api/videos/channels/examplechannel1')
assert response.status_code == HTTPStatus.NOT_FOUND, f'Channel exists: {response.json}'
# Channel directory should be relative to the media directory
assert not pathlib.Path(created['directory']).is_absolute(), \
f'Channel directory is absolute: {created["directory"]}'
# Create it
request, response = api_app.test_client.post('/api/videos/channels', data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.CREATED
location = response.headers['Location']
request, response = api_app.test_client.get(location)
created = response.json['channel']
self.assertIsNotNone(created)
self.assertIsNotNone(created['id'])
# Get the link that was computed
new_channel['link'] = response.json['channel']['link']
assert new_channel['link']
# Channel directory should be relative to the media directory
assert not pathlib.Path(created['directory']).is_absolute(), \
f'Channel directory is absolute: {created["directory"]}'
# Can't create it again
request, response = api_app.test_client.post('/api/videos/channels', data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.BAD_REQUEST
# Get the link that was computed
new_channel['link'] = response.json['channel']['link']
assert new_channel['link']
# Update it
new_channel['name'] = 'Example Channel 2'
new_channel['directory'] = str(new_channel['directory'])
request, response = api_app.test_client.put(location, data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.NO_CONTENT, response.status_code
request, response = api_app.test_client.get(location)
assert response.status_code == HTTPStatus.OK
self.assertDictContains(response.json['channel'], {
'id': 1,
'name': 'Example Channel 2',
'directory': channel_directory,
'match_regex': 'asdf',
'url': 'https://example.com/channel1',
})
# Can't create it again
request, response = api_app.test_client.post('/api/videos/channels', data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.BAD_REQUEST
# Patch it
patch = {'name': 'new name'}
request, response = api_app.test_client.patch(location, data=json.dumps(patch))
assert response.status_code == HTTPStatus.NO_CONTENT, response.status_code
request, response = api_app.test_client.get(location)
assert response.status_code == HTTPStatus.OK
self.assertDictContains(response.json['channel'], {
'id': 1,
'name': 'new name',
'directory': channel_directory,
'match_regex': 'asdf',
'url': 'https://example.com/channel1',
})
# Update it
new_channel['name'] = 'Example Channel 2'
new_channel['directory'] = str(new_channel['directory'])
request, response = api_app.test_client.put(location, data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.NO_CONTENT, response.status_code
request, response = api_app.test_client.get(location)
assert response.status_code == HTTPStatus.OK
self.assertDictContains(response.json['channel'], {
'id': 1,
'name': 'Example Channel 2',
'directory': channel_directory_name,
'match_regex': 'asdf',
'url': 'https://example.com/channel1',
})
# Can't update channel that doesn't exist
request, response = api_app.test_client.put('/api/videos/channels/doesnt_exist',
data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.NOT_FOUND
# Patch it
patch = {'name': 'new name'}
request, response = api_app.test_client.patch(location, data=json.dumps(patch))
assert response.status_code == HTTPStatus.NO_CONTENT, response.status_code
request, response = api_app.test_client.get(location)
assert response.status_code == HTTPStatus.OK
self.assertDictContains(response.json['channel'], {
'id': 1,
'name': 'new name',
'directory': channel_directory_name,
'match_regex': 'asdf',
'url': 'https://example.com/channel1',
})
# Delete the new channel
request, response = api_app.test_client.delete(location)
assert response.status_code == HTTPStatus.NO_CONTENT
# Can't update channel that doesn't exist
request, response = api_app.test_client.put('/api/videos/channels/doesnt_exist',
data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.NOT_FOUND
# Cant delete it again
request, response = api_app.test_client.delete(location)
assert response.status_code == HTTPStatus.NOT_FOUND
# Delete the new channel
request, response = api_app.test_client.delete(location)
assert response.status_code == HTTPStatus.NO_CONTENT
# Cant delete it again
request, response = api_app.test_client.delete(location)
assert response.status_code == HTTPStatus.NOT_FOUND
@wrap_test_db
def test_channel_conflicts(self):
@ -210,15 +213,61 @@ class TestVideoAPI(TestAPI):
assert response.status_code == HTTPStatus.CREATED, response.json
assert location != response.headers['Location']
@wrap_test_db
def test_refresh_no_channel(self):
with get_db_session() as session:
videos_path = pathlib.Path(self.tmp_dir.name) / 'videos'
videos_path.mkdir()
# A regular user-defined channel
channel1_path = videos_path / 'channel1'
channel1_path.mkdir()
vid1 = channel1_path / 'MyChannelName_20000101_abcdefghijk_title1.mp4'
vid1.touch()
vid2 = channel1_path / 'MyChannelName_20000101_abcdefghijk_title2.mp4'
vid2.touch()
channel1 = Channel(name='channel1', link='channel1', directory=channel1_path)
session.add(channel1)
session.commit()
# The special NO CHANNEL directory.
from modules.videos.downloader import get_no_channel_directory
no_channel_path = get_no_channel_directory()
no_channel_path.mkdir()
vid3 = no_channel_path / 'some video.mp4'
vid3.touch()
no_channel_subdir = no_channel_path / 'subdir'
no_channel_subdir.mkdir()
vid4 = no_channel_subdir / 'other video.mp4'
vid4.touch()
api_app.test_client.post('/api/videos/refresh')
videos = session.query(Video).order_by(Video.title).all()
self.assertEqual(len(videos), 4)
# Add some fake videos, they should be removed.
session.add(Video(title='vid5'))
session.add(Video(title='vid6', channel_id=channel1.id))
session.commit()
videos = session.query(Video).order_by(Video.title).all()
self.assertEqual(len(videos), 6)
api_app.test_client.post('/api/videos/refresh')
videos = session.query(Video).order_by(Video.title).all()
self.assertEqual(len(videos), 4)
@wrap_test_db
def test_refresh_videos(self):
# There should be no messages until a refresh is called.
pytest.raises(Empty, refresh_queue.get_nowait)
# Setup a fake channel directory.
with get_db_session() as session, \
tempfile.TemporaryDirectory() as channel_dir:
channel_path = pathlib.Path(channel_dir)
with get_db_session() as session:
channel_path = pathlib.Path(self.tmp_dir.name)
# Files in subdirectories should be found and handled properly.
subdir = channel_path / 'subdir'
@ -243,26 +292,26 @@ class TestVideoAPI(TestAPI):
poster2.touch()
# Create a channel, associate videos with it.
channel = Channel(directory=channel_dir, link='foo', name='foo')
channel = Channel(directory=channel_path, link='foo', name='foo')
session.add(channel)
session.flush()
session.refresh(channel)
video1 = upsert_video(session, vid1, channel)
video2 = upsert_video(session, vid2, channel)
session.commit()
self.assertEqual({i.video_path for i in channel.videos},
{pathlib.Path('subdir/' + vid1.name), pathlib.Path(vid2.name)})
self.assertEqual({str(i.video_path.path) for i in channel.videos},
{f'{channel_path}/subdir/{vid1.name}', f'{channel_path}/{vid2.name}'})
# Poster files were found.
self.assertEqual(video1.poster_path, pathlib.Path('subdir/' + poster1.name))
self.assertEqual(video2.poster_path, pathlib.Path(poster2.name))
self.assertEqual(video1.poster_path.path, subdir / poster1.name)
self.assertEqual(video2.poster_path.path, channel_path / poster2.name)
# Add a bogus file, this should be removed during the refresh
self.assertNotIn('foo', {i.video_path for i in channel.videos})
self.assertNotIn('foo', {i.video_path.path for i in channel.videos})
session.add(Video(video_path='foo', channel_id=channel.id))
session.flush()
session.refresh(channel)
self.assertIn('foo', {str(i.video_path) for i in channel.videos})
self.assertIn(f'{self.tmp_dir.name}/foo', {str(i.video_path.path) for i in channel.videos})
self.assertEqual(len(channel.videos), 3)
# Add a video that isn't in the DB, it should be found and any meta files associated with it
@ -277,10 +326,10 @@ class TestVideoAPI(TestAPI):
# Finally, call the refresh. Again, it should remove the "foo" video, then discover this 3rd video
# file and it's description.
api_app.test_client.post('/api/videos:refresh')
api_app.test_client.post('/api/videos/refresh')
# Bogus file was removed
self.assertNotIn('foo', {i.video_path for i in channel.videos})
self.assertNotIn(f'{self.tmp_dir.name}/foo', {i.video_path.path for i in channel.videos})
# Final channel video list we built
expected = {
@ -290,7 +339,7 @@ class TestVideoAPI(TestAPI):
}
def str_or_none(i):
return str(i) if i else None
return str(i.__json__()) if i else None
self.assertEqual(
{(str_or_none(i.video_path), str_or_none(i.poster_path), str_or_none(i.description_path))
@ -300,10 +349,6 @@ class TestVideoAPI(TestAPI):
assert poster3.is_file(), 'Orphan jpg file was deleted!'
# During the refresh process, messages are pushed to a queue, make sure there are messages there
messages = get_all_messages_in_queue(refresh_queue)
assert 'refresh-started' in [i.get('code') for i in messages]
@wrap_test_db
def test_get_channel_videos(self):
with get_db_session(commit=True) as session:
@ -466,3 +511,93 @@ class TestVideoAPI(TestAPI):
response = do_search('b', 2)
assert [i['id'] for i in response.json['videos']] == [1, 2]
assert response.json['totals']['videos'] == 4
@wrap_test_db
def test_channel_no_download_frequency(self):
"""
A channel does not require a download frequency.
"""
channel_directory = tempfile.TemporaryDirectory(dir=self.tmp_dir.name).name
os.mkdir(channel_directory)
new_channel = dict(
directory=channel_directory,
name='Example Channel 1',
url='https://example.com/channel1',
download_frequency=None,
)
# Create the Channel
request, response = api_app.test_client.post('/api/videos/channels', data=json.dumps(new_channel))
assert response.status_code == HTTPStatus.CREATED
location = response.headers['Location']
# No downloads are scheduled.
spread_channel_downloads()
with get_db_session() as session:
downloads = list(download_manager.get_new_downloads(session))
self.assertEqual(len(downloads), 0)
# Get the Channel
request, response = api_app.test_client.get(location)
channel = response.json['channel']
self.assertEqual(channel['download_frequency'], None)
# Update the Channel with a frequency.
new_channel = dict(
directory=channel['directory'],
name=channel['name'],
url=channel['url'],
download_frequency=10,
)
request, response = api_app.test_client.put(f'/api/videos/channels/{channel["link"]}',
data=json.dumps(new_channel))
self.assertEqual(response.status_code, 204, response.json)
# Remove the frequency.
new_channel = dict(
directory=channel['directory'],
name=channel['name'],
url=channel['url'],
download_frequency=None,
)
request, response = api_app.test_client.put(f'/api/videos/channels/{channel["link"]}',
data=json.dumps(new_channel))
self.assertEqual(response.status_code, 204, response.json)
def test_channel_frequency_update(download_channel, test_client, test_session):
"""
A Channel's Download record is updated when the Channel's frequency is updated.
"""
old_frequency = download_channel.get_download().frequency
assert old_frequency
data = dict(
directory=str(download_channel.directory.path),
name=download_channel.name,
url=download_channel.url,
download_frequency=100,
)
request, response = test_client.put(f'/api/videos/channels/{download_channel.link}', json=data)
assert response.status_code == HTTPStatus.NO_CONTENT, response.json
download = download_channel.get_download()
assert download.frequency == 100
# Only one download
downloads = test_session.query(Download).all()
assert len(list(downloads)) == 1
def test_delete_channel_delete_download(download_channel, test_session):
"""
Deleting a Channel deletes it's Download.
"""
downloads = test_session.query(Download).all()
assert all(i.url == download_channel.url for i in downloads)
delete_channel(download_channel.link)
downloads = test_session.query(Download).all()
assert not list(downloads)

View file

@ -1,21 +0,0 @@
import pathlib
import unittest
from modules.videos.models import Channel
from modules.videos.test.common import create_channel_structure
from wrolpi.db import get_db_session
from wrolpi.test.common import wrap_test_db
class TestChannel(unittest.TestCase):
@wrap_test_db
@create_channel_structure(
{
'Foo': ['vid1.mp4'],
}
)
def test_channel(self, tempdir):
with get_db_session() as session:
channel = session.query(Channel).one()
self.assertIsInstance(channel.directory, pathlib.Path)

View file

@ -12,11 +12,12 @@ from PIL import Image
from sqlalchemy.orm import Session
from wrolpi import before_startup
from wrolpi.common import sanitize_link, logger, CONFIG_PATH, get_config, iterify, chunks, get_media_directory, \
get_absolute_media_path, minimize_dict
from wrolpi.common import logger, CONFIG_PATH, get_config, iterify, chunks, get_media_directory, \
minimize_dict
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.errors import UnknownFile, UnknownDirectory, ChannelNameConflict, ChannelURLConflict, \
ChannelLinkConflict, ChannelDirectoryConflict
ChannelLinkConflict, ChannelDirectoryConflict, ChannelSourceIdConflict
from wrolpi.media_path import MediaPath
from wrolpi.vars import DOCKERIZED, DEFAULT_FILE_PERMISSIONS
from .models import Channel, Video
@ -24,7 +25,7 @@ logger = logger.getChild(__name__)
REQUIRED_OPTIONS = ['name', 'directory']
MINIMUM_CHANNEL_KEYS = {'id', 'name', 'directory', 'url', 'video_count', 'link'}
MINIMUM_INFO_JSON_KEYS = {'description', 'view_count'}
MINIMUM_INFO_JSON_KEYS = {'description', 'view_count', 'webpage_url'}
MINIMUM_VIDEO_KEYS = {'id', 'title', 'upload_date', 'duration', 'channel', 'channel_id', 'favorite', 'size',
'poster_path', 'caption_path', 'video_path', 'info_json', 'channel', 'viewed', 'source_id',
'view_count'}
@ -67,35 +68,38 @@ def import_videos_config():
config = load_channels_config()
with get_db_session(commit=True) as session:
for section in config:
for option in (i for i in REQUIRED_OPTIONS if i not in config[section]):
raise ConfigError(f'Channel "{section}" is required to have "{option}"')
for link in config:
for option in (i for i in REQUIRED_OPTIONS if i not in config[link]):
raise ConfigError(f'Channel "{link}" is required to have "{option}"')
name, directory = config[section]['name'], config[section]['directory']
name, directory = config[link]['name'], config[link]['directory']
link = sanitize_link(name)
matches = session.query(Channel).filter(Channel.link == link)
if not matches.count():
# Channel not yet in the DB, add it
channel = session.query(Channel).filter(Channel.link == link).one_or_none()
if not channel:
# Channel not yet in the DB, add it.
channel = Channel(link=link)
else:
channel = matches.one()
# Only name and directory are required
channel.name = name
channel.directory = str(directory)
channel.calculate_duration = config[section].get('calculate_duration')
channel.download_frequency = config[section].get('download_frequency')
channel.generate_posters = config[section].get('generate_posters')
channel.match_regex = config[section].get('match_regex')
channel.skip_download_videos = list(set(config[section].get('skip_download_videos', {})))
channel.url = config[section].get('url')
channel.calculate_duration = config[link].get('calculate_duration') or channel.calculate_duration
channel.download_frequency = config[link].get('download_frequency') or channel.download_frequency
channel.generate_posters = config[link].get('generate_posters') or channel.generate_posters
channel.match_regex = config[link].get('match_regex') or channel.match_regex
channel.skip_download_videos = list(set(config[link].get('skip_download_videos', {})))
channel.source_id = config[link].get('source_id') or channel.source_id
channel.url = config[link].get('url') or channel.url
if not channel.source_id and channel.url:
# If we can download from a channel, we must have its source_id.
from .downloader import get_channel_source_id
channel.source_id = get_channel_source_id(channel.url)
session.add(channel)
# Set favorite Videos of this Channel.
favorites = config[section].get('favorites', {})
favorites = config[link].get('favorites', {})
if favorites:
videos = session.query(Video).filter(Video.source_id.in_(favorites.keys()))
for video in videos:
@ -107,13 +111,12 @@ def import_videos_config():
VALID_VIDEO_KINDS = {'video', 'caption', 'poster', 'description', 'info_json'}
def get_absolute_video_path(video: Video, kind: str = 'video') -> Path:
def get_absolute_video_path(video: Video, kind: str = 'video') -> MediaPath:
if kind not in VALID_VIDEO_KINDS:
raise Exception(f'Unknown video path kind {kind}')
directory = get_absolute_media_path(video.channel.directory)
path = getattr(video, f'{kind}_path')
if directory and path:
return directory / path
if path:
return path
raise UnknownFile()
@ -123,7 +126,7 @@ get_absolute_video_description = partial(get_absolute_video_path, kind='descript
get_absolute_video_info_json = partial(get_absolute_video_path, kind='info_json')
def get_absolute_video_files(video: Video) -> List[Path]:
def get_absolute_video_files(video: Video) -> List[MediaPath]:
"""
Get all video files that exist.
"""
@ -173,7 +176,7 @@ def any_extensions(filename: str, extensions: Iterable = ()):
match_video_extensions = partial(any_extensions, extensions=VIDEO_EXTENSIONS)
def generate_video_paths(directory: Union[str, pathlib.Path], relative_to=None) -> Tuple[str, pathlib.Path]:
def generate_video_paths(directory: Union[str, pathlib.Path]) -> Tuple[str, pathlib.Path]:
"""
Generate a list of video paths in the provided directory.
"""
@ -182,10 +185,7 @@ def generate_video_paths(directory: Union[str, pathlib.Path], relative_to=None)
for child in directory.iterdir():
if child.is_file() and match_video_extensions(child.name):
child = child.absolute()
if relative_to:
yield child.relative_to(relative_to)
else:
yield child
yield child
elif child.is_dir():
yield from generate_video_paths(child)
@ -232,7 +232,8 @@ def remove_duplicate_video_paths(paths: Iterable[Path]) -> Set[Path]:
yield sorted(paths)[0]
def check_for_channel_conflicts(session: Session, id_=None, url=None, name=None, link=None, directory=None):
def check_for_channel_conflicts(session: Session, id_=None, url=None, name=None, link=None, directory=None,
source_id=None):
"""
Search for any channels that conflict with the provided args, raise a relevant exception if any conflicts are found.
"""
@ -263,6 +264,10 @@ def check_for_channel_conflicts(session: Session, id_=None, url=None, name=None,
conflicts = base_where.filter(Channel.directory == directory)
if list(conflicts):
raise ChannelDirectoryConflict()
if source_id:
conflicts = base_where.filter(Channel.source_id == source_id)
if list(conflicts):
raise ChannelSourceIdConflict()
def verify_config():
@ -295,10 +300,14 @@ def get_matching_directories(path: Union[str, Path]) -> List[str]:
"""
path = str(path)
ignored_directories = {
str(get_media_directory() / 'videos/NO CHANNEL'),
}
if os.path.isdir(path):
# The provided path is a directory, return it's subdirectories, or itself if no subdirectories exist
paths = [os.path.join(path, i) for i in os.listdir(path)]
paths = sorted(i for i in paths if os.path.isdir(i))
paths = sorted(i for i in paths if os.path.isdir(i) and i not in ignored_directories)
if len(paths) == 0:
return [path]
return paths
@ -307,7 +316,8 @@ def get_matching_directories(path: Union[str, Path]) -> List[str]:
paths = os.listdir(head)
paths = [os.path.join(head, i) for i in paths]
pattern = path.lower()
paths = sorted(i for i in paths if os.path.isdir(i) and i.lower().startswith(pattern))
paths = sorted(
i for i in paths if os.path.isdir(i) and i.lower().startswith(pattern) and i not in ignored_directories)
return paths
@ -354,14 +364,12 @@ async def generate_bulk_posters(video_ids: List[int]):
with get_db_session(commit=True) as session:
videos = session.query(Video).filter(Video.id.in_(video_ids))
for video in videos:
video_path = get_absolute_video_path(video)
video_path = video.video_path.path
poster_path = replace_extension(video_path, '.jpg')
if not poster_path.exists():
generate_video_poster(video_path)
channel_dir = get_absolute_media_path(video.channel.directory)
poster_path = poster_path.relative_to(channel_dir)
video.poster_path = str(poster_path)
video.poster_path = poster_path
logger.info('Done generating video posters')
@ -400,10 +408,9 @@ def bulk_validate_posters(video_ids: List[int]):
for video_ids in chunks(video_ids, 10):
for video_id in video_ids:
with get_db_session(commit=True) as session:
video = session.query(Video).filter_by(id=video_id).one()
channel = video.channel
video: Video = session.query(Video).filter_by(id=video_id).one()
poster_path: Path = get_absolute_video_poster(video)
poster_path: Path = video.poster_path.path
new_poster_path = poster_path.with_suffix('.jpg')
if poster_path != new_poster_path and new_poster_path.exists():
@ -423,10 +430,8 @@ def bulk_validate_posters(video_ids: List[int]):
else:
logger.debug(f'Poster was already valid: {new_poster_path}')
channel_dir = get_absolute_media_path(channel.directory)
# Update the video with the new poster path. Mark it as validated.
video.poster_path = str(new_poster_path.relative_to(channel_dir))
video.poster_path = new_poster_path
video.validated_poster = True
logger.info('Done validating video posters.')
@ -465,6 +470,10 @@ async def get_bulk_video_info_json(video_ids: List[int]):
video = session.query(Video).filter_by(id=video_id).one()
logger.debug(f'Getting video info_json data: {video}')
if not video.video_path:
logger.warning(f'Refusing to get info_json for video without a video file: {video}')
continue
try:
info_json = video.get_info_json()
if info_json:
@ -475,8 +484,7 @@ async def get_bulk_video_info_json(video_ids: List[int]):
video.duration = info_json.get('duration')
elif not video.duration:
# As a last resort, get duration from the video file.
video_path = get_absolute_video_path(video)
video.duration = get_video_duration(video_path)
video.duration = get_video_duration(video.video_path.path)
except Exception:
logger.warning(f'Unable to get meta data of {video}', exc_info=True)
@ -493,16 +501,15 @@ async def get_bulk_video_size(video_ids: List[int]):
for video_id in video_ids:
video = session.query(Video).filter_by(id=video_id).one()
logger.debug(f'Getting video size: {video.id} {video.video_path}')
video_path = get_absolute_video_path(video)
size = video_path.stat().st_size
size = video.video_path.path.stat().st_size
video.size = size
logger.info('Done getting video sizes')
def update_view_count(channel_id: int):
"""
Update view_count for all Videos in a channel.
Update view_count for all Videos in a channel using it's info_json file.
"""
with get_db_session() as session:
channel = session.query(Channel).filter_by(id=channel_id).one()

View file

@ -0,0 +1,33 @@
import pytest
from modules.videos.channel.lib import spread_channel_downloads
from modules.videos.models import Channel
from wrolpi.downloader import DownloadFrequency
@pytest.fixture
def simple_channel(test_session, test_directory) -> Channel:
"""
Get a Channel with the minimum properties. This Channel has no download!
"""
channel = Channel(
directory=test_directory,
name='Example Channel 1',
url='https://example.com/channel1',
download_frequency=None, # noqa
link='examplechannel1',
)
test_session.add(channel)
test_session.commit()
return channel
@pytest.fixture
def download_channel(simple_channel) -> Channel:
"""
Get a test Channel that has a download frequency.
"""
# Add a frequency to the test channel, then give it a download.
simple_channel.download_frequency = DownloadFrequency.weekly
spread_channel_downloads()
return simple_channel

View file

@ -1,21 +1,27 @@
#! /usr/bin/env python3
import json
import pathlib
import re
from queue import Queue
from random import shuffle
from typing import Tuple, List, Dict
from abc import ABC
from typing import Tuple, List
from sqlalchemy import or_
from sqlalchemy.orm.exc import NoResultFound
from youtube_dl import YoutubeDL
from yt_dlp import YoutubeDL
from yt_dlp.extractor import YoutubeTabIE
from yt_dlp.utils import UnsupportedError, DownloadError
from wrolpi.common import logger, ProgressReporter, run_after, get_absolute_media_path
from wrolpi.dates import now, today
from wrolpi.common import logger, sanitize_link, extract_domain, \
get_media_directory
from wrolpi.dates import now
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.errors import UnknownChannel, ChannelURLEmpty
from wrolpi.downloader import Downloader, Download
from wrolpi.errors import UnknownChannel, ChannelURLEmpty, UnrecoverableDownloadError
from wrolpi.vars import PYTEST
from .channel.lib import create_channel, get_channel
from .common import load_downloader_config, update_view_count
from .lib import save_channels_config, upsert_video
from .lib import upsert_video, _refresh_videos
from .models import Video, Channel
from .video_url_resolver import video_url_resolver
logger = logger.getChild(__name__)
ydl_logger = logger.getChild('youtube-dl')
@ -24,17 +30,219 @@ YDL = YoutubeDL()
YDL.params['logger'] = ydl_logger
YDL.add_default_info_extractors()
# Channels are handled differently than a single video.
ChannelIEs = {
YoutubeTabIE,
}
def update_channel(channel=None, link: str = None):
PREFERRED_VIDEO_EXTENSION = 'mp4'
PREFERRED_VIDEO_FORMAT = 'best[height=720]'
def get_no_channel_directory():
return get_media_directory() / 'videos/NO CHANNEL'
class ChannelDownloader(Downloader, ABC):
"""
Handling downloading of all Videos in a Channel.
"""
@staticmethod
def valid_url(url) -> bool:
for ie in ChannelIEs:
if ie.suitable(url):
return True
return False
def do_download(self, download: Download):
"""
Update a Channel's catalog, then schedule downloads of every missing video.
"""
with get_db_session() as session:
channel = session.query(Channel).filter_by(url=download.url).one()
update_channel(channel)
domain = extract_domain(download.url)
missing_videos = find_all_missing_videos(channel.link)
with get_db_session(commit=True) as session:
for video_id, source_id, missing_video in missing_videos:
url = video_url_resolver(domain, missing_video)
# Schedule any missing videos for download.
self.manager.create_download(url, session, skip_download=True)
if PYTEST:
self.manager.do_downloads_sync()
else:
self.manager.start_downloads()
return True
class VideoDownloader(Downloader, ABC):
"""
Download a single video. Store the video in it's channel's directory, otherwise store it in `videos/NO CHANNEL`.
"""
@staticmethod
def valid_url(url) -> bool:
"""
Match against all Youtube-DL Info Extractors, except those that match a Channel.
"""
for ie in YDL._ies.values():
if ie.suitable(url) and not ChannelDownloader.valid_url(url):
try:
YDL.extract_info(url, download=False, process=False)
return True
except UnsupportedError:
logger.debug(f'Video downloader extract_info failed for {url}')
return False
except DownloadError:
logger.debug(f'Video downloader extract_info failed for {url}')
return False
return False
def do_download(self, download: Download):
if download.attempts >= 10:
raise UnrecoverableDownloadError('Max download attempts reached')
url = download.url
try:
info = YDL.extract_info(url, download=False, process=False)
except UnsupportedError as e:
# Video wasn't really valid... probably the GenericIE.
raise UnrecoverableDownloadError() from e
channel_name = info.get('channel')
channel_id = info.get('channel_id')
channel = None
if channel_name or channel_id:
channel_url = info.get('channel_url')
channel = get_or_create_channel(channel_id, None, channel_url, channel_name)
# Use the default directory if this video has no channel.
out_dir = get_no_channel_directory()
if channel:
out_dir = channel.directory.path
out_dir.mkdir(exist_ok=True)
try:
video_path, entry = self.prepare_filename(url, out_dir)
# Do the real download.
file_name_format = '%(uploader)s_%(upload_date)s_%(id)s_%(title)s.%(ext)s'
cmd = [
'yt-dlp',
'-cw', # Continue downloads, do not clobber existing files.
'-f', PREFERRED_VIDEO_FORMAT,
'--write-subs',
'--write-auto-subs',
'--write-thumbnail',
'--write-info-json',
'--merge-output-format', PREFERRED_VIDEO_EXTENSION,
'-o', file_name_format,
url,
]
return_code = self.process_runner(url, cmd, out_dir)
if return_code != 0:
raise ValueError(f'youtube-dl exited with {return_code}')
if not video_path.is_file():
raise ValueError(f'Video file could not be found! {video_path}')
except UnrecoverableDownloadError:
raise
except Exception as e:
logger.warning(f'Failed to download "{info["title"]}"', exc_info=e)
if _skip_download(e):
# The video failed to download, and the error will never be fixed. Skip it forever.
source_id = info.get('id')
logger.warning(f'Adding video "{source_id}" to skip list for this channel. WROLPi will not '
f'attempt to download it again.')
with get_db_session(commit=True) as session:
channel = session.query(Channel).filter_by(id=channel.id).one()
channel.add_video_to_skip_list(source_id)
return
with get_db_session(commit=True) as session:
# If the video is from a channel, it will already be in the database.
source_id = entry['id']
existing_video = session.query(Video).filter_by(source_id=source_id).one_or_none()
id_ = existing_video.id if existing_video else None
upsert_video(session, video_path, channel, id_=id_)
# Successful download.
return True
@staticmethod
def prepare_filename(url: str, out_dir: pathlib.Path) -> Tuple[pathlib.Path, dict]:
if not out_dir.is_dir():
raise ValueError(f'Output directory does not exist! {out_dir=}')
# YoutubeDL expects specific options, add onto the default options
config = load_downloader_config()
options = config.copy()
options['outtmpl'] = f'{out_dir}/{config["file_name_format"]}'
# We don't need format when getting the filename.
del options['format']
logger.debug(f'Downloading {url} to {out_dir}')
# Create a new YoutubeDL for the output directory.
ydl = YoutubeDL(options)
ydl.params['logger'] = ydl_logger
ydl.add_default_info_extractors()
# Perform the real download. This saves multiple files.
entry = ydl.extract_info(url, download=False, process=True)
final_filename = pathlib.Path(ydl.prepare_filename(entry))
return final_filename, entry
channel_downloader = ChannelDownloader()
# Videos may match the ChannelDownloader, give it a higher priority.
video_downloader = VideoDownloader(40)
def get_or_create_channel(source_id: str = None, link: str = None, url: str = None, name: str = None) -> Channel:
"""
Attempt to find a Channel using the provided params. The params are in order of reliability.
Creates a new Channel if one cannot be found.
"""
if not link and name:
link = sanitize_link(name)
try:
channel = get_channel(link=link, source_id=source_id, url=url, return_dict=False)
return channel
except UnknownChannel:
pass
channel_directory = get_media_directory() / f'videos/{name}'
data = dict(
name=name,
url=url,
link=link,
domain=extract_domain(url),
directory=str(channel_directory),
source_id=source_id,
)
channel = create_channel(data, return_dict=False)
# Create the directory now that the channel is approved.
channel_directory.mkdir(exist_ok=True)
return channel
def update_channel(channel: Channel):
"""
Connect to the Channel's host website and pull a catalog of all videos. Insert any new videos into the DB.
It is expected that any missing videos will be downloaded later.
"""
with get_db_session() as session:
if not channel:
channel = session.query(Channel).filter_by(link=link).one()
logger.info(f'Downloading video list for {channel.name} at {channel.url} This may take several minutes.')
info = YDL.extract_info(channel.url, download=False, process=False)
if 'url' in info:
@ -63,16 +271,19 @@ def update_channel(channel=None, link: str = None):
logger.warning(f'entries: {entries}')
raise KeyError('No id key for entry!') from e
# In order to store the Video's URL, we will need a quick lookup.
urls = {i['id']: i.get('webpage_url') for i in entries}
with get_db_session(commit=True) as session:
# Get the channel in this new context.
channel = session.query(Channel).filter_by(id=channel.id).one()
channel.info_json = info
channel.info_date = now()
channel.increment_next_download()
channel.source_id = info.get('id')
with get_db_curs() as curs:
# Insert any new videos.
# Get all known videos in this channel.
query = 'SELECT source_id FROM video WHERE channel_id=%s AND source_id IS NOT NULL'
curs.execute(query, (channel.id,))
known_source_ids = {i[0] for i in curs.fetchall()}
@ -82,59 +293,19 @@ def update_channel(channel=None, link: str = None):
logger.info(f'Got {len(new_source_ids)} new videos for channel {channel.name}')
channel_id = channel.id
for source_id in new_source_ids:
session.add(Video(source_id=source_id, channel_id=channel_id))
url = urls.get(source_id)
session.add(Video(source_id=source_id, channel_id=channel_id, url=url))
# Write the Channel's info to a JSON file.
if channel.directory:
info_json_path = channel.directory.path / f'{channel.name}.info.json'
with info_json_path.open('wt') as fh:
json.dump(info, fh, indent=2)
# Update all view counts using the latest from the Channel's info_json.
update_view_count(channel_id)
def update_channels(reporter: ProgressReporter, link: str = None):
"""Update all information for each channel. (No downloads performed)"""
with get_db_session() as session:
if session.query(Channel).count() == 0:
raise UnknownChannel('No channels exist yet')
if link:
try:
channel = session.query(Channel).filter_by(link=link).one()
except NoResultFound:
raise UnknownChannel(f'No channel with link: {link}')
channels = [channel, ]
else:
channels = session.query(Channel).filter(
Channel.url != None, # noqa
Channel.url != '',
or_(
Channel.next_download == None, # noqa
Channel.next_download <= today(),
)
).all()
if len(channels) == 0:
logger.warning(f'All channels are up to date')
reporter.set_progress_total(0, len(channels))
reporter.send_progress(0, 0, f'{len(channels)} channels scheduled for update')
# Randomize downloading of channels.
shuffle(channels)
logger.debug(f'Getting info for {len(channels)} channels')
for idx, channel in enumerate(channels):
reporter.send_progress(0, idx, f'Getting video list for {channel.name}')
try:
update_channel(channel)
except Exception:
logger.critical(f'Unable to get video list for {channel.name}', exc_info=True)
continue
if channels:
reporter.send_progress(0, len(channels), 'Done downloading video lists')
else:
reporter.finish(0, 'Done downloading video lists')
def _find_all_missing_videos(link: str = None) -> List[Tuple]:
"""
Get all Video entries which don't have the required media files (i.e. hasn't been downloaded). Restrict to a
@ -162,11 +333,12 @@ def _find_all_missing_videos(link: str = None) -> List[Tuple]:
WHERE
channel.url IS NOT NULL
AND channel.url != ''
AND source_id IS NOT NULL
AND video.source_id IS NOT NULL
{where}
AND channel_id IS NOT NULL
AND video.channel_id IS NOT NULL
AND (video_path IS NULL OR video_path = '' OR poster_path IS NULL OR poster_path = '')
'''
logger.debug(query)
curs.execute(query, params)
missing_videos = list(curs.fetchall())
return missing_videos
@ -193,6 +365,27 @@ def find_all_missing_videos(link: str = None) -> Tuple[dict, dict]:
# Get all channels while in the db context.
channels = list(channels)
# Check that each channel has some videos. We can't be sure what is missing if we don't know what we have.
for channel in channels:
refresh = False
with get_db_curs() as curs:
curs.execute('''
SELECT
COUNT(v.id)
FROM
channel c
LEFT OUTER JOIN video v on c.id = v.channel_id
WHERE
c.id = %s
''', (channel.id,))
video_count = curs.fetchall()[0]
if video_count == 0:
# No videos for this channel. We can't be sure if any videos are missing without a refresh.
refresh = True
if refresh:
# Refresh this channel's videos.
_refresh_videos([channel.link])
channels = {i.id: i for i in channels}
match_regexen = {i: re.compile(j.match_regex) for i, j in channels.items() if j.match_regex}
@ -221,31 +414,7 @@ def find_all_missing_videos(link: str = None) -> Tuple[dict, dict]:
match_regex: re.compile = match_regexen.get(channel_id)
if not match_regex or (match_regex and missing_video['title'] and match_regex.match(missing_video['title'])):
# No title match regex, or the title matches the regex.
yield channel, id_, missing_video
def download_video(channel: Channel, video: Dict) -> Tuple[pathlib.Path, Dict]:
"""
Download a video (and associated posters/etc) to it's channel's directory.
:param channel:
:param video: A YoutubeDL info entry dictionary
:return:
"""
# YoutubeDL expects specific options, add onto the default options
config = load_downloader_config()
options = dict(config)
directory = get_absolute_media_path(channel.directory)
options['outtmpl'] = f'{directory}/{config["file_name_format"]}'
ydl = YoutubeDL(options)
ydl.add_default_info_extractors()
source_id = video['id']
url = f'https://www.youtube.com/watch?v={source_id}'
entry = ydl.extract_info(url, download=True, process=True)
final_filename = ydl.prepare_filename(entry)
final_filename = pathlib.Path(final_filename)
return final_filename, entry
yield id_, source_id, missing_video
UNRECOVERABLE_ERRORS = {
@ -268,42 +437,6 @@ def _skip_download(error):
return False
@run_after(save_channels_config)
def download_all_missing_videos(reporter: ProgressReporter, link: str = None):
"""Find any videos identified by the info packet that haven't yet been downloaded, download them."""
missing_videos = list(find_all_missing_videos(link))
reporter.set_progress_total(1, len(missing_videos))
reporter.message(1, f'Found {len(missing_videos)} missing videos.')
for idx, (channel, id_, missing_video) in enumerate(missing_videos):
reporter.send_progress(1, idx, f'Downloading {channel.name}: {missing_video["title"]}')
try:
video_path, info_json = download_video(channel, missing_video)
except Exception as e:
logger.warning(f'Failed to download "{missing_video["title"]}"', exc_info=e)
if _skip_download(e):
# The video failed to download, and the error will never be fixed. Skip it forever.
source_id = missing_video.get('id')
logger.warning(f'Adding video "{source_id}" to skip list for this channel. WROLPi will not '
f'attempt to download it again.')
with get_db_session(commit=True) as session:
channel = session.query(Channel).filter_by(id=channel.id).one()
channel.add_video_to_skip_list(source_id)
reporter.error(1, f'Failed to download "{missing_video["title"]}", see server logs...')
continue
with get_db_session(commit=True) as session:
upsert_video(session, video_path, channel, id_=id_, info_json=info_json)
reporter.finish(1, 'All videos are downloaded')
def main(args=None):
"""Find and download any missing videos. Parse any arguments passed by the cmd-line."""
q = Queue()
reporter = ProgressReporter(q, 2)
update_channels(reporter)
download_all_missing_videos(reporter)
return 0
def get_channel_source_id(url: str) -> str:
channel_info = YDL.extract_info(url, download=False, process=False)
return channel_info['id']

View file

@ -2,15 +2,15 @@ import asyncio
import json
import pathlib
import re
from multiprocessing.queues import Queue
from typing import Tuple
from uuid import uuid1
from sqlalchemy.orm import Session
from wrolpi.common import ProgressReporter, save_settings_config, get_absolute_media_path
from wrolpi.common import logger
from wrolpi.common import logger, chunks
from wrolpi.common import save_settings_config
from wrolpi.db import get_db_curs, get_db_session
from wrolpi.vars import PYTEST
from .captions import insert_bulk_captions, process_captions
from .common import generate_bulk_posters, get_bulk_video_info_json, get_bulk_video_size, \
generate_video_paths, remove_duplicate_video_paths, bulk_validate_posters, \
@ -19,6 +19,8 @@ from .models import Channel, Video
logger = logger.getChild(__name__)
DEFAULT_DOWNLOAD_FREQUENCY = 60 * 60 * 24 * 7 # weekly
def refresh_channel_video_captions() -> bool:
with get_db_curs() as curs:
@ -86,6 +88,7 @@ def refresh_channel_info_json() -> bool:
WHERE
v.video_path IS NOT NULL
AND v.info_json_path IS NOT NULL
AND v.info_json_path != ''
AND (v.duration IS NULL OR v.view_count IS NULL)
'''
curs.execute(query)
@ -128,50 +131,38 @@ def process_video_meta_data():
refresh_channel_calculate_size()
def refresh_channel_videos(channel: Channel, reporter: ProgressReporter):
def refresh_channel_videos(channel: Channel):
"""
Find all video files in a channel's directory. Add any videos not in the DB to the DB.
"""
# This function is hard to predict, so we will simply progress in chunks :(
reporter.set_progress_total(1, 7)
reporter.send_progress(1, 0, 'Preparing channel.')
# Set the idempotency key so we can remove any videos not touched during this search
with get_db_curs(commit=True) as curs:
curs.execute('UPDATE video SET idempotency=NULL WHERE channel_id=%s', (channel.id,))
reporter.send_progress(1, 1, 'Finding all videos, checking for duplicates.')
idempotency = str(uuid1())
directory = get_absolute_media_path(channel.directory)
directory = channel.directory.path
# A set of absolute paths that exist in the file system
possible_new_paths = generate_video_paths(directory)
possible_new_paths = remove_duplicate_video_paths(possible_new_paths)
reporter.send_progress(1, 2, 'Matching all videos to the database.')
# Update all videos that match the current video paths
relative_new_paths = [str(i.relative_to(directory)) for i in possible_new_paths]
new_paths = [str(i) for i in possible_new_paths]
with get_db_curs(commit=True) as curs:
query = 'UPDATE video SET idempotency = %s WHERE channel_id = %s AND video_path = ANY(%s) RETURNING video_path'
curs.execute(query, (idempotency, channel.id, relative_new_paths))
curs.execute(query, (idempotency, channel.id, new_paths))
existing_paths = {i for (i,) in curs.fetchall()}
reporter.send_progress(1, 3)
# Get the paths for any video not yet in the DB
# (paths in DB are relative, but we need to pass an absolute path)
new_videos = {p for p in possible_new_paths if str(p.relative_to(directory)) not in existing_paths}
new_videos = {i for i in possible_new_paths if str(i) not in existing_paths}
reporter.send_progress(1, 4, f'Inserting {len(new_videos)} new videos.')
for video_path in new_videos:
for chunk in chunks(new_videos, 20):
with get_db_session(commit=True) as session:
upsert_video(session, pathlib.Path(video_path), channel, idempotency=idempotency)
logger.debug(f'{channel.name}: Added {video_path}')
reporter.send_progress(1, 5, 'Deleting unnecessary video entries.')
for video_path in chunk:
video_path = pathlib.Path(video_path)
upsert_video(session, video_path, channel, idempotency=idempotency)
logger.debug(f'{channel.name}: Added {video_path.name}')
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM video WHERE channel_id=%s AND idempotency IS NULL RETURNING id', (channel.id,))
@ -183,14 +174,50 @@ def refresh_channel_videos(channel: Channel, reporter: ProgressReporter):
logger.info(f'{channel.name}: {len(new_videos)} new videos, {len(existing_paths)} already existed. ')
reporter.send_progress(1, 6, f'Updating video view counts')
update_view_count(channel.id)
reporter.send_progress(1, 7, f'Processed all videos for {channel.name}')
def refresh_no_channel_videos():
"""
Refresh the Videos in the NO CHANNEL directory.
"""
from modules.videos.downloader import get_no_channel_directory
directory = get_no_channel_directory()
if not directory.is_dir():
return
logger.info('Refreshing NO CHANNEL videos')
idempotency = str(uuid1())
# Get all WROLPi compatible videos, remove any duplicates (different formats).
possible_new_paths = generate_video_paths(directory)
possible_new_paths = remove_duplicate_video_paths(possible_new_paths)
new_paths = [str(i) for i in possible_new_paths]
with get_db_curs(commit=True) as curs:
query = 'UPDATE video SET idempotency = %s WHERE video_path = ANY(%s) RETURNING video_path'
curs.execute(query, (idempotency, new_paths))
existing_paths = {i for (i,) in curs.fetchall()}
new_videos = {i for i in possible_new_paths if str(i) not in existing_paths}
for chunk in chunks(new_videos, 20):
with get_db_session(commit=True) as session:
for video_path in chunk:
upsert_video(session, pathlib.Path(video_path), idempotency=idempotency)
logger.debug(f'Added NO CHANNEL video {video_path}')
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM video WHERE channel_id IS NULL AND idempotency IS NULL RETURNING id')
deleted_count = len(curs.fetchall())
if deleted_count:
deleted_status = f'Deleted {deleted_count} video records in NO CHANNEL.'
logger.info(deleted_status)
def _refresh_videos(q: Queue, channel_links: list = None):
def _refresh_videos(channel_links: list = None):
"""
Find any videos in the channel directories and add them to the DB. Delete DB records of any videos not in the
file system.
@ -201,10 +228,6 @@ def _refresh_videos(q: Queue, channel_links: list = None):
"""
logger.info('Refreshing video files')
with get_db_session() as session:
reporter = ProgressReporter(q, 2)
reporter.code('refresh-started')
reporter.set_progress_total(0, session.query(Channel).count())
if channel_links:
channels = session.query(Channel).filter(Channel.link.in_(channel_links))
else:
@ -218,14 +241,15 @@ def _refresh_videos(q: Queue, channel_links: list = None):
raise Exception(f'No channels in DB. Have you created any?')
for idx, channel in enumerate(channels):
reporter.send_progress(0, idx, f'Checking {channel.name} directory for new videos')
refresh_channel_videos(channel, reporter)
refresh_channel_videos(channel)
if not channel_links:
# Refresh NO CHANNEL videos when not refreshing a specific channel.
refresh_no_channel_videos()
# Fill in any missing data for all videos.
process_video_meta_data()
reporter.send_progress(0, len(channels), 'All videos refreshed.')
reporter.code('refresh-complete')
if not PYTEST:
process_video_meta_data()
def get_channels_config(session: Session) -> dict:
@ -319,43 +343,20 @@ async def get_statistics():
return ret
async def get_download_history():
"""
Get the info_date for all channels that have been downloaded. Order by the most recent download first.
"""
query = '''
SELECT
link, name, info_date
FROM
channel
WHERE
info_date IS NOT NULL
ORDER BY channel.info_date DESC
'''
with get_db_curs() as curs:
curs.execute(query)
results = [dict(i) for i in curs.fetchall()]
return results
NAME_PARSER = re.compile(r'(.*?)_((?:\d+?)|(?:NA))_(?:(.{11})_)?(.*)\.'
r'(jpg|webp|flv|mp4|part|info\.json|description|webm|..\.srt|..\.vtt)')
def upsert_video(session: Session, video_path: pathlib.Path, channel: Channel, idempotency: str = None,
skip_captions=False,
id_: str = None, info_json: dict = None) -> Video:
def upsert_video(session: Session, video_path: pathlib.Path, channel: Channel = None, idempotency: str = None,
skip_captions=False, id_: str = None) -> Video:
"""
Insert a video into the DB. Also, find any meta-files near the video file and store them on the video row.
If id_ is provided, update that entry.
"""
channel_dir = get_absolute_media_path(channel.directory)
poster_path, description_path, caption_path, info_json_path = find_meta_files(video_path, relative_to=channel_dir)
# Video paths should be relative to the channel's directory
if video_path.is_absolute():
video_path = video_path.relative_to(channel_dir)
if not video_path.is_absolute():
raise ValueError(f'Video path is not absolute: {video_path}')
poster_path, description_path, caption_path, info_json_path = find_meta_files(video_path)
name_match = NAME_PARSER.match(video_path.name)
_ = upload_date = source_id = title = ext = None
@ -369,28 +370,33 @@ def upsert_video(session: Session, video_path: pathlib.Path, channel: Channel, i
upload_date = None
duration = None
url = None
if info_json_path:
path = (channel_dir / info_json_path).absolute()
try:
with open(path) as fh:
with info_json_path.open('rt') as fh:
json_contents = json.load(fh)
duration = json_contents['duration']
duration = json_contents.get('duration')
url = json_contents.get('webpage_url')
except json.decoder.JSONDecodeError:
logger.warning(f'Failed to load JSON file to get duration: {path}')
logger.warning(f'Failed to load JSON file to get duration: {info_json_path}')
size = video_path.stat().st_size
video_dict = dict(
channel_id=channel.id,
caption_path=str(caption_path) if caption_path else None,
channel_id=channel.id if channel else None,
description_path=str(description_path) if description_path else None,
duration=duration,
ext=ext,
idempotency=idempotency,
info_json_path=str(info_json_path) if info_json_path else None,
poster_path=str(poster_path) if poster_path else None,
size=size,
source_id=source_id,
title=title,
upload_date=upload_date,
url=url,
video_path=str(video_path),
caption_path=str(caption_path) if caption_path else None,
idempotency=idempotency,
info_json_path=str(info_json_path) if info_json_path else None,
duration=duration,
)
if id_:
@ -410,8 +416,7 @@ def upsert_video(session: Session, video_path: pathlib.Path, channel: Channel, i
return video
def find_meta_files(path: pathlib.Path, relative_to=None) -> Tuple[
pathlib.Path, pathlib.Path, pathlib.Path, pathlib.Path]:
def find_meta_files(path: pathlib.Path) -> Tuple[pathlib.Path, pathlib.Path, pathlib.Path, pathlib.Path]:
"""
Find all files that share a file's full path, except their extensions. It is assumed that file with the
same name, but different extension is related to that file. A None will be yielded if the meta file doesn't exist.
@ -429,11 +434,7 @@ def find_meta_files(path: pathlib.Path, relative_to=None) -> Tuple[
for meta_ext in meta_exts:
meta_path = path.with_suffix(meta_ext)
if meta_path.exists():
if relative_to:
yield meta_path.relative_to(relative_to)
break
else:
yield meta_path
break
yield meta_path
break
else:
yield None

View file

@ -1,27 +1,20 @@
import json
from datetime import timedelta, datetime
from datetime import datetime
from pathlib import Path
from typing import Optional
from typing import Optional, List
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, ARRAY, ForeignKey, Computed, types
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, ARRAY, ForeignKey, Computed
from sqlalchemy.orm import relationship, Session
from sqlalchemy.orm.collections import InstrumentedList
from wrolpi.common import Base, tsvector, ModelHelper, PathColumn, get_absolute_media_path
from wrolpi.dates import now, today, TZDateTime
from wrolpi.common import Base, tsvector, ModelHelper, logger
from wrolpi.dates import now, TZDateTime
from wrolpi.db import get_db_curs
from wrolpi.errors import UnknownVideo, NoFrequency, UnknownFile, UnknownDirectory
from wrolpi.downloader import Download
from wrolpi.errors import UnknownVideo, UnknownFile, UnknownDirectory
from wrolpi.media_path import MediaPathType, MediaPath
class ChannelPath(types.TypeDecorator):
impl = types.String
def process_bind_param(self, value, dialect):
return str(value) if value else None
def process_result_value(self, value, dialect):
if value:
return Path(value)
logger = logger.getChild(__name__)
class Video(ModelHelper, Base):
@ -32,12 +25,12 @@ class Video(ModelHelper, Base):
idempotency = Column(String)
# File paths
caption_path = Column(ChannelPath)
description_path = Column(ChannelPath)
caption_path = Column(MediaPathType)
description_path = Column(MediaPathType)
ext = Column(String)
info_json_path = Column(ChannelPath)
poster_path = Column(ChannelPath)
video_path = Column(ChannelPath)
info_json_path = Column(MediaPathType)
poster_path = Column(MediaPathType)
video_path = Column(MediaPathType)
caption = Column(String)
duration = Column(Integer)
@ -49,12 +42,13 @@ class Video(ModelHelper, Base):
validated_poster = Column(Boolean, default=False)
viewed = Column(TZDateTime)
view_count = Column(Integer)
url = Column(String)
textsearch = Column(tsvector, Computed('''to_tsvector('english'::regconfig,
((COALESCE(title, ''::text) || ' '::text) ||
COALESCE(caption, ''::text)))'''))
def __repr__(self):
return f'<Video id={self.id}, title={self.title}, path={self.video_path}, channel={self.channel_id} ' \
return f'<Video id={self.id}, title={self.title}, path={self.video_path.path}, channel={self.channel_id} ' \
f'source_id={self.source_id}>'
def dict(self) -> dict:
@ -83,13 +77,13 @@ class Video(ModelHelper, Base):
"""
Remove all files related to this video. Add it to it's Channel's skip list.
Raises: UnknownFile if a the video has no files.
Raises: UnknownFile if the video has no files.
"""
from modules.videos.common import get_absolute_video_files
video_files = get_absolute_video_files(self)
for path in video_files:
try:
path.unlink()
path.path.unlink()
except FileNotFoundError:
pass
@ -117,13 +111,9 @@ class Video(ModelHelper, Base):
"""
If this Video has an info_json file, return it's contents. Otherwise, return None.
"""
if not self.channel or not self.channel.directory:
return
try:
if self.info_json_path:
path = self.channel.get_relative_path(self.info_json_path)
with open(path, 'rb') as fh:
with open(self.info_json_path.path, 'rb') as fh:
contents = json.load(fh)
return contents
except UnknownFile:
@ -142,12 +132,10 @@ class Video(ModelHelper, Base):
if description:
return description
if self.description_path and self.channel_id:
path = self.channel.get_relative_path(self.description_path)
if path.exists():
with open(path, 'rt') as fh:
contents = fh.read()
return contents
if self.description_path:
with open(self.description_path, 'rt') as fh:
contents = fh.read()
return contents
def get_surrounding_videos(self):
"""
@ -215,6 +203,28 @@ class Video(ModelHelper, Base):
return previous_video, next_video
def __json__(self):
from modules.videos.common import minimize_video_info_json
d = dict(
caption_path=self.caption_path,
channel=self.channel,
channel_id=self.channel_id,
duration=self.duration,
favorite=self.favorite,
id=self.id,
info_json=minimize_video_info_json(self.get_info_json()) if self.info_json_path else None,
poster_path=self.poster_path,
size=self.size,
source_id=self.source_id,
title=self.title,
upload_date=self.upload_date,
url=self.url,
video_path=self.video_path,
view_count=self.view_count,
viewed=self.viewed,
)
return d
class Channel(ModelHelper, Base):
__tablename__ = 'channel'
@ -224,12 +234,12 @@ class Channel(ModelHelper, Base):
idempotency = Column(String)
url = Column(String)
match_regex = Column(String)
directory = Column(PathColumn)
directory: MediaPath = Column(MediaPathType)
skip_download_videos = Column(ARRAY(String))
generate_posters = Column(Boolean)
calculate_duration = Column(Boolean)
download_frequency = Column(Integer)
next_download = Column(Date)
source_id = Column(String)
info_json = Column(JSON)
info_date = Column(Date)
@ -240,9 +250,9 @@ class Channel(ModelHelper, Base):
videos: InstrumentedList = relationship('Video', primaryjoin='Channel.id==Video.channel_id')
def __repr__(self):
return f'<Channel(id={self.id}, name={self.name})>'
return f'<Channel id={self.id}, name={repr(self.name)}>'
def add_video_to_skip_list(self, source_id):
def add_video_to_skip_list(self, source_id: str):
if not source_id:
raise UnknownVideo(f'Cannot skip video with empty source id: {source_id}')
@ -250,36 +260,6 @@ class Channel(ModelHelper, Base):
skip_download_videos.add(source_id)
self.skip_download_videos = skip_download_videos
def increment_next_download(self):
"""
Set the next download predictably during the next download iteration.
For example, two channels that download weekly will need to be downloaded on different days. We want a channel
to always be downloaded on it's day. That may be Monday, or Tuesday, etc.
This is true for all download frequencies (30 days, 90 days, etc.).
The order that channels will be downloaded/distributed will be by `link`.
"""
if not self.download_frequency:
raise NoFrequency(f'{self} does not have a frequency!')
session = Session.object_session(self)
# All the channels that share the my frequency.
channel_group = session.query(self.__class__).filter_by(download_frequency=self.download_frequency)
channel_group = list(channel_group.order_by(self.__class__.link).all())
# My position in the channel group.
index = channel_group.index(self)
# The seconds between each download.
chunk = self.download_frequency // len(channel_group)
# My next download will be distributed by my frequency and my position.
position = chunk * (index + 1)
self.next_download = today() + timedelta(seconds=self.download_frequency + position)
def delete_with_videos(self):
"""
Delete all Video records (but not video files) related to this Channel. Then delete the Channel.
@ -287,6 +267,9 @@ class Channel(ModelHelper, Base):
session = Session.object_session(self)
# Delete the video records, but not the video files!
session.query(Video).filter_by(channel_id=self.id).delete()
if download := self.get_download():
session.delete(download)
session.delete(self)
def update(self, data: dict):
@ -298,35 +281,49 @@ class Channel(ModelHelper, Base):
Retrieve the data about this Channel that should be stored in a config file.
"""
config = dict(
directory=str(self.directory),
calculate_duration=self.calculate_duration,
directory=str(self.directory.path),
download_frequency=self.download_frequency,
favorites={},
generate_posters=self.generate_posters,
match_regex=self.match_regex or '',
name=self.name,
url=self.url or '',
generate_posters=self.generate_posters,
calculate_duration=self.calculate_duration,
skip_download_videos=self.skip_download_videos or [],
download_frequency=self.download_frequency,
next_download=self.next_download,
favorites={},
source_id=self.source_id,
url=self.url or '',
)
session = Session.object_session(self)
favorites = session.query(Video).filter(Video.favorite != None, Video.channel_id == self.id).all()
favorites = session.query(Video).filter(Video.favorite != None, Video.channel_id == self.id).all() # noqa
favorites: List[Video]
if favorites:
config['favorites'] = {i.source_id: {'favorite': i.favorite} for i in favorites}
config['favorites'] = {i.video_path.path.name: {'favorite': i.favorite} for i in favorites}
return config
def get_directory(self):
if self._directory:
return self._directory
self._directory = get_absolute_media_path(self.directory)
return self._directory
def get_relative_path(self, path: Path, exists: bool = True):
channel_dir = self._directory or self.get_directory()
path = channel_dir / path
path = self.directory / path
if exists and not path.exists():
raise FileNotFoundError(f'Channel path {path} does not exist!')
raise FileNotFoundError(f'{path} does not exist!')
return path
def get_download(self) -> Optional[Download]:
"""
Get the Download row for this Channel. If there isn't a Download, return None.
"""
if not self.url:
raise ValueError(f'Channel {self.name} does not have a URL to download!')
session = Session.object_session(self)
download = session.query(Download).filter_by(url=self.url).one_or_none()
return download
def __json__(self):
d = dict(
id=self.id,
name=self.name,
directory=self.directory,
url=self.url,
link=self.link,
)
return d

View file

@ -11,7 +11,7 @@ class ChannelPostRequest:
mkdir = doc.Boolean()
generate_posters = doc.Boolean()
calculate_duration = doc.Boolean()
download_frequency = doc.Integer()
download_frequency = doc.String()
class ChannelPutRequest:
@ -23,7 +23,7 @@ class ChannelPutRequest:
generate_posters = doc.Boolean()
calculate_duration = doc.Boolean()
mkdir = doc.Boolean()
download_frequency = doc.Integer()
download_frequency = doc.String()
class RefreshRequest:
@ -170,13 +170,3 @@ class ChannelStatistics:
class VideosStatisticsResponse:
videos = VideoStatistics
channels = ChannelStatistics
class DownloadHistory:
link = doc.String()
name = doc.String()
info_date = doc.Date()
class DownloadHistoryResponse:
history = doc.List(doc.Object(DownloadHistory))

View file

@ -1,9 +1,10 @@
import pathlib
import tempfile
from functools import wraps
from queue import Queue
from modules.videos.lib import refresh_channel_videos
from modules.videos.models import Channel
from wrolpi.common import ProgressReporter, insert_parameter
from wrolpi.common import insert_parameter, set_test_media_directory
from wrolpi.db import get_db_session
from wrolpi.test.common import wrap_test_db, build_test_directories
@ -37,9 +38,11 @@ def create_channel_structure(structure):
@wraps(func)
@wrap_test_db
def wrapped(*args, **kwargs):
# Dummy queue and reporter to receive messages.
q = Queue()
reporter = ProgressReporter(q, 2)
if args and (test := args[0]) and hasattr(test, 'tmp_dir'):
tmp_dir = pathlib.Path(test.tmp_dir.name)
else:
tmp_dir = pathlib.Path(tempfile.TemporaryDirectory().name)
set_test_media_directory(tmp_dir)
# Convert the channel/video structure to a file structure for the test.
file_structure = []
@ -48,7 +51,7 @@ def create_channel_structure(structure):
file_structure.append(f'{channel}/{path}')
file_structure.append(f'{channel}/')
with build_test_directories(file_structure) as tempdir:
with build_test_directories(file_structure, tmp_dir) as tempdir:
args, kwargs = insert_parameter(func, 'tempdir', tempdir, args, kwargs)
with get_db_session(commit=True) as session:
@ -60,9 +63,12 @@ def create_channel_structure(structure):
with get_db_session(commit=True) as session:
for channel_ in session.query(Channel).all():
refresh_channel_videos(channel_, reporter)
refresh_channel_videos(channel_)
return func(*args, **kwargs)
try:
return func(*args, **kwargs)
finally:
set_test_media_directory(None)
return wrapped

View file

@ -1,15 +1,22 @@
import mock
from pathlib import Path
from modules.videos import captions
from modules.videos.models import Video
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.db import get_db_curs, get_db_context
from wrolpi.test.common import wrap_test_db, TestAPI
from wrolpi.vars import PROJECT_DIR
class TestCaption(TestAPI):
vtt_path1 = (PROJECT_DIR / 'test/example1.en.vtt').absolute()
vtt_path2 = (PROJECT_DIR / 'test/example2.en.vtt').absolute()
def setUp(self) -> None:
super().setUp()
self.vtt_path1 = Path(self.tmp_dir.name) / 'example1.en.vtt'
self.vtt_path1.touch()
self.vtt_path1.write_text((PROJECT_DIR / 'test/example1.en.vtt').read_text())
self.vtt_path2 = Path(self.tmp_dir.name) / 'example2.en.vtt'
self.vtt_path2.touch()
self.vtt_path2.write_text((PROJECT_DIR / 'test/example2.en.vtt').read_text())
def test_example_en_vtt(self):
expected = ['okay welcome to this session this is',
@ -35,25 +42,26 @@ class TestCaption(TestAPI):
@wrap_test_db
def test_process_captions(self):
with get_db_session(commit=True) as session:
video1 = Video(title='scream', caption_path=str(self.vtt_path1))
session.add(video1)
with mock.patch('modules.videos.captions.get_absolute_video_caption', lambda *a: self.vtt_path1):
captions.process_captions(video1)
video2 = Video(title='bar', caption_path=str(self.vtt_path2))
session.add(video2)
with mock.patch('modules.videos.captions.get_absolute_video_caption', lambda *a: self.vtt_path2):
captions.process_captions(video2)
_, session = get_db_context()
video1 = Video(title='scream', caption_path=str(self.vtt_path1))
session.add(video1)
session.commit()
captions.process_captions(video1)
video2 = Video(title='bar', caption_path=str(self.vtt_path2))
session.add(video2)
session.commit()
captions.process_captions(video2)
session.flush()
session.refresh(video1)
session.refresh(video2)
session.commit()
session.flush()
session.refresh(video1)
session.refresh(video2)
# Get the video from the DB
video1 = session.query(Video).filter_by(id=video1.id).one()
self.assertIsNotNone(video1.caption)
video2 = session.query(Video).filter_by(id=video2.id).one()
self.assertIsNotNone(video2.caption)
# Get the video from the DB
video1 = session.query(Video).filter_by(id=video1.id).one()
self.assertIsNotNone(video1.caption)
video2 = session.query(Video).filter_by(id=video2.id).one()
self.assertIsNotNone(video2.caption)
# Search using the tsvector, "sessions" never actually appears in the text, but "session" does
with get_db_curs() as curs:

View file

@ -2,7 +2,7 @@ import pathlib
import subprocess
import tempfile
from unittest import mock
from unittest.mock import Mock
from unittest.mock import Mock, MagicMock
import pytest
from PIL import Image
@ -11,17 +11,17 @@ from modules.videos.models import Channel, Video
from modules.videos.test.common import create_channel_structure
from wrolpi.common import get_absolute_media_path
from wrolpi.db import get_db_session
from wrolpi.test.common import ExtendedTestCase, build_test_directories, wrap_test_db
from wrolpi.test.common import build_test_directories, wrap_test_db, TestAPI
from wrolpi.vars import PROJECT_DIR
from ..common import get_matching_directories, convert_image, bulk_validate_posters, remove_duplicate_video_paths, \
update_view_count, get_video_duration
class TestCommon(ExtendedTestCase):
class TestCommon(TestAPI):
def test_get_absolute_media_path(self):
wrolpi = get_absolute_media_path('videos/wrolpi')
assert str(wrolpi).endswith('wrolpi')
path = get_absolute_media_path('videos')
self.assertTrue(str(path).endswith('videos'))
def test_matching_directories(self):
structure = [
@ -59,13 +59,13 @@ class TestCommon(ExtendedTestCase):
"""
Video duration can be retrieved from the video file.
"""
video_path = PROJECT_DIR / 'test/videos/wrolpi/big_buck_bunny_720p_1mb.mp4'
video_path = PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4'
self.assertEqual(get_video_duration(video_path), 5)
video_path = PROJECT_DIR / 'test/videos/wrolpi/does not exist.mp4'
video_path = PROJECT_DIR / 'test/does not exist.mp4'
self.assertRaises(FileNotFoundError, get_video_duration, video_path)
video_path = str(PROJECT_DIR / 'test/videos/wrolpi/does not exist.mp4')
video_path = str(PROJECT_DIR / 'test/does not exist.mp4')
self.assertRaises(FileNotFoundError, get_video_duration, video_path)
with tempfile.NamedTemporaryFile() as fh:
@ -139,7 +139,7 @@ def test_convert_image():
)
def test_create_db_structure(_structure, paths):
@create_channel_structure(_structure)
def test_func(tempdir):
def test_func(_, tempdir):
assert isinstance(tempdir, pathlib.Path)
for path in paths:
path = (tempdir / path)
@ -154,7 +154,12 @@ def test_create_db_structure(_structure, paths):
assert channel.directory == tempdir / channel_name
assert len(channel.videos) == len([i for i in _structure[channel_name] if i.endswith('mp4')])
test_func()
with tempfile.TemporaryDirectory() as tmp_dir, \
mock.patch('wrolpi.media_path.get_media_directory') as mock_get_directory:
mock_get_directory.return_value = pathlib.Path(tmp_dir)
test_ = MagicMock()
test_.tmp_dir.name = tmp_dir
test_func(test_)
@wrap_test_db
@ -184,10 +189,10 @@ def test_bulk_replace_invalid_posters(tempdir: pathlib.Path):
assert Image.open(webp_fh).format == 'WEBP'
with get_db_session() as session:
vid1 = session.query(Video).filter_by(poster_path='vid1.jpg').one()
vid1 = session.query(Video).filter_by(poster_path=f'{tempdir}/channel1/vid1.jpg').one()
assert vid1.validated_poster is False
vid2 = session.query(Video).filter_by(poster_path='vid2.webp').one()
vid2 = session.query(Video).filter_by(poster_path=f'{tempdir}/channel2/vid2.webp').one()
assert vid2.validated_poster is False
# Convert the WEBP image. convert_image() should only be called once.
@ -201,13 +206,13 @@ def test_bulk_replace_invalid_posters(tempdir: pathlib.Path):
with get_db_session() as session:
# Get the video by ID because it's poster is now a JPEG.
vid2 = session.query(Video).filter_by(id=vid2.id).one()
assert str(vid2.poster_path) == 'vid2.jpg'
assert all('webp' not in str(i.poster_path) for i in session.query(Video).all())
assert str(vid2.poster_path.path).split('/')[-1] == 'vid2.jpg'
assert all('webp' not in str(i.poster_path.path) for i in session.query(Video).all())
assert vid2.validated_poster is True
# Vid1's image was validated, but not converted.
vid1 = session.query(Video).filter_by(id=vid1.id).one()
assert str(vid1.poster_path) == 'vid1.jpg'
assert str(vid1.poster_path.path).split('/')[-1] == 'vid1.jpg'
assert vid1.validated_poster is True
# Old webp was removed
@ -240,9 +245,9 @@ def test_bulk_replace_invalid_posters(tempdir: pathlib.Path):
)
def test_update_view_count(tempdir: pathlib.Path):
def check_view_counts(view_counts):
with get_db_session(commit=True) as session:
with get_db_session() as session_:
for source_id, view_count in view_counts.items():
vid = session.query(Video).filter_by(source_id=source_id).one()
vid = session_.query(Video).filter_by(source_id=source_id).one()
assert vid.view_count == view_count
with get_db_session(commit=True) as session:
@ -251,9 +256,9 @@ def test_update_view_count(tempdir: pathlib.Path):
channel2.info_json = {'entries': [{'id': 'vid2.mp4', 'view_count': 11}, {'id': 'bad_id', 'view_count': 12}]}
channel3.info_json = {'entries': [{'id': 'vid3.mp4', 'view_count': 13}, {'id': 'vid4.mp4', 'view_count': 14}]}
# Use the video_path as a unique source_id
# Use the video file name as a unique source_id
for v in session.query(Video).all():
v.source_id = str(v.video_path)
v.source_id = str(v.video_path.path).split('/')[-1]
# Check all videos are empty.
check_view_counts({'vid1.mp4': None, 'vid2.mp4': None, 'vid3.mp4': None, 'vid4.mp4': None})

View file

@ -1,17 +1,18 @@
import time
from datetime import timedelta
from queue import Queue
import pathlib
from copy import copy
from datetime import datetime
from unittest import mock
from unittest.mock import MagicMock
import pytest
from yt_dlp.utils import UnsupportedError
from modules.videos.downloader import update_channels, find_all_missing_videos, download_video
from modules.videos.channel.lib import spread_channel_downloads
from modules.videos.downloader import find_all_missing_videos, VideoDownloader, \
ChannelDownloader, get_or_create_channel
from modules.videos.models import Channel, Video
from modules.videos.test.common import create_channel_structure
from wrolpi.common import ProgressReporter
from wrolpi.dates import today
from wrolpi.db import get_db_session
from wrolpi.dates import local_timezone
from wrolpi.db import get_db_session, get_db_context
from wrolpi.downloader import DownloadManager, Download, DownloadFrequency
from wrolpi.test.common import wrap_test_db, TestAPI
@ -26,67 +27,13 @@ class FakeYDL:
}
@wrap_test_db
@create_channel_structure(
{
'channel1': ['vid1.mp4'],
'channel2': ['vid2.mp4']
},
)
def test_update_channels(tempdir):
"""
Channels are only updated if their "next_download" has expired, or if they have never been updated.
"""
q = Queue()
reporter = ProgressReporter(q, 2)
with get_db_session(commit=True) as session:
channel1, channel2 = session.query(Channel).order_by(Channel.id).all()
channel1.url = channel2.url = 'some url'
with get_db_session() as session:
channel1, channel2 = session.query(Channel).order_by(Channel.id).all()
with mock.patch('modules.videos.downloader.update_channel') as update_channel:
update_channel: MagicMock
update_channels(reporter)
# Both channels would have been updated.
assert update_channel.call_count == 2
update_channel.assert_any_call(channel1)
update_channel.assert_any_call(channel2)
with get_db_session() as session:
channel1, channel2 = session.query(Channel).order_by(Channel.id).all()
channel1.next_download = today() + timedelta(days=1)
session.commit()
with mock.patch('modules.videos.downloader.update_channel') as update_channel:
update_channel: MagicMock
update_channels(reporter)
# Channel1 is not ready for an update
update_channel.assert_called_once()
update_channel.assert_called_with(channel2)
channel1.next_download = today() + timedelta(days=1)
channel2.next_download = today() + timedelta(days=1)
session.commit()
with mock.patch('modules.videos.downloader.update_channel') as update_channel:
update_channel: MagicMock
update_channels(reporter)
# No channels needed to be updated
assert update_channel.call_count == 0
class TestDownloader(TestAPI):
@wrap_test_db
@create_channel_structure(
{
'channel1': ['vid1.mp4'],
'channel2': ['vid2.mp4']
},
}
)
def test_find_all_missing_videos(self, tempdir):
with get_db_session(commit=True) as session:
@ -106,103 +53,310 @@ class TestDownloader(TestAPI):
self.assertEqual(len(missing), 1)
# The video returned is the one we faked.
channel, id_, entry = missing[0]
self.assertEqual(channel, channel1)
id_, source_id, entry = missing[0]
# Two videos were created for this test already.
self.assertEqual(id_, 3)
# The fake entry we added is regurgitated back.
self.assertEqual(entry, channel1.info_json['entries'][0])
@pytest.mark.skip
def test_download_timeout(self):
class FakeYoutubeDL:
def __init__(self, *a, **kw):
pass
def add_default_info_extractors(self):
pass
example_video_json = {
'age_limit': 0,
'average_rating': 4.6923075,
'categories': ['Howto & Style'],
'channel': 'channel name',
'channel_id': 'channel id',
'channel_url': 'channel url',
'duration': 635,
'extractor': 'youtube',
'extractor_key': 'Youtube',
'id': 'some long id',
'is_live': None,
'like_count': 24,
'subtitles': {},
'title': "The video's title",
'upload_date': '20190707',
'uploader': 'uploader name',
'uploader_id': 'uploader id',
'uploader_url': 'uploader url',
'view_count': 406,
'webpage_url': 'webpage url',
'webpage_url_basename': 'watch',
}
def extract_info(self, *a, **kw):
# Sleep longer than the 1 second timeout.
for i in range(10):
time.sleep(1)
example_channel_json = {
'_type': 'playlist',
'entries': [
{'_type': 'url',
'description': None,
'duration': None,
'id': 'video 1 id',
'ie_key': 'Youtube',
'title': 'video 1 title',
'uploader': None,
'url': 'video 1 url',
'view_count': 58504},
{'_type': 'url',
'description': None,
'duration': None,
'id': 'video 2 id',
'ie_key': 'Youtube',
'title': 'video 2 title',
'uploader': None,
'url': 'video 2 url',
'view_count': 1413},
],
'extractor': 'youtube:tab',
'extractor_key': 'YoutubeTab',
'id': 'some id',
'title': 'channel title',
'webpage_url': 'channel url',
'webpage_url_basename': 'videos',
}
with mock.patch('api.timeout.TEST_TIMEOUT', 1), \
mock.patch('modules.videos.downloader.YoutubeDL', FakeYoutubeDL), \
mock.patch('modules.videos.downloader.get_absolute_media_path'):
channel = Channel()
self.assertRaises(TimeoutError, download_video, channel, {'id': 1})
# {key: 'daily', text: 'Daily', value: 86400},
# {key: 'weekly', text: 'Weekly', value: 604800},
# {key: 'biweekly', text: 'Biweekly', value: 1209600},
# {key: '30days', text: '30 Days', value: 2592000},
# {key: '90days', text: '90 Days', value: 7776000},
class TestVideosDownloaders(TestAPI):
def setUp(self) -> None:
super().setUp()
self.vd = VideoDownloader()
self.cd = ChannelDownloader()
self.mgr = DownloadManager()
self.mgr.register_downloader(self.vd)
self.mgr.register_downloader(self.cd)
self.videos_dir = pathlib.Path(self.tmp_dir.name) / 'videos'
self.videos_dir.mkdir()
def test_video_valid_url(self):
# A specific video can be downloaded.
self.assertTrue(self.vd.valid_url('https://www.youtube.com/watch?v=31jPEBiAC3c'))
# A channel cannot be downloaded.
self.assertFalse(self.vd.valid_url('https://www.youtube.com/c/LearningSelfReliance/videos'))
self.assertFalse(self.vd.valid_url('https://www.youtube.com/c/LearningSelfReliance'))
def test_channel_valid_url(self):
# An entire domain cannot be downloaded.
self.assertFalse(self.cd.valid_url('https://example.com'))
self.assertFalse(self.cd.valid_url('https://youtube.com'))
# Cannot download a single video.
self.assertFalse(self.cd.valid_url('https://www.youtube.com/watch?v=31jPEBiAC3c'))
# Can download entire channels.
self.assertTrue(self.cd.valid_url('https://www.youtube.com/c/LearningSelfReliance/videos'))
self.assertTrue(self.cd.valid_url('https://www.youtube.com/c/LearningSelfReliance'))
# Can download entire playlists.
self.assertTrue(self.cd.valid_url('https://www.youtube.com/playlist?list=PLCdlMQeP-TbG12nkBCt0E96yr3EfwcH4E'))
def test_get_downloader(self):
# The correct Downloader is gotten.
self.assertEqual(self.mgr.get_downloader('https://www.youtube.com/c/LearningSelfReliance/videos'), self.cd)
self.assertEqual(self.mgr.get_downloader('https://www.youtube.com/c/LearningSelfReliance'), self.cd)
self.assertEqual(self.mgr.get_downloader('https://www.youtube.com/watch?v=31jPEBiAC3c'), self.vd)
@staticmethod
def _make_video_files(channel_dir):
video_path = channel_dir / 'video.mp4'
poster_path = channel_dir / 'video.jpg'
description_path = channel_dir / 'video.description'
info_json_path = channel_dir / 'video.info.json'
video_path.touch()
poster_path.touch()
description_path.touch()
info_json_path.touch()
return video_path, poster_path, description_path, info_json_path
@wrap_test_db
@create_channel_structure(
{
'channel1': [], # More channels than days in a week.
'channel2': [],
'channel3': [],
'channel4': [],
'channel5': [],
'channel6': [],
'channel7': [],
'channel8': [],
'channel9': [],
},
)
@mock.patch('modules.videos.downloader.YDL', FakeYDL())
def test_update_channel_order(self, tempdir):
q = Queue()
reporter = ProgressReporter(q, 2)
def test_video_download(self):
channel_dir = self.videos_dir / 'channel name'
channel_dir.mkdir(parents=True)
# All channels are downloaded weekly.
# Video paths
video_path, poster_path, description_path, info_json_path = self._make_video_files(channel_dir)
with get_db_session() as session:
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
mock.patch('modules.videos.downloader.VideoDownloader.prepare_filename') as mock_prepare_filename, \
mock.patch('modules.videos.downloader.VideoDownloader.process_runner') as mock_process_runner:
mock_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
mock_process_runner.return_value = 0
self.mgr.create_download(url, session)
mock_process_runner.assert_called_once()
video_url, _, out_dir = mock_process_runner.call_args[0]
download: Download = session.query(Download).one()
self.assertEqual(video_url, download.url)
channel: Channel = session.query(Channel).one()
self.assertEqual(channel.directory, channel_dir)
video: Video = session.query(Video).one()
self.assertEqual(video.video_path, video_path)
self.assertEqual(video.description_path, description_path)
self.assertEqual(video.poster_path, poster_path)
self.assertEqual(video.info_json_path, info_json_path)
self.assertTrue(video.video_path.path.is_absolute())
self.assertTrue(video.poster_path.path.is_absolute())
self.assertTrue(video.description_path.path.is_absolute())
self.assertTrue(video.info_json_path.path.is_absolute())
@wrap_test_db
def test_video_download_no_channel(self):
_, session = get_db_context()
channel_dir = self.videos_dir / 'NO CHANNEL'
channel_dir.mkdir(parents=True)
# Video paths
video_path, _, _, _ = self._make_video_files(channel_dir)
# Video has no channel
info_json = copy(example_video_json)
del info_json['channel']
del info_json['channel_id']
del info_json['channel_url']
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
mock.patch('modules.videos.downloader.VideoDownloader.prepare_filename') as mock_prepare_filename, \
mock.patch('modules.videos.downloader.VideoDownloader.process_runner') as mock_process_runner:
mock_extract_info.return_value = info_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
mock_process_runner.return_value = 0
self.mgr.create_download(url, session)
mock_process_runner.assert_called_once()
video: Video = session.query(Video).one()
self.assertEqual(str(video.video_path.path), f'{channel_dir}/video.mp4')
@wrap_test_db
def test_channel_download(self):
_, session = get_db_context()
url = 'https://www.youtube.com/c/LearningSelfReliance/videos'
session.add(Channel(url=url, link='foo'))
channel_dir = self.videos_dir / 'NO CHANNEL'
channel_dir.mkdir(parents=True)
video_path, _, _, _ = self._make_video_files(channel_dir)
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
mock.patch('modules.videos.downloader.VideoDownloader.prepare_filename') as mock_prepare_filename, \
mock.patch('modules.videos.downloader.VideoDownloader.process_runner') as mock_process_runner:
mock_extract_info.return_value = example_channel_json
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
mock_process_runner.return_value = 0
self.mgr.create_download(url, session)
# Two videos are in the example channel.
downloads = self.mgr.get_new_downloads(session)
expected = ['https://youtube.com/watch?v=HQ_62YwcA80', 'https://youtube.com/watch?v=4gSz6W4Gv-o']
for download, expected in zip(downloads, expected):
self.assertEqual(download.url, expected)
# Download is skipped for test.
self.assertEqual(download.status, 'new')
@wrap_test_db
def test_invalid_download_url(self):
_, session = get_db_context()
with mock.patch('modules.videos.downloader.VideoDownloader.valid_url') as mock_valid_url, \
mock.patch('modules.videos.downloader.YDL') as mock_ydl:
mock_valid_url.return_value = True
mock_ydl.extract_info.side_effect = UnsupportedError('oops')
self.mgr.create_download('url', session)
@wrap_test_db
@mock.patch('modules.videos.channel.lib.today', lambda: local_timezone(datetime(2020, 1, 1, 0, 0, 0)))
def test_spread_channel_downloads(self):
with get_db_session(commit=True) as session:
channels = list(session.query(Channel).order_by(Channel.link).all())
for channel in channels:
channel.download_frequency = 604800 # one week
channel.url = 'some url'
self.assertEqual(channel.next_download, None)
channel1 = Channel(name='channel1', link='channel1', url='https://example.com/1',
download_frequency=DownloadFrequency.weekly)
channel2 = Channel(name='channel2', link='channel2', url='https://example.com/2',
download_frequency=DownloadFrequency.weekly)
session.add_all([channel1, channel2])
# Channels will download on different days the next week. Except where there are not enough days.
update_channels(reporter)
days = [7, 8, 9, 10, 10, 11, 12, 13, 14]
for days_, channel in zip(days, channels):
nd, expected = channel.next_download, today() + timedelta(days=days_)
self.assertEqual(nd, expected, f'Expected {(nd - expected).days} days for {channel}')
def check_frequencies(expected):
with get_db_session(commit=True) as session:
downloads = list(session.query(Download).all())
self.assertLength(downloads, expected)
for download in downloads:
next_download = expected[download.url]
self.assertEqual(next_download, str(download.next_download),
f'{download.url} frequency {next_download} != {download.next_download}')
# Even though all channels aren't updated, their order is the same.
with get_db_session(commit=True):
channels[0].next_download = None
channels[1].next_download = None
channels[4].next_download = None
channels[8].next_download = None
update_channels(reporter)
for days_, channel in zip(days, channels):
self.assertEqual(channel.next_download, today() + timedelta(days=days_))
# Spread the downloads out.
spread_channel_downloads()
expected = {
'https://example.com/1': '2020-01-08 00:00:00-07:00',
'https://example.com/2': '2020-01-11 12:00:00-07:00',
}
check_frequencies(expected)
# Multiple frequencies are supported.
with get_db_session(commit=True) as session:
frequency_map = {
'channel1': 604800, # weekly
'channel2': 604800,
'channel3': 1209600, # bi-weekly
'channel4': 1209600,
'channel5': 2592000, # 30 days
'channel6': 2592000,
'channel7': 2592000,
'channel8': 2592000,
'channel9': 2592000,
}
for channel in channels:
channel.download_frequency = frequency_map[channel.link]
channel.next_download = None
update_channels(reporter)
days = [10, 14, 21, 28,
36, 42, 48, 54, 60 # 6 days apart = 30 days / 5
]
for days_, channel in zip(days, channels):
nd, expected = channel.next_download, today() + timedelta(days=days_)
self.assertEqual(nd, expected, f'Expected {(nd - expected).days} days for {channel}')
channel3 = Channel(name='channel3', link='channel3', url='https://example.com/3',
download_frequency=DownloadFrequency.daily)
channel4 = Channel(name='channel4', link='channel4', url='https://example.com/4',
download_frequency=DownloadFrequency.weekly)
# 5 and 6 should be ignored.
channel5 = Channel(name='channel5', link='channel5', url='https://example.com/5')
channel6 = Channel(name='channel6', link='channel6', download_frequency=DownloadFrequency.daily)
session.add_all([channel3, channel4, channel5, channel6])
spread_channel_downloads()
expected = {
'https://example.com/1': '2020-01-08 00:00:00-07:00',
'https://example.com/2': '2020-01-10 08:00:00-07:00',
'https://example.com/3': '2020-01-02 00:00:00-07:00',
'https://example.com/4': '2020-01-12 16:00:00-07:00',
}
check_frequencies(expected)
@wrap_test_db
def test_get_or_create_channel(self):
"""
A Channel may need to be created for an arbitrary download. Attempt to use an existing Channel if we can
match it.
"""
with get_db_session(commit=True) as session:
c1 = Channel(name='foo', link='foo', source_id='foo', url='foo')
c2 = Channel(name='bar', link='bar', source_id='bar')
c3 = Channel(name='baz', link='baz', source_id='baz', url='baz')
c4 = Channel(name='qux', link='qux')
session.add_all([c1, c2, c3, c4])
# All existing channels should be used.
tests = [
(dict(source_id='foo'), c1),
(dict(link='foo'), c1),
(dict(url='foo'), c1),
(dict(url='foo', source_id='bar'), c2), # source_id is preferred.
(dict(name='foo', source_id='bar'), c2),
(dict(source_id='bar'), c2),
(dict(source_id='baz'), c3),
(dict(name='qux'), c4),
(dict(link='qux'), c4),
]
for kwargs, expected in tests:
channel = get_or_create_channel(**kwargs)
self.assertEqual(expected.id, channel.id, f'Expected {expected} got {channel}')
# A new channel is created. It will not be automatically downloaded.
channel = get_or_create_channel(source_id='quux', name='quux', url='quux')
self.assertEqual(channel.id, 5)
self.assertEqual(channel.source_id, 'quux')
self.assertEqual(channel.name, 'quux')
self.assertEqual(channel.link, 'quux')
self.assertEqual(channel.url, 'quux')
self.assertIsNone(channel.download_frequency)
# New channel can be retrieved.
self.assertEqual(get_or_create_channel(source_id='quux'), channel)

View file

@ -0,0 +1,74 @@
from datetime import datetime
from unittest import mock
import pytest
from modules.videos.channel import lib
from modules.videos.models import Channel
from wrolpi.dates import local_timezone
from wrolpi.downloader import DownloadFrequency
daily, weekly, days30 = DownloadFrequency.daily, DownloadFrequency.weekly, DownloadFrequency.days30
@pytest.mark.parametrize(
'channels,expected',
[
([Channel(download_frequency=daily, url='https://example.com/1')],
[('https://example.com/1', daily, '2000-01-02 00:00:00-07:00')]
),
# Several different download frequencies.
([
Channel(download_frequency=daily, url='https://example.com/1'),
Channel(download_frequency=daily, url='https://example.com/2'),
Channel(download_frequency=daily, url='https://example.com/3'),
Channel(download_frequency=weekly, url='https://example.com/4'),
Channel(download_frequency=weekly, url='https://example.com/5'),
Channel(download_frequency=days30, url='https://example.com/6'),
],
[
('https://example.com/1', daily, '2000-01-02 00:00:00-07:00'),
('https://example.com/2', daily, '2000-01-02 08:00:00-07:00'),
('https://example.com/3', daily, '2000-01-02 16:00:00-07:00'),
('https://example.com/4', weekly, '2000-01-08 00:00:00-07:00'),
('https://example.com/5', weekly, '2000-01-11 12:00:00-07:00'),
('https://example.com/6', days30, '2000-01-31 00:00:00-07:00'),
]
),
# Multiple downloads in a day.
([Channel(download_frequency=daily, url=f'https://example.com/{i}') for i in range(1, 3)],
[
('https://example.com/1', daily, '2000-01-02 00:00:00-07:00'),
('https://example.com/2', daily, '2000-01-02 12:00:00-07:00'),
]
),
# More downloads in a week than days in a week.
([Channel(download_frequency=weekly, url=f'https://example.com/{i}') for i in range(1, 9)],
[
('https://example.com/1', weekly, '2000-01-08 00:00:00-07:00'),
('https://example.com/2', weekly, '2000-01-08 21:00:00-07:00'),
('https://example.com/3', weekly, '2000-01-09 18:00:00-07:00'),
('https://example.com/4', weekly, '2000-01-10 15:00:00-07:00'),
('https://example.com/5', weekly, '2000-01-11 12:00:00-07:00'),
('https://example.com/6', weekly, '2000-01-12 09:00:00-07:00'),
('https://example.com/7', weekly, '2000-01-13 06:00:00-07:00'),
('https://example.com/8', weekly, '2000-01-14 03:00:00-07:00'),
]
),
# Any Channel without a URL AND frequency is ignored.
([
Channel(download_frequency=daily, url='https://example.com/1'),
Channel(download_frequency=daily),
Channel(url='this should be ignored'),
],
[('https://example.com/1', daily, '2000-01-02 00:00:00-07:00')]
),
]
)
@mock.patch('modules.videos.channel.lib.today', lambda: local_timezone(datetime(2000, 1, 1, 0, 0, 0)))
def test_spread_by_frequency(channels, expected):
result = lib._spread_by_frequency(channels)
for result, (expected_url, expected_frequency, expected_dt) in zip(result, expected):
assert result['url'] == expected_url
assert result['frequency'] == expected_frequency
assert str(result['next_download']) == expected_dt

View file

@ -0,0 +1,15 @@
import pytest
from modules.videos.video_url_resolver import video_url_resolver
@pytest.mark.parametrize(
'domain,entry,expected',
[
('youtube.com', {'url': 'foo'}, 'https://youtube.com/watch?v=foo'),
('youtube.com', {'webpage_url': 'https://youtube.com/watch?v=foo'}, 'https://youtube.com/watch?v=foo'),
('youtube.com', {'url': 'https://youtube.com/watch?v=foo'}, 'https://youtube.com/watch?v=foo'),
]
)
def test_video_url_resolver(domain, entry, expected):
assert video_url_resolver(domain, entry) == expected

View file

@ -5,12 +5,12 @@ from sanic.request import Request
from wrolpi.common import logger, wrol_mode_check, run_after, get_media_directory, \
get_relative_to_media_directory
from wrolpi.root_api import json_response
from wrolpi.db import get_db_session
from wrolpi.errors import ValidationError, InvalidOrderBy
from wrolpi.root_api import json_response
from wrolpi.schema import validate_doc, JSONErrorResponse
from .lib import get_video, VIDEO_ORDERS, DEFAULT_VIDEO_ORDER, video_search, get_video_for_app
from ..common import get_matching_directories, get_allowed_limit, minimize_video
from ..common import get_matching_directories, get_allowed_limit
from ..lib import save_channels_config
from ..schema import VideoResponse, VideoSearchRequest, VideoSearchResponse, \
DirectoriesResponse, DirectoriesRequest
@ -55,7 +55,7 @@ async def search(_: Request, data: dict):
videos, videos_total = video_search(search_str, offset, limit, channel_link, order_by, favorites)
ret = {'videos': list(map(minimize_video, videos)), 'totals': {'videos': videos_total}}
ret = {'videos': list(videos), 'totals': {'videos': videos_total}}
return json_response(ret)

View file

@ -31,9 +31,9 @@ def get_video_for_app(video_id: int) -> Tuple[dict, Optional[dict], Optional[dic
video.set_viewed()
previous_video, next_video = video.get_surrounding_videos()
video = video.get_minimize()
previous_video = previous_video.get_minimize() if previous_video else None
next_video = next_video.get_minimize() if next_video else None
video = video.__json__()
previous_video = previous_video.__json__() if previous_video else None
next_video = next_video.__json__() if next_video else None
return video, previous_video, next_video
@ -139,7 +139,7 @@ def video_search(
results = session.query(Video).filter(Video.id.in_(ranked_ids)).all()
results = sorted(results, key=lambda r: ranked_ids.index(r.id))
results = [i.get_minimize() for i in results]
results = [i.__json__() for i in results]
return results, total

View file

@ -3,7 +3,6 @@ from datetime import timedelta
from http import HTTPStatus
from json import dumps
from unittest import mock
from unittest.mock import MagicMock
from uuid import uuid4
from modules.videos.models import Channel, Video
@ -102,13 +101,13 @@ class TestVideoFunctions(TestAPI):
self.assertError(resp, HTTPStatus.FORBIDDEN, API_ERRORS[WROLModeEnabled]['code'])
# Can't refresh or download
_, resp = api_app.test_client.post('/api/videos:refresh')
_, resp = api_app.test_client.post('/api/videos/refresh')
self.assertError(resp, HTTPStatus.FORBIDDEN, API_ERRORS[WROLModeEnabled]['code'])
_, resp = api_app.test_client.post('/api/videos:download')
_, resp = api_app.test_client.post('/api/videos/download')
self.assertError(resp, HTTPStatus.FORBIDDEN, API_ERRORS[WROLModeEnabled]['code'])
# THE REST OF THESE METHODS ARE ALLOWED
_, resp = api_app.test_client.post('/api/videos:favorite', data=favorite)
_, resp = api_app.test_client.post('/api/videos/favorite', data=favorite)
self.assertEqual(resp.status_code, HTTPStatus.OK)
@wrap_test_db
@ -164,33 +163,3 @@ class TestVideoFunctions(TestAPI):
self.assertEqual(session.query(Video).count(), 2)
self.assertFalse((tempdir / 'channel1/vid1.mp4').is_file())
self.assertFalse((tempdir / 'channel2/vid2.mp4').is_file())
def test_events(self):
request, response = api_app.test_client.get('/api/events')
self.assertOK(response)
self.assertGreater(len(response.json['events']), 1)
self.assertFalse(any(i['is_set'] for i in response.json['events']))
calls = []
async def fake_refresh_videos(*a, **kw):
calls.append((a, kw))
with mock.patch('modules.videos.api.refresh_videos', fake_refresh_videos), \
mock.patch('modules.videos.api.refresh_event') as refresh_event:
refresh_event: MagicMock
# Cannot start a second refresh while one is running.
refresh_event.is_set.return_value = True
request, response = api_app.test_client.post('/api/videos:refresh')
self.assertCONFLICT(response)
# Refresh is started, a stream is created
refresh_event.is_set.return_value = False
request, response = api_app.test_client.post('/api/videos:refresh')
self.assertOK(response)
self.assertEqual(response.json['code'], 'stream-started')
stream_url: str = response.json['stream_url']
assert stream_url.startswith('ws://')
assert calls == [((None,), {})]
refresh_event.set.assert_called()

View file

@ -0,0 +1,24 @@
formatters = {
'youtube.com': lambda i: f'https://youtube.com/watch?v={i["url"]}',
'www.youtube.com': lambda i: f'https://youtube.com/watch?v={i["url"]}',
}
def video_url_resolver(domain: str, entry: dict) -> str:
"""
Guess the full URL of a Youtube-DL entry.
"""
if 'webpage_url' in entry:
return entry['webpage_url']
if entry.get('url', '').startswith('https://'):
return entry['url']
if entry.get('url', '').startswith('http://'):
return entry['url']
try:
formatter = formatters[domain]
return formatter(entry)
except KeyError:
# Unable to resolve, try the URL as is.
return entry['url']

View file

@ -8,11 +8,12 @@ psycopg2-binary==2.8.6
pytz==2021.1
PyYAML==5.4
requests==2.26.0
Sanic-Cors==0.10.0.post3
Sanic-Cors==1.0.1
sanic-openapi==0.6.2
sanic==20.12.0
sanic-testing==0.7.0
sanic==21.9.1
SQLAlchemy==1.3.22
srt==3.4.1
websockets==8.1
websockets==10.1
webvtt-py==0.4.6
youtube-dl==2021.6.6
yt-dlp==2021.12.1

View file

@ -1,9 +1,48 @@
import inspect
import multiprocessing
BEFORE_STARTUP_FUNCTIONS = []
def before_startup(func: callable):
"""
Run a callable before startup of the WROLPi API.
Run a callable before startup of the WROLPi API. This will be called (and blocked on) once.
"""
BEFORE_STARTUP_FUNCTIONS.append(func)
return func
def after_startup(func: callable):
"""
Run a function after the startup of the WROLPi Sanic API. This will be run for each process!
"""
from .root_api import api_app
api_app.after_server_start(func)
return func
def limit_concurrent(limit: int):
"""
Wrapper that limits the amount of concurrently running functions.
"""
sema = multiprocessing.Semaphore(value=limit)
def wrapper(func: callable):
if inspect.iscoroutinefunction(func):
async def wrapped(*a, **kw):
acquired = sema.acquire(block=False)
if not acquired:
return
return await func(*a, **kw)
return wrapped
else:
def wrapped(*a, **kw):
acquired = sema.acquire(block=False)
if not acquired:
return
return func(*a, **kw)
return wrapped
return wrapper

View file

@ -3,19 +3,17 @@ import collections
import inspect
import json
import logging
import os
import pathlib
import queue
import re
import string
from copy import deepcopy
from datetime import datetime, date
from functools import wraps
from itertools import islice
from multiprocessing import Event, Queue
from pathlib import Path
from typing import Union, Callable, Tuple, Dict, Mapping, List, Iterable
from urllib.parse import urlunsplit
from urllib.parse import urlunsplit, urlparse
import yaml
from cachetools import cached, TTLCache
@ -25,11 +23,10 @@ from sqlalchemy import types
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.ext.declarative import declarative_base
from wrolpi.errors import WROLModeEnabled, UnknownDirectory
from wrolpi.vars import CONFIG_PATH, EXAMPLE_CONFIG_PATH, PUBLIC_HOST, PUBLIC_PORT, LAST_MODIFIED_DATE_FORMAT, \
PROJECT_DIR
from wrolpi.errors import WROLModeEnabled
from wrolpi.vars import CONFIG_PATH, EXAMPLE_CONFIG_PATH, PUBLIC_HOST, PUBLIC_PORT, PROJECT_DIR, PYTEST, MODULES_DIR
logger = logging.getLogger(__name__)
logger = logging.getLogger()
ch = logging.StreamHandler()
formatter = logging.Formatter('[%(asctime)s] [%(name)s] [%(levelname)s] %(message)s')
ch.setFormatter(formatter)
@ -165,107 +162,6 @@ def get_sanic_url(scheme: str = 'http', path: str = None, query: list = None, fr
return unparsed
def make_progress_calculator(total):
"""
Create a function that calculates the percentage of completion.
"""
def progress_calculator(current) -> int:
if current >= total:
# Progress doesn't make sense, just return 100
return 100
percent = int((current / total) * 100)
return percent
return progress_calculator
class ProgressReporter:
"""
I am used to consistently send messages and progress(s) to a Websocket Feed.
"""
def __init__(self, q: Queue, progress_count: int = 1):
self.queue: Queue = q
self.progresses = [{'percent': 0, 'total': 0, 'value': 0} for _ in range(progress_count)]
self.calculators = [lambda _: None for _ in range(progress_count)]
def _update(self, idx: int, **kwargs):
if 'message' in kwargs and kwargs['message'] is None:
# Message can't be cleared.
kwargs.pop('message')
self.progresses[idx].update(kwargs)
def _send(self, code: str = None):
msg = dict(
progresses=deepcopy(self.progresses)
)
if code:
msg['code'] = code
self.queue.put(msg)
def message(self, idx: int, msg: str, code: str = None):
self._update(idx, message=msg)
self._send(code)
def code(self, code: str):
self._send(code)
def error(self, idx: int, msg: str = None):
self.message(idx, msg, 'error')
def set_progress_total(self, idx: int, total: int):
self.progresses[idx]['total'] = total
self.calculators[idx] = make_progress_calculator(total)
def send_progress(self, idx: int, value: int, msg: str = None):
kwargs = dict(value=value, percent=self.calculators[idx](value), message=msg)
self._update(idx, **kwargs)
self._send()
def finish(self, idx: int, msg: str = None):
kwargs = dict(percent=100, message=msg)
if self.progresses[idx]['total'] == 0:
kwargs.update(dict(value=1, total=1))
else:
kwargs.update(dict(value=self.progresses[idx]['total']))
self._update(idx, **kwargs)
self._send()
class FileNotModified(Exception):
pass
def get_modified_time(path: Union[Path, str]) -> datetime:
"""
Return a datetime object containing the os modification time of the provided path.
"""
modified = datetime.utcfromtimestamp(os.path.getmtime(str(path)))
return modified
def get_last_modified_headers(request_headers: dict, path: Union[Path, str]) -> dict:
"""
Get a dict containing the Last-Modified header for the provided path. If If-Modified-Since is in the provided
request headers, then this will raise a FileNotModified exception, which should be handled by
`handle_FileNotModified`.
"""
last_modified = get_modified_time(path)
modified_since = request_headers.get('If-Modified-Since')
if modified_since:
modified_since = datetime.strptime(modified_since, LAST_MODIFIED_DATE_FORMAT)
if last_modified >= modified_since:
raise FileNotModified()
last_modified = last_modified.strftime(LAST_MODIFIED_DATE_FORMAT)
headers = {'Last-Modified': last_modified}
return headers
def get_example_config() -> dict:
config_path = EXAMPLE_CONFIG_PATH
with open(str(config_path), 'rt') as fh:
@ -442,6 +338,9 @@ def run_after(after: callable, *args, **kwargs) -> callable:
return synchronous_after(*a, **kw)
def wrapper(func: callable):
if PYTEST:
return func
if inspect.iscoroutinefunction(func):
@wraps(func)
async def wrapped(*a, **kw):
@ -468,7 +367,12 @@ MEDIA_DIRECTORY = None
def set_test_media_directory(path):
global TEST_MEDIA_DIRECTORY
TEST_MEDIA_DIRECTORY = pathlib.Path(path) if path else None
if isinstance(path, pathlib.Path):
TEST_MEDIA_DIRECTORY = path
elif path:
TEST_MEDIA_DIRECTORY = pathlib.Path(path)
else:
TEST_MEDIA_DIRECTORY = None
def get_media_directory() -> Path:
@ -478,6 +382,9 @@ def get_media_directory() -> Path:
global TEST_MEDIA_DIRECTORY
global MEDIA_DIRECTORY
if PYTEST and not TEST_MEDIA_DIRECTORY:
raise ValueError('No test media directory set during testing!!')
if isinstance(TEST_MEDIA_DIRECTORY, pathlib.Path):
return TEST_MEDIA_DIRECTORY
@ -505,12 +412,9 @@ def get_absolute_media_path(path: str) -> Path:
:raises UnknownDirectory: the directory/path doesn't exist
"""
media_directory = get_media_directory()
if not path:
raise ValueError(f'Cannot combine empty path with {media_directory}')
path = media_directory / path
if not path.exists():
raise UnknownDirectory(f'path={path}')
raise ValueError('Path cannot be empty!')
path = get_media_directory() / path
return path
@ -543,3 +447,35 @@ def make_media_directory(path: str):
media_dir = get_media_directory()
path = media_dir / str(path)
path.mkdir(parents=True)
def extract_domain(url):
"""
Extract the domain from a URL. Remove leading www.
>>> extract_domain('https://www.example.com/foo')
'example.com'
"""
parsed = urlparse(url)
domain = parsed.netloc
domain = domain.decode() if hasattr(domain, 'decode') else domain
if domain.startswith('www.'):
# Remove leading www.
domain = domain[4:]
return domain
def import_modules():
"""
Import all WROLPi Modules in the modules directory. Raise an ImportError if there are no modules.
"""
try:
modules = [i.name for i in MODULES_DIR.iterdir() if i.is_dir() and not i.name.startswith('_')]
for module in modules:
module = f'modules.{module}.api'
logger.debug(f'Importing {module}')
__import__(module, globals(), locals(), [], 0)
except ImportError as e:
logger.fatal('No modules could be found!', exc_info=e)
raise
return modules

View file

@ -1,28 +1,66 @@
"""
Fixtures for Pytest tests.
"""
import os
import pathlib
import tempfile
from typing import Tuple
from unittest import mock
from uuid import uuid1
import pytest
from sqlalchemy.orm import Session
import yaml
from sanic_testing.testing import SanicTestClient
from sqlalchemy.engine import Engine, create_engine
from sqlalchemy.orm import Session, sessionmaker
from wrolpi.db import postgres_engine
from wrolpi.test.common import test_db
from wrolpi.common import set_test_media_directory, Base, get_example_config
from wrolpi.db import postgres_engine, get_db_args
from wrolpi.root_api import BLUEPRINTS, api_app
os.environ['DB_PORT'] = '54321'
@pytest.fixture
def get_test_db_engine():
suffix = str(uuid1()).replace('-', '')
db_name = f'wrolpi_testing_{suffix}'
conn = postgres_engine.connect()
conn.execute(f'DROP DATABASE IF EXISTS {db_name}')
conn.execute(f'CREATE DATABASE {db_name}')
conn.execute('commit')
test_args = get_db_args(db_name)
test_engine = create_engine('postgresql://{user}:{password}@{host}:{port}/{dbname}'.format(**test_args))
return test_engine
def test_db() -> Tuple[Engine, Session]:
"""
Create a unique SQLAlchemy engine/session for a test.
"""
test_engine = get_test_db_engine()
if test_engine.engine.url.database == 'wrolpi':
raise ValueError('Refusing the test on wrolpi database!')
# Create all tables. No need to check if they exist because this is a test DB.
Base.metadata.create_all(test_engine, checkfirst=False)
session = sessionmaker(bind=test_engine)()
return test_engine, session
@pytest.fixture()
def test_session() -> Session:
"""
Pytest Fixture to get a test database session.
"""
test_engine, session = test_db()
def fake_get_db_context():
def fake_get_db_session():
"""Get the testing db"""
return test_engine, session
try:
with mock.patch('wrolpi.db._get_db_session', fake_get_db_context):
with mock.patch('wrolpi.db._get_db_session', fake_get_db_session):
yield session
finally:
session.rollback()
@ -30,3 +68,45 @@ def test_session() -> Session:
test_engine.dispose()
conn = postgres_engine.connect()
conn.execute(f'DROP DATABASE IF EXISTS {test_engine.engine.url.database}')
@pytest.fixture(autouse=True)
def test_directory():
"""
Overwrite the media directory with a temporary directory.
"""
tmp_dir = tempfile.TemporaryDirectory()
tmp_path = pathlib.Path(tmp_dir.name)
set_test_media_directory(tmp_path)
return tmp_path
@pytest.fixture(autouse=True)
def test_config():
"""
Create a test config based off the example config.
"""
test_config_path = tempfile.NamedTemporaryFile(mode='rt')
with mock.patch('wrolpi.vars.CONFIG_PATH', test_config_path.name):
config = get_example_config()
with open(test_config_path.name, 'wt') as fh:
fh.write(yaml.dump(config))
yield
ROUTES_ATTACHED = False
@pytest.fixture(autouse=True)
def test_client() -> SanicTestClient:
"""
Get a Sanic Test Client with all default routes attached.
"""
global ROUTES_ATTACHED
if ROUTES_ATTACHED is False:
# Attach any blueprints for the test.
for bp in BLUEPRINTS:
api_app.blueprint(bp)
ROUTES_ATTACHED = True
return api_app.test_client

View file

@ -85,9 +85,12 @@ class TZDateTime(types.TypeDecorator):
value = value.astimezone(timezone.utc)
elif isinstance(value, str) and '-' in value:
value = datetime.fromisoformat(value)
value = value.astimezone(timezone.utc)
return value
def process_result_value(self, value, dialect):
def process_result_value(self, value: datetime, dialect):
if value is not None:
# Assume the DB timestamp is UTC if not specified.
value = value.replace(tzinfo=pytz.utc) if not value.tzinfo else value
value = local_timezone(value)
return value

View file

@ -1,5 +1,5 @@
from contextlib import contextmanager
from typing import ContextManager
from typing import ContextManager, Tuple
import psycopg2
import sqlalchemy.exc
@ -51,7 +51,9 @@ LOGGED_ARGS = False
def _get_db_session():
"""This function allows the database to be wrapped during testing. See: api.test.common.wrap_test_db"""
"""
This function allows the database to be wrapped during testing. See: api.test.common.wrap_test_db
"""
global LOGGED_ARGS
if LOGGED_ARGS is False:
# Print the DB args for troubleshooting.
@ -64,10 +66,19 @@ def _get_db_session():
return engine, session
def get_db_context() -> Tuple[create_engine, Session]:
"""
Get a DB engine and session.
"""
return _get_db_session()
@contextmanager
def get_db_session(commit: bool = False) -> ContextManager[Session]:
"""Context manager that creates a DB session. This will automatically rollback changes, unless `commit` is True."""
local_engine, session = _get_db_session()
"""
Context manager that creates a DB session. This will automatically rollback changes, unless `commit` is True.
"""
_, session = get_db_context()
try:
yield session
if commit:
@ -83,8 +94,10 @@ def get_db_session(commit: bool = False) -> ContextManager[Session]:
@contextmanager
def get_db_curs(commit: bool = False):
"""Context manager that yields a DictCursor to execute raw SQL statements."""
local_engine, session = _get_db_session()
"""
Context manager that yields a DictCursor to execute raw SQL statements.
"""
local_engine, session = get_db_context()
connection = local_engine.raw_connection()
curs = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)
try:

527
wrolpi/downloader.py Normal file
View file

@ -0,0 +1,527 @@
import asyncio
import multiprocessing
import pathlib
import subprocess
from collections import defaultdict
from datetime import timedelta
from enum import Enum
from functools import partial
from operator import attrgetter
from typing import List, Optional
from cachetools import cached, TTLCache
from sqlalchemy import Column, Integer, String
from sqlalchemy.orm import Session
from wrolpi.common import Base, ModelHelper, logger, iterify
from wrolpi.dates import TZDateTime, now
from wrolpi.db import get_db_session, get_db_curs, get_db_context
from wrolpi.errors import InvalidDownload, UnrecoverableDownloadError
from wrolpi.vars import PYTEST
logger = logger.getChild(__name__)
DOWNLOAD_IN_PROGRESS = multiprocessing.Semaphore()
DEFAULT_RETRY_FREQUENCY = timedelta(hours=1)
class DownloadFrequency(int, Enum):
hourly = 3600
daily = 86400
weekly = 604800
biweekly = 1209600
days30 = 2592000
days90 = 7776000
class Download(ModelHelper, Base):
"""
Model that is used to schedule downloads.
"""
__tablename__ = 'download'
id = Column(Integer, primary_key=True)
url = Column(String, nullable=False)
attempts = Column(Integer, default=0)
frequency = Column(Integer)
last_successful_download = Column(TZDateTime)
next_download = Column(TZDateTime)
status = Column(String, default='new')
def __repr__(self):
if self.next_download or self.frequency:
return f'<Download id={self.id} status={self.status} url={repr(self.url)} ' \
f'next_download={self.next_download} frequency={self.frequency} attempts={self.attempts}>'
return f'<Download id={self.id} status={self.status} url={repr(self.url)} attempts={self.attempts}>'
def increment_next_download(self):
if self.frequency:
delta = timedelta(seconds=self.frequency)
else:
delta = DEFAULT_RETRY_FREQUENCY
self.next_download = now() + delta
def renew(self, reset_attempts: bool = False):
"""
Mark this Download as "new" so it will be retried.
"""
self.status = 'new'
if reset_attempts:
self.attempts = 0
def defer(self):
"""
Download should be tried again after a time.
"""
self.status = 'deferred'
self.increment_next_download()
def fail(self):
"""
Download should not be attempted again. A recurring Download will raise an error.
"""
if self.frequency:
raise ValueError('Recurring download should not be failed.')
self.status = 'failed'
def started(self):
"""
Mark this Download as in progress.
"""
self.attempts += 1
self.status = 'pending'
def complete(self):
"""
Mark this Download as successfully downloaded.
"""
self.status = 'complete'
self.last_successful_download = now()
if self.frequency:
self.increment_next_download()
class Downloader:
def __init__(self, priority: int = 50):
"""
Lower `priority` means the downloader will be checked first. Valid priority: 0-100
Downloaders of equal priority will be used at random.
"""
if not 0 <= priority <= 100:
raise ValueError(f'Priority of {priority} of {self} is invalid!')
self.priority = priority
self._kill = multiprocessing.Event()
self._manager: DownloadManager = None # noqa
download_manager.register_downloader(self)
@staticmethod
def valid_url(url) -> bool:
raise NotImplementedError()
def do_download(self, download: Download):
raise NotImplementedError()
@property
def manager(self):
if self._manager is None:
raise NotImplementedError('This needs to be registered, see DownloadManager.register_downloader')
return self._manager
@manager.setter
def manager(self, value):
self._manager = value
def kill(self):
"""
Kill the running download for this Downloader.
"""
if not self._kill.is_set():
self._kill.set()
def clear(self):
"""
Clear any "kill" request for this Downloader.
"""
if self._kill.is_set():
self.clear()
def process_runner(self, url: str, cmd: List[str], cwd: pathlib.Path, **kwargs) -> int:
"""
Run a subprocess using the provided arguments. This process can be killed by the Download Manager.
"""
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd, **kwargs)
pid = proc.pid
logger.debug(f'{self} launched download process {pid=} for {url}')
try:
while True:
try:
proc.wait(timeout=1.0)
# Download finished.
break
except subprocess.TimeoutExpired:
pass
if self._kill.is_set():
logger.warning(f'Killing download {pid=}')
proc.kill()
proc.poll()
raise UnrecoverableDownloadError(f'Download {pid=} was killed!')
finally:
self._kill.clear()
# Output all logs from the process.
# TODO is there a way to stream this output while the process is running?
logger.debug(f'Download exited with {proc.returncode}')
outs, errs = proc.communicate()
for line in outs.decode().splitlines(): # noqa
logger.debug(line)
for line in errs.decode().splitlines(): # noqa
logger.error(line)
return proc.returncode
class DownloadManager:
priority_sorter = partial(sorted, key=attrgetter('priority'))
def __init__(self):
self.instances = tuple()
def register_downloader(self, instance: Downloader):
if not isinstance(instance, Downloader):
raise ValueError(f'Invalid downloader cannot be registered! {instance=}')
if instance in self.instances:
raise ValueError(f'Downloader already registered! {instance=}')
instance.manager = self
instances = (*self.instances, instance)
self.instances = tuple(self.priority_sorter(instances))
@cached(cache=TTLCache(maxsize=100, ttl=600))
def get_downloader(self, url: str):
for i in self.instances:
if i.valid_url(url):
return i
raise InvalidDownload(f'Invalid URL {url=}')
def valid_url(self, url: str) -> bool:
try:
self.get_downloader(url)
return True
except InvalidDownload:
return False
@staticmethod
def get_new_downloads(session: Session) -> List[Download]:
"""
Get all "new" downloads. This method fetches the first download each iteration, so it will fetch downloads
that were created after calling it.
"""
last = None
while True:
download = session.query(Download).filter_by(status='new').order_by(Download.id).first()
if not download:
return
if download == last:
# Got the last download again. Is something wrong?
return
last = download
yield download
def get_or_create_download(self, url: str, session: Session) -> Download:
"""
Get a Download by its URL, if it cannot be found create one.
"""
download = self.get_download(session, url=url)
if not download:
download = Download(url=url, status='new')
session.add(download)
session.flush()
return download
def create_download(self, url: str, session, skip_download: bool = False, reset_attempts: bool = False) -> Download:
"""
Schedule a URL for download. If the URL failed previously, it will be retried.
"""
# Make sure the URL can be downloaded.
self.get_downloader(url)
download = self.get_or_create_download(url, session)
# Download may have failed, try again.
download.renew(reset_attempts=reset_attempts)
session.commit()
if skip_download is True:
return download
if PYTEST:
# Download now (for testing).
self._do_downloads(session)
else:
# Start downloading ASAP.
self.start_downloads()
return download
def create_downloads(self, urls: List[str], session: Session = None, skip_download: bool = False,
reset_attempts: bool = False) -> List[Download]:
"""
Schedule all URLs for download. If one cannot be downloaded, none will be added.
"""
if not session:
_, session = get_db_context()
downloads = []
# Verify that all URLs can be downloaded.
for url in urls:
self.get_downloader(url)
with session.transaction:
for url in urls:
download = self.get_or_create_download(url, session)
# Download may have failed, try again.
download.renew(reset_attempts=reset_attempts)
downloads.append(download)
if skip_download is True:
return downloads
if PYTEST:
# Download now (for testing).
self._do_downloads(session)
else:
# Start downloading ASAP.
self.start_downloads()
return downloads
def _do_downloads(self, session=None):
# This is a long-running function, lets get a session that can be used for a long time.
if not session:
_, session = get_db_context()
try:
downloads = self.get_new_downloads(session)
if not downloads:
logger.info('No new downloads')
return
logger.info(f'Doing downloads.')
download_count = 0
for download in downloads:
download_count += 1
download_id = download.id
downloader = self.get_downloader(download.url)
downloader.clear()
download.started()
session.commit()
success = None
failure = None
try:
success = downloader.do_download(download)
except UnrecoverableDownloadError as e:
# Download failed and should not be retried.
logger.warning(f'UnrecoverableDownloadError for {download.url}', exc_info=e)
failure = True
except Exception as e:
logger.warning(f'Failed to download {download.url}. Will be tried again later.', exc_info=e)
# Long-running, get the download again.
with get_db_session(commit=True) as session_:
download = self.get_download(session_, id_=download_id)
if failure:
download.fail()
elif success:
download.complete()
else:
download.defer()
session_.commit()
logger.info(f'Done doing {download_count} downloads.')
finally:
session.close()
def do_downloads_sync(self):
acquired = DOWNLOAD_IN_PROGRESS.acquire(block=False)
if not acquired:
return
try:
self._do_downloads()
finally:
DOWNLOAD_IN_PROGRESS.release()
async def do_downloads(self):
return self.do_downloads_sync()
def start_downloads(self):
"""
Start an async task to do downloads. This does nothing when testing!
"""
if not PYTEST:
asyncio.create_task(self.do_downloads())
@staticmethod
def reset_downloads():
"""
Set any incomplete Downloads to `new` so they will be retried.
"""
with get_db_curs(commit=True) as curs:
curs.execute("UPDATE download SET status='new' WHERE status='pending' OR status='deferred'")
def recurring_download(self, url: str, frequency: int, skip_download: bool = False) -> Download:
"""
Schedule a recurring download.
"""
with get_db_session(commit=True) as session:
download = self.create_download(url, session=session, skip_download=True)
download.frequency = frequency
if skip_download is False and PYTEST:
self.do_downloads_sync()
elif skip_download is False:
self.start_downloads()
return download
DOWNLOAD_SORT = ('pending', 'failed', 'new', 'deferred', 'complete')
@classmethod
@iterify(list)
def _downloads_sorter(cls, downloads: List[Download]):
"""
Downloads should be displayed to the user by status.
"""
grouped_by_statuses = defaultdict(lambda: [])
for download in downloads:
grouped_by_statuses[download.status].append(download)
# Yield the downloads in the order defined above.
for sort in cls.DOWNLOAD_SORT:
yield from grouped_by_statuses[sort] # converted to a list by iterify()
def get_recurring_downloads(self, session: Session = None, limit: int = None):
"""
Get all Downloads that will be downloaded in the future.
"""
if not session:
_, session = get_db_context()
query = session.query(Download).filter(
Download.frequency != None # noqa
).order_by(
Download.next_download, # Sort recurring by which will occur first, then by frequency.
Download.frequency, # Sort by frequency if the next_download is the same.
)
if limit:
query = query.limit(limit)
downloads = query.all()
downloads = self._downloads_sorter(downloads)
return downloads
def get_once_downloads(self, session: Session = None, limit: int = None):
"""
Get all Downloads that will not reoccur.
"""
if not session:
_, session = get_db_context()
query = session.query(Download).filter(
Download.frequency == None # noqa
).order_by(
Download.last_successful_download.desc(),
Download.id,
)
if limit:
query = query.limit(limit)
downloads = query.all()
downloads = self._downloads_sorter(downloads)
return downloads
def renew_recurring_downloads(self, session: Session = None):
"""
Mark any recurring downloads that are due for download as "new". Start a download.
"""
now_ = now()
if not session:
_, session = get_db_context()
recurring = self.get_recurring_downloads(session)
renewed = False
for download in recurring:
if download.next_download <= now_:
download.renew()
renewed = True
if renewed:
session.commit()
self.start_downloads()
@staticmethod
def get_downloads(session: Session) -> List[Download]:
downloads = session.query(Download).all()
return list(downloads)
@staticmethod
def get_download(session: Session, url: str = None, id_: int = None) -> Optional[Download]:
"""
Attempt to find a Download by its URL or by its id.
"""
query = session.query(Download)
if url:
return query.filter_by(url=url).one_or_none()
elif id:
return query.filter_by(id=id_).one_or_none()
def delete_download(self, download_id: int, session: Session = None):
"""
Delete a Download. Returns True if a Download was deleted, otherwise return False.
"""
if not session:
_, session = get_db_context()
download = self.get_download(session, id_=download_id)
if download:
session.delete(download)
session.commit()
return True
return False
def kill_download(self, download_id: int):
"""
Fail a Download, if it is pending, kill the Downloader so the download stops.
"""
with get_db_session(commit=True) as session:
download = self.get_download(session, id_=download_id)
downloader = self.get_downloader(download.url)
logger.warning(f'Killing download {download_id} in {downloader}')
if download.status == 'pending':
downloader.kill()
download.fail()
FINISHED_STATUSES = ('complete', 'failed')
def delete_old_once_downloads(self):
"""
Delete all once-downloads that have expired. Do not delete downloads that are new, or should be tried again.
"""
with get_db_session(commit=True) as session:
downloads = self.get_once_downloads(session)
one_month = now() - timedelta(days=30)
for download in downloads:
if download.status in self.FINISHED_STATUSES and download.last_successful_download < one_month:
session.delete(download)
# The global DownloadManager. This should be used everywhere!
download_manager = DownloadManager()

View file

@ -36,6 +36,10 @@ class ChannelDirectoryConflict(APIError):
pass
class ChannelSourceIdConflict(APIError):
pass
class SearchEmpty(APIError):
pass
@ -124,6 +128,14 @@ class InvalidArchive(APIError):
pass
class InvalidDownload(APIError):
pass
class UnrecoverableDownloadError(APIError):
pass
error_codes = iter(range(1, 1000))
API_ERRORS = {
@ -272,4 +284,19 @@ API_ERRORS = {
'message': 'The archive is invalid. See server logs.',
'status': HTTPStatus.INTERNAL_SERVER_ERROR,
},
InvalidDownload: {
'code': next(error_codes),
'message': 'The URL cannot be downloaded.',
'status': HTTPStatus.BAD_REQUEST,
},
ChannelSourceIdConflict: {
'code': next(error_codes),
'message': 'Channel with source id already exists',
'status': HTTPStatus.INTERNAL_SERVER_ERROR,
},
UnrecoverableDownloadError: {
'code': next(error_codes),
'message': 'Download experienced an error which cannot be recovered. Download will be deleted.',
'status': HTTPStatus.INTERNAL_SERVER_ERROR,
}
}

View file

@ -51,6 +51,12 @@ class MediaPath:
def __eq__(self, other):
return self._path == other
def __truediv__(self, other):
return self._path / other
def startswith(self, other):
return str(self._path).startswith(other)
class MediaPathType(types.TypeDecorator): # noqa
impl = types.String

View file

@ -1,5 +1,6 @@
import json
import logging
import json
import multiprocessing
import re
from datetime import datetime, date
from decimal import Decimal
@ -8,18 +9,20 @@ from http import HTTPStatus
from pathlib import Path
from pytz import UnknownTimeZoneError
from sanic import Sanic, response, Blueprint
from sanic import Sanic, response, Blueprint, __version__ as sanic_version
from sanic.request import Request
from sanic.response import HTTPResponse
from sanic_cors import CORS
from sanic_openapi import swagger_blueprint
from wrolpi.common import EVENTS, set_sanic_url_parts, logger, get_config, wrol_mode_enabled, save_settings_config, \
Base
from wrolpi.common import set_sanic_url_parts, logger, get_config, wrol_mode_enabled, save_settings_config, \
Base, get_media_directory, wrol_mode_check
from wrolpi.dates import set_timezone
from wrolpi.downloader import download_manager
from wrolpi.errors import WROLModeEnabled, InvalidTimezone
from wrolpi.media_path import MediaPath
from wrolpi.schema import RegexRequest, RegexResponse, SettingsRequest, SettingsResponse, EchoResponse, \
EventsResponse, validate_doc
validate_doc, DownloadRequest
from wrolpi.vars import DOCKERIZED
logger = logger.getChild(__name__)
@ -38,10 +41,9 @@ CORS(
}
)
root_api = Blueprint('Root API', url_prefix='/api')
root_api = Blueprint('RootAPI', url_prefix='/api')
# Attach the Sanic OpenAPI blueprint to the API App. This will generate our API docs.
BLUEPRINTS = [swagger_blueprint, root_api, ]
BLUEPRINTS = [root_api, ]
def get_blueprint(name: str, url_prefix: str) -> Blueprint:
@ -57,29 +59,29 @@ def add_blueprint(bp: Blueprint):
BLUEPRINTS.append(bp)
def run_webserver(host: str, port: int, workers: int = 8):
def run_webserver(loop, host: str, port: int, workers: int = 8):
set_sanic_url_parts(host, port)
# Attach all blueprints after they have been defined.
for bp in BLUEPRINTS:
api_app.blueprint(bp)
# Sanic should match our logging level.
debug = logger.level == logging.DEBUG
# TODO remove the auto reload when development is stable
api_app.run(host, port, workers=workers, debug=debug, auto_reload=True)
kwargs = dict(host=host, port=port, workers=workers, auto_reload=DOCKERIZED)
logger.debug(f'Running Sanic {sanic_version} with kwargs {kwargs}')
return api_app.run(**kwargs)
def init_parser(parser):
# Called by WROLPI's main() function
parser.add_argument('-H', '--host', default=DEFAULT_HOST, help='What network interface to connect webserver')
parser.add_argument('-p', '--port', default=DEFAULT_PORT, type=int, help='What port to connect webserver')
parser.add_argument('-w', '--workers', default=4, type=int, help='How many web workers to run')
parser.add_argument('-w', '--workers', default=multiprocessing.cpu_count(), type=int,
help='How many web workers to run')
def main(args):
run_webserver(args.host, args.port, args.workers)
return 0
def main(loop, args):
return run_webserver(loop, args.host, args.port, args.workers)
index_html = '''
@ -122,16 +124,6 @@ async def echo(request: Request):
return response.json(ret)
@root_api.get('/events')
@validate_doc(
summary='Get a list of event feeds',
produces=EventsResponse,
)
def events(_: Request):
e = [{'name': name, 'is_set': event.is_set()} for (name, event) in EVENTS]
return response.json({'events': e})
@root_api.route('/settings', methods=['GET', 'OPTIONS'])
@validate_doc(
summary='Get WROLPi settings',
@ -180,23 +172,72 @@ def valid_regex(_: Request, data: dict):
return response.json({'valid': False, 'regex': data['regex']}, HTTPStatus.BAD_REQUEST)
@root_api.post('/download')
@validate_doc(
summary='Download the many URLs that are provided.',
consumes=DownloadRequest,
)
@wrol_mode_check
async def post_download(request: Request, data: dict):
# URLs are provided in a textarea, lets split all lines.
urls = [i.strip() for i in str(data['urls']).strip().splitlines()]
download_manager.create_downloads(urls)
return response.empty()
@root_api.get('/download')
@validate_doc(
summary='Get all Downloads that need to be processed.',
)
async def get_downloads(request: Request):
data = dict(
recurring_downloads=download_manager.get_recurring_downloads(),
once_downloads=download_manager.get_once_downloads(limit=1000),
)
return json_response(data)
@root_api.post('/download/<download_id:int>/kill')
async def kill_download(request: Request, download_id: int):
download_manager.kill_download(download_id)
return response.empty()
@root_api.delete('/download/<download_id:integer>')
@wrol_mode_check
async def delete_download(request: Request, download_id: int):
deleted = download_manager.delete_download(download_id)
return response.empty(HTTPStatus.NO_CONTENT if deleted else HTTPStatus.NOT_FOUND)
class CustomJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime):
return obj.timestamp()
elif isinstance(obj, date):
return datetime(obj.year, obj.month, obj.day).timestamp()
elif isinstance(obj, Decimal):
return str(obj)
elif isinstance(obj, Base):
if hasattr(obj, 'dict'):
return obj.dict()
elif isinstance(obj, Path):
return str(obj)
elif hasattr(obj, '__json__'):
return obj.__json__()
return super(CustomJSONEncoder, self).default(obj)
try:
if hasattr(obj, '__json__'):
# Get __json__ before others.
return obj.__json__()
elif isinstance(obj, datetime):
return obj.timestamp()
elif isinstance(obj, date):
return datetime(obj.year, obj.month, obj.day).timestamp()
elif isinstance(obj, Decimal):
return str(obj)
elif isinstance(obj, Base):
if hasattr(obj, 'dict'):
return obj.dict()
elif isinstance(obj, Path):
return str(obj)
elif isinstance(obj, MediaPath):
media_directory = get_media_directory()
path = obj.path.relative_to(media_directory)
if str(path) == '.':
return ''
return str(path)
return super(CustomJSONEncoder, self).default(obj)
except Exception as e:
logger.fatal(f'Failed to JSON encode {obj}', exc_info=e)
raise
@wraps(response.json)

View file

@ -178,6 +178,10 @@ class EventsResponse:
events = doc.List(EventObject)
class DownloadRequest:
urls = doc.String()
class Trinary(Field):
"""
A field for API docs. Can be True/False/None.

View file

@ -6,54 +6,22 @@ from contextlib import contextmanager
from functools import partialmethod
from http import HTTPStatus
from itertools import zip_longest
from queue import Empty
from shutil import copyfile
from typing import List, Tuple, Optional
from uuid import uuid1
from typing import List, Optional
import mock
import websockets
import yaml
from sanic_openapi.api import Response
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
from sqlalchemy.orm import Session, sessionmaker
from wrolpi.common import EXAMPLE_CONFIG_PATH, get_config, Base, QUEUES, set_test_media_directory
from wrolpi.db import postgres_engine, get_db_args
from wrolpi.root_api import BLUEPRINTS, api_app
from wrolpi.common import EXAMPLE_CONFIG_PATH, get_config, set_test_media_directory, get_media_directory
from wrolpi.conftest import ROUTES_ATTACHED, test_db, test_client # noqa
from wrolpi.db import postgres_engine
from wrolpi.vars import PROJECT_DIR
TEST_CONFIG_PATH = tempfile.NamedTemporaryFile(mode='rt', delete=False)
def get_test_db_engine():
suffix = str(uuid1()).replace('-', '')
db_name = f'wrolpi_testing_{suffix}'
conn = postgres_engine.connect()
conn.execute(f'DROP DATABASE IF EXISTS {db_name}')
conn.execute(f'CREATE DATABASE {db_name}')
conn.execute('commit')
test_args = get_db_args(db_name)
test_engine = create_engine('postgresql://{user}:{password}@{host}:{port}/{dbname}'.format(**test_args))
return test_engine
def test_db() -> Tuple[Engine, Session]:
"""
Create a unique SQLAlchemy engine/session for a test.
"""
test_engine = get_test_db_engine()
if test_engine.engine.url.database == 'wrolpi':
raise ValueError('Refusing the test on wrolpi database!')
# Create all tables. No need to check if they exist because this is a test DB.
Base.metadata.create_all(test_engine, checkfirst=False)
session = sessionmaker(bind=test_engine)()
return test_engine, session
def wrap_test_db(func):
"""
Wrap a test so that when calling wrolpi.common.get_db, it returns a testing database cloned from the api
@ -63,12 +31,12 @@ def wrap_test_db(func):
def wrapped(*a, **kw):
test_engine, session = test_db()
def fake_get_db_context():
def fake_get_db_session():
"""Get the testing db"""
return test_engine, session
try:
with mock.patch('wrolpi.db._get_db_session', fake_get_db_context):
with mock.patch('wrolpi.db._get_db_session', fake_get_db_session):
# Run the test.
result = func(*a, **kw)
return result
@ -82,44 +50,32 @@ def wrap_test_db(func):
return wrapped
def get_all_messages_in_queue(q):
"""Get all messages in a Queue without waiting."""
messages = []
while True:
try:
msg = q.get_nowait()
messages.append(msg)
except Empty:
break
return messages
class PytestCase:
"""
Replicate unittest.TestCase methods to bridge the gap between it and Pytest.
"""
@staticmethod
def assertGreater(a, b, msg: str = None):
def assertGreater(a, b, msg: str = None): # noqa
assert a > b, msg
@staticmethod
def assertLess(a, b, msg: str = None):
def assertLess(a, b, msg: str = None): # noqa
assert a < b, msg
@staticmethod
def assertEqual(a, b, msg: str = None):
def assertEqual(a, b, msg: str = None): # noqa
assert a == b, msg or f'{a} != {b}'
@staticmethod
def assertRaises(exception, func, *args, **kwargs):
def assertRaises(exception, func, *args, **kwargs): # noqa
try:
func(*args, **kwargs)
except exception:
pass
@classmethod
def assertDictContains(cls, d1: dict, d2: dict):
def assertDictContains(cls, d1: dict, d2: dict): # noqa
if hasattr(d1, '__dict__'):
d1 = d1.__dict__
if hasattr(d2, '__dict__'):
@ -134,13 +90,13 @@ class PytestCase:
else:
assert d1[k2] == d2[k2], f'{k2} of value "{d1[k2]}" does not equal {d2[k2]} in dict 1'
def assertError(self, response, http_status: int, code=None):
def assertError(self, response, http_status: int, code=None): # noqa
self.assertEqual(response.status_code, http_status)
if code:
self.assertEqual(response.json['code'], code)
@staticmethod
def assertTruth(value, expected):
def assertTruth(value, expected): # noqa
"""
Check that a value is Truthy or Falsy.
"""
@ -161,6 +117,10 @@ class PytestCase:
raise ValueError('d2 is None')
self.assertTruth(d1[d2_key], d2[d2_key])
@staticmethod
def assertLength(a, b): # noqa
assert len(a) == len(b), f'{len(a)=} != {len(b)=}'
class ExtendedTestCase(PytestCase, unittest.TestCase):
"""
@ -169,9 +129,6 @@ class ExtendedTestCase(PytestCase, unittest.TestCase):
pass
ROUTES_ATTACHED = False
class TestAPI(ExtendedTestCase):
def setUp(self) -> None:
@ -188,21 +145,9 @@ class TestAPI(ExtendedTestCase):
with open(TEST_CONFIG_PATH.name, 'wt') as fh:
fh.write(yaml.dump(config))
@classmethod
def setUpClass(cls) -> None:
global ROUTES_ATTACHED
if ROUTES_ATTACHED is False:
# Attach any blueprints for the test.
for bp in BLUEPRINTS:
api_app.blueprint(bp)
ROUTES_ATTACHED = True
def tearDown(self) -> None:
self.config_path_patch.stop()
set_test_media_directory(None)
# Clear out any messages in queues
for q in QUEUES:
get_all_messages_in_queue(q)
def assertHTTPStatus(self, response: Response, status: int):
self.assertEqual(response.status_code, status)
@ -229,7 +174,7 @@ def wrap_media_directory(path: Optional[pathlib.Path] = None):
@contextmanager
def build_test_directories(paths: List[str]) -> pathlib.Path:
def build_test_directories(paths: List[str], tmp_dir: pathlib.Path = None) -> pathlib.Path:
"""
Create directories based on the provided structure.
@ -247,23 +192,20 @@ def build_test_directories(paths: List[str]) -> pathlib.Path:
channel2/vid2.mp4
channel2/vid2.en.vtt
"""
dir_ = get_config().get('media_directory')
dir_ = pathlib.Path(dir_).absolute()
with tempfile.TemporaryDirectory(dir=dir_) as temp_dir:
root = pathlib.Path(temp_dir)
tmp_dir = tmp_dir or get_media_directory()
directories = filter(lambda i: i.endswith('/'), paths)
for directory in directories:
(root / directory).mkdir(parents=True)
directories = filter(lambda i: i.endswith('/'), paths)
for directory in directories:
(tmp_dir / directory).mkdir(parents=True)
files = filter(lambda i: not i.endswith('/'), paths)
for file in files:
file = root / file
parents = file.parents
parents[0].mkdir(parents=True, exist_ok=True)
(root / file).touch()
files = filter(lambda i: not i.endswith('/'), paths)
for file in files:
file = tmp_dir / file
parents = file.parents
parents[0].mkdir(parents=True, exist_ok=True)
(tmp_dir / file).touch()
yield root.absolute()
yield tmp_dir.absolute()
async def get_all_ws_messages(ws) -> List[dict]:

View file

@ -1,74 +0,0 @@
import json
from http import HTTPStatus
from unittest import mock
from unittest.mock import MagicMock
import pytest
from sanic_openapi import swagger_blueprint
from wrolpi.root_api import api_app
from wrolpi.test.common import TestAPI
class TestRootAPI(TestAPI):
def test_index(self):
"""
Index should have some details in an HTML response
"""
request, response = api_app.test_client.get('/')
assert response.status_code == HTTPStatus.OK
assert b'html' in response.body
def test_echo(self):
"""
Echo should send back a JSON object with our request details.
"""
request, response = api_app.test_client.get('/api/echo?hello=world')
assert response.status_code == HTTPStatus.OK
self.assertDictContains(response.json['form'], {})
assert response.json['method'] == 'GET', 'Method was not GET'
assert response.json['json'] is None, 'JSON was not empty'
assert response.json['args'] == {'hello': ['world']}
data = {'foo': 'bar'}
request, response = api_app.test_client.post('/api/echo', data=json.dumps(data))
assert response.status_code == HTTPStatus.OK
self.assertDictContains(response.json['form'], {})
assert response.json['method'] == 'POST', 'Method was not POST'
assert response.json['json'] == data, 'JSON was not data'
assert response.json['args'] == {}
def test_valid_regex(self):
"""
The endpoint should only return valid if the regex is valid.
"""
data = {'regex': 'foo'}
request, response = api_app.test_client.post('/api/valid_regex', data=json.dumps(data))
assert response.status_code == HTTPStatus.OK
assert json.loads(response.body) == {'valid': True, 'regex': 'foo'}
data = {'regex': '.*(title match).*'}
request, response = api_app.test_client.post('/api/valid_regex', data=json.dumps(data))
assert response.status_code == HTTPStatus.OK
assert json.loads(response.body) == {'valid': True, 'regex': '.*(title match).*'}
data = {'regex': '.*(missing parenthesis.*'}
request, response = api_app.test_client.post('/api/valid_regex', data=json.dumps(data))
assert response.status_code == HTTPStatus.BAD_REQUEST
assert json.loads(response.body) == {'valid': False, 'regex': '.*(missing parenthesis.*'}
@pytest.mark.xfail
def test_swagger(self):
"""
Swagger can be generated. This test is just to assure that our spec generation hasn't broken.
"""
# An API request is required to set things up (apparently).
api_app.test_client.get('/')
from sanic.response import json as json_
json_(swagger_blueprint._spec)
def test_get_settings(self):
request, response = api_app.test_client.get('/api/settings')
self.assertOK(response)

View file

@ -1,15 +1,14 @@
import unittest
from datetime import date, datetime
from queue import Queue
import pytest
from sanic_openapi import doc
from wrolpi.common import combine_dicts, insert_parameter, ProgressReporter, date_range
from wrolpi.common import combine_dicts, insert_parameter, date_range
from wrolpi.dates import set_timezone, now
from wrolpi.errors import NoBodyContents, MissingRequiredField, ExcessJSONFields, InvalidTimezone
from wrolpi.schema import validate_data
from wrolpi.test.common import build_test_directories, wrap_test_db
from wrolpi.test.common import build_test_directories, wrap_media_directory
class Model:
@ -114,42 +113,43 @@ def test_combine_dicts(data, expected):
def test_build_video_directories():
structure = [
'channel1/vid1.mp4',
]
with build_test_directories(structure) as tempdir:
assert (tempdir / 'channel1').is_dir()
assert (tempdir / 'channel1/vid1.mp4').is_file()
with wrap_media_directory():
structure = [
'channel1/vid1.mp4',
]
with build_test_directories(structure) as tempdir:
assert (tempdir / 'channel1').is_dir()
assert (tempdir / 'channel1/vid1.mp4').is_file()
structure = [
'channel2/',
'channel2.1/channel2.2/',
]
with build_test_directories(structure) as tempdir:
assert (tempdir / 'channel2').is_dir()
assert (tempdir / 'channel2.1/channel2.2').is_dir()
structure = [
'channel2/',
'channel2.1/channel2.2/',
]
with build_test_directories(structure) as tempdir:
assert (tempdir / 'channel2').is_dir()
assert (tempdir / 'channel2.1/channel2.2').is_dir()
structure = [
'channel3/vid1.mp4',
'channel3/vid2.mp4',
'channel4/vid1.mp4',
'channel4/vid1.en.vtt',
'channel5/',
]
with build_test_directories(structure) as tempdir:
assert (tempdir / 'channel3/vid1.mp4').is_file()
assert (tempdir / 'channel3').is_dir()
assert (tempdir / 'channel3/vid2.mp4').is_file()
assert (tempdir / 'channel4/vid1.mp4').is_file()
assert (tempdir / 'channel4/vid1.en.vtt').is_file()
assert (tempdir / 'channel5').is_dir()
structure = [
'channel3/vid1.mp4',
'channel3/vid2.mp4',
'channel4/vid1.mp4',
'channel4/vid1.en.vtt',
'channel5/',
]
with build_test_directories(structure) as tempdir:
assert (tempdir / 'channel3/vid1.mp4').is_file()
assert (tempdir / 'channel3').is_dir()
assert (tempdir / 'channel3/vid2.mp4').is_file()
assert (tempdir / 'channel4/vid1.mp4').is_file()
assert (tempdir / 'channel4/vid1.en.vtt').is_file()
assert (tempdir / 'channel5').is_dir()
structure = [
'channel6/subdirectory/vid1.mp4',
]
with build_test_directories(structure) as tempdir:
assert (tempdir / 'channel6/subdirectory').is_dir()
assert (tempdir / 'channel6/subdirectory/vid1.mp4').is_file()
structure = [
'channel6/subdirectory/vid1.mp4',
]
with build_test_directories(structure) as tempdir:
assert (tempdir / 'channel6/subdirectory').is_dir()
assert (tempdir / 'channel6/subdirectory/vid1.mp4').is_file()
def test_insert_parameter():
@ -195,146 +195,6 @@ def test_insert_parameter():
pytest.raises(TypeError, insert_parameter, func, 'bar', 'bar', (1,), {})
@pytest.mark.parametrize(
'totals,progresses,messages',
[
(
[25],
[],
[
{'progresses': [dict(message='foo', percent=0, total=25, value=0)]},
{'code': 'bar', 'progresses': [dict(message='foo', percent=0, total=25, value=0)]},
{'code': 'error', 'progresses': [dict(message='baz', percent=0, total=25, value=0)]},
],
),
(
[25],
[
(0, 1),
(0, 2),
(0, 4),
(0, 8),
(0, 16),
(0, 25),
],
[
{'progresses': [dict(message='foo', percent=0, total=25, value=0)]},
{'code': 'bar', 'progresses': [dict(message='foo', percent=0, total=25, value=0)]},
{'progresses': [dict(message='foo', percent=4, total=25, value=1)]},
{'progresses': [dict(message='foo', percent=8, total=25, value=2)]},
{'progresses': [dict(message='foo', percent=16, total=25, value=4)]},
{'progresses': [dict(message='foo', percent=32, total=25, value=8)]},
{'progresses': [dict(message='foo', percent=64, total=25, value=16)]},
{'progresses': [dict(message='foo', percent=100, total=25, value=25)]},
{'code': 'error', 'progresses': [dict(message='baz', percent=100, total=25, value=25)]},
]
),
(
[25, 10],
[
(1, 3),
(1, 8),
(0, 18),
(1, 10),
(0, 30), # This is higher than the total of 25, percent should be 100.
],
[
{'progresses': [
dict(message='foo', percent=0, total=25, value=0),
dict(percent=0, total=10, value=0),
]},
{'code': 'bar', 'progresses': [
dict(message='foo', percent=0, total=25, value=0),
dict(percent=0, total=10, value=0),
]},
{'progresses': [
dict(message='foo', percent=0, total=25, value=0),
dict(percent=30, total=10, value=3),
]},
{'progresses': [
dict(message='foo', percent=0, total=25, value=0),
dict(percent=80, total=10, value=8),
]},
{'progresses': [
dict(message='foo', percent=72, total=25, value=18),
dict(percent=80, total=10, value=8),
]},
{'progresses': [
dict(message='foo', percent=72, total=25, value=18),
dict(percent=100, total=10, value=10),
]},
{'progresses': [
dict(message='foo', percent=100, total=25, value=30),
dict(percent=100, total=10, value=10),
]},
{
'code': 'error',
'progresses': [
dict(message='baz', percent=100, total=25, value=30),
dict(percent=100, total=10, value=10),
]
},
]
),
]
)
def test_feed_reporter(totals, progresses, messages):
q = Queue()
reporter = ProgressReporter(q, len(totals))
for idx, total in enumerate(totals):
reporter.set_progress_total(idx, total)
reporter.message(0, 'foo')
reporter.code('bar')
for progress in progresses:
reporter.send_progress(*progress)
reporter.error(0, 'baz')
count = 0
while not q.empty():
received = q.get_nowait()
try:
message = messages.pop(0)
except IndexError:
raise AssertionError(f'Queue ({count}) had excess message: {received}')
assert message == received, f'Message {count} did not match'
count += 1
def test_feed_reporter_finish():
q = Queue()
reporter = ProgressReporter(q)
reporter.set_progress_total(0, 50)
reporter.send_progress(0, 0)
reporter.send_progress(0, 20)
reporter.finish(0, 'completed')
expected = [
{'progresses': [{'percent': 0, 'total': 50, 'value': 0}]},
{'progresses': [{'percent': 40, 'total': 50, 'value': 20}]},
{'progresses': [{'message': 'completed', 'percent': 100, 'value': 50, 'total': 50}]},
]
count = 0
while not q.empty():
received = q.get_nowait()
try:
message = expected.pop(0)
except IndexError:
raise AssertionError(f'Queue ({count}) had excess message: {received}')
assert message == received, f'Message {count} did not match'
count += 1
class TestCommon(unittest.TestCase):
def test_date_range(self):

View file

@ -1,8 +1,10 @@
from datetime import datetime, timedelta
from sqlalchemy import Column, Integer
from wrolpi.common import Base
from wrolpi.dates import TZDateTime
from wrolpi.db import get_db_session
from wrolpi.dates import TZDateTime, local_timezone, strftime_ms
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.test.common import TestAPI, wrap_test_db
@ -17,11 +19,19 @@ class TestTable(Base):
class TestDates(TestAPI):
def assert_raw_datetime(self, expected_datetime: str):
with get_db_curs() as curs:
curs.execute('SELECT * FROM test_table')
dt_ = curs.fetchone()['dt']
self.assertEqual(strftime_ms(dt_), expected_datetime)
@wrap_test_db
def test_TZDateTime(self):
with get_db_session(commit=True) as session:
# TZDateTime can be None
session.add(TestTable())
tt = TestTable()
session.add(tt)
self.assertIsNone(tt.dt)
with get_db_session(commit=True) as session:
tt = session.query(TestTable).one()
@ -30,14 +40,22 @@ class TestDates(TestAPI):
# Store a timezone naive string, it should be assumed this is a local time.
tt.dt = '2021-10-05 16:20:10.346823'
# DB actually contains a UTC timestamp.
self.assert_raw_datetime('2021-10-05 22:20:10.346823')
with get_db_session(commit=True) as session:
tt = session.query(TestTable).one()
self.assertEqual(str(tt.dt), '2021-10-05 16:20:10.346823-06:00')
# Datetime is unchanged and localized.
self.assertEqual(tt.dt, local_timezone(datetime(2021, 10, 5, 16, 20, 10, 346823)))
# Store a timezone string.
tt.dt = '2021-10-05 00:20:10.346824-08:00'
# Increment by one hour.
tt.dt += timedelta(seconds=60 * 60)
self.assertEqual(tt.dt, local_timezone(datetime(2021, 10, 5, 17, 20, 10, 346823)))
with get_db_session() as session:
# DB is incremented by one hour.
self.assert_raw_datetime('2021-10-05 23:20:10.346823')
with get_db_session(commit=True) as session:
tt = session.query(TestTable).one()
# Timezone is preserved.
self.assertEqual(str(tt.dt), '2021-10-05 08:20:10.346824-06:00')
# Datetime is incremented by an hour.
self.assertEqual(tt.dt, local_timezone(datetime(2021, 10, 5, 17, 20, 10, 346823)))

View file

@ -0,0 +1,303 @@
from datetime import datetime, timedelta
from itertools import zip_longest
from unittest import mock
from unittest.mock import MagicMock
from wrolpi import downloader
from wrolpi.dates import local_timezone
from wrolpi.db import get_db_context
from wrolpi.downloader import Downloader, Download
from wrolpi.errors import UnrecoverableDownloadError, InvalidDownload
from wrolpi.test.common import wrap_test_db, TestAPI
class PermissiveDownloader(Downloader):
"""
A testing Downloader which always says it's valid.
"""
def valid_url(self, url: str):
return True
class HTTPDownloader(Downloader):
"""
A testing Downloader which says its valid when a URL starts with http/https
"""
def valid_url(self, url: str):
return url.startswith('https://') or url.startswith('http://')
class TestDownloader(TestAPI):
def setUp(self) -> None:
super().setUp()
self.mgr = downloader.DownloadManager()
PermissiveDownloader.do_download = MagicMock()
HTTPDownloader.do_download = MagicMock()
def test_register_downloader(self):
"""
Downloaders can be registered and have specific priorities.
"""
self.assertEqual(self.mgr.instances, tuple())
http_downloader = HTTPDownloader()
self.mgr.register_downloader(http_downloader)
self.assertEqual(self.mgr.instances, (http_downloader,))
# PermissiveDownloader is first priority.
permissive_downloader = PermissiveDownloader(priority=0)
self.mgr.register_downloader(permissive_downloader)
self.assertEqual(self.mgr.instances, (permissive_downloader, http_downloader))
self.assertRaises(ValueError, self.mgr.register_downloader, http_downloader)
self.assertRaises(ValueError, self.mgr.register_downloader, permissive_downloader)
def test_valid_url(self):
# No downloaders available.
self.assertFalse(self.mgr.valid_url('foo'))
self.assertFalse(self.mgr.valid_url('https://example.com'))
http_downloader = HTTPDownloader()
self.mgr.register_downloader(http_downloader)
self.assertFalse(self.mgr.valid_url('foo'))
self.assertTrue(self.mgr.valid_url('https://example.com'))
self.assertEqual(self.mgr.get_downloader('https://example.com'), http_downloader)
# Last priority.
permissive_downloader = PermissiveDownloader(priority=100)
self.mgr.register_downloader(permissive_downloader)
self.assertTrue(self.mgr.valid_url('foo'))
self.assertEqual(self.mgr.get_downloader('foo'), permissive_downloader)
self.assertTrue(self.mgr.valid_url('https://example.com'))
self.assertEqual(self.mgr.get_downloader('https://example.com'), http_downloader)
@wrap_test_db
def test_ensure_download(self):
_, session = get_db_context()
self.assertEqual(len(self.mgr.get_downloads(session)), 0)
http_downloader = HTTPDownloader()
http_downloader.do_download.return_value = False
self.mgr.register_downloader(http_downloader)
self.mgr.create_download('https://example.com', session)
download = self.mgr.get_download(session, 'https://example.com')
self.assertEqual(download.url, 'https://example.com')
self.assertEqual(len(self.mgr.get_downloads(session)), 1)
self.assertIsNotNone(self.mgr.get_download(session, 'https://example.com'))
self.assertIsNone(self.mgr.get_download(session, 'https://example.com/not downloading'))
@wrap_test_db
def test_do_downloads(self):
_, session = get_db_context()
http_downloader = HTTPDownloader()
http_downloader.do_download.return_value = True
self.mgr.register_downloader(http_downloader)
permissive_downloader = PermissiveDownloader(priority=100)
permissive_downloader.do_download.return_value = False # returns a failure
self.mgr.register_downloader(permissive_downloader)
# https is handled by the HTTP Downloader.
self.mgr.create_download('https://example.com', session)
http_downloader.do_download.assert_called_once()
permissive_downloader.do_download.assert_not_called()
self.assertIsNotNone(self.mgr.get_download(session, 'https://example.com'))
http_downloader.do_download.reset_mock()
# try the permissive download, which returns a failure.
self.mgr.create_download('foo', session)
http_downloader.do_download.assert_not_called()
permissive_downloader.do_download.assert_called_once()
download = self.mgr.get_download(session, 'foo')
self.assertEqual(download.attempts, 1)
# try again
self.mgr.create_download('foo', session)
self.mgr._do_downloads(session)
download = self.mgr.get_download(session, 'foo')
self.assertEqual(download.attempts, 2)
# finally success
permissive_downloader.do_download.return_value = True
self.mgr.create_download('foo', session)
self.mgr._do_downloads(session)
download = self.mgr.get_download(session, 'foo')
self.assertEqual(download.status, 'complete')
# No downloads left.
self.assertEqual(list(self.mgr.get_new_downloads(session)), [])
@wrap_test_db
def test_max_attempts(self):
"""
A Download will only be attempted so many times, it will eventually be deleted.
"""
_, session = get_db_context()
http_downloader = HTTPDownloader()
http_downloader.do_download.return_value = False
self.mgr.register_downloader(http_downloader)
self.mgr.create_download('https://example.com', session)
download = session.query(Download).one()
self.assertEqual(download.attempts, 1)
self.mgr.create_download('https://example.com', session)
download = session.query(Download).one()
self.assertEqual(download.attempts, 2)
# There are no further attempts.
http_downloader.do_download.side_effect = UnrecoverableDownloadError()
self.mgr.create_download('https://example.com', session)
download = session.query(Download).one()
self.assertEqual(download.attempts, 3)
self.assertEqual(download.status, 'failed')
@wrap_test_db
def test_recurring_downloads(self):
"""
A recurring Download should be downloaded repeatedly forever.
"""
_, session = get_db_context()
http_downloader = HTTPDownloader()
http_downloader.do_download.return_value = True
http_downloader.do_download: MagicMock # noqa
self.mgr.register_downloader(http_downloader)
# Download every hour.
self.mgr.recurring_download('https://example.com', 3600, skip_download=True)
# One download is scheduled.
downloads = self.mgr.get_new_downloads(session)
self.assertEqual(
[(i.url, i.frequency) for i in downloads],
[('https://example.com', 3600)]
)
with mock.patch('wrolpi.downloader.now') as mock_now:
now = local_timezone(datetime(2020, 1, 1, 0, 0, 0))
mock_now.return_value = now
# Download is processed and successful, no longer "new".
self.mgr.do_downloads_sync()
http_downloader.do_download.assert_called_once()
self.assertEqual(list(self.mgr.get_new_downloads(session)), [])
downloads = list(self.mgr.get_recurring_downloads(session))
self.assertEqual(len(downloads), 1)
download = downloads[0]
expected = local_timezone(datetime(2020, 1, 1, 1, 0, 0))
self.assertEqual(download.next_download, expected)
self.assertEqual(download.last_successful_download, now)
# Download is not due for an hour.
self.mgr.renew_recurring_downloads(session)
self.assertEqual(list(self.mgr.get_new_downloads(session)), [])
self.assertEqual(download.last_successful_download, now)
# Download is due an hour later.
mock_now.return_value = local_timezone(datetime(2020, 1, 1, 1, 0, 0))
self.mgr.renew_recurring_downloads(session)
downloads = list(self.mgr.get_new_downloads(session))
self.assertEqual(len(downloads), 1)
download = downloads[0]
# Download is "new" but has not been done a second time.
self.assertEqual(download.next_download, expected)
self.assertEqual(download.last_successful_download, now)
# Try the download, but it fails.
http_downloader.do_download.reset_mock()
http_downloader.do_download.return_value = False
self.mgr.do_downloads_sync()
http_downloader.do_download.assert_called_once()
download = session.query(Download).one()
# Download is deferred, last successful download remains the same.
self.assertEqual(download.status, 'deferred')
self.assertEqual(download.last_successful_download, now)
# Download should be retried after the DEFAULT_RETRY_FREQUENCY.
self.assertEqual(download.next_download, expected + timedelta(hours=1))
# Try the download again, it finally succeeds.
http_downloader.do_download.reset_mock()
now = local_timezone(datetime(2020, 1, 1, 3, 0, 0))
mock_now.return_value = now
http_downloader.do_download.return_value = True
self.mgr.renew_recurring_downloads(session)
self.mgr.do_downloads_sync()
http_downloader.do_download.assert_called_once()
download = session.query(Download).one()
self.assertEqual(download.status, 'complete')
self.assertEqual(download.last_successful_download, now)
self.assertEqual(download.next_download, local_timezone(datetime(2020, 1, 1, 4, 0, 0)))
@wrap_test_db
@mock.patch('wrolpi.downloader.now', lambda: local_timezone(datetime(2020, 6, 5, 0, 0)))
def test_delete_old_once_downloads(self):
"""
Once-downloads over a month old should be deleted.
"""
permissive_downloader = PermissiveDownloader(priority=0)
self.mgr.register_downloader(permissive_downloader)
_, session = get_db_context()
# Recurring downloads should not be deleted.
d1 = self.mgr.create_download('https://example.com/1', session, skip_download=True)
d2 = self.mgr.create_download('https://example.com/2', session, skip_download=True)
d1.frequency = 1
d2.frequency = 1
d2.started()
# Should be deleted.
d3 = self.mgr.create_download('https://example.com/3', session, skip_download=True)
d4 = self.mgr.create_download('https://example.com/4', session, skip_download=True)
d3.complete()
d4.complete()
d3.last_successful_download = local_timezone(datetime(2020, 1, 1, 0, 0, 0))
d4.last_successful_download = local_timezone(datetime(2020, 5, 1, 0, 0, 0))
# Not a month old.
d5 = self.mgr.create_download('https://example.com/5', session, skip_download=True)
d5.last_successful_download = local_timezone(datetime(2020, 6, 1, 0, 0, 0))
# An old, but pending download should not be deleted.
d6 = self.mgr.create_download('https://example.com/6', session, skip_download=True)
d6.last_successful_download = local_timezone(datetime(2020, 4, 1, 0, 0, 0))
d6.started()
self.mgr.delete_old_once_downloads()
# Two old downloads are deleted.
downloads = self.mgr._downloads_sorter(session.query(Download).all())
expected = [
dict(url='https://example.com/2', frequency=1, attempts=1, status='pending'),
dict(url='https://example.com/6', attempts=1, status='pending'),
dict(url='https://example.com/1', frequency=1, attempts=0, status='new'),
dict(url='https://example.com/5', attempts=0, status='new'),
]
for download, expected in zip_longest(downloads, expected):
self.assertDictContains(download.dict(), expected)
@wrap_test_db
def test_create_downloads(self):
"""
Multiple downloads can be scheduled using DownloadManager.create_downloads.
"""
http_downloader = HTTPDownloader(priority=0)
self.mgr.register_downloader(http_downloader)
_, session = get_db_context()
# Both URLs are valid and are scheduled.
self.mgr.create_downloads(['https://example.com/1', 'https://example.com/2'], skip_download=True)
downloads = self.mgr.get_downloads(session)
self.assertEqual({i.url for i in downloads}, {'https://example.com/1', 'https://example.com/2'})
# One URL is bad, neither should be scheduled.
self.assertRaises(InvalidDownload, self.mgr.create_downloads, ['https://example.com/3', 'bad url should fail'],
skip_download=True)
downloads = self.mgr.get_downloads(session)
self.assertEqual({i.url for i in downloads}, {'https://example.com/1', 'https://example.com/2'})

View file

@ -4,6 +4,7 @@ import tempfile
import unittest
from unittest import mock
from wrolpi.common import set_test_media_directory
from wrolpi.media_path import MediaPath
from wrolpi.root_api import CustomJSONEncoder
@ -12,11 +13,13 @@ class TestMediaPath(unittest.TestCase):
def setUp(self):
self.test_dir = tempfile.TemporaryDirectory()
set_test_media_directory(pathlib.Path(self.test_dir.name))
path = pathlib.Path(self.test_dir.name).absolute()
self.patch = mock.patch('wrolpi.media_path.get_media_directory', lambda: path)
self.patch.start()
def tearDown(self):
set_test_media_directory(None)
self.test_dir.cleanup()
self.patch.stop()

View file

@ -0,0 +1,156 @@
import json
from http import HTTPStatus
from itertools import zip_longest
from wrolpi.dates import strptime
from wrolpi.db import get_db_session
from wrolpi.downloader import Download
from wrolpi.root_api import api_app
from wrolpi.test.common import TestAPI, wrap_test_db
class TestRootAPI(TestAPI):
def test_index(self):
"""
Index should have some details in an HTML response
"""
request, response = api_app.test_client.get('/')
assert response.status_code == HTTPStatus.OK
assert b'html' in response.body
def test_echo(self):
"""
Echo should send back a JSON object with our request details.
"""
request, response = api_app.test_client.get('/api/echo?hello=world')
assert response.status_code == HTTPStatus.OK
self.assertDictContains(response.json['form'], {})
assert response.json['method'] == 'GET', 'Method was not GET'
assert response.json['json'] is None, 'JSON was not empty'
assert response.json['args'] == {'hello': ['world']}
data = {'foo': 'bar'}
request, response = api_app.test_client.post('/api/echo', data=json.dumps(data))
assert response.status_code == HTTPStatus.OK
self.assertDictContains(response.json['form'], {})
assert response.json['method'] == 'POST', 'Method was not POST'
assert response.json['json'] == data, 'JSON was not data'
assert response.json['args'] == {}
def test_valid_regex(self):
"""
The endpoint should only return valid if the regex is valid.
"""
data = {'regex': 'foo'}
request, response = api_app.test_client.post('/api/valid_regex', data=json.dumps(data))
assert response.status_code == HTTPStatus.OK
assert json.loads(response.body) == {'valid': True, 'regex': 'foo'}
data = {'regex': '.*(title match).*'}
request, response = api_app.test_client.post('/api/valid_regex', data=json.dumps(data))
assert response.status_code == HTTPStatus.OK
assert json.loads(response.body) == {'valid': True, 'regex': '.*(title match).*'}
data = {'regex': '.*(missing parenthesis.*'}
request, response = api_app.test_client.post('/api/valid_regex', data=json.dumps(data))
assert response.status_code == HTTPStatus.BAD_REQUEST
assert json.loads(response.body) == {'valid': False, 'regex': '.*(missing parenthesis.*'}
def test_get_settings(self):
request, response = api_app.test_client.get('/api/settings')
self.assertOK(response)
@wrap_test_db
def test_get_downloads(self):
request, response = api_app.test_client.get('/api/download')
self.assertDictContains(response.json, {'once_downloads': [], 'recurring_downloads': []})
with get_db_session(commit=True) as session:
d1 = Download(url='https://example.com/1', status='complete')
d2 = Download(url='https://example.com/2', status='pending')
d3 = Download(url='https://example.com/3', status='deferred')
d4 = Download(url='https://example.com/4', status='deferred')
d5 = Download(url='https://example.com/5', status='new')
d6 = Download(url='https://example.com/6', status='deferred', frequency=60)
d7 = Download(url='https://example.com/7', status='pending', frequency=60)
session.add_all([d1, d2, d3, d4, d5, d6, d7])
request, response = api_app.test_client.get('/api/download')
expected = [
{'url': 'https://example.com/2', 'status': 'pending'},
{'url': 'https://example.com/5', 'status': 'new'},
{'url': 'https://example.com/3', 'status': 'deferred'},
{'url': 'https://example.com/4', 'status': 'deferred'},
{'url': 'https://example.com/1', 'status': 'complete'},
]
for download, expected in zip_longest(response.json['once_downloads'], expected):
self.assertDictContains(download, expected)
expected = [
{'url': 'https://example.com/7', 'status': 'pending'},
{'url': 'https://example.com/6', 'status': 'deferred'},
]
for download, expected in zip_longest(response.json['recurring_downloads'], expected):
self.assertDictContains(download, expected)
@wrap_test_db
def test_downloads_sorter(self):
with get_db_session(commit=True) as session:
downloads = [
dict(status='complete', last_successful_download=strptime('2020-01-01 00:00:01')),
dict(status='complete', last_successful_download=strptime('2020-01-01 00:00:03')),
dict(status='complete', last_successful_download=strptime('2020-01-01 00:00:02')),
# last_successful_download are equal, so id is used next.
dict(status='pending', last_successful_download=strptime('2020-01-01 00:00:01'), url='1'),
dict(status='pending', last_successful_download=strptime('2020-01-01 00:00:01'), url='2'),
dict(status='pending', last_successful_download=strptime('2020-01-01 00:00:04')),
dict(status='failed'),
dict(status='failed', last_successful_download=strptime('2020-01-01 00:00:01')),
]
for download in downloads:
session.add(Download(url=download.pop('url', 'https://example.com'), **download))
expected = [
dict(status='pending', last_successful_download=strptime('2020-01-01 00:00:04').timestamp()),
dict(status='pending', last_successful_download=strptime('2020-01-01 00:00:01').timestamp(), url='1'),
dict(status='pending', last_successful_download=strptime('2020-01-01 00:00:01').timestamp(), url='2'),
dict(status='failed'),
dict(status='failed', last_successful_download=strptime('2020-01-01 00:00:01').timestamp()),
dict(status='complete', last_successful_download=strptime('2020-01-01 00:00:03').timestamp()),
dict(status='complete', last_successful_download=strptime('2020-01-01 00:00:02').timestamp()),
dict(status='complete', last_successful_download=strptime('2020-01-01 00:00:01').timestamp()),
]
request, response = api_app.test_client.get('/api/download')
once_downloads = response.json['once_downloads']
for d1, d2 in zip_longest(expected, once_downloads):
self.assertDictContains(d2, d1)
@wrap_test_db
def test_downloads_sorter_recurring(self):
with get_db_session(commit=True) as session:
downloads = [
dict(status='complete', frequency=1),
dict(status='complete', frequency=2, next_download=strptime('2020-01-01 00:00:01')),
dict(status='complete', frequency=2, next_download=strptime('2020-01-01 00:00:02')),
dict(status='pending', frequency=1),
dict(status='pending', frequency=4),
dict(status='failed', frequency=1),
dict(status='failed', frequency=1),
]
for download in downloads:
session.add(Download(url='https://example.com', **download))
expected = [
dict(status='pending', frequency=1),
dict(status='pending', frequency=4),
dict(status='failed', frequency=1),
dict(status='failed'),
dict(status='complete', frequency=2, next_download=strptime('2020-01-01 00:00:01').timestamp()),
dict(status='complete', frequency=2, next_download=strptime('2020-01-01 00:00:02').timestamp()),
dict(status='complete', frequency=1),
]
request, response = api_app.test_client.get('/api/download')
recurring_downloads = response.json['recurring_downloads']
for d1, d2 in zip_longest(expected, recurring_downloads):
self.assertDictContains(d2, d1)