rt/wrolpi
1
0
Fork 0
mirror of https://github.com/lrnselfreliance/wrolpi.git synced 2026-10-25 18:40:06 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

Fixing JS warnings.

Browse source 
Handling all combinations of toggle in light and dark.

Changing CPUUsageProgress colors to be more readable in dark mode.

Moving theme handling into Theme.js.  Importing element wrappers from Theme.js to handle element theming.

Changing video menu to dark mode.

Creating ThemeContext to share Dark Mode setting.

Removing duplicate code for search functions.

Displaying X when downloads are disabled.

Using react context to share settings across components.

Refusing to start workers when WROL Mode is enabled.

Changing warning to debug.

Queuing recurring downloads first.

Adding Video.url index.

Adding special handling for youtube shorts to videos.

Reworking Downloads to have one worker per process.

Testing creating downloads and getting downloaders.

Testing video validation and poster generation.  Testing video statistics.

Increasing inventory test coverage.

Adding pytest.ini to automatically detect async test.

Adding more archive test coverage.

Testing Archive CRUD.

Catching and logging when encountering error during indexing.

Using FileIcon when Archive has no screenshot.

A Video will be assigned to a Channel by its info json first, then to its directory.

Including suffix in File json.

Adding index for File.suffix.

Gathering suffix during indexing.

Passing entire file down to FileIcon.

Moving logging up.

Returning File.associated in file json.

Associated files are last in file search.

Removing duplicate refreshing code by creating new _refresh_directory function.

Changing File.idempotency to a datetime to avoid having to clear idempotency.

Truncating indexes to avoid error with large files.

Don't overwrite video data when info json is empty.

Using already_downloaded for channel videos.

Awaiting refresh in Channel.refresh.

Testing that videos already downloaded aren't downloaded again.

Adding indexes for video/archive referenced paths.

Changing grid width for mobile or computer.

Adding button to refresh specific directories.

Restoring Channel edit link.

Fixing caption path in video_modeler.

Adding ZIP files to mimetype colors.

Using model instead of first file's model.

Fetching File models in batches to avoid issuing a SELECT for every File with a model.

Flushing generated poster file.

Getting Channel in VideoPage, relying on Video.channl_id, falling back to URL's channel_id.

Attaching Video to it's Channel during download.

Using ExternalCardLink for ArchivePage singlefile link.

Validating and indexing videos after download.

Handling glob when file name contains matching characters.

Reworking and testing split_path_stem_and_suffix and glob_shared_stem.

Adding associated video files to DB during upsert.

Adding pending downloads to dashboard.

Hiding status when download is open.

Handling when load is empty.

Adding load statistics to Dashboard.

Limiting concurrent after_refresh functions.

Handling when no onSelect is declared.

Adding ImageRow for FileTable.

Moving image date into Card.Meta.

Using ExternalLink for FileCard.

Using ExternalLink for ImageCard.

Handling when error occurs during refresh of a directory.  Committing during refresh to avoid losing all files if an error occurs.

Removing api health check to avoid spamming logs.

Creating ImageCard to display image posters when they aren't too large.

Removing archive.domain_id not null.

Reworking file refresh to process all files in a single directory.

Bumping version for major file rework.

No longer checking for video corruption.

Adding duration by default to speed up tests.

Using register_indexer for all indexers.

Adding CPU usage progress bar in Welcome.

Modifying TextIndexer to ignore large text files.

Counting all files during refresh.

Adding refresh_logger to clarify logging.  Checking for wrol_mode before refresh functions.

Removing unused idempotency from video/channel tables.

Using magic module to speed up mimetype identification.

All tests pass.

Archive tests pass.

Changing video get endpoint to return 'file' to avoid ambiguity.

Video tests pass.

Simplifying regex filtering.

Using regex to ignore case when filtering.

Converting all channel classes to functions.

Restoring channel refresh and download buttons.

Matching format of Archive and Video pages.

Adding sorting to Archives search.

Reworking video orders for file search.  Using SearchInput for videos search.

Archives can be searched on the Archives page.  Domains can be filtered.

Reworking Archive search to use Files.  Videos, Archives and Global search use the FilesView.

Videos are claimed by their Channel based on the Channel's directory.  Prev and Next video cards are displayed on the VideoPage.

Creating useVideo for reworked VideoWrapper and VideoPage.

Adding Paginator to FilesView.

Moving file refresh button out of manage tab.  Removing Files page tabs.  Limiting refresh to one at a time.

Adding class for card icon to match poster height.

Video and Archives always re-claim their files on refresh.

Archive files are indexed and claimed.

FileView table is compact and striped.

Videos page uses FileView.

Updated navlinks to router v6.

Removing ! character from file names.

Disabling select button when cards are visible.

Adding show/hide button for FileTable selection.

The Archive Page is displayed.

Adding mimetype dropdown.

Upgrading to React Router v6 for useSearchParams.

Validating video during modeling.  Removing screenshot during corruption check because it was very slow.  Updating app to use video files as search results.

Updating video search to return Files.

Video and WROLPi tests pass.

Creating function to refresh files within a channel's directory.

Removing troubleshooting print.  Updating result order in test.

Reworking video_factory to handle a specific video file location.

Returning video model on file search.

Getting session using video_file.

Screenshotting when checking for video corruption.

Reducing ffmpeg calls by gathering duration during poster generation.

Moving video file matching into model.

VideoIndexer is moved into video module.  Video files are validated.

Moving files module into wrolpi in preparation to use it in modules.

Title and contents are indexed for text file.

Files are indexed on refresh by using their indexer.

Zip files are indexed using their contents.

Testing multiple zip files.

Adding unique to file.path.  Checking that the correct indexer is used.

Adding index to file.modification_datetime and file.size.

Handling file info gathering in File.do_index.  Only reading contents of file if file.indexed is False.

Video model is found during file refresh.  A Video record is created that has the associated files.

Videos without associated files are found.

Videos are found using registered video_modeler() rather than in refresh code.

Removing old video columns.

Handling video info json files.

Trying to import each module individually.  Will continue to import modules even if one fails.

Reworking refresh so it performs less DB calls.

Converting file.path to primary key.

file.path is always a pathlib.Path.

Updating video_factory to use new file system, testing it.

Bump sanic from 21.9.1 to 21.12.2 in /docker/archive

Bumps [sanic](https://github.com/sanic-org/sanic) from 21.9.1 to 21.12.2.
- [Release notes](https://github.com/sanic-org/sanic/releases)
- [Changelog](https://github.com/sanic-org/sanic/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/sanic-org/sanic/compare/v21.9.1...v21.12.2)

---
updated-dependencies:
- dependency-name: sanic
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
This commit is contained in:
dependabot[bot] 2022-08-06 05:22:07 +00:00 • committed by lrnselfreliance
parent 1ede5f3f55
commit d4ed5fa5ce
98 changed files with 7441 additions and 5257 deletions
Download patch file Download diff file Expand all files Collapse all files

216
alembic/versions/70db5ba8763d_files_rework.py Normal file
View file

@ -0,0 +1,216 @@
"""Files rework. Unifying files for Videos and Archives.
Revision ID: 70db5ba8763d
Revises: aac864072193
Create Date: 2022-07-27 20:11:30.990396
"""
import os
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = '70db5ba8763d'
down_revision = 'aac864072193'
branch_labels = None
depends_on = None
DOCKERIZED = True if os.environ.get('DOCKER', '').lower().startswith('t') else False
def upgrade():
bind = op.get_bind()
session = Session(bind=bind)
# Recreate the file table. Use file.path as the primary key.
session.execute('DROP TABLE IF EXISTS file')
session.execute('''CREATE TABLE file (
path TEXT PRIMARY KEY,
idempotency TIMESTAMP WITH TIME ZONE,
indexed BOOLEAN DEFAULT FALSE,
mimetype TEXT,
suffix TEXT,
model TEXT,
associated BOOLEAN DEFAULT FALSE,
modification_datetime TIMESTAMP WITH TIME ZONE,
size BIGINT,
title TEXT,
a_text TEXT,
b_text TEXT,
c_text TEXT,
d_text TEXT,
textsearch tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, COALESCE(a_text, '')), 'A'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, '')), 'B'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, '')), 'C'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(d_text, '')), 'D'::"char")
) STORED
)''')
session.execute('CREATE INDEX file_mimetype_idx ON file USING BTREE(mimetype)')
session.execute('CREATE INDEX file_textsearch_idx ON file USING GIN(textsearch)')
session.execute('CREATE INDEX file_modification_datetime_idx ON file USING BTREE(modification_datetime)')
session.execute('CREATE INDEX file_size_idx ON file USING BTREE(size)')
session.execute('CREATE INDEX file_model_idx ON file USING BTREE(model)')
session.execute('CREATE INDEX file_idempotency_idx ON file USING BTREE(idempotency)')
session.execute('CREATE INDEX file_suffix_idx ON file USING BTREE(suffix)')
# Videos are fundamentally different, replace the video table.
session.execute('DROP TABLE IF EXISTS video')
session.execute('''
CREATE TABLE video (
id BIGSERIAL PRIMARY KEY,
title TEXT,
source_id TEXT,
channel_id INTEGER REFERENCES channel(id),
upload_date TIMESTAMPTZ,
duration INTEGER,
favorite TIMESTAMPTZ,
size BIGINT,
viewed TIMESTAMPTZ,
view_count INTEGER,
url TEXT,
censored BOOLEAN DEFAULT FALSE,
validated BOOLEAN DEFAULT FALSE,
modification_datetime TIMESTAMPTZ,
video_path TEXT REFERENCES file(path),
info_json_path TEXT REFERENCES file(path),
poster_path TEXT REFERENCES file(path),
caption_path TEXT REFERENCES file(path)
)''')
session.execute('''
CREATE INDEX video_caption_path_idx ON video(caption_path);
CREATE INDEX video_info_json_path_idx ON video(info_json_path);
CREATE INDEX video_poster_path_idx ON video(poster_path);
CREATE INDEX video_video_path_idx ON video(video_path);
CREATE INDEX video_url_idx ON video(url);
''')
session.execute('ALTER TABLE channel DROP COLUMN IF EXISTS idempotency')
session.execute('DELETE FROM archive') # noqa
session.execute('DELETE FROM domains') # noqa
session.execute('DROP TABLE IF EXISTS archive')
session.execute('DROP TABLE IF EXISTS domains')
session.execute('''
CREATE TABLE domains (
id SERIAL PRIMARY KEY,
domain TEXT NOT NULL,
directory TEXT
)''')
session.execute('''
CREATE TABLE archive (
id SERIAL PRIMARY KEY,
domain_id INTEGER REFERENCES domains(id),
title TEXT,
archive_datetime timestamptz,
url TEXT,
validated BOOLEAN DEFAULT FALSE,
singlefile_path TEXT REFERENCES file(path),
readability_path TEXT REFERENCES file(path),
readability_json_path TEXT REFERENCES file(path),
readability_txt_path TEXT REFERENCES file(path),
screenshot_path TEXT REFERENCES file(path)
)''')
session.execute('''
CREATE INDEX archive_singlefile_path ON archive(singlefile_path);
CREATE INDEX archive_readability_path ON archive(readability_path);
CREATE INDEX archive_readability_json_path ON archive(readability_json_path);
CREATE INDEX archive_readability_txt_path ON archive(readability_txt_path);
CREATE INDEX archive_screenshot_path ON archive(screenshot_path);
''')
def downgrade():
bind = op.get_bind()
session = Session(bind=bind)
session.execute('''
DROP INDEX IF EXISTS archive_singlefile_path;
DROP INDEX IF EXISTS archive_readability_path;
DROP INDEX IF EXISTS archive_readability_json_path;
DROP INDEX IF EXISTS archive_readability_txt_path;
DROP INDEX IF EXISTS archive_screenshot_path;
''')
session.execute('DELETE FROM archive') # noqa
session.execute('''
ALTER TABLE archive DROP COLUMN IF EXISTS singlefile_path;
ALTER TABLE archive DROP COLUMN IF EXISTS readability_path;
ALTER TABLE archive DROP COLUMN IF EXISTS readability_json_path;
ALTER TABLE archive DROP COLUMN IF EXISTS readability_txt_path;
ALTER TABLE archive DROP COLUMN IF EXISTS screenshot_path;
ALTER TABLE archive DROP COLUMN IF EXISTS associated;
''')
session.execute('''
ALTER TABLE archive ADD COLUMN IF NOT EXISTS singlefile_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_json_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS readability_txt_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS screenshot_path TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS contents TEXT;
ALTER TABLE archive ADD COLUMN IF NOT EXISTS status TEXT;
''')
session.execute('ALTER TABLE archive ALTER COLUMN domain_id SET NOT NULL')
session.execute('ALTER TABLE channel ADD COLUMN IF NOT EXISTS idempotency TEXT')
session.execute('DELETE FROM video') # noqa
session.execute('''
ALTER TABLE video DROP COLUMN IF EXISTS caption_path;
ALTER TABLE video DROP COLUMN IF EXISTS info_json_path;
ALTER TABLE video DROP COLUMN IF EXISTS poster_path;
ALTER TABLE video DROP COLUMN IF EXISTS video_path;
ALTER TABLE video DROP COLUMN IF EXISTS video_path;
''')
session.execute('''
ALTER TABLE video ADD COLUMN IF NOT EXISTS idempotency TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS caption_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS description_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS info_json_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS poster_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS video_path TEXT;
ALTER TABLE video ADD COLUMN IF NOT EXISTS caption TEXT;
''')
session.execute('''
DROP INDEX IF EXISTS video_caption_path_idx;
DROP INDEX IF EXISTS video_info_json_path_idx;
DROP INDEX IF EXISTS video_poster_path_idx;
DROP INDEX IF EXISTS video_video_path_idx;
DROP INDEX IF EXISTS video_url_idx;
''')
session.execute('''
ALTER TABLE video
ADD COLUMN textsearch tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, title), 'A') ||
setweight(to_tsvector('english'::regconfig, caption), 'D')
) STORED
''')
session.execute('CREATE INDEX IF NOT EXISTS video_textsearch_idx ON video USING GIN(textsearch)')
session.execute('DROP INDEX IF EXISTS file_mimetype_idx')
session.execute('DROP INDEX IF EXISTS file_textsearch_idx')
session.execute('DROP INDEX IF EXISTS file_modification_datetime_idx')
session.execute('DROP INDEX IF EXISTS file_size_idx')
session.execute('DROP INDEX IF EXISTS file_model_idx')
session.execute('DROP INDEX IF EXISTS file_idempotency_idx')
session.execute('DROP INDEX IF EXISTS file_suffix_idx')
# Restore the old file table.
session.execute('DROP TABLE IF EXISTS file')
session.execute('''CREATE TABLE file (
id SERIAL PRIMARY KEY,
path TEXT,
title TEXT,
mimetype TEXT,
size BIGINT,
modification_datetime TIMESTAMP WITH TIME ZONE,
idempotency TEXT DEFAULT NULL,
textsearch tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('english'::regconfig, title), 'A'::"char")
) STORED
)''')
session.execute('CREATE UNIQUE INDEX file_path_idx ON file USING BTREE(path)')
session.execute('CREATE INDEX file_mimetype_idx ON file USING BTREE(mimetype)')
session.execute('CREATE INDEX file_textsearch_idx ON file USING GIN(textsearch)')

2
app/package.json
View file

@ -11,7 +11,7 @@
"react-dropzone": "^11.4.2",
"react-keyed-file-browser": "^1.14.0",
"react-qr-code": "^2.0.7",
"react-router-dom": "^5.2.0",
"react-router-dom": "^6.3.0",
"react-scripts": "^3.4.1",
"react-semantic-toasts": "^0.6.5",
"react-timezone-select": "0.8.4",

6
app/public/index.html
View file

@ -7,9 +7,9 @@
<meta name="theme-color" content="#000000"/>
<meta
name="description"
content="Web site created using create-react-app"
content="WROLPi is a self-contained collection of software (on a Raspberry Pi) to help you survive the world Without Rule of Law."
/>
<!--<link rel="apple-touch-icon" href="logo192.png" />-->
<link rel="apple-touch-icon" href="%PUBLIC_URL%/logo512.png"/>
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
@ -26,7 +26,7 @@
-->
<title>WROLPi - Bring your personal internet, off-grid</title>
</head>
<body style="margin: 0px">
<body style="margin: 0">
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--

BIN
app/public/logo192.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 8.4 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Before After
Before After

BIN
app/public/logo512.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 103 KiB

Before After
Before After

4
app/public/manifest.json
View file

@ -1,6 +1,6 @@
{
"short_name": "React App",
"name": "Create React App Sample",
"short_name": "WROLPi",
"name": "WROLPi web interface",
"icons": [
{
"src": "favicon.ico",

65
app/src/App.css
View file

@ -46,3 +46,68 @@ pre.wrap-text {
.center-me {
margin: 0 auto;
}
div.inverted {
background-color: #1B1C1D;
}
.toggle {
position: relative;
display: inline-block;
width: 49px;
height: 23px;
}
.toggle input {
opacity: 0;
width: 0;
height: 0;
}
.slider {
position: absolute;
cursor: pointer;
top: 0;
left: 0;
right: 0;
bottom: 0;
background-color: rgb(166, 166, 166);
-webkit-transition: .4s;
transition: .4s;
border-radius: 34px;
}
.slider:before {
position: absolute;
content: "";
height: 22px;
width: 22px;
left: 0;
bottom: 0;
background-color: white;
-webkit-transition: .4s;
transition: .4s;
border-radius: 50%;
}
.slider.disabled:before {
background-color: #CCCCCC;
}
input:checked + .slider {
background-color: #2185d0;
}
input.inverted:checked + .slider.inverted {
background-color: rgba(33, 133, 208, 0.4);
}
input.disabled + .slider {
background-color: rgb(147, 147, 147);
}
input:checked + .slider:before {
-webkit-transform: translateX(26px);
-ms-transform: translateX(26px);
transform: translateX(26px);
}

203
app/src/App.js
View file

@ -1,60 +1,103 @@
import React from 'react';
import React, {useContext, useEffect, useState} from 'react';
import './App.css';
import {NavBar} from "./components/Nav";
import {Route, Switch, useHistory} from "react-router-dom";
import {VideosPreview, VideosRoute, VideoWrapper} from "./components/Videos";
import Admin from "./components/Admin";
import {Container, Divider, Header} from "semantic-ui-react";
import {Link, Route, Routes} from "react-router-dom";
import {VideosRoute, VideoWrapper} from "./components/Videos";
import Admin, {CPUUsageProgress} from "./components/Admin";
import {Container, Divider} from "semantic-ui-react";
import 'semantic-ui-offline/semantic.min.css';
import {SemanticToastContainer} from 'react-semantic-toasts';
import 'react-semantic-toasts/styles/react-semantic-alert.css';
import {AppsRoute} from "./components/Apps";
import {InventoryRoute} from "./components/Inventory";
import {ArchiveRoute, ArchivesList} from "./components/Archive";
import {Downloads, Saver} from "./components/Upload";
import {MoreButton, PageContainer, SearchInput} from "./components/Common";
import {FilesRoute} from "./components/Files";
import {useSearch, useUp, useVersion} from "./hooks/customHooks";
import {ArchiveRoute} from "./components/Archive";
import {DownloadMenu} from "./components/Upload";
import {LoadStatistic, PageContainer, SearchInput, useTitle} from "./components/Common";
import {FilesRoute, FilesSearchView} from "./components/Files";
import {useSearchFiles, useSettingsInterval, useStatus} from "./hooks/customHooks";
import {MapRoute} from "./components/Map";
import {darkTheme, lightTheme, SettingsContext, ThemeContext} from "./contexts/contexts";
import {Header, Segment, Statistic, StatisticGroup} from "./components/Theme";
function Welcome() {
const {searchStr, setSearchStr, archives, videos} = useSearch();
const history = useHistory();
useUp();
function Dashboard() {
useTitle('Dashboard');
let body = (
<Downloads/>
);
const {searchStr, setSearchStr} = useSearchFiles();
const [downloadOpen, setDownloadOpen] = useState(false);
const onDownloadOpen = (name) => setDownloadOpen(!!name);
const {wrol_mode} = useContext(SettingsContext);
const downloads = <DownloadMenu onOpen={onDownloadOpen}/>;
// Only show dashboard parts if not searching.
let body;
if (searchStr) {
let archiveMore = () => {
history.push(`/archive?q=${searchStr}`);
};
let videosMore = () => {
history.push(`/videos?q=${searchStr}&o=rank`);
};
body = (<>
<Header as='h2'>Archives</Header>
<ArchivesList archives={archives} searchStr={searchStr}/>
<MoreButton onClick={archiveMore} disabled={!archives || archives.length === 0}/>
<Divider/>
<Header as='h2'>Videos</Header>
<VideosPreview videos={videos}/>
<MoreButton onClick={videosMore} disabled={!videos || videos.length === 0}/>
</>);
body = <FilesSearchView showLimit={true} showSelect={true} showSelectButton={true}/>;
} else {
body = <>
{!wrol_mode && downloads}
{/* Hide Status when user is starting a download */}
{!downloadOpen && <DashboardStatus/>}
</>;
}
return (
<PageContainer>
<SearchInput initValue={searchStr} onSubmit={setSearchStr} size='big' placeholder='Search Everywhere...'/>
<SearchInput clearable
searchStr={searchStr}
onSubmit={setSearchStr}
size='large'
placeholder='Search Everywhere...'
actionIcon='search'
style={{marginBottom: '2em'}}
/>
{body}
</PageContainer>
)
}
function DashboardStatus() {
const {status} = useStatus();
let percent = 0;
let load = {};
let cores = 0;
let pending_downloads = '?';
if (status && status['cpu_info']) {
percent = status['cpu_info']['percent'];
load = status['load'];
cores = status['cpu_info']['cores'];
pending_downloads = status['downloads']['pending_downloads'];
}
const {download_manager_disabled, download_manager_stopped} = useContext(SettingsContext);
if (pending_downloads === 0 && (download_manager_disabled || download_manager_stopped)) {
pending_downloads = 'x';
}
return <Segment>
<Link to='/admin/status'>
<Header as='h2'>Status</Header>
<CPUUsageProgress value={percent} label='CPU Usage'/>
<Header as='h3'>Load</Header>
<StatisticGroup size='mini'>
<LoadStatistic label='1 Minute' value={load['minute_1']} cores={cores}/>
<LoadStatistic label='5 Minute' value={load['minute_5']} cores={cores}/>
<LoadStatistic label='15 Minute' value={load['minute_15']} cores={cores}/>
</StatisticGroup>
</Link>
<Divider/>
<Link to='/admin'>
<StatisticGroup size='mini'>
<Statistic label='Downloads' value={pending_downloads}/>
</StatisticGroup>
</Link>
</Segment>;
}
function PageNotFound() {
return (
<Container fluid>
@ -65,35 +108,75 @@ function PageNotFound() {
}
function Footer() {
const version = useVersion();
const {t} = useContext(ThemeContext);
const {version} = useContext(SettingsContext);
return <Container textAlign='center' style={{marginTop: '1.5em', marginBottom: '1em'}}>
WROLPi v{version} <a href='https://github.com/lrnselfreliance/wrolpi'>GitHub</a>
<span {...t}>WROLPi v{version} <a href='https://github.com/lrnselfreliance/wrolpi'>GitHub</a></span>
</Container>
}
export default function App() {
const {settings} = useSettingsInterval();
const [i, setI] = useState({});
const [s, setS] = useState({});
const [t, setT] = useState({});
const [theme, setTheme] = useState();
const setDarkTheme = () => {
console.debug('setDarkTheme');
setI({inverted: true});
setS({style: {backgroundColor: '#1B1C1D', color: '#dddddd'}});
setT({style: {color: '#dddddd'}});
setTheme(darkTheme);
document.body.style.background = '#1B1C1D';
}
const setLightTheme = () => {
console.debug('setLightTheme');
setI({inverted: undefined});
setS({});
setT({});
setTheme(lightTheme);
document.body.style.background = '#FFFFFF';
}
useEffect(() => {
window.matchMedia('(prefers-color-scheme: dark)').matches && setDarkTheme();
window.matchMedia('(prefers-color-scheme: dark)').addEventListener(
'change', (e) => e.matches && setDarkTheme());
window.matchMedia('(prefers-color-scheme: light)').addEventListener(
'change', (e) => e.matches && setLightTheme());
}, []);
const themeValue = {i, s, t, theme, setDarkTheme, setLightTheme};
return (
<>
<header>
<NavBar/>
</header>
<>
<Switch>
<Route path='/videos/video/:video_id' exact component={VideoWrapper}/>
<Route path='/videos/channel/:channel_id/video/:video_id' exact component={VideoWrapper}/>
<Route path="/" exact={true} component={Welcome}/>
<Route path="/videos" component={VideosRoute}/>
<Route path="/admin" component={Admin}/>
<Route path="/apps" component={AppsRoute}/>
<Route path="/inventory" component={InventoryRoute}/>
<Route path='/archive' component={ArchiveRoute}/>
<Route path='/map' component={MapRoute}/>
<Route path='/files' component={FilesRoute}/>
<Route component={PageNotFound}/>
</Switch>
</>
<SemanticToastContainer position="top-right"/>
<Footer/>
</>
<ThemeContext.Provider value={themeValue}>
<SettingsContext.Provider value={settings}>
<header>
<NavBar/>
</header>
<>
<Routes>
<Route path='/videos/video/:videoId' exact element={<VideoWrapper/>}/>
<Route path='/videos/channel/:channelId/video/:videoId' exact element={<VideoWrapper/>}/>
<Route path="/" exact element={<Dashboard/>}/>
<Route path="/videos/*" element={<VideosRoute/>}/>
<Route path="/admin/*" element={<Admin/>}/>
<Route path="/apps/*" element={<AppsRoute/>}/>
<Route path="/inventory/*" element={<InventoryRoute/>}/>
<Route path='/archive/*' element={<ArchiveRoute/>}/>
<Route path='/map/*' element={<MapRoute/>}/>
<Route path='/files/*' element={<FilesRoute/>}/>
<Route element={<PageNotFound/>}/>
</Routes>
</>
<SemanticToastContainer position="top-right"/>
<Footer/>
</SettingsContext.Provider>
</ThemeContext.Provider>
);
}

96
app/src/api.js
View file

@ -21,7 +21,7 @@ function timeoutPromise(ms, promise) {
})
}
async function apiCall(url, method, body, ms = 10000) {
async function apiCall(url, method, body, ms = 20000) {
let init = {method};
if (body !== undefined) {
init.body = JSON.stringify(body);
@ -33,7 +33,7 @@ async function apiCall(url, method, body, ms = 10000) {
try {
// await the response or error.
let response = await timeoutPromise(ms, promise);
if (200 <= response.status < 300) {
if (200 <= response.status && response.status < 300) {
// Request was successful.
return response;
}
@ -44,7 +44,7 @@ async function apiCall(url, method, body, ms = 10000) {
toast({
type: 'error',
title: 'WROL Mode is enabled!',
description: 'This functionality is not enabled while WROL Mode is enabled!',
description: 'This functionality is disabled while WROL Mode is enabled!',
time: 5000,
});
}
@ -97,8 +97,8 @@ export async function createChannel(channel) {
return await apiPost(`${VIDEOS_API}/channels`, channel);
}
export async function deleteChannel(id) {
return apiDelete(`${VIDEOS_API}/channels/${id}`);
export async function deleteChannel(channelId) {
return apiDelete(`${VIDEOS_API}/channels/${channelId}`);
}
export async function getChannels() {
@ -120,24 +120,25 @@ export async function getChannel(id) {
return (await response.json())['channel'];
}
export async function searchVideos(offset, limit, channel_id, searchStr, order_by, filters) {
export async function searchVideos(offset, limit, channelId, searchStr, order_by, filters = []) {
// Build a search query to retrieve a list of videos from the API
offset = offset || 0;
limit = limit || DEFAULT_LIMIT;
let body = {offset, limit, filters};
offset = parseInt(offset || 0);
limit = parseInt(limit || DEFAULT_LIMIT);
let body = {offset, limit, filters, order_by: order_by || 'rank'};
if (searchStr) {
body.search_str = searchStr;
body['search_str'] = searchStr;
}
if (channel_id) {
body.channel_id = channel_id;
if (channelId) {
body['channel_id'] = parseInt(channelId);
}
body.order_by = order_by ? order_by : 'rank'
console.debug('searching videos', body);
let response = await apiPost(`${VIDEOS_API}/search`, body);
if (response.status === 200) {
let data = await response.json();
return [data['videos'], data['totals']['videos']];
return [data['files'], data['totals']['files']];
} else {
toast({
type: 'error',
@ -145,13 +146,14 @@ export async function searchVideos(offset, limit, channel_id, searchStr, order_b
description: 'Failed to search videos. See server logs.',
time: 5000,
});
return [[], 0];
}
}
export async function getVideo(video_id) {
let response = await apiGet(`${VIDEOS_API}/video/${video_id}`);
let data = await response.json();
return [data['video'], data['prev'], data['next']];
return [data['file'], data['prev'], data['next']];
}
export async function deleteVideo(video_id) {
@ -221,7 +223,6 @@ export async function getDownloads() {
}
}
export async function validateRegex(regex) {
let response = await apiPost(`${API_URI}/valid_regex`, {regex});
return (await response.json())['valid'];
@ -247,18 +248,8 @@ export async function getStatistics() {
}
}
export async function refresh() {
let response = await apiPost(`${VIDEOS_API}/refresh`);
return await response.json();
}
export async function download() {
let response = await apiPost(`${VIDEOS_API}/download`);
return await response.json();
}
export async function downloadChannel(id) {
let url = `${VIDEOS_API}/download/${id}`;
export async function downloadChannel(channelId) {
let url = `${VIDEOS_API}/channels/download/${channelId}`;
let response = await apiPost(url);
let json = await response.json();
if (response.status === 400 && json['code'] === 30) {
@ -271,8 +262,8 @@ export async function downloadChannel(id) {
}
}
export async function refreshChannel(id) {
let url = `${VIDEOS_API}/refresh/${id}`;
export async function refreshChannel(channelId) {
let url = `${VIDEOS_API}/channels/refresh/${channelId}`;
await fetch(url, {method: 'POST'});
}
@ -381,7 +372,6 @@ export async function getItems(inventoryId) {
}
export async function saveItem(inventoryId, item) {
item = emptyToNull(item);
return await apiPost(`${API_URI}/inventory/${inventoryId}/item`, item);
}
@ -399,7 +389,7 @@ export async function deleteArchive(archive_id) {
return await apiDelete(`${API_URI}/archive/${archive_id}`);
}
export async function searchArchives(offset, limit, domain = null, searchStr = null) {
export async function searchArchives(offset, limit, domain, searchStr, order) {
// Build a search query to retrieve a list of videos from the API
offset = parseInt(offset || 0);
limit = parseInt(limit || DEFAULT_LIMIT);
@ -410,7 +400,11 @@ export async function searchArchives(offset, limit, domain = null, searchStr = n
if (searchStr) {
body['search_str'] = searchStr;
}
if (order) {
body['order_by'] = order;
}
console.debug('searching archives', body);
let response = await apiPost(`${ARCHIVES_API}/search`, body);
if (response.status === 200) {
let data = await response.json();
@ -425,10 +419,6 @@ export async function searchArchives(offset, limit, domain = null, searchStr = n
}
}
export async function refreshArchives() {
return await apiPost(`${ARCHIVES_API}/refresh`);
}
export async function fetchDomains() {
let response = await apiGet(`${ARCHIVES_API}/domains`);
if (response.status === 200) {
@ -448,7 +438,7 @@ export async function getArchive(archiveId) {
const response = await apiGet(`${ARCHIVES_API}/${archiveId}`);
if (response.status === 200) {
const data = await response.json();
return [data['archive'], data['alternatives']];
return [data['file'], data['alternatives']];
} else {
toast({
type: 'error',
@ -502,8 +492,15 @@ const replaceFileDatetimes = (files) => {
return files;
}
export async function filesSearch(offset, limit, searchStr) {
export async function filesSearch(offset, limit, searchStr, mimetype, model) {
const body = {search_str: searchStr, offset: parseInt(offset), limit: parseInt(limit)};
if (mimetype) {
body['mimetype'] = mimetype;
}
if (model) {
body['model'] = model;
}
console.debug('searching files', body);
const response = await apiPost(`${API_URI}/files/search`, body);
if (response.status === 200) {
@ -518,6 +515,7 @@ export async function filesSearch(offset, limit, searchStr) {
description: 'Cannot search files. See server logs.',
time: 5000,
});
return [null, null];
}
}
@ -525,6 +523,21 @@ export async function refreshFiles() {
return await apiPost(`${API_URI}/files/refresh`);
}
export async function refreshDirectoryFiles(directory) {
let body = {directory};
try {
return await apiPost(`${API_URI}/files/refresh/directory`, body);
} catch (e) {
console.error(e);
toast({
type: 'error',
title: 'Unable to refresh directory',
description: 'Cannot refresh the files in the directory. See server logs.',
time: 5000,
});
}
}
export async function getFiles(directories) {
let body = {directories: directories || []};
let response = await apiPost(`${API_URI}/files`, body);
@ -538,13 +551,6 @@ export async function deleteFile(file) {
await apiPost(`${API_URI}/files/delete`, body);
}
export async function getVersion() {
let response = await apiGet(`${API_URI}/settings`);
if (response.status === 200) {
let data = await response.json();
return data['version'];
}
}
export async function getHotspotStatus() {
let response = await getSettings();

450
app/src/components/Admin.js
View file

@ -1,24 +1,23 @@
import React from 'react';
import React, {useContext} from 'react';
import {
Button,
Checkbox,
Confirm,
Container,
Dimmer,
Divider,
Form,
Header,
Icon,
Loader,
Modal,
Placeholder, Progress, Statistic,
Table
PlaceholderLine,
TableBody,
TableCell,
TableHeader,
TableHeaderCell,
TableRow
} from "semantic-ui-react";
import {
clearCompletedDownloads,
clearFailedDownloads,
getDownloads,
getSettings, getStatus,
getSettings,
getStatus,
killDownload,
postDownload,
saveSettings
@ -26,18 +25,38 @@ import {
import TimezoneSelect from 'react-timezone-select';
import {
DisableDownloadsToggle,
ExternalCardLink,
HelpPopup,
HotspotToggle, humanFileSize,
HotspotToggle,
humanFileSize,
LoadStatistic,
PageContainer,
secondsToDate,
secondsToFrequency,
TabLinks,
textEllipsis,
ThrottleToggle,
Toggle,
WROLModeMessage
} from "./Common";
import {Route} from "react-router-dom";
import {Route, Routes} from "react-router-dom";
import QRCode from "react-qr-code";
import {ThemeContext} from "../contexts/contexts";
import {useDownloads} from "../hooks/customHooks";
import {
Button,
Form,
FormField,
FormGroup,
FormInput,
Header,
Loader,
Placeholder,
Progress,
Statistic,
StatisticGroup,
Table
} from "./Theme";
class Settings extends React.Component {
@ -150,54 +169,53 @@ class Settings extends React.Component {
const qrButton = <Button icon style={{marginBottom: '1em'}}><Icon name='qrcode' size='big'/></Button>;
return (
<Container fluid>
return <ThemeContext.Consumer>
{({i, t}) => <Container fluid>
<WROLModeMessage content='Settings are disabled because WROL Mode is enabled.'/>
<HotspotToggle/>
<br/>
<ThrottleToggle style={{marginTop: '0.5em'}}/>
<ThrottleToggle/>
<Divider/>
<Header as='h3'>
The settings for your WROLPi.
</Header>
<p className="text-muted">
<p {...t}>
Any changes will be written to <i>config/wrolpi.yaml</i>.
</p>
<Form id="settings" onSubmit={this.handleSubmit}>
<Checkbox toggle
style={{marginTop: '0.5em', marginBottom: '0.5em'}}
label='Download on Startup'
disabled={disabled || download_on_startup === null}
checked={download_on_startup === true}
onChange={(e, d) => this.handleInputChange(e, 'download_on_startup', d.checked)}
/>
<div style={{margin: '0.5em'}}>
<Toggle
label='Download on Startup'
disabled={disabled || download_on_startup === null}
checked={download_on_startup === true}
onChange={(checked) => this.handleInputChange(null, 'download_on_startup', checked)}
/>
</div>
<br/>
<Checkbox toggle
style={{marginTop: '0.5em', marginBottom: '0.5em'}}
label='WiFi Hotspot on Startup'
disabled={disabled || hotspot_on_startup === null}
checked={hotspot_on_startup === true}
onChange={(e, d) => this.handleInputChange(e, 'hotspot_on_startup', d.checked)}
/>
<div style={{margin: '0.5em'}}>
<Toggle
label='WiFi Hotspot on Startup'
disabled={disabled || hotspot_on_startup === null}
checked={hotspot_on_startup === true}
onChange={(checked) => this.handleInputChange(null, 'hotspot_on_startup', checked)}
/>
</div>
<br/>
<div style={{margin: '0.5em'}}>
<Toggle
label='CPU Power-save on Startup'
disabled={disabled || throttle_on_startup === null}
checked={throttle_on_startup === true}
onChange={(checked) => this.handleInputChange(null, 'throttle_on_startup', checked)}
/>
</div>
<Checkbox toggle
style={{marginTop: '0.5em', marginBottom: '0.5em'}}
label='CPU Power-save on Startup'
disabled={disabled || throttle_on_startup === null}
checked={throttle_on_startup === true}
onChange={(e, d) => this.handleInputChange(e, 'throttle_on_startup', d.checked)}
/>
<Form.Group inline>
<Form.Input
<FormGroup inline>
<FormInput
label={<>
<b>Download Timeout</b>
<b {...t}>Download Timeout</b>
<HelpPopup content='Downloads will be stopped after this many seconds have elapsed.
Downloads will never timeout if this is empty.'/>
</>}
@ -205,30 +223,30 @@ class Settings extends React.Component {
disabled={disabled || download_timeout === null}
onChange={(e, d) => this.handleTimeoutChange(e, 'download_timeout', d.value)}
/>
</Form.Group>
</FormGroup>
<Form.Group inline>
<Form.Input
<FormGroup inline>
<FormInput
label='Hotspot SSID'
value={hotspot_ssid}
disabled={disabled || hotspot_ssid === null}
onChange={(e, d) =>
this.setState({hotspot_ssid: d.value}, this.handleHotspotChange)}
/>
<Form.Input
<FormInput
label='Hotspot Password'
disabled={disabled || hotspot_password === null}
value={hotspot_password}
onChange={(e, d) =>
this.setState({hotspot_password: d.value}, this.handleHotspotChange)}
/>
<Form.Input
<FormInput
label='Hotspot Device'
disabled={disabled || hotspot_password === null}
value={hotspot_device}
onChange={(e, d) => this.handleInputChange(e, 'hotspot_device', d.value)}
/>
</Form.Group>
</FormGroup>
<Modal closeIcon
onClose={() => this.setState({qrOpen: false})}
@ -244,25 +262,26 @@ class Settings extends React.Component {
</Modal.Content>
</Modal>
<Form.Field>
<FormField>
<label>Timezone</label>
<TimezoneSelect
value={timezone}
onChange={(v) => this.handleInputChange(null, 'timezone', v)}
onChange={i => this.handleInputChange(null, 'timezone', i)}
/>
</Form.Field>
</FormField>
<Button color="blue" type="submit" disabled={disabled}>
Save
</Button>
<Dimmer active={pending} inverted>
<Dimmer active={pending}>
<Loader active={pending} size='large'/>
</Dimmer>
</Form>
</Container>
)
}
</ThemeContext.Consumer>
}
}
@ -284,13 +303,13 @@ class WROLMode extends React.Component {
}
}
toggleWROLMode = async () => {
toggleWROLMode = async (checked) => {
// Handle WROL Mode toggling by itself so that other settings are not modified.
let config = {
wrol_mode: !this.state.WROLMode,
wrol_mode: checked,
}
await saveSettings(config);
this.setState({disabled: !this.state.WROLMode, WROLMode: !this.state.WROLMode});
this.setState({disabled: !this.state.WROLMode, WROLMode: checked});
}
render() {
@ -298,39 +317,28 @@ class WROLMode extends React.Component {
return <Loader active inline='centered'/>
}
return (
<Container fluid>
return <ThemeContext.Consumer>
{({i, t}) => <Container fluid>
<Header as="h1">WROL Mode</Header>
<h4>
<Header as='h4'>
Enable read-only mode. No content can be deleted or modified. Enable this when the SHTF and you
want to prevent any potential loss of data.
</h4>
<p>
</Header>
<p {...t}>
Note: User settings and favorites can still be modified.
</p>
<Checkbox toggle
checked={this.state.WROLMode}
onChange={this.toggleWROLMode}
label={this.state.WROLMode ? 'WROL Mode Enabled' : 'WROL Mode Disabled'}
<Toggle
checked={this.state.WROLMode}
onChange={this.toggleWROLMode}
label={this.state.WROLMode ? 'WROL Mode Enabled' : 'WROL Mode Disabled'}
/>
</Container>
)
</Container>}
</ThemeContext.Consumer>
}
}
function LoadStatistic({label, value, cores}) {
let color;
const quarter = cores / 4;
if (cores && value >= (quarter * 3)) {
color = 'red';
} else if (cores && value >= (quarter * 2)) {
color = 'yellow';
}
return <Statistic label={label} value={parseFloat(value).toFixed(1)} color={color}/>;
}
function DriveInfo({used, size, percent, mount}) {
let color;
if (percent >= 90) {
@ -359,20 +367,20 @@ function CPUTemperatureStatistic({value}) {
return <Statistic label='Temp C°' value={value} color={color}/>
}
function CPUUsageProgress({value, label}) {
export function CPUUsageProgress({value, label}) {
if (value === null) {
return <Progress progress={0} color='grey' label='Average CPU Usage ERROR' disabled/>
}
let color = 'grey';
let color = 'black';
if (value >= 90) {
color = 'red';
} else if (value >= 70) {
color = 'orange';
color = 'brown';
} else if (value >= 50) {
color = 'yellow';
color = 'orange';
} else if (value >= 30) {
color = 'olive';
color = 'green';
}
return <Progress percent={value} progress color={color} label={label}/>
}
@ -407,34 +415,35 @@ class Status extends React.Component {
}
render() {
let body = <Loader active inline='centered'>Loading system status</Loader>;
if (this.state.status) {
const {cpu_info, load, drives} = this.state.status;
const {minute_1, minute_5, minute_15} = load;
const {percent, cores, temperature} = cpu_info;
body = (<>
<CPUUsageProgress value={percent} label='CPU Usage'/>
return <ThemeContext.Consumer>
{({i}) => <>
<CPUUsageProgress value={percent} label='CPU Usage'/>
<CPUTemperatureStatistic value={temperature}/>
<Statistic label='Cores' value={cores || '?'}/>
<Statistic.Group>
<LoadStatistic label='1 Minute Load' value={minute_1} cores={cores}/>
<LoadStatistic label='5 Minute Load' value={minute_5} cores={cores}/>
<LoadStatistic label='15 Minute Load' value={minute_15} cores={cores}/>
</Statistic.Group>
<CPUTemperatureStatistic value={temperature}/>
<Statistic label='Cores' value={cores || '?'}/>
<StatisticGroup>
<LoadStatistic label='1 Minute Load' value={minute_1} cores={cores}/>
<LoadStatistic label='5 Minute Load' value={minute_5} cores={cores}/>
<LoadStatistic label='15 Minute Load' value={minute_15} cores={cores}/>
</StatisticGroup>
<Header as='h2'>Drive Usage</Header>
{drives.map(DriveInfo)}
</>);
<Header as='h2'>Drive Usage</Header>
{drives.map((drive) => <DriveInfo key={drive['mount']} {...drive}/>)}
</>}
</ThemeContext.Consumer>
}
return (
<Container fluid style={{marginBottom: '5em'}}>
{body}
return <ThemeContext.Consumer>
{({i}) => <Container fluid style={{marginBottom: '5em'}}>
<Loader active inline='centered'>Loading system status</Loader>;
</Container>
)
}
</ThemeContext.Consumer>
}
}
@ -529,15 +538,15 @@ class DownloadRow extends React.Component {
</Modal.Actions>
</Modal>;
return (
<Table.Row positive={positive}>
<Table.Cell><a href={url} target='_blank'>{textEllipsis(url, 50)}</a></Table.Cell>
<Table.Cell>{secondsToFrequency(frequency)}</Table.Cell>
<Table.Cell>{last_successful_download ? secondsToDate(last_successful_download) : null}</Table.Cell>
<Table.Cell>{secondsToDate(next_download)}</Table.Cell>
<Table.Cell>{error && errorModal}{location && <a href={location}>View</a>}</Table.Cell>
</Table.Row>
);
return <TableRow positive={positive}>
<TableCell>
<ExternalCardLink to={url}>{textEllipsis(url, 50)}</ExternalCardLink>
</TableCell>
<TableCell>{secondsToFrequency(frequency)}</TableCell>
<TableCell>{last_successful_download ? secondsToDate(last_successful_download) : null}</TableCell>
<TableCell>{secondsToDate(next_download)}</TableCell>
<TableCell>{error && errorModal}{location && <a href={location}>View</a>}</TableCell>
</TableRow>
}
}
@ -587,14 +596,14 @@ class StoppableRow extends React.Component {
let {stopOpen, startOpen, errorModalOpen} = this.state;
let completedAtCell = last_successful_download ? secondsToDate(last_successful_download) : null;
let buttonCell = <Table.Cell/>;
let buttonCell = <TableCell/>;
let positive = false;
let negative = false;
let warning = false;
if (status === 'pending' || status === 'new') {
positive = status === 'pending';
buttonCell = (
<Table.Cell>
<TableCell>
<Button
onClick={this.openStop}
color='red'
@ -606,13 +615,13 @@ class StoppableRow extends React.Component {
onCancel={this.closeStop}
onConfirm={this.handleStop}
/>
</Table.Cell>
</TableCell>
);
} else if (status === 'failed' || status === 'deferred') {
negative = status === 'failed';
warning = status === 'deferred';
buttonCell = (
<Table.Cell>
<TableCell>
<Button
onClick={this.openStop}
color='red'
@ -635,13 +644,13 @@ class StoppableRow extends React.Component {
onCancel={this.closeStart}
onConfirm={this.handleStart}
/>
</Table.Cell>
</TableCell>
);
} else if (status === 'complete' && location) {
buttonCell = (
<Table.Cell>
<TableCell>
<a href={location}>View</a>
</Table.Cell>
</TableCell>
);
}
if (error) {
@ -665,138 +674,97 @@ class StoppableRow extends React.Component {
}
return (
<Table.Row positive={positive} negative={negative} warning={warning}>
<Table.Cell><a href={url} target='_blank'>{textEllipsis(url, 50)}</a></Table.Cell>
<Table.Cell>{status}</Table.Cell>
<Table.Cell>{completedAtCell}</Table.Cell>
<TableRow positive={positive} negative={negative} warning={warning}>
<TableCell>
<ExternalCardLink to={url}>{textEllipsis(url, 50)}</ExternalCardLink>
</TableCell>
<TableCell>{status}</TableCell>
<TableCell>{completedAtCell}</TableCell>
{buttonCell}
</Table.Row>
</TableRow>
);
}
}
class Downloads extends React.Component {
constructor(props) {
super(props);
this.state = {
once_downloads: null,
recurring_downloads: null,
pending_downloads: null,
stopOpen: false,
disabled: false,
};
function Downloads() {
const {t} = useContext(ThemeContext);
const {onceDownloads, recurringDownloads, fetchDownloads} = useDownloads();
const tablePlaceholder = <Placeholder>
<PlaceholderLine/>
<PlaceholderLine/>
</Placeholder>;
const stoppableHeader = (
<TableHeader>
<TableRow>
<TableHeaderCell>URL</TableHeaderCell>
<TableHeaderCell>Status</TableHeaderCell>
<TableHeaderCell>Completed At</TableHeaderCell>
<TableHeaderCell>Control</TableHeaderCell>
</TableRow>
</TableHeader>
);
const nonStoppableHeader = (
<TableHeader>
<TableRow>
<TableHeaderCell>URL</TableHeaderCell>
<TableHeaderCell>Download Frequency</TableHeaderCell>
<TableHeaderCell>Last Successful Download</TableHeaderCell>
<TableHeaderCell>Next Download</TableHeaderCell>
<TableHeaderCell>View</TableHeaderCell>
</TableRow>
</TableHeader>
);
let onceTable = tablePlaceholder;
if (onceDownloads && onceDownloads.length === 0) {
onceTable = <p {...t}>No downloads are scheduled.</p>
} else if (onceDownloads) {
onceTable = (
<>
<ClearCompleteDownloads callback={fetchDownloads}/>
<ClearFailedDownloads callback={fetchDownloads}/>
<Table>
{stoppableHeader}
<TableBody>
{onceDownloads.map((i) =>
<StoppableRow fetchDownloads={fetchDownloads} key={i.id} {...i}/>
)}
</TableBody>
</Table>
</>);
}
async componentDidMount() {
await this.fetchDownloads();
this.intervalId = setInterval(this.fetchDownloads, 1000 * 10);
let recurringTable = tablePlaceholder;
if (recurringDownloads && recurringDownloads.length === 0) {
recurringTable = <p {...t}>No recurring downloads are scheduled.</p>
} else if (recurringDownloads) {
recurringTable = <Table>
{nonStoppableHeader}
<TableBody>
{recurringDownloads.map((i) => <DownloadRow key={i.id} {...i}/>)}
</TableBody>
</Table>;
}
componentWillUnmount() {
clearInterval(this.intervalId);
}
return <>
<WROLModeMessage content='Downloads are disabled because WROL Mode is enabled.'/>
<DisableDownloadsToggle/>
<Header as='h1'>Downloads</Header>
{onceTable}
closeStop = () => {
this.setState({stopOpen: false});
}
openStop = () => {
this.setState({stopOpen: true});
}
fetchDownloads = async () => {
try {
let data = await getDownloads();
this.setState({
once_downloads: data.once_downloads,
recurring_downloads: data.recurring_downloads,
pending_downloads: data.pending_downloads,
});
} catch (e) {
console.error(e);
}
}
render() {
let tablePlaceholder = (
<Placeholder>
<Placeholder.Line/>
<Placeholder.Line/>
</Placeholder>
);
let stoppableHeader = (
<Table.Header>
<Table.Row>
<Table.HeaderCell>URL</Table.HeaderCell>
<Table.HeaderCell>Status</Table.HeaderCell>
<Table.HeaderCell>Completed At</Table.HeaderCell>
<Table.HeaderCell>Control</Table.HeaderCell>
</Table.Row>
</Table.Header>
);
let nonStoppableHeader = (
<Table.Header>
<Table.Row>
<Table.HeaderCell>URL</Table.HeaderCell>
<Table.HeaderCell>Download Frequency</Table.HeaderCell>
<Table.HeaderCell>Last Successful Download</Table.HeaderCell>
<Table.HeaderCell>Next Download</Table.HeaderCell>
<Table.HeaderCell>View</Table.HeaderCell>
</Table.Row>
</Table.Header>
);
let onceTable = tablePlaceholder;
if (this.state.once_downloads !== null && this.state.once_downloads.length === 0) {
onceTable = <p>No downloads are scheduled.</p>
} else if (this.state.once_downloads !== null) {
onceTable = (
<>
<ClearCompleteDownloads callback={() => this.fetchDownloads()}/>
<ClearFailedDownloads callback={() => this.fetchDownloads()}/>
<Table>
{stoppableHeader}
<Table.Body>
{this.state.once_downloads.map((i) =>
<StoppableRow {...i} fetchDownloads={this.fetchDownloads} key={i.id}/>
)}
</Table.Body>
</Table>
</>);
}
let recurringTable = tablePlaceholder;
if (this.state.recurring_downloads !== null && this.state.recurring_downloads.length === 0) {
recurringTable = <p>No recurring downloads are scheduled.</p>
} else if (this.state.recurring_downloads !== null) {
recurringTable = (<Table>
{nonStoppableHeader}
<Table.Body>
{this.state.recurring_downloads.map((i) => <DownloadRow {...i} key={i.id}/>)}
</Table.Body>
</Table>);
}
return (
<div>
<WROLModeMessage content='Downloads are disabled because WROL Mode is enabled.'/>
<DisableDownloadsToggle/>
<Header as='h1'>Downloads</Header>
{onceTable}
<Header as='h1'>Recurring Downloads</Header>
{recurringTable}
</div>
);
}
<Header as='h1'>Recurring Downloads</Header>
{recurringTable}
</>
}
export default function Admin(props) {
const links = [
{text: 'Downloads', to: '/admin', exact: true, key: 'admin'},
{text: 'Downloads', to: '/admin', key: 'admin', end: true},
{text: 'Settings', to: '/admin/settings', key: 'settings'},
{text: 'Status', to: '/admin/status', key: 'status'},
{text: 'WROL Mode', to: '/admin/wrol', key: 'wrol'},
@ -805,10 +773,12 @@ export default function Admin(props) {
return (
<PageContainer>
<TabLinks links={links}/>
<Route path='/admin' exact component={Downloads}/>
<Route path='/admin/settings' exact component={Settings}/>
<Route path='/admin/status' exact component={Status}/>
<Route path='/admin/wrol' exact component={WROLMode}/>
<Routes>
<Route path='/' exact element={<Downloads/>}/>
<Route path='settings' exact element={<Settings/>}/>
<Route path='status' exact element={<Status/>}/>
<Route path='wrol' exact element={<WROLMode/>}/>
</Routes>
</PageContainer>
)

190
app/src/components/Apps.js
View file

@ -1,17 +1,17 @@
import React from 'react';
import {Button, Divider, Segment, TextArea} from "semantic-ui-react";
import {Route} from "react-router-dom";
import React, {useContext} from 'react';
import {Divider, SegmentGroup} from "semantic-ui-react";
import {Route, Routes} from "react-router-dom";
import "../static/wrolpi.css";
import {decryptOTP, encryptOTP} from "../api";
import {PageContainer} from "./Common";
import {ThemeContext} from "../contexts/contexts";
import {Button, Header, Segment, TextArea} from "./Theme";
class Encrypt extends React.Component {
constructor(props) {
super(props);
this.state = {
otp: '',
plaintext: '',
ciphertext: '',
otp: '', plaintext: '', ciphertext: '',
}
}
@ -30,39 +30,37 @@ class Encrypt extends React.Component {
}
render() {
return (
<>
<h2>Encrypt</h2>
<Segment.Group>
<Segment>
<h3>Key</h3>
<TextArea
name='otp'
className='otp'
value={this.state.otp}
onChange={this.handleInputChange}
placeholder='The random letters from your One-Time Pad'
/>
</Segment>
<Segment>
<h3>Plaintext</h3>
<TextArea
name='plaintext'
className='otp'
value={this.state.plaintext}
onChange={this.handleInputChange}
placeholder='The message you want to send'
/>
</Segment>
<Segment>
<h3>Ciphertext</h3>
<pre>{this.state.ciphertext || 'Enter your message above'}</pre>
</Segment>
</Segment.Group>
<br/>
<Button onClick={this.handleSubmit}>Encrypt</Button>
</>
)
return <>
<Header as='h2'>Encrypt</Header>
<SegmentGroup>
<Segment>
<h3>Key</h3>
<TextArea
name='otp'
className='otp'
value={this.state.otp}
onChange={this.handleInputChange}
placeholder='The random letters from your One-Time Pad'
/>
</Segment>
<Segment>
<h3>Plaintext</h3>
<TextArea
name='plaintext'
className='otp'
value={this.state.plaintext}
onChange={this.handleInputChange}
placeholder='The message you want to send'
/>
</Segment>
<Segment>
<h3>Ciphertext</h3>
<pre>{this.state.ciphertext || 'Enter your message above'}</pre>
</Segment>
</SegmentGroup>
<br/>
<Button onClick={this.handleSubmit}>Encrypt</Button>
</>
}
}
@ -70,9 +68,7 @@ class Decrypt extends React.Component {
constructor(props) {
super(props);
this.state = {
otp: '',
plaintext: '',
ciphertext: '',
otp: '', plaintext: '', ciphertext: '',
}
}
@ -91,70 +87,66 @@ class Decrypt extends React.Component {
}
render() {
return (
<>
<h2>Decrypt</h2>
<Segment.Group>
<Segment>
<h3>Key</h3>
<TextArea
name='otp'
className='otp'
value={this.state.otp}
onChange={this.handleInputChange}
placeholder='The random letters from your One-Time Pad'
/>
</Segment>
<Segment>
<h3>Ciphertext</h3>
<TextArea
name='ciphertext'
className='otp'
value={this.state.ciphertext}
onChange={this.handleInputChange}
placeholder='The message you received'
/>
</Segment>
<Segment>
<h3>Plaintext</h3>
<pre>{this.state.plaintext || 'Enter the encrypted message above'}</pre>
</Segment>
</Segment.Group>
<br/>
<Button onClick={this.handleSubmit}>Decrypt</Button>
</>
)
return <>
<Header as='h2'>Decrypt</Header>
<SegmentGroup>
<Segment>
<h3>Key</h3>
<TextArea
name='otp'
className='otp'
value={this.state.otp}
onChange={this.handleInputChange}
placeholder='The random letters from your One-Time Pad'
/>
</Segment>
<Segment>
<h3>Ciphertext</h3>
<TextArea
name='ciphertext'
className='otp'
value={this.state.ciphertext}
onChange={this.handleInputChange}
placeholder='The message you received'
/>
</Segment>
<Segment>
<h3>Plaintext</h3>
<pre>{this.state.plaintext || 'Enter the encrypted message above'}</pre>
</Segment>
</SegmentGroup>
<br/>
<Button onClick={this.handleSubmit}>Decrypt</Button>
</>
}
}
class OTP extends React.Component {
render() {
let newPadURL = `http://${window.location.host}/api/otp/html`;
let cheatSheetURL = `${process.env.PUBLIC_URL}/one-time-pad-cheat-sheet.pdf`;
function OTP() {
const {t} = useContext(ThemeContext);
return (
<div>
<h1>One-Time Pad</h1>
<h4>One-Time Pads can be used to encrypt your communications. This can be done by hand (yes, really) or
in this app.</h4>
<p>These messages are never stored and cannot be retrieved.</p>
<Button primary href={newPadURL}>Generate New Pad</Button>
<Button secondary href={cheatSheetURL}>Cheat Sheet PDF</Button>
let newPadURL = `http://${window.location.host}/api/otp/html`;
let cheatSheetURL = `${process.env.PUBLIC_URL}/one-time-pad-cheat-sheet.pdf`;
<Divider/>
<Encrypt/>
return <>
<Header as='h1'>One-Time Pad</Header>
<Header as='h4'>One-Time Pads can be used to encrypt your communications. This can be done by hand
(yes, really) or in this app.</Header>
<p {...t}>These messages are never stored and cannot be retrieved.</p>
<Button primary href={newPadURL}>Generate New Pad</Button>
<Button secondary href={cheatSheetURL}>Cheat Sheet PDF</Button>
<Divider/>
<Decrypt/>
</div>
)
}
<Divider/>
<Encrypt/>
<Divider/>
<Decrypt/>
</>
}
export function AppsRoute(props) {
return (
<PageContainer>
<Route path='/apps/otp' exact component={OTP}/>
</PageContainer>
)
return (<PageContainer>
<Routes>
<Route path='otp' exact element={<OTP/>}/>
</Routes>
</PageContainer>)
}

377
app/src/components/Archive.js
View file

@ -1,52 +1,71 @@
import React, {useState} from "react";
import {Card, Confirm, Container, Header, Image, Loader, Placeholder, Segment} from "semantic-ui-react";
import React, {useContext, useState} from "react";
import {
CardGroupCentered,
ClearButton,
Card,
Confirm,
Container,
Image,
Input,
PlaceholderHeader,
PlaceholderLine,
TableBody,
TableCell,
TableHeader,
TableHeaderCell,
TableRow
} from "semantic-ui-react";
import {
CardLink,
defaultSearchOrder,
ExternalCardLink,
FileIcon,
HelpHeader,
mimetypeColor,
PageContainer,
Paginator,
SearchInput,
TabLinks,
textEllipsis,
uploadDate,
WROLModeMessage
uploadDate
} from "./Common";
import {deleteArchive, postDownload, refreshArchives} from "../api";
import Button from "semantic-ui-react/dist/commonjs/elements/Button";
import {Link, Route, Switch, useHistory} from "react-router-dom";
import {ArchivePlaceholder} from "./Placeholder";
import Table from "semantic-ui-react/dist/commonjs/collections/Table";
import {deleteArchive, postDownload} from "../api";
import {Link, Route, Routes, useNavigate, useParams} from "react-router-dom";
import Message from "semantic-ui-react/dist/commonjs/collections/Message";
import {useArchive, useArchives, useDomains} from "../hooks/customHooks";
import {useArchive, useDomains, useQuery, useSearchArchives} from "../hooks/customHooks";
import {FileCards, FilesView} from "./Files";
import Grid from "semantic-ui-react/dist/commonjs/collections/Grid";
import Dropdown from "semantic-ui-react/dist/commonjs/modules/Dropdown";
import _ from "lodash";
import {ThemeContext} from "../contexts/contexts";
import {Button, Header, Loader, Placeholder, Segment, Table} from "./Theme";
function ArchivePage(props) {
function ArchivePage() {
const [deleteOpen, setDeleteOpen] = useState(false);
const [syncOpen, setSyncOpen] = useState(false);
const history = useHistory();
const archiveId = props.match.params.archive_id;
const [archive, alternatives] = useArchive(archiveId);
const navigate = useNavigate();
const {archiveId} = useParams();
const {archiveFile, alternatives} = useArchive(archiveId);
if (archive === null) {
if (archiveFile === null) {
return <Segment><Loader active/></Segment>;
}
if (archive === undefined) {
if (archiveFile === undefined) {
return <>
<Header as='h2'>Unknown archive</Header>
This archive does not exist
</>
}
const {archive} = archiveFile;
const singlefileUrl = archive.singlefile_path ? `/media/${encodeURIComponent(archive.singlefile_path)}` : null;
const screenshotUrl = archive.screenshot_path ? `/media/${encodeURIComponent(archive.screenshot_path)}` : null;
const readabilityUrl = archive.readability_path ? `/media/${encodeURIComponent(archive.readability_path)}` : null;
const singlefileButton = <a href={singlefileUrl} target='_blank'>
const singlefileButton = <ExternalCardLink to={singlefileUrl}>
<Button content='View' color='blue'/>
</a>;
</ExternalCardLink>;
const readabilityLink = readabilityUrl ?
<a href={readabilityUrl} target='_blank'><Button content='Article'/></a> :
<ExternalCardLink to={readabilityUrl}><Button content='Article'/></ExternalCardLink> :
<Button disabled content='Article'/>;
const screenshot = screenshotUrl ?
@ -85,22 +104,26 @@ function ArchivePage(props) {
let alternativesList = <Loader active/>;
if (alternatives && alternatives.length > 0) {
alternativesList = <ArchivesList archives={alternatives}/>;
alternativesList = <FileCards files={alternatives}/>;
} else if (alternatives && alternatives.length === 0) {
alternativesList = <p>No alternatives available</p>;
}
return (
<>
<Button icon='arrow left' content='Back' onClick={() => history.goBack()}/>
<Button icon='arrow left' content='Back' onClick={() => navigate(-1)}/>
<Segment>
{screenshot}
<a href={singlefileUrl} target='_blank' className='no-link-underscore card-link'>
<ExternalCardLink to={singlefileUrl}>
<Header as='h2'>{textEllipsis(archive.title || archive.url, 100)}</Header>
</a>
</ExternalCardLink>
<Header as='h3'>{uploadDate(archive.archive_datetime)}</Header>
<p><a href={archive.url} target='_blank'>{textEllipsis(archive.url, 100)}</a></p>
<p>
<ExternalCardLink to={archive.url}>
{textEllipsis(archive.url, 100)}
</ExternalCardLink>
</p>
{singlefileButton}
{readabilityLink}
@ -119,32 +142,41 @@ function ArchivePage(props) {
)
}
function ArchiveCard({archive, setDomain}) {
export function ArchiveCard({file}) {
const {s} = useContext(ThemeContext);
const {archive} = file;
const imageSrc = archive.screenshot_path ? `/media/${encodeURIComponent(archive.screenshot_path)}` : null;
const singlefileUrl = archive.singlefile_path ? `/media/${encodeURIComponent(archive.singlefile_path)}` : null;
const domain = archive.domain.domain;
let screenshot = <center className='card-icon'><FileIcon file={file}/></center>;
if (imageSrc) {
screenshot = <Image src={imageSrc} wrapped style={{position: 'relative', width: '100%'}}/>;
}
const domain = archive.domain ? archive.domain.domain : null;
const domainUrl = `/archive?domain=${domain}`;
return (
<Card>
<a href={singlefileUrl} target='_blank' className='no-link-underscore card-link'>
<Image src={imageSrc} wrapped style={{position: 'relative', width: '100%'}}/>
</a>
<Card color={mimetypeColor(file.mimetype)} {...s}>
<div>
<ExternalCardLink to={singlefileUrl}>
{screenshot}
</ExternalCardLink>
</div>
<Card.Content>
<Card.Header>
<Container textAlign='left'>
<a href={singlefileUrl} target='_blank' className='no-link-underscore card-link'>
<p>{textEllipsis(archive.title || archive.url, 100)}</p>
</a>
<ExternalCardLink to={singlefileUrl}>
{textEllipsis(archive.title || archive.url, 100)}
</ExternalCardLink>
</Container>
</Card.Header>
{setDomain &&
<a onClick={() => setDomain(domain)} className="no-link-underscore card-link">
{textEllipsis(domain, 40)}
</a>
}
<Card.Meta>
{domain &&
<CardLink to={domainUrl}>
<p {...s}>{domain}</p>
</CardLink>}
<Card.Meta {...s}>
<p>
{uploadDate(archive.archive_datetime)}
</p>
@ -161,158 +193,145 @@ function ArchiveCard({archive, setDomain}) {
)
}
function ArchiveCards({archives, setDomain}) {
return (
<CardGroupCentered>
{archives.map((i) => {
return <ArchiveCard
key={i['id']}
archive={i}
setDomain={setDomain}
/>
})}
</CardGroupCentered>
)
}
export function ArchivesList({archives, searchStr, setDomain}) {
if (archives === null || archives === undefined) {
// Fetching archives.
return <ArchivePlaceholder/>;
} else if (archives.length === 0 && searchStr) {
// Search with no results.
return <p>No archives found! Is your search too restrictive?</p>;
} else if (archives.length === 0) {
// No archives fetched.
return <Message>
<Message.Header>No archives</Message.Header>
<Message.Content>Have you archived any webpages?</Message.Content>
</Message>;
} else {
// Archives fetched successfully!
return <ArchiveCards
archives={archives}
setDomain={setDomain}
/>;
}
}
export function Archives() {
const {archives, setPage, totalPages, searchStr, activePage, setSearchStr, domain, setDomain, search} =
useArchives(20);
let domainClearButton = null;
if (domain) {
domainClearButton = <ClearButton
onClick={() => setDomain(null)}
style={{marginBottom: '1em'}}
label={`Domain: ${domain}`}
/>;
}
return (
<>
<SearchInput initValue={searchStr} onSubmit={setSearchStr}/>
{domainClearButton}
<ArchivesList
archives={archives}
searchStr={searchStr}
setDomain={setDomain}
/>
<div style={{marginTop: '3em', textAlign: 'center'}}>
<Paginator activePage={activePage} totalPages={totalPages} onPageChange={setPage}/>
</div>
</>
)
}
class ManageArchives extends React.Component {
async refresh() {
await refreshArchives();
}
render() {
return (<>
<WROLModeMessage content='Cannot modify Archives'/>
<Button secondary
id='refresh_archives'
onClick={this.refresh}>
Refresh Archive Files
</Button>
<label htmlFor='refresh_archives'>
Find any new archive files. Remove Archives which no longer have files.
</label>
</>)
}
}
function Domains() {
export function Domains() {
const [domains] = useDomains();
const row = (domain) => {
return <Table.Row key={domain['domain']}>
<Table.Cell>
<Link to={`/archive?domain=${domain['domain']}`}>
{domain['domain']}
</Link>
</Table.Cell>
<Table.Cell>{domain['url_count']}</Table.Cell>
</Table.Row>
}
if (domains && domains.length > 0) {
return (
<>
<Table celled>
<Table.Header>
<Table.Row>
<Table.HeaderCell>Domain</Table.HeaderCell>
<Table.HeaderCell>Archives</Table.HeaderCell>
</Table.Row>
</Table.Header>
<Table.Body>
{domains.map(row)}
</Table.Body>
</Table>
</>
)
}
const [searchStr, setSearchStr] = useState('');
if (domains === null) {
return (<>
<Placeholder>
<Placeholder.Header>
<Placeholder.Line/>
<Placeholder.Line/>
</Placeholder.Header>
<PlaceholderHeader>
<PlaceholderLine/>
<PlaceholderLine/>
</PlaceholderHeader>
</Placeholder>
</>)
}
return (<>
<Message>
} else if (!domains || domains.length === 0) {
return <Message>
<Message.Header>No domains yet.</Message.Header>
<Message.Content>Archive some webpages!</Message.Content>
</Message>
</>)
</Message>;
}
let filteredDomains = domains;
if (searchStr) {
const re = new RegExp(_.escapeRegExp(searchStr), 'i');
filteredDomains = domains.filter(i => re.test(i['domain']));
}
const row = ({domain, url_count}) => {
return <TableRow key={domain}>
<TableCell>
<Link to={`/archive?domain=${domain}`}>
{domain}
</Link>
</TableCell>
<TableCell>{url_count}</TableCell>
</TableRow>
}
return <>
<Input
icon='search'
value={searchStr}
placeholder='Search...'
onChange={(e, {value}) => setSearchStr(value)}
/>
<Table celled>
<TableHeader>
<TableRow>
<TableHeaderCell>Domain</TableHeaderCell>
<TableHeaderCell>Archives</TableHeaderCell>
</TableRow>
</TableHeader>
<TableBody>
{filteredDomains.map(row)}
</TableBody>
</Table>
</>;
}
function Archives() {
const [domains] = useDomains();
let filterOptions = [];
if (domains && domains.length > 0) {
domains.forEach(i => {
filterOptions = [...filterOptions, {text: i['domain'], key: i['domain'], value: i['domain']}]
});
}
const {searchParams, updateQuery} = useQuery();
export function ArchiveRoute(props) {
let searchOrder = '-date';
if (searchParams.get('order')) {
// Use whatever order the user specified.
searchOrder = searchParams.get('order');
} else if (searchParams.get('q')) {
// User used a search_str
searchOrder = defaultSearchOrder;
}
const domain = searchParams.get('domain');
const {archives, limit, setLimit, totalPages, activePage, setPage, searchStr, setSearchStr, setOrderBy} =
useSearchArchives(24, domain, searchOrder);
const setView = (value) => updateQuery({view: value});
const view = searchParams.get('view');
const setDomain = (value) => updateQuery({'domain': value});
const archiveOrders = [
{key: '-date', value: '-date', text: 'Newest'},
{key: 'date', value: 'date', text: 'Oldest'},
];
const menuColumns = (<>
<Grid.Column width={5}>
<SearchInput clearable
actionIcon='search'
searchStr={searchStr}
onSubmit={setSearchStr}
/>
</Grid.Column>
<Grid.Column width={5}>
<Dropdown selection fluid
placeholder='Sort by...'
value={searchOrder}
options={archiveOrders}
onChange={(e, {value}) => setOrderBy(value)}
/>
</Grid.Column>
</>);
return <FilesView
files={archives}
limit={limit}
setLimit={setLimit}
activePage={activePage}
totalPages={totalPages}
showLimit={true}
showSelect={true}
view={view}
setView={setView}
setPage={setPage}
filterOptions={filterOptions}
activeFilters={domain}
setFilters={setDomain}
filterPlaceholder='Domain'
menuColumns={menuColumns}
menuColumnsCount={2}
/>
}
export function ArchiveRoute() {
const links = [
{text: 'Archives', to: '/archive', exact: true, key: 'archive'},
{text: 'Domains', to: '/archive/domains', key: 'domains'},
{text: 'Manage', to: '/archive/manage', key: 'manage'},
{text: 'Archives', to: '/archive', end: true},
{text: 'Domains', to: '/archive/domains'},
];
return <PageContainer>
<TabLinks links={links}/>
<Switch>
<Route path='/archive/domains' exact component={Domains}/>
<Route path='/archive/manage' exact component={ManageArchives}/>
<Route path='/archive/:archive_id' exact component={ArchivePage}/>
<Route path='/archive' exact component={Archives}/>
</Switch>
<Routes>
<Route path='/' element={<Archives/>}/>
<Route path='domains' element={<Domains/>}/>
<Route path=':archiveId' element={<ArchivePage/>}/>
</Routes>
</PageContainer>
}

970
app/src/components/Channels.js
View file

File diff suppressed because it is too large Load diff

745
app/src/components/Common.js
View file

@ -1,24 +1,11 @@
import React, {useEffect, useState} from "react";
import {
Card,
Checkbox,
Container,
Form,
Header,
Icon,
Image,
Input,
Menu,
Pagination,
Popup,
Responsive
} from 'semantic-ui-react';
import React, {useContext, useEffect, useState} from "react";
import {Card, Container, Image, Input, Pagination, Popup, Responsive, TableCell, TableRow} from 'semantic-ui-react';
import {Link, NavLink} from "react-router-dom";
import Button from "semantic-ui-react/dist/commonjs/elements/Button";
import Grid from "semantic-ui-react/dist/commonjs/collections/Grid";
import Message from "semantic-ui-react/dist/commonjs/collections/Message";
import {getSettings} from "../api";
import {useDownloaders, useHotspot, useThrottle} from "../hooks/customHooks";
import {getSettings} from "../api";
import {darkTheme, SettingsContext, ThemeContext} from "../contexts/contexts";
import {Button, Form, Header, Icon, Menu, Statistic} from "./Theme";
export const API_URI = `http://${window.location.host}/api`;
export const VIDEOS_API = `${API_URI}/videos`;
@ -26,65 +13,21 @@ export const ARCHIVES_API = `${API_URI}/archive`;
export const OTP_API = `${API_URI}/otp`;
export const DEFAULT_LIMIT = 20;
export default class OldPaginator extends React.Component {
state = {
boundaryRange: 1,
siblingRange: 2,
showEllipsis: true,
showFirstAndLastNav: false,
showPreviousAndNextNav: true,
}
handlePaginationChange = (e, {activePage}) => {
this.props.changePage(activePage);
}
render() {
const {
boundaryRange,
siblingRange,
showEllipsis,
showFirstAndLastNav,
showPreviousAndNextNav,
} = this.state;
return (
<Pagination
activePage={this.props.activePage}
boundaryRange={boundaryRange}
onPageChange={this.handlePaginationChange}
size='mini'
siblingRange={siblingRange}
totalPages={this.props.totalPages}
// Heads up! All items are powered by shorthands, if you want to hide one of them, just pass `null` as value
ellipsisItem={showEllipsis ? undefined : null}
firstItem={showFirstAndLastNav ? undefined : null}
lastItem={showFirstAndLastNav ? undefined : null}
prevItem={showPreviousAndNextNav ? undefined : null}
nextItem={showPreviousAndNextNav ? undefined : null}
/>
)
}
}
export function Paginator({activePage, onPageChange, totalPages, showFirstAndLast}) {
export function Paginator({activePage, onPageChange, totalPages, showFirstAndLast, size = 'mini'}) {
const handlePageChange = (e, {activePage}) => {
onPageChange(activePage);
}
return (
<Pagination
activePage={activePage}
boundaryRange={1}
onPageChange={handlePageChange}
size='mini'
siblingRange={2}
totalPages={totalPages}
firstItem={showFirstAndLast ? undefined : null}
lastItem={showFirstAndLast ? undefined : null}
/>
)
return (<Pagination
activePage={activePage}
boundaryRange={1}
onPageChange={handlePageChange}
size={size}
siblingRange={2}
totalPages={totalPages}
firstItem={showFirstAndLast ? undefined : null}
lastItem={showFirstAndLast ? undefined : null}
/>)
}
export function secondsToDuration(video) {
@ -114,7 +57,7 @@ export function Duration({video}) {
}
export function uploadDate(d) {
// Convert a date integer to a human readable date format.
// Convert a date integer to a human-readable date format.
let upload_date = <></>;
if (d) {
upload_date = new Date(d * 1000);
@ -123,69 +66,137 @@ export function uploadDate(d) {
return upload_date;
}
export function secondsToDateTime(seconds) {
var t = new Date(1970, 0, 1); // Epoch
t.setSeconds(seconds);
return t;
export function CardLink({to, newTab = false, ...props}) {
const {t} = useContext(ThemeContext);
props = {...props, ...t};
props = newTab === true ? {...props, target: '_blank', rel: 'noopener noreferrer'} : props;
return <Link to={to} className="no-link-underscore card-link" {...props}>
{props.children}
</Link>
}
export function VideoCard({video}) {
export function ExternalCardLink({to, ...props}) {
const {t} = useContext(ThemeContext);
return <a href={to} target='_blank' rel='noopener noreferrer' className='no-link-underscore card-link' {...t}>
{props.children}
</a>
}
export function VideoCard({file}) {
const {video} = file;
const {s} = useContext(ThemeContext);
let video_url = `/videos/video/${video.id}`;
let upload_date = uploadDate(video.upload_date);
const upload_date = uploadDate(video.upload_date);
// A video may not have a channel.
let channel = video.channel ? video.channel : null;
const channel = video.channel ? video.channel : null;
let channel_url = null;
if (channel) {
channel_url = `/videos/channel/${channel.id}/video`;
video_url = `/videos/channel/${channel.id}/video/${video.id}`;
}
let poster_url = video.poster_path ? `/media/${encodeURIComponent(video.poster_path)}` : null;
const poster_url = video.poster_path ? `/media/${encodeURIComponent(video.poster_path)}` : null;
let imageLabel = null;
if (video.favorite) {
imageLabel = {corner: 'left', icon: 'heart', color: 'green'};
}
let poster = <center className='card-icon'><FileIcon file={file}/></center>;
if (poster_url) {
poster = <Image wrapped
src={poster_url}
label={imageLabel}
style={{position: 'relative', width: '100%'}}
/>;
}
return (
<Card>
<Link to={video_url}>
<Image wrapped
src={poster_url}
label={imageLabel}
style={{position: 'relative', width: '100%'}}
/>
</Link>
<Duration video={video}/>
<Card.Content>
<Card.Header>
<Container textAlign='left'>
<Link to={video_url} className="no-link-underscore card-link">
<p>{textEllipsis(video.title || video.stem || video.video_path, 100)}</p>
</Link>
</Container>
</Card.Header>
<Card.Description>
<Container textAlign='left'>
{channel &&
<Link to={channel_url} className="no-link-underscore card-link">
<b>{channel.name}</b>
</Link>}
<p>{upload_date}</p>
</Container>
</Card.Description>
</Card.Content>
</Card>
)
return (<Card color={mimetypeColor(file.mimetype)}>
<Link to={video_url}>
{poster}
</Link>
<Duration video={video}/>
<Card.Content {...s}>
<Card.Header>
<Container textAlign='left'>
<Link to={video_url} className="no-link-underscore card-link">
<p {...s}>{textEllipsis(video.title || video.stem || video.video_path, 100)}</p>
</Link>
</Container>
</Card.Header>
<Card.Description>
<Container textAlign='left'>
{channel && <Link to={channel_url} className="no-link-underscore card-link">
<b {...s}>{channel.name}</b>
</Link>}
<p {...s}>{upload_date}</p>
</Container>
</Card.Description>
</Card.Content>
</Card>)
}
export function VideoCards({videos}) {
return (
<CardGroupCentered>
{videos.map((v) => {
return <VideoCard key={v['id']} video={v}/>
})}
</CardGroupCentered>
)
export function VideoRow({file, checkbox}) {
const {video} = file;
let video_url = `/videos/video/${video.id}`;
const poster_url = video.poster_path ? `/media/${encodeURIComponent(video.poster_path)}` : null;
let poster;
let imageLabel = video.favorite ? {corner: 'left', icon: 'heart', color: 'green'} : null;
if (poster_url) {
poster = <CardLink to={video_url}>
<Image wrapped
src={poster_url}
label={imageLabel}
width='50px'
/>
</CardLink>
} else {
poster = <FileIcon file={file} size='large'/>;
}
return (<TableRow>
{checkbox}
<TableCell>
<center>
{poster}
</center>
</TableCell>
<TableCell>
<CardLink to={video_url}>
{textEllipsis(video.title || video.stem || video.video_path, 100)}
</CardLink>
</TableCell>
</TableRow>)
}
export function ArchiveRow({file, checkbox}) {
const {archive} = file;
const archiveUrl = `/archive/${archive.id}`;
const posterUrl = archive.screenshot_path ? `/media/${encodeURIComponent(archive.screenshot_path)}` : null;
let poster;
if (posterUrl) {
poster = <CardLink to={archiveUrl}>
<Image wrapped src={posterUrl} width='50px'/>
</CardLink>;
} else {
poster = <FileIcon file={file} size='large'/>;
}
return (<TableRow>
{checkbox}
<TableCell>
<center>{poster}</center>
</TableCell>
<TableCell>
<CardLink to={archiveUrl}>
{textEllipsis(archive.title || archive.stem, 100)}
</CardLink>
</TableCell>
</TableRow>)
}
export function RequiredAsterisk() {
@ -195,43 +206,33 @@ export function RequiredAsterisk() {
export let defaultVideoOrder = '-upload_date';
export let defaultSearchOrder = 'rank';
export let videoOrders = [
{key: '-upload_date', value: '-upload_date', text: 'Newest', title: 'Newest Videos'},
{key: 'upload_date', value: 'upload_date', text: 'Oldest', title: 'Oldest Videos'},
{key: '-duration', value: '-duration', text: 'Longest', title: 'Longest Videos'},
{key: 'duration', value: 'duration', text: 'Shortest', title: 'Shortest Videos'},
{key: '-viewed', value: '-viewed', text: 'Recently viewed', title: 'Recently Viewed Videos'},
{key: '-size', value: '-size', text: 'Largest', title: 'Largest Videos'},
{key: 'size', value: 'size', text: 'Smallest', title: 'Smallest Videos'},
{key: '-view_count', value: '-view_count', text: 'Most Views', title: 'Most Views'},
{key: 'view_count', value: 'view_count', text: 'Least Views', title: 'Least Views'},
{key: '-modification_datetime', value: '-modification_datetime', text: 'Newest File', title: 'Newest File'},
{key: 'modification_datetime', value: 'modification_datetime', text: 'Oldest File', title: 'Oldest File'},
]
export let videoOrders = [{key: '-upload_date', value: '-upload_date', text: 'Newest'}, {
key: 'upload_date', value: 'upload_date', text: 'Oldest'
}, {key: '-duration', value: '-duration', text: 'Longest'}, {
key: 'duration', value: 'duration', text: 'Shortest'
}, {key: '-viewed', value: '-viewed', text: 'Recently viewed'}, {
key: '-size', value: '-size', text: 'Largest'
}, {key: 'size', value: 'size', text: 'Smallest'}, {
key: '-view_count', value: '-view_count', text: 'Most Views'
}, {key: 'view_count', value: 'view_count', text: 'Least Views'}, {
key: '-modification_datetime', value: '-modification_datetime', text: 'Newest File'
}, {key: 'modification_datetime', value: 'modification_datetime', text: 'Oldest File'},]
export let searchOrders = [
{key: 'rank', value: 'rank', text: 'Search Rank', title: 'Search Results'},
];
export let searchOrders = [{key: 'rank', value: 'rank', text: 'Search Rank', title: 'Search Results'},];
export const frequencyOptions = [
{key: 'daily', text: 'Daily', value: 86400},
{key: 'weekly', text: 'Weekly', value: 604800},
{key: 'biweekly', text: 'Biweekly', value: 1209600},
{key: '30days', text: '30 Days', value: 2592000},
{key: '90days', text: '90 Days', value: 7776000},
];
export const frequencyOptions = [{key: 'daily', text: 'Daily', value: 86400}, {
key: 'weekly', text: 'Weekly', value: 604800
}, {key: 'biweekly', text: 'Biweekly', value: 1209600}, {
key: '30days', text: '30 Days', value: 2592000
}, {key: '90days', text: '90 Days', value: 7776000},];
export const rssFrequencyOptions = [
{key: 'once', text: 'Once', value: 0},
{key: 'hourly', text: 'Hourly', value: 3600},
{key: '3hours', text: '3 hours', value: 10800},
{key: '12hours', text: '12 hours', value: 43200},
{key: 'daily', text: 'Daily', value: 86400},
{key: 'weekly', text: 'Weekly', value: 604800},
{key: 'biweekly', text: 'Biweekly', value: 1209600},
{key: '30days', text: '30 Days', value: 2592000},
{key: '90days', text: '90 Days', value: 7776000},
];
export const rssFrequencyOptions = [{key: 'once', text: 'Once', value: 0}, {
key: 'hourly', text: 'Hourly', value: 3600
}, {key: '3hours', text: '3 hours', value: 10800}, {key: '12hours', text: '12 hours', value: 43200}, {
key: 'daily', text: 'Daily', value: 86400
}, {key: 'weekly', text: 'Weekly', value: 604800}, {key: 'biweekly', text: 'Biweekly', value: 1209600}, {
key: '30days', text: '30 Days', value: 2592000
}, {key: '90days', text: '90 Days', value: 7776000},];
export function secondsToFrequency(seconds) {
for (let i = 0; i < Object.keys(rssFrequencyOptions).length; i++) {
@ -343,20 +344,13 @@ export class APIForm extends React.Component {
errors[errorName] = true;
}
this.setState({
error: true,
errors: errors,
message_content: content,
message_header: header,
success: false,
error: true, errors: errors, message_content: content, message_header: header, success: false,
});
}
setSuccess = (header, content) => {
this.setState({
error: false,
message_content: content,
message_header: header,
success: true,
error: false, message_content: content, message_header: header, success: true,
});
}
@ -407,18 +401,13 @@ export class APIForm extends React.Component {
setLoading = () => {
this.setState({
disabled: true,
error: false,
errors: this.getEmptyErrors(),
loading: true,
success: false,
disabled: true, error: false, errors: this.getEmptyErrors(), loading: true, success: false,
})
}
clearLoading = () => {
this.setState({
disabled: false,
loading: false,
disabled: false, loading: false,
})
}
@ -470,8 +459,7 @@ export function arraysEqual(a, b) {
export function scrollToTop() {
window.scrollTo({
top: 0,
behavior: "auto"
top: 0, behavior: "auto"
});
}
@ -488,95 +476,54 @@ export function changePageHistory(history, location, activePage, searchStr, sear
search = `${search}&f=${filters}`;
}
history.push({
pathname: location.pathname,
search: search,
pathname: location.pathname, search: search,
});
scrollToTop();
}
export function ClearButton({onClick, style, label, icon = 'close'}) {
return <Button icon
labelPosition='right'
onClick={onClick}
style={style}
>
{label}
<Icon name={icon}/>
</Button>
}
export function SearchInput({
searchStr,
onSubmit,
size = 'small',
placeholder,
action,
actionIcon,
clearable,
...props
}) {
let [value, setValue] = useState(searchStr || '');
export function SearchInput({initValue, onSubmit, size, placeholder = 'Search...'}) {
let [value, setValue] = useState(initValue || '');
size = size || 'small';
const buttonClick = () => {
const handleClearSearch = (e) => {
e.preventDefault();
// Clear the input when the "clear" button is clicked, search again.
setValue('');
onSubmit('');
}
// Button is "search" when input is dirty, otherwise it is "clear".
let button = (
<Button icon onClick={buttonClick} style={{marginLeft: '0.5em'}} type='reset' size={size}>
<Icon name='close'/>
</Button>
);
if (!initValue || initValue !== value) {
button = (
<Button icon style={{marginLeft: '0.5em'}} type='submit' disabled={value === ''} size={size}>
<Icon name='search'/>
</Button>
);
}
const localOnSubmit = (e) => {
// Send the value up when submitting.
e.preventDefault();
onSubmit(value);
}
return (
<Form onSubmit={localOnSubmit}>
<Form.Group inline style={{marginBottom: '1em'}}>
<Input
type='text'
placeholder={placeholder}
onChange={(e) => setValue(e.target.value)}
value={value}
size={size}
/>
{button}
</Form.Group>
</Form>
)
}
if ((action || actionIcon) && searchStr && searchStr === value && clearable) {
action = <Button icon='close' size={size} onClick={handleClearSearch}/>;
} else if (actionIcon) {
action = <Button icon={actionIcon} size={size}/>;
} else if (action) {
action = <Button size={size}>{action}</Button>;
}
export function ExternalLink(props) {
return (
<Link to={props.to} target='_blank' className={props.className} rel='noopener noreferrer'>
{props.children}
</Link>
)
}
export function MoreButton(props) {
return (
<Grid columns={1} textAlign='center'>
<Grid.Row>
<Grid.Column>
<Button
onClick={props.onClick}
size='big'
style={{marginTop: '1em'}}
disabled={props.disabled}
>
{props.children}
More
</Button>
</Grid.Column>
</Grid.Row>
</Grid>
)
return <Form onSubmit={localOnSubmit} {...props}>
<Input
placeholder={placeholder}
type='text'
onChange={(e) => setValue(e.target.value)}
value={value}
size={size}
action={action}
/>
</Form>
}
export const useWROLMode = () => {
@ -601,16 +548,18 @@ export const useWROLMode = () => {
}
export function WROLModeMessage({content}) {
let enabled = useWROLMode();
if (enabled) {
const settings = useContext(SettingsContext);
const wrol_mode = settings ? settings.wrol_mode : null;
if (wrol_mode) {
return <Message icon='lock' header='WROL Mode Enabled' content={content}/>
}
return null;
}
// Thanks https://www.npmjs.com/package/text-ellipsis
export function textEllipsis(str, maxLength, {side = "end", ellipsis = "..."} = {}) {
if (str.length > maxLength) {
export function textEllipsis(str, maxLength = 100, {side = "end", ellipsis = "..."} = {}) {
if (str && str.length > maxLength) {
switch (side) {
case "start":
return ellipsis + str.slice(-(maxLength - ellipsis.length));
@ -623,55 +572,47 @@ export function textEllipsis(str, maxLength, {side = "end", ellipsis = "..."} =
}
export function TabLinks({links}) {
return (
<Menu tabular>
{links.map((link) =>
<NavLink
to={link.to}
className={'item'}
exact={link.exact || false}
style={{padding: '1em'}}
key={link.key}
>
{link.text}
</NavLink>
)}
</Menu>
)
return (<Menu tabular>
{links.map((link) => <NavLink
to={link.to}
className='item'
style={{padding: '1em'}}
key={link.to}
end={link.end === true ? true : null}
>
{link.text}
</NavLink>)}
</Menu>)
}
export function PageContainer(props) {
return (
<>
<Responsive minWidth={770}>
<Container fluid style={{marginTop: '1em', padding: '1em'}}>
{props.children}
</Container>
</Responsive>
<Responsive maxWidth={769}>
<Container style={{marginTop: '1em', padding: 0}}>
{props.children}
</Container>
</Responsive>
</>
)
return (<>
<Responsive minWidth={770}>
<Container fluid style={{marginTop: '1em', padding: '1em'}}>
{props.children}
</Container>
</Responsive>
<Responsive maxWidth={769}>
<Container style={{marginTop: '1em', padding: 0}}>
{props.children}
</Container>
</Responsive>
</>)
}
export function CardGroupCentered(props) {
return (
<>
<Responsive minWidth={770}>
<Card.Group>
{props.children}
</Card.Group>
</Responsive>
<Responsive maxWidth={769}>
<Card.Group centered>
{props.children}
</Card.Group>
</Responsive>
</>
)
return (<div style={{marginTop: '1em'}}>
<Responsive minWidth={770}>
<Card.Group>
{props.children}
</Card.Group>
</Responsive>
<Responsive maxWidth={769}>
<Card.Group centered>
{props.children}
</Card.Group>
</Responsive>
</div>)
}
export function HelpPopup({icon, size, content, position}) {
@ -679,64 +620,109 @@ export function HelpPopup({icon, size, content, position}) {
content={content}
size={size || null}
position={position || 'left center'}
trigger={
<Icon circular link name={icon || 'question'} size='small'
style={{marginLeft: '0.25em', marginRight: '0.25em'}}
/>}
trigger={<Icon circular link name={icon || 'question'} size='small'
style={{marginLeft: '0.25em', marginRight: '0.25em'}}
/>}
/>
}
export function HelpHeader({icon, headerSize, iconSize, headerContent, popupContent}) {
return (
<div className='inline-header'>
<Header as={headerSize || 'h2'}>{headerContent}</Header>
<span>
return (<div className='inline-header'>
<Header as={headerSize || 'h2'}>{headerContent}</Header>
<span>
<HelpPopup content={popupContent} size={iconSize} icon={icon}/>
</span>
</div>
)
</div>)
}
export function HotspotToggle(props) {
export function HotspotToggle() {
let {on, setHotspot} = useHotspot();
const disabled = on === null;
return (
<>
<Checkbox toggle {...props}
label='WiFi Hotspot'
disabled={disabled}
checked={on === true}
onChange={(e, data) => setHotspot(data.checked)}
/>
{disabled && <HelpPopup content='Hotspot is not supported on this server'/>}
</>
);
return (<div style={{margin: '0.5em'}}>
<Toggle
label='WiFi Hotspot'
disabled={disabled}
checked={on === true}
onChange={(e, data) => setHotspot(data.checked)}
/>
{disabled && <HelpPopup content='Hotspot is not supported on this server'/>}
</div>);
}
export function ThrottleToggle(props) {
export function ThrottleToggle() {
let {on, setThrottle} = useThrottle();
const disabled = on === null;
return (
<>
<Checkbox toggle {...props}
label='CPU Power-save'
disabled={disabled}
checked={on === true}
onChange={(e, data) => setThrottle(data.checked)}
/>
{disabled && <HelpPopup content='CPU Power-save is not supported on this server'/>}
</>
);
return (<div style={{margin: '0.5em'}}>
<Toggle
label='CPU Power-save'
disabled={disabled}
checked={on === true}
onChange={(e, data) => setThrottle(data.checked)}
/>
{disabled && <HelpPopup content='CPU Power-save is not supported on this server'/>}
</div>);
}
export function Toggle({label, checked, disabled, onChange, icon}) {
// Custom toggle because Semantic UI does not handle inverted labels correctly.
const {i, t} = useContext(ThemeContext);
const inverted = i && i['inverted'] === true ? 'inverted' : '';
let style = {marginLeft: '1em'};
if (t && t.style) {
style = {...t.style, ...style};
}
disabled = disabled === true ? 'disabled' : '';
let inputClassName = `${disabled} ${inverted}`;
let sliderClassName = `${disabled} ${inverted} slider`;
if (inverted) {
style['color'] = '#dddddd';
}
if (disabled && inverted) {
style['color'] = '#888888';
}
let onMouseUp;
if (onChange) {
onMouseUp = (e) => {
if (disabled) {
return;
}
e.preventDefault();
if (onChange) {
onChange(!checked);
}
}
}
if (icon) {
icon = <Icon name={icon}/>
}
return <>
<div className='toggle' onMouseUp={onMouseUp}>
<input type="checkbox" className={inputClassName} checked={checked} onChange={onMouseUp}/>
<span className={sliderClassName}></span>
</div>
<span style={style}>
{icon}
{label}
</span>
</>
}
export function DisableDownloadsToggle() {
let {on, setDownloads} = useDownloaders();
return <Checkbox toggle
label={on ? 'Downloading Enabled' : 'Downloading Disabled'}
disabled={on === null}
checked={on === true}
onChange={(e, data) => setDownloads(data.checked)}
/>;
return <Form>
<Toggle toggle
label={on ? 'Downloading Enabled' : 'Downloading Disabled'}
disabled={on === null}
checked={on === true}
onChange={setDownloads}
/>
</Form>;
}
export function emptyToNull(obj) {
@ -750,10 +736,95 @@ export function emptyToNull(obj) {
return obj;
}
export function minutesToTimestamp(minutes) {
minutes = Math.round(minutes);
const hours = String(Math.floor(minutes / 60)).padStart(1, '0');
minutes = minutes - (hours * 60);
minutes = String(minutes).padStart(2, '0');
return `${hours}:${minutes}`
export function mimetypeColor(mimetype) {
try {
if (mimetype === 'application/pdf') {
return 'red'
} else if (mimetype && mimetype.startsWith('video/')) {
return 'blue'
} else if (mimetype && mimetype.startsWith('image/')) {
return 'pink'
} else if (mimetype && mimetype.startsWith('text/html')) {
return 'green'
} else if (mimetype && mimetype.startsWith('application/zip')) {
return 'purple'
}
} catch (e) {
console.error(e);
console.error('Unable to choose mimetype color');
}
return 'grey'
}
export function FileIcon({file, disabled = true, size = 'huge', ...props}) {
// Default to a grey file icon.
const {mimetype} = file;
props['name'] = 'file';
props['color'] = mimetypeColor(mimetype);
if (mimetype) {
if (mimetype.startsWith('text/html') || mimetype.startsWith('application/json')) {
props['name'] = 'file code';
} else if (mimetype.startsWith('application/pdf')) {
props['name'] = 'file pdf';
} else if (mimetype.startsWith('text/plain')) {
props['name'] = 'file text';
} else if (mimetype.startsWith('image/')) {
props['name'] = 'image';
} else if (mimetype.startsWith('video/')) {
props['name'] = 'film';
} else if (mimetype.startsWith('application/zip')) {
props['name'] = 'file archive';
} else if (mimetype.startsWith('application/x-iso9660-image')) {
props['name'] = 'dot circle';
}
}
return <Icon disabled={disabled} size={size} {...props}/>
}
export function LoadStatistic({label, value, cores, ...props}) {
let color;
const quarter = cores / 4;
if (cores && value >= (quarter * 3)) {
color = 'red';
} else if (cores && value >= (quarter * 2)) {
color = 'orange';
}
return <Statistic label={label} value={value ? parseFloat(value).toFixed(1) : '?'} color={color}
{...props}/>;
}
export function DarkModeToggle() {
const {theme, setDarkTheme, setLightTheme} = useContext(ThemeContext);
const darkMode = theme === darkTheme;
const toggleDarkMode = () => {
if (darkMode) {
setLightTheme();
} else {
setDarkTheme();
}
}
return <>
<Toggle checked={darkMode} onChange={toggleDarkMode} icon={darkMode ? 'moon' : 'sun'}/>
</>
}
export function useTitle(title) {
const documentDefined = typeof document !== 'undefined';
const originalTitle = React.useRef(documentDefined ? document.title : null);
useEffect(() => {
if (!documentDefined) {
return;
}
const newTitle = `${title} - WROLPi`
if (title && document.title !== newTitle) {
document.title = newTitle;
}
return () => {
document.title = originalTitle.current;
}
}, []);
}

563
app/src/components/Files.js
View file

@ -1,11 +1,41 @@
import React from "react";
import React, {useContext} from "react";
import FileBrowser from 'react-keyed-file-browser';
import {Button, Confirm, Header, Icon, Modal} from "semantic-ui-react";
import {
Card,
Checkbox,
Confirm,
Divider,
Dropdown,
Image, PlaceholderLine,
TableBody,
TableCell,
TableHeader,
TableHeaderCell,
TableRow,
} from "semantic-ui-react";
import 'react-keyed-file-browser/dist/react-keyed-file-browser.css';
import {deleteFile, refreshFiles} from "../api";
import {Paginator, humanFileSize, PageContainer, SearchInput, TabLinks} from "./Common";
import {useBrowseFiles, useSearchFiles} from "../hooks/customHooks";
import {Route} from "react-router-dom";
import {deleteFile, refreshDirectoryFiles, refreshFiles} from "../api";
import {
ArchiveRow,
CardGroupCentered,
ExternalCardLink,
FileIcon,
humanFileSize,
mimetypeColor,
PageContainer,
Paginator,
textEllipsis,
uploadDate,
VideoCard,
VideoRow
} from "./Common";
import {useBrowseFiles, useQuery, useSearchFiles} from "../hooks/customHooks";
import {Route, Routes} from "react-router-dom";
import {CardPlacholder} from "./Placeholder";
import {ArchiveCard} from "./Archive";
import Grid from "semantic-ui-react/dist/commonjs/collections/Grid";
import {ThemeContext} from "../contexts/contexts";
import {Button, Icon, Placeholder, Segment, Table} from "./Theme";
const icons = {
File: <Icon name='file'/>,
@ -16,42 +46,33 @@ const icons = {
};
export function Files() {
const {searchFiles, setPage, totalPages, activePage, searchStr, setSearchStr} = useSearchFiles(50);
const {t} = useContext(ThemeContext);
const {browseFiles, setOpenFolders, fetchFiles} = useBrowseFiles();
const [deleteOpen, setDeleteOpen] = React.useState(false);
const [previewOpen, setPreviewOpen] = React.useState(false);
const [selectedFile, setSelectedFile] = React.useState(null);
const [selectedDirectory, setSelectedDirectory] = React.useState(null);
const handleFolderChange = async (file, browserProps) => {
let openFolders = Object.keys(browserProps.openFolders);
setOpenFolders(openFolders);
}
const onDownloadFile = async () => {
// Open file in new tab/window.
window.open(`/media/${selectedFile.key}`);
}
const onSelectFile = async (file) => {
setSelectedFile(file);
openPreview();
const onSelect = async (file) => {
// A user can only select one file or one directory.
if (file.key.endsWith('/')) {
setSelectedFile(null);
setSelectedDirectory(file);
} else {
setSelectedDirectory(null);
setSelectedFile(file);
}
}
const onDeleteFile = async () => {
await deleteFile(selectedFile.key);
await fetchFiles();
closePreview();
}
const openPreview = () => {
setPreviewOpen(true);
}
const closePreview = () => {
setPreviewOpen(false);
setSelectedFile(null);
setDeleteOpen(false);
}
const openDelete = () => {
@ -62,95 +83,427 @@ export function Files() {
setDeleteOpen(false);
}
let selectedModal = <></>;
if (selectedFile) {
selectedModal = (
<Modal closeIcon
open={previewOpen}
onClose={closePreview}
>
<Modal.Header><Icon name='file'/>File Preview</Modal.Header>
<Modal.Content>
<Header>{selectedFile.name}</Header>
<p>{humanFileSize(selectedFile.size)}</p>
</Modal.Content>
<Modal.Actions>
<Button color='red' onClick={openDelete} floated='left'>Delete</Button>
<Confirm
open={deleteOpen}
content={`Are you sure you want to delete ${selectedFile.key}`}
onCancel={closeDelete}
onConfirm={onDeleteFile}
/>
<Button color='green' onClick={onDownloadFile}>Download</Button>
</Modal.Actions>
</Modal>);
const clearSelection = async (e) => {
e.preventDefault();
setOpenFolders(null);
setSelectedDirectory(null);
setSelectedFile(null);
await fetchFiles();
}
let browser;
let pagination = null;
if (searchStr) {
browser = (<FileBrowser
showActionBar={false}
canFilter={false}
files={searchFiles}
icons={icons}
onSelectFile={onSelectFile}
detailRenderer={() => <></>} // Hide the preview that the 3rd party provided.
/>);
pagination = (
<div style={{marginTop: '3em', textAlign: 'center'}}>
<Paginator
activePage={activePage}
changePage={setPage}
totalPages={totalPages}
/>
</div>);
let clearButton = <Button
onClick={clearSelection}
disabled={!!!selectedFile && !!!selectedDirectory}>
Clear
</Button>;
let buttons;
if (selectedFile) {
buttons = <>
<ExternalCardLink to={`/media/${selectedFile.key}`}>
<Button primary>Open</Button>
</ExternalCardLink>
<Confirm
open={deleteOpen}
onCancel={closeDelete}
onConfirm={onDeleteFile}
content={`Are you sure you want to delete: ${selectedFile.key}`}
confirmButton='Delete'
/>
<Button floated='right' color='red' onClick={openDelete}>Delete</Button>
</>;
} else if (selectedDirectory) {
buttons = <>
<DirectoryFilesRefreshButton directory={selectedDirectory.key}/>
</>;
} else {
browser = (<FileBrowser
showActionBar={false}
canFilter={false}
files={browseFiles}
icons={icons}
onFolderOpen={handleFolderChange}
onFolderClose={handleFolderChange}
onSelectFile={onSelectFile}
detailRenderer={() => <></>} // Hide the preview that the 3rd party provided.
/>);
buttons = <FilesRefreshButton/>;
}
return <>
<SearchInput initValue={searchStr} onSubmit={setSearchStr}/>
{selectedModal}
{browser}
{pagination}
{clearButton}
{buttons}
<Divider/>
<div {...t}>
<FileBrowser
showActionBar={false}
canFilter={false}
files={browseFiles}
icons={icons}
onFolderOpen={handleFolderChange}
onFolderClose={handleFolderChange}
onSelect={onSelect}
detailRenderer={() => <></>} // Hide the preview that the 3rd party provided.
/>
</div>
</>
}
function ManageFiles() {
return (<>
<Button secondary
id='refresh_files'
onClick={refreshFiles}>
Refresh Files
</Button>
<label htmlFor='refresh_files'>
Find and index any new files.
</label>
</>)
function ImageCard({file}) {
const {s} = useContext(ThemeContext);
const url = `/media/${encodeURIComponent(file.path)}`;
let poster = <center className='card-icon'><FileIcon file={file}/></center>;
if (file.size && file.size < 50000000) {
// Image is less than 5mb, use it.
poster = <Image wrapped
src={url}
style={{position: 'relative', width: '100%'}}
/>;
}
return <Card color={mimetypeColor(file.mimetype)}>
<ExternalCardLink to={url}>
{poster}
</ExternalCardLink>
<Card.Content {...s}>
<Card.Header>
<ExternalCardLink to={url} className='no-link-underscore card-link'>
<p>{textEllipsis(file.title || file.stem || file.path, 100)}</p>
</ExternalCardLink>
</Card.Header>
<Card.Meta {...s}>
<p>{uploadDate(file.modified / 1000)}</p>
</Card.Meta>
<Card.Description {...s}>
<p>{humanFileSize(file.size)}</p>
</Card.Description>
</Card.Content>
</Card>
}
export function FilesRoute(props) {
const links = [
{text: 'Files', to: '/files', exact: true, key: 'files'},
{text: 'Manage', to: '/files/manage', exact: true, key: 'manage'},
];
function FileCard({file}) {
const {s} = useContext(ThemeContext);
if (file.model === 'video' && file['video']) {
return <VideoCard key={file['path']} file={file}/>;
} else if (file.model === 'archive' && file['archive']) {
return <ArchiveCard key={file['path']} file={file}/>;
} else if (file.mimetype && file.mimetype.startsWith('image/')) {
return <ImageCard key={file['path']} file={file}/>;
}
const url = `/media/${encodeURIComponent(file.path)}`;
const color = mimetypeColor(file.mimetype);
return <Card color={color}>
<ExternalCardLink to={url}>
<center className='card-icon'>
<FileIcon file={file}/>
</center>
</ExternalCardLink>
<Card.Content {...s}>
<Card.Header>
<ExternalCardLink to={url}>
{textEllipsis(file.title || file.stem || file.path)}
</ExternalCardLink>
</Card.Header>
</Card.Content>
</Card>
}
export function FileCards({files}) {
if (files && files.length > 0) {
return <CardGroupCentered>
{files.map((i) => <FileCard key={i['path']} file={i}/>)}
</CardGroupCentered>
} else if (files && files.length === 0) {
return <p>No files found.</p>
} else {
return <CardGroupCentered><CardPlacholder/></CardGroupCentered>
}
}
function ImageRow({file, checkbox}) {
const url = `/media/${encodeURIComponent(file.path)}`;
let poster = <FileIcon file={file} size='large'/>;
if (file.size && file.size < 50000000) {
// Image is less than 5mb, use it.
poster = <Image wrapped src={url} width='50px'/>;
}
return (
<PageContainer>
<TabLinks links={links}/>
<Route path='/files' exact component={Files}/>
<Route path='/files/manage' exact component={ManageFiles}/>
</PageContainer>
);
<Table.Row>
{checkbox}
<Table.Cell>
<center>
{poster}
</center>
</Table.Cell>
<Table.Cell>
<ExternalCardLink to={url}>
<p>{textEllipsis(file.title || file.stem || file.path, 100)}</p>
</ExternalCardLink>
</Table.Cell>
</Table.Row>
)
}
function FileRow({file, selectOn, onSelect}) {
let checkbox;
if (selectOn) {
const localOnSelect = (e, {checked}) => {
try {
onSelect(file.key, checked);
} catch (e) {
console.error('No onSelect declared');
}
};
checkbox = <Table.Cell>
<Checkbox onChange={localOnSelect}/>
</Table.Cell>
}
if (file.model === 'video' && 'video' in file) {
return <VideoRow key={file['path']} file={file} checkbox={checkbox}/>;
} else if (file.model === 'archive' && 'archive' in file) {
return <ArchiveRow key={file['path']} file={file} checkbox={checkbox}/>;
} else if (file.mimetype && file.mimetype.startsWith('image/')) {
return <ImageRow key={file['path']} file={file} checkbox={checkbox}/>;
}
const onDownloadFile = async () => window.open(`/media/${file.key}`);
return <TableRow>
{checkbox}
<TableCell>
<center><FileIcon file={file} size='large'/></center>
</TableCell>
<TableCell onClick={() => onDownloadFile()}>
{textEllipsis(file.title)}
</TableCell>
</TableRow>
}
export function FileTable({files, selectOn, onSelect}) {
if (files && files.length > 0) {
return (
<Table unstackable striped compact='very'>
<TableHeader>
<TableRow>
{selectOn && <TableHeaderCell/>}
<TableHeaderCell>Poster</TableHeaderCell>
<TableHeaderCell>Title</TableHeaderCell>
</TableRow>
</TableHeader>
<TableBody>
{files.map(i => <FileRow key={i['path']} file={i} selectOn={selectOn} onSelect={onSelect}/>)}
</TableBody>
</Table>
)
} else if (files && files.length === 0) {
return <Segment><p>No results!</p></Segment>
} else {
return <Placeholder>
<PlaceholderLine/>
<PlaceholderLine/>
<PlaceholderLine/>
</Placeholder>
}
}
export function FilesView({
files,
limit,
setLimit,
showLimit = false,
activePage,
totalPages,
view,
setView,
showView = true,
showSelect = false,
onSelect,
setPage,
menuColumnsCount,
menuColumns,
filterOptions,
activeFilters,
setFilters,
multipleFilters,
filterPlaceholder,
}) {
let selectButton;
const [selectOn, setSelectOn] = React.useState(false);
const toggleSelectOn = () => setSelectOn(!selectOn);
const selectButtonDisabled = view !== 'list';
if (showSelect && selectOn) {
selectButton = <Button active disabled={selectButtonDisabled} icon='checkmark box'
onClick={toggleSelectOn}/>;
} else if (showSelect) {
selectButton = <Button icon='checkmark box' disabled={selectButtonDisabled}
onClick={toggleSelectOn}/>;
}
let viewButton;
if (showView) {
if (view === 'list') {
viewButton = <Button icon='th' onClick={() => setView('cards')}/>;
} else {
viewButton = <Button icon='browser' onClick={() => setView('list')}/>;
}
}
let limitDropdown;
const limitOptions = [
{key: 12, value: 12, text: '12'},
{key: 24, value: 24, text: '24'},
{key: 48, value: 48, text: '48'},
{key: 96, value: 96, text: '96'},
];
if (showLimit) {
limitDropdown = <Dropdown
compact
selection
placeholder='Limit'
options={limitOptions}
value={parseInt(limit || limitOptions[0]['value'])}
onChange={(e, {value}) => setLimit(value)}
/>
}
const paginator = <center style={{marginTop: '2em'}}>
<Paginator activePage={activePage} totalPages={totalPages} onPageChange={setPage}/>
</center>;
let body;
if (view === 'list') {
body = <FileTable files={files} selectOn={selectOn} onSelect={onSelect}/>;
} else {
body = <FileCards files={files}/>;
}
const filtersDropdown = <Dropdown selection clearable search
multiple={multipleFilters || undefined}
options={filterOptions}
placeholder={filterPlaceholder || 'Filters'}
onChange={(e, {value}) => setFilters(value)}
style={{marginLeft: '0.3em'}}
value={activeFilters}
/>;
return <>
<Grid columns={menuColumnsCount || 1} stackable>
<Grid.Column mobile={8} computer={6}>
{selectButton}
{viewButton}
{limitDropdown}
{filterOptions && filtersDropdown}
</Grid.Column>
{menuColumns}
</Grid>
{body}
{paginator}
</>
}
export function FilesSearchView({
showView = true,
showLimit = false,
showSelect = false,
emptySearch = false,
filterOptions,
mimetype,
model,
onSelect,
setFilters,
}) {
const {searchFiles, limit, setLimit, totalPages, activePage, setPage} =
useSearchFiles(24, emptySearch, mimetype, model);
const {searchParams, updateQuery} = useQuery();
const setView = (value) => updateQuery({view: value});
const view = searchParams.get('view');
const setMimetype = (value) => updateQuery({'mimetype': value});
filterOptions = filterOptions || [
{key: 'video', text: 'Video', value: 'video'},
{key: 'archive', text: 'Archive', value: 'text/html'},
{key: 'pdf', text: 'PDF', value: 'application/pdf'},
{key: 'zip', text: 'ZIP', value: 'application/zip'},
{key: 'image', text: 'Image', value: 'image'},
];
return <FilesView
files={searchFiles}
limit={limit}
setLimit={setLimit}
showLimit={showLimit}
activePage={activePage}
totalPages={totalPages}
view={view}
setView={setView}
showView={showView}
showSelect={showSelect}
onSelect={onSelect}
setPage={setPage}
filterOptions={filterOptions}
setFilters={setFilters || setMimetype}
/>
}
function FilesRefreshButton() {
const {t} = useContext(ThemeContext);
const [loading, setLoading] = React.useState(false);
const [name, setName] = React.useState('refresh');
const handleClick = async () => {
setLoading(true);
await refreshFiles();
setLoading(false);
setName('check');
}
return <>
<Button icon
labelPosition='left'
loading={loading}
id='refresh_files'
onClick={handleClick}>
<Icon name={name}/>
Refresh Files
</Button>
<label htmlFor='refresh_files' {...t}>
Find and index all files in the media directory.
</label>
</>;
}
function DirectoryFilesRefreshButton({directory}) {
const {t} = useContext(ThemeContext);
const [loading, setLoading] = React.useState(false);
const [name, setName] = React.useState('refresh');
const handleClick = async () => {
setLoading(true);
await refreshDirectoryFiles(directory);
setLoading(false);
setName('check');
}
return <>
<Button icon primary
labelPosition='left'
loading={loading}
id='refresh_files'
onClick={handleClick}>
<Icon name={name}/>
Refresh Directory
</Button>
<label htmlFor='refresh_files' {...t}>
Find and index all files in the directory.
</label>
</>;
}
export function FilesRoute() {
return (<PageContainer>
<Routes>
<Route path='/' exact element={<Files/>}/>
</Routes>
</PageContainer>);
}

877
app/src/components/Inventory.js
View file

File diff suppressed because it is too large Load diff

129
app/src/components/Map.js
View file

@ -1,10 +1,19 @@
import React from "react";
import {HelpPopup, humanFileSize, PageContainer, TabLinks, WROLModeMessage} from "./Common";
import {Route} from "react-router-dom";
import {Route, Routes} from "react-router-dom";
import {getMapImportStatus, importMapFiles} from "../api";
import Table from "semantic-ui-react/dist/commonjs/collections/Table";
import {Button, Checkbox, Icon, Loader, Placeholder} from "semantic-ui-react";
import {
Checkbox,
PlaceholderLine,
TableBody,
TableCell,
TableFooter,
TableHeader,
TableHeaderCell,
TableRow
} from "semantic-ui-react";
import Message from "semantic-ui-react/dist/commonjs/collections/Message";
import {Button, Icon, Loader, Placeholder, Table} from "./Theme";
class ManageMap extends React.Component {
constructor(props) {
@ -76,28 +85,28 @@ class ManageMap extends React.Component {
let ref = React.createRef();
const {size, path, imported, time_to_import} = pbf;
let sizeCells = (<>
<Table.Cell>{humanFileSize(size)}</Table.Cell>
<Table.Cell>{humanFileSize(size * 25)}</Table.Cell>
<Table.Cell>{time_to_import}</Table.Cell>
<TableCell>{humanFileSize(size)}</TableCell>
<TableCell>{humanFileSize(size * 25)}</TableCell>
<TableCell>{time_to_import}</TableCell>
</>);
return <Table.Row key={path}>
<Table.Cell collapsing>
return <TableRow key={path}>
<TableCell collapsing>
<Checkbox
disabled={disabled}
ref={ref}
onChange={(e, data) => this.handleCheckbox(data.checked, pbf)}
/>
</Table.Cell>
<Table.Cell>
</TableCell>
<TableCell>
<a href={`/media/${path}`}>{path}</a>
</Table.Cell>
<Table.Cell>
</TableCell>
<TableCell>
{path === this.state.importing ?
<Loader active inline size='mini'/> :
imported ? 'yes' : 'no'}
</Table.Cell>
{size !== null ? sizeCells : <Table.Cell colSpan={3}/>}
</Table.Row>
</TableCell>
{size !== null ? sizeCells : <TableCell colSpan={3}/>}
</TableRow>
}
render() {
@ -170,19 +179,19 @@ class ManageMap extends React.Component {
let rows;
if (!files) {
// Fetch request is not complete.
rows = <Table.Row>
<Table.Cell/>
<Table.Cell colSpan={6}>
rows = <TableRow>
<TableCell/>
<TableCell colSpan={6}>
<Placeholder>
<Placeholder.Line/>
<Placeholder.Line/>
<PlaceholderLine/>
<PlaceholderLine/>
</Placeholder>
</Table.Cell>
</Table.Row>;
</TableCell>
</TableRow>;
} else if (files.length === 0) {
rows = <Table.Row>
<Table.Cell/><Table.Cell colSpan={5}>No PBF map files were found in <b>map/pbf</b></Table.Cell>
</Table.Row>;
rows = <TableRow>
<TableCell/><TableCell colSpan={5}>No PBF map files were found in <b>map/pbf</b></TableCell>
</TableRow>;
} else {
rows = files.map(i => this.tableRow(i, disabled));
}
@ -191,37 +200,35 @@ class ManageMap extends React.Component {
content='Upon importing, a PBF file will consume more disk space than the original file.'/>;
let timeHelpPopup = <HelpPopup content='Estimated for a Raspberry Pi 4'/>;
return (
<PageContainer>
<WROLModeMessage content='Cannot modify Map'/>
{dockerWarning}
{downloadMessage}
{importingLoader}
<Table>
<Table.Header>
<Table.Row>
<Table.HeaderCell/>
<Table.HeaderCell>PBF File</Table.HeaderCell>
<Table.HeaderCell>Imported</Table.HeaderCell>
<Table.HeaderCell>Size</Table.HeaderCell>
<Table.HeaderCell>Space Required {spaceHelpPopup}</Table.HeaderCell>
<Table.HeaderCell>Time to Import {timeHelpPopup}</Table.HeaderCell>
</Table.Row>
</Table.Header>
<Table.Body>
{rows}
</Table.Body>
<Table.Footer>
<Table.Row>
<Table.HeaderCell/>
<Table.HeaderCell colSpan={5}>
{importButton}
</Table.HeaderCell>
</Table.Row>
</Table.Footer>
</Table>
</PageContainer>
);
return <PageContainer>
<WROLModeMessage content='Cannot modify Map'/>
{dockerWarning}
{downloadMessage}
{importingLoader}
<Table>
<TableHeader>
<TableRow>
<TableHeaderCell/>
<TableHeaderCell>PBF File</TableHeaderCell>
<TableHeaderCell>Imported</TableHeaderCell>
<TableHeaderCell>Size</TableHeaderCell>
<TableHeaderCell>Space Required {spaceHelpPopup}</TableHeaderCell>
<TableHeaderCell>Time to Import {timeHelpPopup}</TableHeaderCell>
</TableRow>
</TableHeader>
<TableBody>
{rows}
</TableBody>
<TableFooter>
<TableRow>
<TableHeaderCell/>
<TableHeaderCell colSpan={5}>
{importButton}
</TableHeaderCell>
</TableRow>
</TableFooter>
</Table>
</PageContainer>
}
}
@ -241,13 +248,15 @@ function MapApp() {
export function MapRoute() {
const links = [
{text: 'Map', to: '/map', exact: true, key: 'map'},
{text: 'Manage', to: '/map/manage', exact: true, key: 'manage'},
{text: 'Map', to: '/map', key: 'map', end: true},
{text: 'Manage', to: '/map/manage', key: 'manage'},
];
return <div style={{marginTop: '2em'}}>
<TabLinks links={links}/>
<Route path='/map' exact component={MapApp}/>
<Route path='/map/manage' exact component={ManageMap}/>
<Routes>
<Route path='/' exact element={<MapApp/>}/>
<Route path='manage' exact element={<ManageMap/>}/>
</Routes>
</div>
}

23
app/src/components/Nav.js
View file

@ -1,6 +1,9 @@
import React from "react";
import React, {useContext} from "react";
import {NavLink} from "react-router-dom";
import {Dropdown, Menu, Responsive} from "semantic-ui-react";
import {SettingsContext, ThemeContext} from "../contexts/contexts";
import Icon from "semantic-ui-react/dist/commonjs/elements/Icon";
import {DarkModeToggle} from "./Common";
const responsiveWidth = 500;
@ -43,11 +46,12 @@ function MenuLink(props) {
}
if (!props.link.links) {
const exact = props.link.exact ? {exact: null} : {};
return (
<NavLink
className={classes}
exact={props.link.exact || false}
to={props.link.to}
{...exact}
>
{props.link.text}
</NavLink>
@ -60,10 +64,15 @@ function MenuLink(props) {
}
export function NavBar() {
const settings = useContext(SettingsContext);
const wrol_mode = settings ? settings.wrol_mode : null;
const topNavText = wrol_mode ? <>WROLPi&nbsp; <Icon name='lock'/></> : 'WROLPi';
const {i} = useContext(ThemeContext);
return (
<Menu>
<Menu {...i}>
{/*Always show WROLPi home button*/}
<MenuLink link={{to: '/', text: 'WROLPi', exact: true}}/>
<MenuLink link={{to: '/', text: topNavText, exact: true}}/>
{/*Show the links in a menu when on desktop*/}
{links.map((link) => {
@ -72,11 +81,17 @@ export function NavBar() {
)
})}
<Responsive minWidth={responsiveWidth} as={Menu.Menu} position="right">
<div style={{margin: '0.8em'}}>
<DarkModeToggle/>
</div>
{rightLinks.map((link) => <MenuLink link={link} key={link.key}/>)}
</Responsive>
{/*Show the menu items in a dropdown when on mobile*/}
<Responsive as={Menu.Menu} maxWidth={responsiveWidth - 1} position='right'>
<div style={{margin: '0.8em'}}>
<DarkModeToggle/>
</div>
<Dropdown item icon="bars">
<Dropdown.Menu>
{collapsedLinks.map((link) =>

61
app/src/components/Placeholder.js
View file

@ -1,21 +1,29 @@
import {Card, Placeholder} from "semantic-ui-react";
import React from "react";
import {Card, PlaceholderImage, PlaceholderLine} from "semantic-ui-react";
import React, {useContext} from "react";
import {ThemeContext} from "../contexts/contexts";
import {Placeholder} from "./Theme";
export function CardPlacholder() {
const {s} = useContext(ThemeContext);
return <Card>
<Placeholder>
<PlaceholderImage rectangular/>
</Placeholder>
<Card.Content {...s}>
<Placeholder>
<PlaceholderLine/>
<PlaceholderLine/>
<PlaceholderLine/>
</Placeholder>
</Card.Content>
</Card>
}
export function VideoPlaceholder() {
return (
<Card.Group doubling stackable>
<Card>
<Placeholder>
<Placeholder.Image rectangular/>
</Placeholder>
<Card.Content>
<Placeholder>
<Placeholder.Line/>
<Placeholder.Line/>
<Placeholder.Line/>
</Placeholder>
</Card.Content>
</Card>
<CardPlacholder/>
</Card.Group>
)
}
@ -23,29 +31,8 @@ export function VideoPlaceholder() {
export function ChannelPlaceholder() {
return (
<Placeholder>
<Placeholder.Line length='long'/>
<Placeholder.Line length='short'/>
<PlaceholderLine length='long'/>
<PlaceholderLine length='short'/>
</Placeholder>
)
}
export function ArchivePlaceholder() {
return (
<Card.Group doubling stackable>
<Card>
<Placeholder>
<Placeholder.Image rectangular/>
</Placeholder>
<Card.Content>
<Placeholder>
<Placeholder.Line/>
<Placeholder.Line/>
</Placeholder>
<Placeholder style={{height: 40, width: 40}}>
<Placeholder.Image rectangular/>
</Placeholder>
</Card.Content>
</Card>
</Card.Group>
)
}

154
app/src/components/Theme.js Normal file
View file

@ -0,0 +1,154 @@
import React, {useContext} from 'react';
import {ThemeContext} from "../contexts/contexts";
import {
Accordion as ACCORDION,
Button as BUTTON,
Form as FORM,
FormField as FORM_FIELD,
FormGroup as FORM_GROUP,
FormInput as FORM_INPUT,
Header as HEADER,
Icon as ICON,
Loader as LOADER,
Menu as MENU,
Placeholder as PLACEHOLDER,
Popup as POPUP,
Progress as PROGRESS,
Segment as SEGMENT,
Statistic as STATISTIC,
StatisticGroup as STATISTIC_GROUP,
Tab as TAB,
Table as TABLE,
TabPane as TAB_PANE,
TextArea as TEXTAREA
} from "semantic-ui-react";
// Simple wrappers for Semantic UI elements to use the current theme.
const invertedNull = (props) => {
if (props['inverted'] === true) {
props['inverted'] = null;
}
return props;
}
export function Button(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <BUTTON {...props}/>
}
export function Accordion(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <ACCORDION {...props}/>
}
export function Header(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <HEADER {...props}/>
}
export function Form(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <FORM {...props}/>
}
export function FormField(props) {
const {i} = useContext(ThemeContext);
props = invertedNull({...i, ...props});
return <FORM_FIELD {...props}/>
}
export function FormGroup(props) {
const {i} = useContext(ThemeContext);
props = invertedNull({...i, ...props});
return <FORM_GROUP {...props}/>
}
export function FormInput(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <FORM_INPUT {...props}/>
}
export function Icon(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <ICON {...props}/>
}
export function Loader(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <LOADER {...props}/>
}
export function Menu(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <MENU {...props}/>
}
export function Placeholder(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <PLACEHOLDER {...props}/>
}
export function Popup(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <POPUP {...props}/>
}
export function Progress(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <PROGRESS {...props}/>
}
export function Segment(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <SEGMENT {...props}/>
}
export function Statistic(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <STATISTIC {...props}/>
}
export function StatisticGroup(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <STATISTIC_GROUP {...props}/>
}
export function Tab(props) {
const {i} = useContext(ThemeContext);
props = invertedNull({...i, ...props});
return <TAB {...props}/>
}
export function TabPane(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <TAB_PANE {...props}/>
}
export function Table(props) {
const {i} = useContext(ThemeContext);
props = {...i, ...props};
return <TABLE {...props}/>
}
export function TextArea(props) {
const {i} = useContext(ThemeContext);
props = invertedNull({...i, ...props});
return <TEXTAREA {...props}/>
}

200
app/src/components/Upload.js
View file

@ -1,9 +1,11 @@
import React, {useState} from "react";
import {Button, Form, Header, Loader, TextArea} from "semantic-ui-react";
import {getDownloaders, postDownload} from "../api";
import {frequencyOptions, rssFrequencyOptions, WROLModeMessage} from "./Common";
import Icon from "semantic-ui-react/dist/commonjs/elements/Icon";
import Message from "semantic-ui-react/dist/commonjs/collections/Message";
import {ThemeContext} from "../contexts/contexts";
import {Button, Form, FormInput, Header, Loader, TextArea} from "./Theme";
import {FormDropdown} from "semantic-ui-react";
const validUrl = /^(http|https):\/\/[^ "]+$/;
@ -55,22 +57,22 @@ class Downloader extends React.Component {
let disabled = !this.state.urls || !this.state.valid || this.state.pending || !this.state.downloader;
const {header} = this.props;
return (
<Form onSubmit={this.submitDownload}>
<WROLModeMessage content='Downloading is disabled while WROL Mode is enabled'/>
<Header as='h3'>{header}</Header>
<Form.Field>
return <ThemeContext.Consumer>
{({i}) => (
<Form onSubmit={this.submitDownload}>
<WROLModeMessage content='Downloading is disabled while WROL Mode is enabled'/>
<Header as='h3'>{header}</Header>
<TextArea required
placeholder={'Enter one URL per line'}
name='urls'
onChange={this.handleInputChange}
value={this.state.urls}
/>
</Form.Field>
<Button content='Cancel' onClick={this.props.clearSelected}/>
<Button primary style={{marginTop: '1em'}} disabled={disabled}>Download</Button>
</Form>
)
<Button content='Cancel' onClick={this.props.clearSelected}/>
<Button primary style={{marginTop: '1em'}} disabled={disabled}>Download</Button>
</Form>
)}
</ThemeContext.Consumer>
}
}
@ -134,42 +136,41 @@ class ChannelDownload extends React.Component {
in the future.</Message.Content>
</Message>);
return (
<Form>
<Header as='h4'><Icon name='video'/> Channel / Playlist</Header>
<Form.Input
required
label='URL'
placeholder='https://example.com/channel/videos'
value={url}
error={error}
onChange={this.handleUrlChange}
/>
<Form.Dropdown
required
selection
label='Download Frequency'
name='download_frequency'
placeholder='Frequency'
options={this.freqOptions}
value={frequency}
selected={frequency}
onChange={this.handleFrequencyChange}
/>
return <Form>
<WROLModeMessage content='Downloading is disabled while WROL Mode is enabled'/>
<Header as='h4'><Icon name='video'/> Channel / Playlist</Header>
<FormInput
required
label='URL'
placeholder='https://example.com/channel/videos'
value={url}
error={error}
onChange={this.handleUrlChange}
/>
<FormDropdown
required
selection
label='Download Frequency'
name='download_frequency'
placeholder='Frequency'
options={this.freqOptions}
value={frequency}
selected={frequency}
onChange={this.handleFrequencyChange}
/>
{frequency === 0 && onceMessage}
{frequency === 0 && onceMessage}
<Button content='Cancel' onClick={this.props.clearSelected}/>
<Button
color='blue'
content='Download'
onClick={this.handleSubmit}
disabled={buttonDisabled}
/>
<Loader active={pending}/>
{success && <Icon name='check'/>}
</Form>
)
<Button content='Cancel' onClick={this.props.clearSelected}/>
<Button
color='blue'
content='Download'
onClick={this.handleSubmit}
disabled={buttonDisabled}
/>
<Loader active={pending}/>
{success && <Icon name='check'/>}
</Form>
}
}
@ -236,63 +237,68 @@ class RSSDownload extends ChannelDownload {
in the future.</Message.Content>
</Message>);
return (
<Form>
<Header as='h4'><Icon name='rss'/> RSS Feed</Header>
<Form.Input
required
label='URL'
placeholder='https://example.com/feed'
value={url}
error={error}
onChange={this.handleUrlChange}
/>
<Form.Dropdown
required
selection
label='Download Frequency'
name='download_frequency'
placeholder='Frequency'
options={this.freqOptions}
value={frequency}
selected={frequency}
onChange={this.handleFrequencyChange}
/>
{frequency === 0 && onceMessage}
<Form.Dropdown selection required
name='sub_downloader'
label='Downloader'
options={this.state.downloaders}
placeholder='Select a downloader'
onChange={this.handleInputChange}
/>
return <Form>
<WROLModeMessage content='Downloading is disabled while WROL Mode is enabled'/>
<Header as='h4'><Icon name='rss'/> RSS Feed</Header>
<FormInput
required
label='URL'
placeholder='https://example.com/feed'
value={url}
error={error}
onChange={this.handleUrlChange}
/>
<FormDropdown
required
selection
label='Download Frequency'
name='download_frequency'
placeholder='Frequency'
options={this.freqOptions}
value={frequency}
selected={frequency}
onChange={this.handleFrequencyChange}
/>
{frequency === 0 && onceMessage}
<FormDropdown selection required
name='sub_downloader'
label='Downloader'
options={this.state.downloaders}
placeholder='Select a downloader'
onChange={this.handleInputChange}
/>
<Button content='Cancel' onClick={this.props.clearSelected}/>
<Button
color='blue'
content='Download'
onClick={this.handleSubmit}
disabled={buttonDisabled}
/>
<Loader active={pending}/>
{success && <Icon name='check'/>}
</Form>
)
<Button content='Cancel' onClick={this.props.clearSelected}/>
<Button
color='blue'
content='Download'
onClick={this.handleSubmit}
disabled={buttonDisabled}
/>
<Loader active={pending}/>
{success && <Icon name='check'/>}
</Form>
}
}
export function Downloads() {
export function DownloadMenu({onOpen}) {
const [downloader, setDownloader] = useState();
const localOnOpen = (name) => {
setDownloader(name);
onOpen(name)
}
let body = (<>
<Button content='Videos' onClick={() => setDownloader('video')}/>
<Button content='Archive' onClick={() => setDownloader('archive')}/>
<Button content='Channel/Playlist' onClick={() => setDownloader('video_channel')}/>
<Button content='RSS Feed' onClick={() => setDownloader('rss')}/>
<Button content='Videos' onClick={() => localOnOpen('video')} style={{marginBottom: '1em'}}/>
<Button content='Archive' onClick={() => localOnOpen('archive')} style={{marginBottom: '1em'}}/>
<Button content='Channel/Playlist' onClick={() => localOnOpen('video_channel')}
style={{marginBottom: '1em'}}/>
<Button content='RSS Feed' onClick={() => localOnOpen('rss')} style={{marginBottom: '1em'}}/>
</>);
function clearSelected() {
setDownloader(null);
localOnOpen(null);
body = null;
}
@ -307,10 +313,8 @@ export function Downloads() {
body = downloader in downloaders ? downloaders[downloader] : body;
return (
<>
<Header as='h2'>Download</Header>
{body}
</>
)
return <>
<Header as='h2'>Download</Header>
{body}
</>
}

222
app/src/components/VideoPlayer.js
View file

@ -1,40 +1,57 @@
import React, {useState} from 'react';
import Icon from "semantic-ui-react/dist/commonjs/elements/Icon";
import React, {useContext, useState} from 'react';
import {deleteVideo, favoriteVideo} from "../api";
import Button from "semantic-ui-react/dist/commonjs/elements/Button";
import {Link} from "react-router-dom";
import {humanFileSize, humanNumber, secondsToTimestamp, uploadDate, VideoCard} from "./Common";
import {Confirm, Segment, Tab} from "semantic-ui-react";
import {Link, useNavigate, useParams} from "react-router-dom";
import {humanFileSize, humanNumber, PageContainer, secondsToTimestamp, uploadDate, useTitle, VideoCard} from "./Common";
import {Confirm} from "semantic-ui-react";
import Grid from "semantic-ui-react/dist/commonjs/collections/Grid";
import Container from "semantic-ui-react/dist/commonjs/elements/Container";
import {VideoPlaceholder} from "./Placeholder";
import {useChannel} from "../hooks/customHooks";
import {darkTheme, ThemeContext} from "../contexts/contexts";
import {Button, Header, Icon, Segment, Tab, TabPane} from "./Theme";
const MEDIA_PATH = '/media';
function VideoPage(props) {
let {video, channel} = props;
function VideoPage({videoFile, prevFile, nextFile, ...props}) {
const {theme} = useContext(ThemeContext);
const navigate = useNavigate();
let [deleteOpen, setDeleteOpen] = useState(false);
async function handleDeleteVideo(video_id) {
useTitle(videoFile === null || videoFile === undefined ? null : videoFile.video.title);
let video;
if (videoFile) {
video = videoFile.video;
}
// Get the Video's channel, fallback to the URL's channel id.
const {channelId} = useParams();
const {channel} = useChannel(video && video.channel_id ? video.channel_id : channelId);
if (videoFile === null) {
return <VideoPlaceholder/>;
}
if (videoFile === undefined) {
return <PageContainer><p>Could not find video</p></PageContainer>;
}
let favorite = video.favorite;
const handleDeleteVideo = async (video_id) => {
try {
await deleteVideo(video_id)
} catch (e) {
setDeleteOpen(false);
throw e;
}
props.history.goBack();
navigate(-1);
}
let videoUrl = `${MEDIA_PATH}/${encodeURIComponent(video.video_path)}`;
let posterUrl = null;
if (video.poster_path) {
posterUrl = `${MEDIA_PATH}/${encodeURIComponent(video.poster_path)}`;
}
let captionsUrl = null;
if (video.caption_path) {
captionsUrl = `${MEDIA_PATH}/${encodeURIComponent(video.caption_path)}`;
}
let posterUrl = video.poster_path ? `${MEDIA_PATH}/${encodeURIComponent(video.poster_path)}` : null;
let captionsUrl = video.caption_path ? `${MEDIA_PATH}/${encodeURIComponent(video.caption_path)}` : null;
let description = 'No description available.';
let viewCount = video.view_count;
@ -43,41 +60,33 @@ function VideoPage(props) {
viewCount = viewCount || video.info_json['view_count'];
}
let [favorite, setFavorite] = React.useState(video.favorite);
let handleFavorite = async (e) => {
e.preventDefault();
let new_favorite = await favoriteVideo(video.id, !!!favorite);
setFavorite(new_favorite);
favorite = await favoriteVideo(video.id, !!!favorite);
}
let favorite_button;
if (favorite) {
favorite_button = (
<Button color='green' style={{'margin': '0.5em'}}
onClick={handleFavorite}>
<Icon name='heart'/>
Unfavorite
</Button>
);
favorite_button = (<Button color='green' style={{'margin': '0.5em'}} onClick={handleFavorite}>
<Icon name='heart'/>
Unfavorite
</Button>);
} else {
favorite_button = (
<Button basic color='green' style={{'margin': '0.5em'}}
onClick={handleFavorite}>
<Icon name='heart'/>
Favorite
</Button>
);
favorite_button = (<Button basic color='green' style={{'margin': '0.5em'}} onClick={handleFavorite}>
<Icon name='heart'/>
Favorite
</Button>);
}
let descriptionPane = {
menuItem: 'Description', render: () => <Tab.Pane>
menuItem: 'Description', render: () => <TabPane>
<pre className="wrap-text">
{description || 'No description'}
</pre>
</Tab.Pane>
</TabPane>
};
let aboutPane = {
menuItem: 'About', render: () => <Tab.Pane>
menuItem: 'About', render: () => <TabPane>
<h3>Size</h3>
<p>{video.size ? humanFileSize(video.size) : '???'}</p>
@ -95,89 +104,88 @@ function VideoPage(props) {
<h3>File Modification Time</h3>
<p>{video.modification_datetime ? secondsToTimestamp(video.modification_datetime) : null}</p>
</Tab.Pane>
</TabPane>
}
let captionsPane = {
menuItem: 'Captions', render: () => <Tab.Pane>
menuItem: 'Captions', render: () => <TabPane>
<pre>{video.caption}</pre>
</Tab.Pane>
</TabPane>
};
let tabPanes = [descriptionPane, aboutPane, captionsPane];
const tabMenu = theme === darkTheme ? {inverted: true, attached: true} : {attached: true};
return (
<>
<Container>
<Button
style={{marginTop: '1em', marginBottom: '1em'}}
onClick={() => props.history.goBack()}
>
<Icon name='left arrow'/>
Back
</Button>
</Container>
<video controls
autoPlay={props.autoplay !== undefined ? props.autoplay : true}
poster={posterUrl}
id="player"
playsInline={true}
style={{maxWidth: '100%'}}
return (<>
<Container>
<Button
style={{marginTop: '1em', marginBottom: '1em'}}
onClick={() => navigate(-1)}
>
<source src={videoUrl} type="video/mp4"/>
<track kind="captions" label="English captions" src={captionsUrl} srcLang="en" default/>
</video>
<Icon name='left arrow'/>
Back
</Button>
</Container>
<Container style={{marginTop: '1em'}}>
<Segment>
<video controls
autoPlay={props.autoplay !== undefined ? props.autoplay : true}
poster={posterUrl}
id="player"
playsInline={true}
style={{maxWidth: '100%'}}
>
<source src={videoUrl} type="video/mp4"/>
<track kind="captions" label="English captions" src={captionsUrl} srcLang="en" default/>
</video>
<h2>{video.title || video.video_path}</h2>
{video.upload_date && <h3>{uploadDate(video.upload_date)}</h3>}
<h3>
{channel && <Link to={`/videos/channel/${channel.id}/video`}>
{channel.name}
</Link>}
</h3>
<Container style={{marginTop: '1em'}}>
<Segment>
<p>
{favorite_button}
<a href={videoUrl}>
<Button style={{margin: '0.5em'}}>
<Icon name='download'/>
Download
</Button>
</a>
<Button
color='red'
onClick={() => setDeleteOpen(true)}
style={{margin: '0.5em'}}
>Delete</Button>
<Confirm
open={deleteOpen}
content='Are you sure you want to delete this video? All files related to this video will be deleted. It will not be downloaded again!'
confirmButton='Delete'
onCancel={() => setDeleteOpen(false)}
onConfirm={() => handleDeleteVideo(video.id)}
/>
</p>
</Segment>
<Header as='h2'>{video.title || video.video_path}</Header>
{video.upload_date && <h3>{uploadDate(video.upload_date)}</h3>}
<h3>
{channel && <Link to={`/videos/channel/${channel.id}/video`}>
{channel.name}
</Link>}
</h3>
<Tab panes={tabPanes}/>
<p>
{favorite_button}
<a href={videoUrl}>
<Button style={{margin: '0.5em'}}>
<Icon name='download'/>
Download
</Button>
</a>
<Button
color='red'
onClick={() => setDeleteOpen(true)}
style={{margin: '0.5em'}}
>Delete</Button>
<Confirm
open={deleteOpen}
content='Are you sure you want to delete this video? All files related to this video will be deleted. It will not be downloaded again!'
confirmButton='Delete'
onCancel={() => setDeleteOpen(false)}
onConfirm={() => handleDeleteVideo(video.id)}
/>
</p>
</Segment>
<Grid columns={2} stackable>
<Grid.Row>
<Grid.Column textAlign='left'>
{props.prev && <><h3>Older</h3><VideoCard video={props.prev}/></>}
</Grid.Column>
<Grid.Column textAlign='left'>
{props.next && <><h3>Newer</h3><VideoCard video={props.next}/></>}
</Grid.Column>
</Grid.Row>
</Grid>
</Container>
</>
)
<Tab menu={tabMenu} panes={tabPanes}/>
<Grid columns={2} stackable>
<Grid.Row>
<Grid.Column textAlign='left'>
{prevFile && <><Header as='h3'>Older</Header><VideoCard file={prevFile}/></>}
</Grid.Column>
<Grid.Column textAlign='left'>
{nextFile && <><Header as='h3'>Newer</Header><VideoCard file={nextFile}/></>}
</Grid.Column>
</Grid.Row>
</Grid>
</Container>
</>)
}
export default VideoPage;

728
app/src/components/Videos.js
View file

@ -1,619 +1,193 @@
import React from 'react';
import {Link, Route} from "react-router-dom";
import OldPaginator, {
changePageHistory,
DEFAULT_LIMIT,
import {Link, Route, Routes, useParams} from "react-router-dom";
import {
defaultSearchOrder,
defaultVideoOrder,
humanFileSize,
PageContainer,
scrollToTop,
searchOrders,
secondsToString,
SearchInput,
TabLinks,
VideoCards,
videoOrders,
WROLModeMessage
useTitle,
videoOrders
} from "./Common"
import VideoPage from "./VideoPlayer";
import {download, getChannel, getStatistics, getVideo, refresh, searchVideos} from "../api";
import {
Button,
Dropdown,
Form,
Grid,
Header,
Icon,
Input,
Loader,
Modal,
Radio,
Segment,
Statistic
} from "semantic-ui-react";
import * as QueryString from 'query-string';
import Container from "semantic-ui-react/dist/commonjs/elements/Container";
import {Dropdown, PlaceholderHeader, PlaceholderLine, StatisticLabel, StatisticValue} from "semantic-ui-react";
import {Channels, EditChannel, NewChannel} from "./Channels";
import {VideoPlaceholder} from "./Placeholder";
import {useChannel, useQuery, useSearchVideos, useVideo, useVideoStatistics} from "../hooks/customHooks";
import {FilesView} from "./Files";
import Grid from "semantic-ui-react/dist/commonjs/collections/Grid";
import Icon from "semantic-ui-react/dist/commonjs/elements/Icon";
import {Header, Loader, Placeholder, Segment, Statistic, StatisticGroup} from "./Theme";
class ManageVideos extends React.Component {
export function VideoWrapper() {
const {videoId} = useParams();
const {videoFile, prevFile, nextFile} = useVideo(videoId);
scrollToTop();
constructor(props) {
super(props);
this.state = {
streamUrl: null,
mostRecentDownload: '',
}
}
download = async (e) => {
e.preventDefault();
let response = await download();
if (response.stream_url) {
this.setState({streamUrl: response.stream_url});
}
}
refresh = async (e) => {
e.preventDefault();
let response = await refresh();
if (response.stream_url) {
this.setState({streamUrl: response.stream_url});
}
}
render() {
return (
<Container fluid>
<WROLModeMessage content='Cannot modify Videos'/>
<p>
<Button secondary
id='refresh_videos'
onClick={this.refresh}
>
Refresh Video Files
</Button>
<label htmlFor='refresh_videos'>
Find any new video files. Remove any Videos which no longer have files.
</label>
</p>
</Container>
)
}
return <VideoPage videoFile={videoFile} prevFile={prevFile} nextFile={nextFile} autoplay={true}/>
}
export class VideoWrapper extends React.Component {
function Videos({filter}) {
const {channelId} = useParams();
const {searchParams, updateQuery} = useQuery();
constructor(props) {
super(props);
this.state = {
video: null,
prev: null,
next: null,
channel: null,
no_channel: null,
}
let searchOrder = defaultVideoOrder;
if (searchParams.get('order')) {
// Use whatever order the user specified.
searchOrder = searchParams.get('order');
} else if (searchParams.get('q')) {
// User used a search_str
searchOrder = defaultSearchOrder;
}
async componentDidMount() {
await this.fetchVideo();
let filtersEnabled = searchParams.getAll('filter');
filtersEnabled = filter ? [...filtersEnabled, filter] : filtersEnabled;
const {searchStr, setSearchStr, videos, activePage, setPage, limit, setLimit, totalPages, setOrderBy} =
useSearchVideos(null, channelId, searchOrder, filtersEnabled);
const setView = (value) => updateQuery({view: value});
const view = searchParams.get('view');
const {channel} = useChannel(channelId);
const filterOptions = [
{text: 'Favorites', key: 'favorite', value: 'favorite'},
{text: 'Censored', key: 'censored', value: 'censored'},
];
const setFilters = (value) => updateQuery({filter: value});
const menuColumns = <>
<Grid.Column mobile={8} computer={5}>
<SearchInput clearable searchStr={searchStr} onSubmit={setSearchStr} actionIcon='search'/>
</Grid.Column>
<Grid.Column mobile={6} computer={5}>
<Dropdown selection fluid
size='large'
placeholder='Sort by...'
value={searchOrder}
options={videoOrders}
onChange={(e, {value}) => setOrderBy(value)}
/>
</Grid.Column>
</>;
let header;
if (channel && channel.name) {
const editLink = `/videos/channel/${channelId}/edit`;
header = <>
<Header as='h1'>
{channel.name}
<Link to={editLink}>
<Icon name='edit' style={{marginLeft: '0.5em'}}/>
</Link>
</Header>
</>;
} else if (channelId) {
header = <Placeholder style={{marginBottom: '1em'}}>
<PlaceholderHeader>
<PlaceholderLine/>
</PlaceholderHeader>
</Placeholder>;
}
async fetchVideo() {
// Get and display the Video specified in the Router match
let [video, prev, next] = await getVideo(this.props.match.params.video_id);
let channel_id = this.props.match.params.channel_id;
let channel = channel_id ? await getChannel(channel_id) : null;
let no_channel = false;
if (!channel) {
no_channel = true;
}
this.setState({video, prev, next, channel, no_channel}, scrollToTop);
}
async componentDidUpdate(prevProps, prevState) {
if (prevProps.match.params.video_id !== this.props.match.params.video_id) {
// Clear the current video so that it will change, even if the video is playing.
this.setState({video: null, prev: null, next: null, channel: null});
await this.fetchVideo();
}
}
render() {
if (this.state.video && (this.state.no_channel || this.state.channel)) {
return <VideoPage {...this.state} history={this.props.history} autoplay={true}/>
} else {
return <VideoPlaceholder/>
}
}
}
export function VideosPreview({videos}) {
let body = <VideoPlaceholder/>
if (videos && videos.length === 0) {
body = <p>No videos available.</p>
} else if (videos && videos.length > 0) {
body = <VideoCards videos={videos}/>
}
return <>
{body}
{header}
<FilesView
files={videos}
view={view}
limit={limit}
activePage={activePage}
totalPages={totalPages}
showView={true}
showLimit={true}
showSelect={true}
setView={setView}
setLimit={setLimit}
setPage={setPage}
filterOptions={filterOptions}
activeFilters={filtersEnabled}
setFilters={setFilters}
multipleFilters={true}
menuColumnsCount={3}
menuColumns={menuColumns}
>
</FilesView>
</>
}
function FilterModal(props) {
/*
Expecting props.filters = [
{label: 'Favorites', name: 'favorite', value: false},
{label: 'Censored', name: 'censored', value: false},
function Statistics() {
const {statistics} = useVideoStatistics();
const {videos, historical, channels} = statistics;
if (!videos) {
return <Loader active inline='centered'/>
}
const videoNames = [
{key: 'videos', label: 'Downloaded Videos'},
{key: 'favorites', label: 'Favorite Videos'},
{key: 'sum_size', label: 'Total Size'},
{key: 'max_size', label: 'Largest Video'},
{key: 'week', label: 'Downloads Past Week'},
{key: 'month', label: 'Downloads Past Month'},
{key: 'year', label: 'Downloads Past Year'},
{key: 'sum_duration', label: 'Total Duration'},
];
*/
let inputs = props.filters.map((i) =>
<Form.Input>
<Radio toggle
checked={i.value}
label={i.label}
onClick={!i.disabled ? () => props.toggleFilter(i.name) : null}
disabled={i.disabled}
/>
</Form.Input>
)
const historicalNames = [
{key: 'average_count', label: 'Average Monthly Downloads'},
{key: 'average_size', label: 'Average Monthly Usage'},
];
const channelNames = [
{key: 'channels', label: 'Channels'},
];
const buildSegment = (title, names, stats) => {
return <Segment>
<Header textAlign='center' as='h1'>{title}</Header>
<StatisticGroup>
{names.map(
({key, label}) =>
<Statistic key={key} style={{margin: '2em'}}>
<StatisticValue>{stats[key]}</StatisticValue>
<StatisticLabel>{label}</StatisticLabel>
</Statistic>
)}
</StatisticGroup>
</Segment>
}
return (
<Modal closeIcon open={props.open} onClose={props.onClose} onOpen={props.onOpen} size='mini'>
<Modal.Header>{props.header}</Modal.Header>
<Modal.Content>
<Modal.Description>
<Form>
{inputs}
</Form>
</Modal.Description>
</Modal.Content>
<Modal.Actions>
{props.applyButton && <Button color='blue'>Apply</Button>}
<Button color='black' onClick={props.onClose}>Close</Button>
</Modal.Actions>
</Modal>
)
}
class Videos extends React.Component {
constructor(props) {
super(props);
const query = QueryString.parse(this.props.location.search);
let activePage = query.page ? parseInt(query.page) : 1; // First page is 1 by default, of course.
let searchStr = query.q || '';
let searchOrder = query.o || defaultVideoOrder;
let filtersEnabled = query.f && query.f.length > 0 ? query.f.split(',') : [];
let filters = [
{
label: 'Favorites',
name: 'favorite',
value: filtersEnabled.indexOf('favorite') >= 0 || props.filter === 'favorite',
disabled: props.filter === 'favorite',
},
{
label: 'Censored',
name: 'censored',
value: filtersEnabled.indexOf('censored') >= 0
},
];
this.state = {
activePage,
channel: null,
filters,
filtersEnabled,
filtersOpen: false,
header: '',
limit: DEFAULT_LIMIT,
next: null,
prev: null,
queryStr: searchStr,
searchOrder,
searchStr: '',
totalPages: null,
videoOrders: searchStr === '' ? videoOrders : searchOrders,
videos: null,
};
}
async componentDidMount() {
if (this.props.match.params.channel_id) {
await this.fetchChannel();
} else {
await this.fetchVideos();
}
this.setHeader();
}
async componentDidUpdate(prevProps, prevState, snapshot) {
let params = this.props.match.params;
let channelChanged = params.channel_id !== prevProps.match.params.channel_id;
let pageChanged = (
prevState.activePage !== this.state.activePage ||
prevState.searchOrder !== this.state.searchOrder ||
prevState.queryStr !== this.state.queryStr
);
if (channelChanged) {
await this.fetchChannel();
} else if (pageChanged) {
this.applyStateToHistory();
// TODO this causes a double search request. This will be fixed with a custom hook.
await this.fetchVideos();
}
}
setHeader() {
let header = '';
if (this.props.header) {
header = this.props.header;
} else if (this.state.channel) {
header = this.state.channel.name;
} else {
// Find the matching header from the search orders.
for (let i = 0; i < this.state.videoOrders.length; i++) {
let item = this.state.videoOrders[i];
if (item.value === this.state.searchOrder) {
header = item.title;
}
}
}
this.setState({header: header});
}
async fetchChannel() {
// Get and display the channel specified in the Router match
let channel_id = this.props.match.params.channel_id;
let channel = null;
if (channel_id) {
channel = await getChannel(channel_id);
}
this.setState({
channel,
title: channel ? channel.title : '',
offset: 0,
total: null,
videos: null,
video: null,
},
this.fetchVideos);
}
enabledFilters = () => {
let filters = this.state.filters;
let filtersArr = [];
for (let i = 0; i < filters.length; i++) {
let filter = filters[i];
if (filter['value'] === true) {
filtersArr = filtersArr.concat([filter['name']]);
}
}
return filtersArr;
}
async fetchVideos() {
this.setState({videos: null});
let offset = this.state.limit * this.state.activePage - this.state.limit;
let channel_id = this.state.channel ? this.state.channel.id : null;
let {queryStr, searchOrder, limit} = this.state;
// Pass any enabled filters to the search.
let filtersArr = this.enabledFilters();
try {
let [videos, total] = await searchVideos(offset, limit, channel_id, queryStr, searchOrder, filtersArr);
let totalPages = Math.round(total / this.state.limit) || 1;
this.setState({videos, totalPages});
} catch (e) {
console.error(e);
}
}
changePage = async (activePage) => {
this.setState({activePage});
}
clearSearch = async () => {
this.setState({searchStr: '', queryStr: '', searchOrder: defaultVideoOrder, activePage: 1});
}
applyStateToHistory = () => {
let {history, location} = this.props;
let {activePage, queryStr, searchOrder} = this.state;
let filters = this.enabledFilters();
changePageHistory(history, location, activePage, queryStr, searchOrder, filters);
}
handleSearch = async (e) => {
e && e.preventDefault();
this.setState({activePage: 1, searchOrder: defaultSearchOrder, queryStr: this.state.searchStr},
this.applyStateToHistory);
}
handleInputChange = (event, {name, value}) => {
this.setState({[name]: value});
}
changeSearchOrder = (event, {value}) => {
this.setState({searchOrder: value, activePage: 1}, this.applyStateToHistory);
}
toggleFilter = (name) => {
let filters = this.state.filters;
let filtersEnabled = [];
for (let i = 0; i < filters.length; i++) {
let filter = filters[i];
if (filter.name === name) {
filter.value = !filter.value;
}
if (filter.value) {
filtersEnabled = filtersEnabled.concat([filter.name]);
}
}
this.setState({filters, filtersEnabled});
}
applyFilters = async () => {
this.applyStateToHistory();
await this.fetchVideos();
}
closeFilters = () => {
this.setState({filtersOpen: false}, this.applyFilters);
}
openFilters = () => {
this.setState({filtersOpen: true});
}
render() {
let {
activePage,
channel,
queryStr,
searchOrder,
searchStr,
header,
totalPages,
videoOrders,
videos,
filtersEnabled,
} = this.state;
let body = <Container fluid style={{marginTop: '1em'}}><VideoPlaceholder/></Container>;
if (videos && videos.length === 0) {
// API didn't send back any videos, tell the user what to do.
if (this.props.filter === 'favorites') {
body = <p>You haven't tagged any videos as favorite.</p>;
} else {
// default empty body.
body = <p>No videos retrieved. Have you downloaded videos yet?</p>;
}
} else if (videos) {
body = <VideoCards videos={videos}/>;
}
let pagination = null;
if (totalPages) {
pagination = (
<div style={{marginTop: '3em', textAlign: 'center'}}>
<OldPaginator
activePage={activePage}
changePage={this.changePage}
totalPages={totalPages}
/>
</div>
);
}
let clearSearchButton = (
<Button icon labelPosition='right' onClick={this.clearSearch}>
Search: {queryStr}
<Icon name='close'/>
</Button>
);
return (
<Container fluid>
<Grid columns={4} stackable>
<Grid.Column textAlign='left' width={6}>
<h1>
{header}
{
channel &&
<>
&nbsp;
&nbsp;
<Link to={`/videos/channel/${channel.id}/edit`}>
<Icon name="edit"/>
</Link>
</>
}
</h1>
{queryStr && clearSearchButton}
</Grid.Column>
<Grid.Column width={1}>
<Button icon onClick={this.openFilters} color={filtersEnabled.length > 0 ? 'black' : null}>
<Icon name='filter'/>
</Button>
<FilterModal
applyFilters={this.applyFilters}
open={this.state.filtersOpen}
onClose={this.closeFilters}
header='Filter Videos'
filters={this.state.filters}
toggleFilter={this.toggleFilter}
applyButton={null}
/>
</Grid.Column>
<Grid.Column textAlign='right' width={5}>
<Form onSubmit={this.handleSearch}>
<Input
fluid
icon='search'
placeholder='Search...'
name="searchStr"
value={searchStr}
onChange={this.handleInputChange}/>
</Form>
</Grid.Column>
<Grid.Column width={4}>
<Dropdown
size='large'
placeholder='Sort by...'
selection
fluid
name='searchOrder'
onChange={this.changeSearchOrder}
value={searchOrder}
options={videoOrders}
disabled={searchOrder === defaultSearchOrder}
/>
</Grid.Column>
</Grid>
<Container fluid>
{body}
</Container>
{pagination}
</Container>
)
}
}
function ManageTab(props) {
return (
<Container fluid>
<ManageVideos/>
<Statistics/>
</Container>
<>
{buildSegment('Videos', videoNames, videos)}
{buildSegment('Historical Video', historicalNames, historical)}
{buildSegment('Channels', channelNames, channels)}
</>
)
}
export function VideosRoute(props) {
useTitle('Videos');
const links = [
{text: 'Videos', to: '/videos', exact: true, key: 'videos'},
{text: 'Videos', to: '/videos', key: 'videos', end: true},
{text: 'Favorites', to: '/videos/favorites', key: 'favorites'},
{text: 'Channels', to: '/videos/channel', key: 'channel'},
{text: 'Manage', to: '/videos/manage', key: 'manage'},
{text: 'Statistics', to: '/videos/statistics', key: 'statistics'},
];
return (
<PageContainer>
<TabLinks links={links}/>
<Route path='/videos' exact
component={(i) =>
<Videos
match={i.match}
history={i.history}
location={i.location}
/>}
/>
<Route path='/videos/favorites' exact
component={(i) =>
<Videos
match={i.match}
history={i.history}
location={i.location}
filter='favorite'
header='Favorite Videos'
/>
}/>
<Route path='/videos/channel' exact component={Channels}/>
<Route path='/videos/manage' exact component={ManageTab}/>
<Route path='/videos/channel/new' exact component={NewChannel}/>
<Route path='/videos/channel/:channel_id/edit' exact
component={(i) =>
<EditChannel
match={i.match}
history={i.history}
/>
}
/>
<Route path='/videos/channel/:channel_id/video' exact component={Videos}/>
<Routes>
<Route path='/' exact element={<Videos/>}/>
<Route path='favorites' exact element={<Videos filter='favorite' header='Favorite Videos'/>}/>
<Route path='channel' exact element={<Channels/>}/>
<Route path='statistics' exact element={<Statistics/>}/>
<Route path='channel/new' exact element={<NewChannel/>}/>
<Route path='channel/:channelId/edit' exact element={<EditChannel/>}/>
<Route path='channel/:channelId/video' exact element={<Videos/>}/>
</Routes>
</PageContainer>
)
}
class Statistics extends React.Component {
constructor(props) {
super(props);
this.state = {
videos: null,
historical: null,
channels: null,
};
this.videoNames = [
{key: 'videos', label: 'Downloaded Videos'},
{key: 'favorites', label: 'Favorite Videos'},
{key: 'sum_size', label: 'Total Size'},
{key: 'max_size', label: 'Largest Video'},
{key: 'week', label: 'Downloads Past Week'},
{key: 'month', label: 'Downloads Past Month'},
{key: 'year', label: 'Downloads Past Year'},
{key: 'sum_duration', label: 'Total Duration'},
];
this.historicalNames = [
{key: 'average_count', label: 'Average Monthly Downloads'},
{key: 'average_size', label: 'Average Monthly Usage'},
];
this.channelNames = [
{key: 'channels', label: 'Channels'},
];
}
async componentDidMount() {
await this.fetchStatistics();
}
async fetchStatistics() {
try {
let stats = await getStatistics();
stats.videos.sum_duration = secondsToString(stats.videos.sum_duration);
stats.videos.sum_size = humanFileSize(stats.videos.sum_size, true);
stats.videos.max_size = humanFileSize(stats.videos.max_size, true);
stats.historical.average_size = humanFileSize(stats.historical.average_size, true);
this.setState({...stats});
} catch (e) {
console.error(e);
}
}
buildSegment(title, names, stats) {
return <Segment secondary>
<Header textAlign='center' as='h1'>{title}</Header>
<Statistic.Group>
{names.map(
({key, label}) =>
<Statistic key={key} style={{margin: '2em'}}>
<Statistic.Value>{stats[key]}</Statistic.Value>
<Statistic.Label>{label}</Statistic.Label>
</Statistic>
)}
</Statistic.Group>
</Segment>
}
render() {
if (this.state.videos) {
return (
<>
{this.buildSegment('Videos', this.videoNames, this.state.videos)}
{this.buildSegment('Historical Video', this.historicalNames, this.state.historical)}
{this.buildSegment('Channels', this.channelNames, this.state.channels)}
</>
)
} else {
return <Loader active inline='centered'/>
}
}
}

12
app/src/contexts/contexts.js Normal file
View file

@ -0,0 +1,12 @@
import React from "react";
export const darkTheme = 'dark';
export const lightTheme = 'light';
export const SettingsContext = React.createContext({});
export const ThemeContext = React.createContext({
theme: null,
i: {},
setTheme: () => {
},
});

609
app/src/hooks/customHooks.js
View file

@ -2,13 +2,20 @@ import {useEffect, useState} from "react";
import {
fetchDomains,
filesSearch,
getAPIStatus,
getArchive,
getChannel,
getChannels,
getDirectories,
getDownloaders,
getDownloads,
getFiles,
getHotspotStatus,
getInventory,
getSettings,
getStatistics,
getStatus,
getThrottleStatus,
getVersion,
getVideo,
killDownloads,
searchArchives,
searchVideos,
@ -16,64 +23,16 @@ import {
setThrottle,
startDownloads
} from "../api";
import {useHistory} from "react-router-dom";
import {createSearchParams, useSearchParams} from "react-router-dom";
import {toast} from "react-semantic-toasts";
import {enumerate, humanFileSize, secondsToString} from "../components/Common";
export const useSearchParam = (key, defaultValue = null) => {
// Get a window.location.search param.
const startingValue = new URLSearchParams(window.location.search).get(key);
const [value, setValue] = useState(startingValue || defaultValue);
const history = useHistory();
useEffect(() => {
const params = new URLSearchParams(window.location.search);
params.delete(key);
if (value) {
params.append(key, value);
}
history.push({search: params.toString()});
}, [value, history, key])
return [value, setValue];
const calculatePage = (offset, limit) => {
return offset && limit ? Math.round((offset / limit) + 1) : 1;
}
export const useSearch = () => {
let [searchStr, setSearchStr] = useSearchParam('q');
const [archives, setArchives] = useState();
const [videos, setVideos] = useState();
const localSearchArchives = async (term) => {
setArchives(null);
try {
const [archives] = await searchArchives(0, 6, null, term);
setArchives(archives);
} catch (e) {
console.error(e);
setArchives([]);
}
}
const localSearchVideos = async (term) => {
setVideos(null);
try {
const [videos] = await searchVideos(0, 6, null, term);
setVideos(videos);
} catch (e) {
console.error(e);
setVideos([]);
}
}
useEffect(() => {
if (searchStr) {
localSearchArchives(searchStr);
localSearchVideos(searchStr)
}
}, [searchStr]);
return {searchStr, setSearchStr, archives, videos}
const calculateTotalPages = (total, limit) => {
return total && limit ? Math.round(total / limit) + 1 : 1;
}
export const useDomains = () => {
@ -100,71 +59,18 @@ export const useDomains = () => {
return [domains, total];
}
export const useArchives = ({defaultLimit = 20}) => {
let [limit, setLimit] = useSearchParam('l', defaultLimit);
let [offset, setOffset] = useSearchParam('o');
let [searchStr, setSearchStr] = useSearchParam('q');
let [domain, setDomain] = useSearchParam('domain');
const [archives, setArchives] = useState(null);
const [totalPages, setTotalPages] = useState(0);
const [activePage, setActivePage] = useState((offset / limit) + 1);
const search = async () => {
setArchives(null);
try {
const [archives, total] = await searchArchives(offset, limit, domain, searchStr);
setTotalPages(Math.floor(total / limit) + 1);
setArchives(archives);
} catch (e) {
console.error(e);
setTotalPages(0);
setArchives([]);
}
}
useEffect(() => {
search();
}, [offset, limit, domain, searchStr]);
const setPage = (i) => {
i = parseInt(i);
let l = parseInt(limit);
setOffset((l * i) - l);
setActivePage(i);
}
return {
archives,
totalPages,
setTotalPages,
offset,
setOffset,
setPage,
limit,
setLimit,
domain,
setDomain,
searchStr,
setSearchStr,
activePage,
setActivePage,
search,
};
}
export const useArchive = (archiveId) => {
const [archive, setArchive] = useState(null);
const [archiveFile, setArchiveFile] = useState(null);
const [alternatives, setAlternatives] = useState(null);
const fetchArchive = async () => {
try {
const [a, alt] = await getArchive(archiveId);
setArchive(a);
const [file, alt] = await getArchive(archiveId);
setArchiveFile(file);
setAlternatives(alt);
} catch (e) {
console.error(e);
setArchive(undefined);
setArchiveFile(undefined);
}
}
@ -172,69 +78,328 @@ export const useArchive = (archiveId) => {
fetchArchive();
}, [archiveId]);
return [archive, alternatives];
return {archiveFile, alternatives};
}
export const useVersion = () => {
const [version, setVersion] = useState('');
export const useQuery = () => {
const [searchParams, setSearchParams] = useSearchParams();
const fetchVersion = async () => {
const setQuery = (obj) => {
setSearchParams(createSearchParams(obj), {replace: true});
}
const updateQuery = (obj) => {
const newQuery = {};
for (const entry of searchParams.entries()) {
newQuery[entry[0]] = entry[1];
}
Object.entries(obj).forEach(([key, value]) => {
if (value === undefined || value === null || value === '') {
delete newQuery[key];
} else {
newQuery[key] = value;
}
})
setQuery(newQuery);
}
return {searchParams, setSearchParams, setQuery, updateQuery}
}
export const usePages = (defaultLimit) => {
const {searchParams, updateQuery} = useQuery();
const offset = searchParams.get('o') || 0;
const limit = searchParams.get('l') || defaultLimit || 24;
const [activePage, setActivePage] = useState(calculatePage(offset, limit));
const setLimit = (value) => {
setPage(1);
updateQuery({l: value, o: null});
}
const setPage = (value) => {
setActivePage(value);
value = value - 1; // Page really starts as 0.
updateQuery({o: value * limit});
}
return {offset, limit, setLimit, activePage, setPage};
}
export const useSearchArchives = (defaultLimit, domain, order_by) => {
const {offset, limit, setLimit, activePage, setPage} = usePages(defaultLimit)
const {searchParams, updateQuery} = useQuery();
const searchStr = searchParams.get('q') || '';
const order = searchParams.get('order') || order_by;
const [archives, setArchives] = useState();
const [totalPages, setTotalPages] = useState(0);
const localSearchArchives = async () => {
setArchives(null);
setTotalPages(0);
try {
let version = await getVersion();
setVersion(version);
let [archives_, total] = await searchArchives(offset, limit, domain, searchStr, order);
setArchives(archives_);
setTotalPages(calculateTotalPages(total, limit));
} catch (e) {
console.error(e);
toast({
type: 'error',
title: 'Unexpected server response',
description: 'Could not get archives',
time: 5000,
});
setArchives([]);
}
}
useEffect(() => {
fetchVersion();
})
localSearchArchives();
}, [searchStr, limit, domain, order, activePage]);
return version;
const setSearchStr = (value) => {
updateQuery({q: value, o: 0, order: undefined});
}
const setOrderBy = (value) => {
setPage(1);
updateQuery({order: value});
}
return {
archives,
limit,
setLimit,
offset,
order,
setOrderBy,
totalPages,
activePage,
setPage,
searchStr,
setSearchStr,
}
}
export const useSearchFiles = ({defaultLimit = 50}) => {
let [limit, setLimit] = useSearchParam('l', defaultLimit);
let [offset, setOffset] = useSearchParam('o');
let [searchStr, setSearchStr] = useSearchParam('q');
export const useSearchVideos = (defaultLimit, channelId, order_by, filters) => {
const {searchParams, updateQuery} = useQuery();
const {offset, limit, setLimit, activePage, setPage} = usePages(defaultLimit)
const searchStr = searchParams.get('q') || '';
const order = searchParams.get('order') || order_by;
const [searchFiles, setSearchFiles] = useState([]);
const [videos, setVideos] = useState();
const [totalPages, setTotalPages] = useState(0);
const [activePage, setActivePage] = useState(1);
const localSearchFiles = async () => {
setSearchFiles([]);
const localSearchVideos = async () => {
setVideos(null);
setTotalPages(0);
if (searchStr) {
try {
let [videos_, total] = await searchVideos(offset, limit, channelId, searchStr, order, filters);
setVideos(videos_);
setTotalPages(calculateTotalPages(total, limit));
} catch (e) {
console.error(e);
toast({
type: 'error',
title: 'Unexpected server response',
description: 'Could not get videos',
time: 5000,
});
setVideos([]);
}
}
useEffect(() => {
localSearchVideos();
}, [searchStr, limit, channelId, offset, order_by, filters.join('')]);
const setSearchStr = (value) => {
updateQuery({q: value, o: 0, order: undefined});
}
const setOrderBy = (value) => {
setPage(1);
updateQuery({order: value});
}
return {
videos,
totalPages,
limit,
offset,
order,
searchStr,
activePage,
setPage,
setLimit,
setSearchStr,
setOrderBy,
}
}
export const useVideo = (videoId) => {
const [videoFile, setVideoFile] = useState(null);
const [prevFile, setPrevFile] = useState(null);
const [nextFile, setNextFile] = useState(null);
const fetchVideo = async () => {
try {
const [v, p, n] = await getVideo(videoId);
setVideoFile(v);
setPrevFile(p);
setNextFile(n);
} catch (e) {
console.error(e);
setVideoFile(undefined);
setPrevFile(undefined);
setNextFile(undefined);
toast({
type: 'error',
title: 'Unexpected server response',
description: 'Could not get the video',
time: 5000,
})
}
}
useEffect(() => {
fetchVideo();
}, [videoId]);
return {videoFile, prevFile, nextFile};
}
export const useChannel = (channel_id) => {
const emptyChannel = {
name: '',
directory: '',
mkdir: false,
url: '',
download_frequency: '',
match_regex: '',
};
const [fetched, setFetched] = useState(false);
const [channel, setChannel] = useState(emptyChannel);
const [original, setOriginal] = useState({});
const localGetChannel = async () => {
if (channel_id) {
try {
let [files, total] = await filesSearch(offset, limit, searchStr);
setSearchFiles(files);
setTotalPages(Math.floor(total / limit) + 1);
const c = await getChannel(channel_id);
// Prevent controlled to uncontrolled.
c['url'] = c['url'] || '';
c['download_frequency'] = c['download_frequency'] || '';
c['match_regex'] = c['match_regex'] || '';
setChannel(c);
setOriginal(c);
setFetched(true);
} catch (e) {
console.error(e);
toast({
type: 'error',
title: 'Unexpected server response',
description: 'Could not get files',
description: 'Could not get Channel',
time: 5000,
});
}
} else {
setChannel(emptyChannel);
}
}
useEffect(() => {
localGetChannel();
}, [channel_id])
const changeValue = (name, value) => {
setChannel({...channel, [name]: value});
}
return {channel, changeValue, original, fetched, fetchChannel: localGetChannel};
}
export const useChannels = () => {
const [channels, setChannels] = useState(null);
const fetchChannels = async () => {
setChannels(null);
try {
const c = await getChannels();
setChannels(c);
} catch (e) {
console.error(e);
toast({
type: 'error',
title: 'Unexpected server response',
description: 'Could not get channels',
time: 5000,
});
}
}
useEffect(() => {
fetchChannels();
}, []);
return {channels, fetchChannels}
}
export const useSearchFiles = (defaultLimit = 48, emptySearch = false, mimetype, model) => {
const {searchParams, updateQuery} = useQuery();
const limit = searchParams.get('l') || defaultLimit;
const offset = searchParams.get('o') || 0;
const searchStr = searchParams.get('q');
const mimetype_ = searchParams.get('mimetype');
const model_ = searchParams.get('model');
const [searchFiles, setSearchFiles] = useState(null);
const [totalPages, setTotalPages] = useState(0);
const [activePage, setActivePage] = useState(calculatePage(offset, limit));
const localSearchFiles = async () => {
if (!emptySearch && !searchStr) {
return;
}
setSearchFiles(null);
setTotalPages(0);
try {
let [files, total] = await filesSearch(
offset, limit, searchStr, mimetype || mimetype_, model || model_);
setSearchFiles(files);
setTotalPages(calculateTotalPages(total, limit));
} catch (e) {
console.error(e);
toast({
type: 'error',
title: 'Unexpected server response',
description: 'Could not get files',
time: 5000,
});
}
}
useEffect(() => {
localSearchFiles();
}, [searchStr, limit, offset, activePage]);
}, [searchStr, limit, offset, activePage, mimetype, mimetype_, model, model_]);
const setPage = (i) => {
i = parseInt(i);
let l = parseInt(limit);
setOffset((l * i) - l);
updateQuery({o: (l * i) - l})
setActivePage(i);
}
return {searchFiles, totalPages, limit, setLimit, setOffset, searchStr, setSearchStr, activePage, setPage};
const setSearchStr = (value) => {
updateQuery({q: value, o: null});
}
const setLimit = (value) => {
setActivePage(1);
updateQuery({l: value, o: 0});
}
return {searchFiles, totalPages, limit, searchStr, mimetype, setSearchStr, activePage, setPage, setLimit};
}
export const useBrowseFiles = () => {
@ -293,6 +458,31 @@ export const useHotspot = () => {
return {on, setOn, setHotspot: localSetHotspot};
}
export const useDownloads = () => {
const [onceDownloads, setOnceDownloads] = useState();
const [recurringDownloads, setRecurringDownloads] = useState();
const fetchDownloads = async () => {
try {
const data = await getDownloads();
setOnceDownloads(data['once_downloads']);
setRecurringDownloads(data['recurring_downloads']);
} catch (e) {
console.error(e);
}
}
useEffect(() => {
fetchDownloads();
const interval = setInterval(() => {
fetchDownloads();
}, 1000 * 3);
return () => clearInterval(interval);
}, []);
return {onceDownloads, recurringDownloads, fetchDownloads}
}
export const useDownloaders = () => {
const [on, setOn] = useState(null);
const [downloaders, setDownloaders] = useState(null);
@ -356,21 +546,146 @@ export const useThrottle = () => {
return {on, setOn, setThrottle: localSetThrottle};
}
export const useUp = () => {
// Checks that the API is up.
const fetchAPIStatus = async () => {
let status = await getAPIStatus();
if (status === false) {
export const useDirectories = (defaultDirectory) => {
const [directories, setDirectories] = useState();
const [directory, setDirectory] = useState(defaultDirectory);
const fetchDirectories = async () => {
if (defaultDirectory && directory === '') {
setDirectory(defaultDirectory);
}
try {
setDirectories(await getDirectories(directory));
} catch (e) {
toast({
type: 'error',
title: 'Error!',
description: `API did not respond. Check the server's status.`,
title: 'Unexpected server response',
description: 'Could not get directories',
time: 5000,
});
}
}
useEffect(() => {
fetchAPIStatus();
}, []);
fetchDirectories();
}, [directory, defaultDirectory]);
return {directory, directories, setDirectory};
}
export const useSettings = () => {
const [settings, setSettings] = useState({});
const fetchSettings = async () => {
try {
setSettings(await getSettings())
} catch (e) {
setSettings({});
toast({
type: 'error',
title: 'Unexpected server response',
description: 'Could not get directories',
time: 5000,
});
}
}
useEffect(() => {
fetchSettings();
}, []);
return {settings, fetchSettings};
}
export const useSettingsInterval = () => {
const {settings, fetchSettings} = useSettings();
useEffect(() => {
const interval = setInterval(() => {
fetchSettings();
}, 1000 * 3);
return () => clearInterval(interval);
}, []);
return {settings, fetchSettings};
}
export const useStatus = () => {
const [status, setStatus] = useState({});
const fetchStatus = async () => {
try {
const s = await getStatus();
setStatus(s);
} catch (e) {
console.error(e);
toast({
type: 'error',
title: 'Unexpected server response',
description: 'Could not get server status',
time: 5000,
});
}
}
useEffect(() => {
fetchStatus();
const interval = setInterval(() => {
// console.log('fetch');
fetchStatus();
}, 1000 * 5);
return () => clearInterval(interval);
}, []);
return {status, fetchStatus}
}
export const useVideoStatistics = () => {
const [statistics, setStatistics] = useState({});
const fetchStatistics = async () => {
try {
let stats = await getStatistics();
stats.videos.sum_duration = secondsToString(stats.videos.sum_duration);
stats.videos.sum_size = humanFileSize(stats.videos.sum_size, true);
stats.videos.max_size = humanFileSize(stats.videos.max_size, true);
stats.historical.average_size = humanFileSize(stats.historical.average_size, true);
setStatistics(stats);
} catch (e) {
console.error(e);
}
}
useEffect(() => {
fetchStatistics();
}, []);
return {statistics, fetchStatistics}
}
export const useInventory = (inventoryId) => {
const [byCategory, setByCategory] = useState();
const [bySubcategory, setBySubcategory] = useState();
const [byName, setByName] = useState();
const fetchInventory = async () => {
if (!inventoryId) {
return
}
try {
const inventory = await getInventory(inventoryId);
setByCategory(enumerate(inventory['by_category']));
setBySubcategory(enumerate(inventory['by_subcategory']));
setByName(enumerate(inventory['by_name']));
} catch (e) {
console.error(e);
}
}
useEffect(() => {
fetchInventory();
}, [inventoryId]);
return {byCategory, bySubcategory, byName, fetchInventory}
}

6
app/src/static/wrolpi.css
View file

@ -26,3 +26,9 @@ textarea.otp {
.inline-header span {
margin-left: 1em;
}
.card-icon {
min-height: 163px;
text-align: center;
padding-top: 53px;
}

4
docker-compose.yml
View file

@ -29,9 +29,9 @@ services:
ports:
- ${REACT_APP_API-127.0.0.1:8081}:8081
command: '-vv api --host 0.0.0.0'
healthcheck:
test: [ 'CMD-SHELL', 'curl http://127.0.0.1:8081/api/echo' ]
user: '${UID-1000}:${GID-1000}'
# healthcheck:
# test: [ 'CMD-SHELL', 'curl http://127.0.0.1:8081/api/echo' ]
archive:
build:

50
main.py
View file

@ -9,8 +9,8 @@ import pytz
from sanic import Sanic
from sanic.signals import Event
from wrolpi import root_api, BEFORE_STARTUP_FUNCTIONS, after_startup, limit_concurrent, admin
from wrolpi.common import logger, get_config, import_modules, check_media_directory
from wrolpi import root_api, BEFORE_STARTUP_FUNCTIONS, after_startup, admin
from wrolpi.common import logger, get_config, import_modules, check_media_directory, limit_concurrent, wrol_mode_enabled
from wrolpi.dates import set_timezone
from wrolpi.downloader import download_manager
from wrolpi.vars import PROJECT_DIR, DOCKERIZED, PYTEST
@ -174,33 +174,49 @@ async def set_log_level(args):
@after_startup
@limit_concurrent(1)
def periodic_downloads(app: Sanic, loop):
"""A simple function that perpetually calls downloader_manager.do_downloads() after sleeping."""
async def periodic_downloads(app: Sanic, loop):
"""
A simple function that perpetually calls downloader_manager.do_downloads() after sleeping.
Limited to only one process.
"""
# Set all downloads to new.
download_manager.reset_downloads()
config = get_config()
if config.wrol_mode:
logger.warning(f'Not starting download worker because WROL Mode is enabled.')
download_manager.kill()
return
if config.download_on_startup is False:
logger.warning(f'Not starting download worker because Downloads are disabled on startup.')
download_manager.kill()
if wrol_mode_enabled:
logger.warning(f'Not starting download manager because WROL Mode is enabled.')
download_manager.disable()
return
logger.info('Starting download manager.')
config = get_config()
if config.download_on_startup is False:
logger.warning(f'Not starting download manager because Downloads are disabled on startup.')
download_manager.disable()
return
download_manager.enable()
async def _periodic_download():
download_manager.start_workers()
await download_manager.do_downloads()
await asyncio.sleep(60)
app.add_task(_periodic_download()) # noqa
await asyncio.sleep(30)
await app.add_task(_periodic_download()) # noqa
app.add_task(_periodic_download())
await app.add_task(_periodic_download())
@after_startup
async def start_workers(app: Sanic, loop):
"""All Sanic processes have their own Download workers."""
if wrol_mode_enabled():
logger.warning(f'Not starting download workers because WROL Mode is enabled.')
download_manager.stop()
return
download_manager.start_workers()
@root_api.api_app.signal(Event.SERVER_SHUTDOWN_BEFORE)
@limit_concurrent(1)
def handle_server_shutdown(*args, **kwargs):
"""Stop downloads when server is shutting down."""
if not PYTEST:

145
modules/archive/__init__.py
View file

@ -1,20 +1,28 @@
import json
import pathlib
from abc import ABC
from typing import Tuple, Optional
from typing import Tuple, Dict, List
from sqlalchemy.orm import Session
from wrolpi.common import logger
from wrolpi.db import optional_session
from wrolpi.common import logger, register_modeler, register_after_refresh, limit_concurrent
from wrolpi.db import optional_session, get_db_session
from wrolpi.downloader import Downloader, Download, DownloadResult
from wrolpi.errors import UnrecoverableDownloadError
from wrolpi.files.indexers import register_indexer, Indexer
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from . import lib
from .api import bp # noqa
from .models import Archive
from .lib import match_archive_files
from .models import Archive, Domain
PRETTY_NAME = 'Archive'
logger = logger.getChild(__name__)
__all__ = ['ArchiveDownloader', 'archive_downloader']
class ArchiveDownloader(Downloader, ABC):
name = 'archive'
@ -38,9 +46,134 @@ class ArchiveDownloader(Downloader, ABC):
return DownloadResult(success=True, location=f'/archive/{archive.id}')
@optional_session
def already_downloaded(self, url: str, session: Session = None) -> bool:
return bool(session.query(Archive).filter_by(url=url).count())
def already_downloaded(self, urls: List[str], session: Session = None) -> List:
archives = list(session.query(Archive).filter(Archive.url.in_(urls)))
return archives
# Archive downloader is the last downloader which should be used.
archive_downloader = ArchiveDownloader(priority=100)
@register_modeler
def archive_modeler(groups: Dict[str, List[File]], session: Session):
archive_groups = {}
for stem, group in groups.items():
if any(lib.is_singlefile_file(i.path) for i in group):
archive_groups[stem] = group
count = 0
for stem, group in archive_groups.items():
readability_file, singlefile_file, readability_json_file, readability_txt_file, screenshot_file = \
match_archive_files(group)
archive: Archive = Archive.find_by_path(singlefile_file.path, session)
if not archive:
archive = Archive(singlefile_file=singlefile_file)
session.add(archive)
archive.readability_file = readability_file
archive.readability_json_file = readability_json_file
archive.readability_txt_file = readability_txt_file
archive.screenshot_file = screenshot_file
if archive.readability_file:
archive.readability_file.associated = True
archive.readability_file.do_index()
if archive.readability_json_file:
archive.readability_json_file.associated = True
archive.readability_json_file.do_index()
if archive.readability_txt_file:
archive.readability_txt_file.associated = True
archive.readability_txt_file.do_index()
if archive.screenshot_file:
archive.screenshot_file.associated = True
archive.screenshot_file.do_index()
archive.singlefile_file.model = Archive.__tablename__
archive.singlefile_file.do_index()
archive.validate()
# Remove this group, it will not be processed by other modelers.
del groups[stem]
count += 1
if not count % 100:
# Commit occasionally.
session.commit()
@register_after_refresh
@limit_concurrent(1)
def archive_cleanup():
# Process all Archives that have not been validated. Delete any that no longer exist, or are not real Archives.
# Validate all other Archives.
logger.info('Searching for invalid Archives')
offset = 0
limit = 100
while True:
with get_db_session(commit=True) as session:
archives: List[Archive] = session.query(Archive).order_by(Archive.id).limit(limit).offset(offset)
processed = 0
deleted = 0
for archive in archives:
processed += 1
if not lib.is_singlefile_file(archive.singlefile_path):
# Archive is not a real archive.
archive.singlefile_file.model = None
session.delete(archive)
deleted += 1
if processed < limit:
# Validated all archives.
break
offset += (limit - deleted)
with get_db_session(commit=True) as session:
# Remove any Domains without any Archives.
domain_ids = [i[0] for i in session.execute('SELECT DISTINCT domain_id FROM archive') if i[0]]
deleted = 0
for domain in session.query(Domain):
if domain.id not in domain_ids:
deleted += 1
session.delete(domain)
@register_indexer('text/html')
class ArchiveIndexer(Indexer, ABC):
@staticmethod
def get_title(file: File):
singlefile_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
readability_json_path = singlefile_path.with_suffix('.readability.json')
if readability_json_path.is_file():
with readability_json_path.open('rt') as fh:
try:
return json.load(fh)['title']
except Exception:
if not PYTEST:
logger.warning(f'Archive readability json file exists, but cannot get title. {file}')
return None
@staticmethod
def get_article(file: File):
singlefile_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
readability_txt_path = singlefile_path.with_suffix('.readability.txt')
if readability_txt_path.is_file():
return readability_txt_path.read_text()
@classmethod
def create_index(cls, file: File) -> Tuple:
"""
Index an Archive file and it's associated files. This mimetype matches non-archive (non-singlefile) html files,
so this index will return empty if a readability file is found.
a = title
b = <empty>
c = <empty>
d = readability article
"""
if file.path.name.endswith('.readability.html'):
# Readability files are not the primary Archive file.
return None, None, None, None
a = cls.get_title(file)
d = cls.get_article(file)
return a, None, None, d

15
modules/archive/api.py
View file

@ -1,4 +1,3 @@
import asyncio
from http import HTTPStatus
from sanic import response, Request
@ -22,8 +21,9 @@ logger = logger.getChild(__name__)
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
async def get_archive(_: Request, archive_id: int):
archive = lib.get_archive(archive_id=archive_id)
alternatives = archive.alternatives
return json_response({'archive': archive, 'alternatives': alternatives})
archive_file = archive.singlefile_file.__json__()
alternatives = [i.singlefile_file.__json__() for i in archive.alternatives]
return json_response({'file': archive_file, 'alternatives': alternatives})
@bp.delete('/<archive_id:int>')
@ -35,13 +35,6 @@ async def delete_archive(_: Request, archive_id: int):
return response.empty()
@bp.post('/refresh')
@wrol_mode_check
async def refresh_archives(_: Request):
asyncio.ensure_future(lib.refresh_archives())
return response.empty()
@bp.get('/domains')
async def fetch_domains(_: Request):
domains = lib.get_domains()
@ -65,6 +58,6 @@ async def search_archives(_: Request, body: schema.ArchiveSearchRequest):
limit = archive_limit_limiter(body.limit)
offset = body.offset or 0
archives, total = lib.search(search_str, domain, limit, offset)
archives, total = lib.archive_search(search_str, domain, limit, offset, body.order_by)
ret = dict(archives=archives, totals=dict(archives=total))
return json_response(ret)

50
modules/archive/conftest.py
View file

@ -8,6 +8,7 @@ import pytest
from modules.archive.lib import archive_strftime
from modules.archive.models import Archive, Domain
from wrolpi.dates import local_timezone
from wrolpi.files.models import File
@pytest.fixture
@ -18,7 +19,7 @@ def archive_directory(test_directory) -> pathlib.Path:
@pytest.fixture
def archive_factory(test_session, archive_directory):
def archive_factory(test_session, archive_directory, make_files_structure):
def time_generator():
timestamp = datetime.datetime(2000, 1, 1, 0, 0, 0)
while True:
@ -27,12 +28,12 @@ def archive_factory(test_session, archive_directory):
now = time_generator()
def _(domain: str = None, url: str = None, title: str = 'NA', contents: str = None) -> Archive:
def _(domain: str = None, url: str = None, title: str = 'NA', contents: str = None,
singlefile_contents: str = None) -> Archive:
if domain:
assert '/' not in domain
title = title or str(uuid4())
url = url or f'https://example.com/{title}'
domain_dir = domain or title
domain_dir = archive_directory / domain_dir
domain_dir.mkdir(exist_ok=True)
@ -40,19 +41,25 @@ def archive_factory(test_session, archive_directory):
archive_datetime = local_timezone(next(now))
timestamp = archive_strftime(archive_datetime)
singlefile_path = domain_dir / f'{timestamp}_{title}.html'
screenshot_path = domain_dir / f'{timestamp}_{title}.png'
readability_path = domain_dir / f'{timestamp}_{title}.readability.html'
readability_txt_path = domain_dir / f'{timestamp}_{title}.readability.txt'
readability_json_path = domain_dir / f'{timestamp}_{title}.readability.json'
with readability_json_path.open('wt') as fh:
json.dump({'url': url}, fh)
json_contents = {}
if title != 'NA':
json_contents['title'] = title
if url:
json_contents['url'] = url
json_contents = json.dumps(json_contents)
singlefile_path, screenshot_path, readability_path, readability_txt_path, readability_json_path = \
make_files_structure({
str(domain_dir / f'{timestamp}_{title}.html'): singlefile_contents or '<html></html>',
str(domain_dir / f'{timestamp}_{title}.png'): None,
str(domain_dir / f'{timestamp}_{title}.readability.html'): '<html></html>',
str(domain_dir / f'{timestamp}_{title}.readability.txt'): contents,
str(domain_dir / f'{timestamp}_{title}.readability.json'): json_contents,
})
if domain:
domain = test_session.query(Domain).filter_by(domain=domain).one_or_none()
if not domain:
if not domain and domain_dir.name != 'NA':
domain = Domain(
domain=domain_dir.name,
directory=domain_dir,
@ -61,23 +68,18 @@ def archive_factory(test_session, archive_directory):
test_session.flush([domain])
archive = Archive(
readability_json_path=readability_json_path,
readability_path=readability_path,
readability_txt_path=readability_txt_path,
screenshot_path=screenshot_path,
singlefile_path=singlefile_path,
readability_json_file=File(path=readability_json_path) if readability_json_path else None,
readability_file=File(path=readability_path) if readability_path else None,
readability_txt_file=File(path=readability_txt_path) if readability_txt_path else None,
screenshot_file=File(path=screenshot_path) if screenshot_path else None,
singlefile_file=File(path=singlefile_path, model='archive'),
title=title,
url=url,
domain_id=domain.id,
contents=contents,
domain_id=domain.id if domain else None,
archive_datetime=archive_datetime,
)
test_session.add(archive)
test_session.commit()
for path in archive.my_paths():
path.touch()
archive.singlefile_file.do_index()
return archive

224
modules/archive/lib.py
View file

@ -7,21 +7,22 @@ import subprocess
import tempfile
from dataclasses import dataclass
from datetime import datetime
from functools import partial
from itertools import groupby
from typing import Iterator, Optional, Tuple, List
import aiohttp
from bs4 import BeautifulSoup
from selenium import webdriver
from sqlalchemy.orm import Session
from modules.archive.models import Domain, Archive
from wrolpi.cmd import which
from wrolpi.common import get_media_directory, logger, chunks, extract_domain, chdir, escape_file_name, walk, \
aiohttp_post
from wrolpi.common import get_media_directory, logger, extract_domain, chdir, escape_file_name, walk, \
aiohttp_post, match_paths_to_suffixes, iterify
from wrolpi.dates import now, Seconds, local_timezone
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models, optional_session
from wrolpi.errors import InvalidDomain, UnknownURL, InvalidArchive
from wrolpi.db import get_db_session, get_db_curs, optional_session
from wrolpi.errors import UnknownArchive, InvalidOrderBy
from wrolpi.files.lib import handle_search_results
from wrolpi.files.models import File
from wrolpi.vars import DOCKERIZED, PYTEST
logger = logger.getChild(__name__)
@ -284,12 +285,11 @@ async def do_archive(url: str) -> Archive:
archive = Archive(
title=title,
archive_datetime=now(),
singlefile_path=archive_files.singlefile,
readability_path=archive_files.readability,
readability_json_path=archive_files.readability_json,
readability_txt_path=archive_files.readability_txt,
screenshot_path=archive_files.screenshot,
contents=readability_txt,
singlefile_file=File(path=archive_files.singlefile, model='archive'),
readability_file=File(path=archive_files.readability) if archive_files.readability else None,
readability_json_file=File(path=archive_files.readability_json) if archive_files.readability_json else None,
readability_txt_file=File(path=archive_files.readability_txt) if archive_files.readability_txt else None,
screenshot_file=File(path=archive_files.screenshot) if archive_files.screenshot else None,
url=url,
domain_id=domain.id,
)
@ -323,19 +323,12 @@ def get_title_from_html(html: str, url: str = None) -> str:
logger.info(f'Unable to extract title {url}')
def get_domain(session, domain: str) -> Domain:
domain_ = session.query(Domain).filter_by(domain=domain).one_or_none()
if not domain_:
raise InvalidDomain(f'Invalid domain: {domain}')
return domain_
@optional_session
def get_archive(session, archive_id: int) -> Archive:
"""Get an Archive."""
archive = session.query(Archive).filter_by(id=archive_id).one_or_none()
if not archive:
raise InvalidArchive(f'Invalid archive id: {archive_id}')
raise UnknownArchive(f'Could not find archive with id: {archive_id}')
return archive
@ -344,7 +337,7 @@ def delete_archive(archive_id: int):
with get_db_session(commit=True) as session:
archive: Archive = session.query(Archive).filter_by(id=archive_id).one_or_none()
if not archive:
raise UnknownURL(f'Unknown Archive with id: {archive_id}')
raise UnknownArchive(f'Unknown Archive with id: {archive_id}')
# Delete any files associated with this URL.
archive.delete()
@ -476,95 +469,13 @@ def is_archive_file(path: pathlib.Path) -> bool:
return path.is_file() and path.suffix.lower() in ARCHIVE_SUFFIXES and bool(ARCHIVE_MATCHER.match(path.name))
def _refresh_archives():
def is_singlefile_file(path: pathlib.Path) -> bool:
"""
Search the Archives directory for archive files, update the database if new files are found. Remove any orphan
URLs or Domains.
Archive singlefile files are expected to start with the following: %Y-%m-%d-%H-%M-%S
they must end with .html, but never with .readability.html.
"""
archive_directory = get_archive_directory()
# TODO remove this later when everyone has migrated their files.
migrate_archive_files()
singlefile_paths = set()
for domain_directory in filter(lambda i: i.is_dir(), archive_directory.iterdir()):
logger.debug(f'Refreshing directory: {domain_directory}')
all_archives_files = filter(is_archive_file, walk(domain_directory))
archive_groups = group_archive_files(all_archives_files)
archive_groups = list(archive_groups)
archive_count = 0
for chunk in chunks(archive_groups, 20):
with get_db_session(commit=True) as session:
for dt, archive_files in chunk:
archive_count += 1
singlefile_paths.add(str(archive_files.singlefile))
upsert_archive(dt, archive_files, session)
if archive_count:
logger.info(f'Inserted/updated {archive_count} archives in {domain_directory}')
singlefile_paths = list(singlefile_paths)
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM archive WHERE singlefile_path != ALL(%s)', (singlefile_paths,))
with get_db_curs(commit=True) as curs:
stmt = '''
DELETE FROM domains WHERE
id NOT IN (select distinct domain_id from archive)
RETURNING domains.id
'''
curs.execute(stmt)
domains = list(map(dict, curs.fetchall()))
logger.info(f'Deleted {len(domains)} Domains')
with get_db_session(commit=True) as session:
# Get all archives that have no text contents, but have txt files.
archives = session.query(Archive).filter(
Archive.contents == None, # noqa
Archive.readability_txt_path != None,
).all()
for archive in archives:
with archive.readability_txt_path.path.open('rt') as fh:
archive.contents = fh.read()
async def refresh_archives():
_refresh_archives()
def upsert_archive(dt: str, archive_files: ArchiveFiles, session: Session):
"""Get or create an Archive, and it's URL/Domain. If it already exists, update it with these new files."""
# Extract the URL from the JSON. Fail if this is not possible.
try:
with archive_files.readability_json.open() as fh:
json_contents = json.load(fh)
url = json_contents['url']
title = json_contents.get('title')
except Exception as e:
raise InvalidArchive() from e
domain = get_or_create_domain(session, url)
# Get the existing Archive, or create a new one.
archive = session.query(Archive).filter_by(singlefile_path=archive_files.singlefile).one_or_none()
if not archive:
# No archive matches this singlefile_path, create a new one.
archive = Archive(url=url, domain_id=domain.id)
session.add(archive)
session.flush()
if not archive.title and title:
archive.title = title
if not archive.title and archive_files.singlefile:
# As a last resort, get the title from the HTML.
archive.title = get_title_from_html(archive_files.singlefile.read_text(), url)
# Update the archive with the files that we have.
archive.archive_datetime = dt
archive.singlefile_path = archive_files.singlefile
archive.readability_path = archive_files.readability
archive.readability_txt_path = archive_files.readability_txt
archive.readability_json_path = archive_files.readability_json
archive.screenshot_path = archive_files.screenshot
return path.is_file() and path.suffix.lower() == '.html' and not path.name.endswith('.readability.html') \
and bool(ARCHIVE_MATCHER.match(path.name))
def get_domains():
@ -581,46 +492,69 @@ def get_domains():
return domains
def search(search_str: str, domain: str, limit: int, offset: int) -> Tuple[List[Archive], int]:
with get_db_curs() as curs:
columns = 'id, COUNT(*) OVER() AS total'
params = dict(offset=offset, limit=limit)
wheres = ''
order_by = '1 DESC'
ARCHIVE_ORDERS = {
'date': 'a.archive_datetime ASC, LOWER(a.singlefile_path) ASC',
'-date': 'a.archive_datetime DESC NULLS LAST, LOWER(a.singlefile_path) DESC',
'rank': '2 DESC, a.archive_datetime DESC',
'-rank': '2 ASC, a.archive_datetime ASC',
}
if search_str:
columns = 'id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
params['search_str'] = search_str
wheres += '\nAND textsearch @@ websearch_to_tsquery(%(search_str)s)'
# highest rank, then most recent
order_by = '2 DESC, archive_datetime DESC'
if domain:
curs.execute('SELECT id FROM domains WHERE domain=%s', (domain,))
try:
domain_id = curs.fetchone()[0]
except TypeError:
# No domains match the provided domain.
return [], 0
params['domain_id'] = domain_id
wheres += '\nAND domain_id = %(domain_id)s'
def archive_search(search_str: str, domain: str, limit: int, offset: int, order_by: str) -> Tuple[List[Archive], int]:
wheres = []
params = dict(offset=offset, limit=limit)
order = '1 DESC'
# TODO handle different Archive search orders.
stmt = f'''
SELECT {columns}
FROM archive
WHERE
singlefile_path IS NOT NULL AND singlefile_path != ''
{wheres}
ORDER BY {order_by}
OFFSET %(offset)s
LIMIT %(limit)s
'''
curs.execute(stmt, params)
results = [dict(i) for i in curs.fetchall()]
total = results[0]['total'] if results else 0
ranked_ids = [i['id'] for i in results]
if search_str:
columns = 'f.path, ts_rank(f.textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
params['search_str'] = search_str
wheres.append('\nAND textsearch @@ websearch_to_tsquery(%(search_str)s)')
# highest rank, then most recent
if not order_by:
order = ARCHIVE_ORDERS['rank']
else:
# No search_str provided, get path and total only.
columns = 'f.path, COUNT(*) OVER() AS total'
results = get_ranked_models(ranked_ids, Archive)
if order_by:
try:
order = ARCHIVE_ORDERS[order_by]
except KeyError:
raise InvalidOrderBy(f'Invalid order by: {order_by}')
if domain:
params['domain'] = domain
wheres.append('AND domain_id = (select id from domains where domains.domain = %(domain)s)')
wheres = '\n'.join(wheres)
stmt = f'''
SELECT {columns}
FROM archive a
LEFT JOIN file f on f.path = a.singlefile_path
WHERE
a.singlefile_path != ''
{wheres}
ORDER BY {order}
OFFSET %(offset)s
LIMIT %(limit)s
'''
logger.warning(stmt, params)
results, total = handle_search_results(stmt, params)
return results, total
match_archive_paths = partial(match_paths_to_suffixes, suffix_groups=(
('.readability.html',),
('.html',),
('.readability.json',),
('.readability.txt',),
('.jpg', '.jpeg', '.webp', '.png'),
))
@iterify(tuple)
def match_archive_files(files: List[File]) -> Tuple[File, File, File, File, File]:
archive_paths = match_archive_paths([i.path for i in files])
for path in archive_paths:
yield next((i for i in files if i.path == path), None)

156
modules/archive/models.py
View file

@ -1,13 +1,22 @@
import datetime
import json
import pathlib
import re
from typing import Generator
from sqlalchemy import Column, Integer, String, ForeignKey, Computed
from sqlalchemy.orm import relationship, Session
import pytz
from sqlalchemy import Column, Integer, String, ForeignKey, Boolean
from sqlalchemy.orm import relationship, Session, validates
from sqlalchemy.orm.collections import InstrumentedList
from wrolpi.common import ModelHelper, Base, tsvector
from wrolpi.common import ModelHelper, Base, logger
from wrolpi.dates import TZDateTime
from wrolpi.errors import InvalidArchive
from wrolpi.files.models import File
from wrolpi.media_path import MediaPathType
logger = logger.getChild(__name__)
class Archive(Base, ModelHelper):
__tablename__ = 'archive'
@ -16,34 +25,40 @@ class Archive(Base, ModelHelper):
domain_id = Column(Integer, ForeignKey('domains.id'))
domain = relationship('Domain', primaryjoin='Archive.domain_id==Domain.id')
singlefile_path = Column(MediaPathType)
readability_path = Column(MediaPathType)
readability_json_path = Column(MediaPathType)
readability_txt_path = Column(MediaPathType)
screenshot_path = Column(MediaPathType)
url = Column(String)
title = Column(String)
archive_datetime = Column(TZDateTime)
contents = Column(String)
validated = Column(Boolean)
textsearch = Column(tsvector, Computed('''setweight(to_tsvector('english'::regconfig, title), 'A') ||
setweight(to_tsvector('english'::regconfig, contents), 'D')'''))
# Associated Files.
singlefile_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
singlefile_file: File = relationship('File', primaryjoin='Archive.singlefile_path==File.path')
readability_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
readability_file: File = relationship('File', primaryjoin='Archive.readability_path==File.path')
readability_json_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
readability_json_file: File = relationship('File', primaryjoin='Archive.readability_json_path==File.path')
readability_txt_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
readability_txt_file: File = relationship('File', primaryjoin='Archive.readability_txt_path==File.path')
screenshot_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
screenshot_file: File = relationship('File', primaryjoin='Archive.screenshot_path==File.path')
def __repr__(self):
return f'<Archive id={self.id} url={self.url} singlefile={self.singlefile_path}>'
if self.domain:
return f'<Archive id={self.id} url={self.url} singlefile={repr(str(self.singlefile_path))} ' \
f'domain={self.domain.domain}>'
return f'<Archive id={self.id} url={self.url} singlefile={repr(str(self.singlefile_path))}>'
def my_paths(self) -> Generator:
if self.singlefile_path:
yield self.singlefile_path.path
yield self.singlefile_path
if self.readability_path:
yield self.readability_path.path
yield self.readability_path
if self.readability_json_path:
yield self.readability_json_path.path
yield self.readability_json_path
if self.readability_txt_path:
yield self.readability_txt_path.path
yield self.readability_txt_path
if self.screenshot_path:
yield self.screenshot_path.path
yield self.screenshot_path
def unlink(self):
"""
@ -73,6 +88,7 @@ class Archive(Base, ModelHelper):
readability_txt_path=self.readability_txt_path,
screenshot_path=self.screenshot_path,
singlefile_path=self.singlefile_path,
stem=self.singlefile_path.stem if self.singlefile_path else None,
title=self.title,
url=self.url,
)
@ -96,11 +112,103 @@ class Archive(Base, ModelHelper):
Get a list of Archives that share my URL.
"""
session = Session.object_session(self)
i = list(session.query(Archive).filter(
alternatives = list(session.query(Archive).filter(
Archive.id != self.id,
Archive.url == self.url,
).order_by(Archive.archive_datetime))
return i
return alternatives
def my_files(self) -> Generator[pathlib.Path, None, None]:
"""Return all paths of this Archive. Returns nothing if all paths are None.""" # noqa
if self.singlefile_path:
yield self.singlefile_path
if self.readability_path:
yield self.readability_path
if self.readability_txt_path:
yield self.readability_txt_path
if self.readability_json_path:
yield self.readability_json_path
if self.screenshot_path:
yield self.screenshot_path
@staticmethod
def find_by_path(path, session: Session) -> Base:
archive = session.query(Archive).filter_by(singlefile_path=path).one_or_none()
return archive
def read_singlefile_data(self):
"""
Read the start of the singlefile (if any) and decode the Archive information.
"""
match_url = re.compile(r'^\s+?url:\s+?(http.*)', re.MULTILINE)
match_date = re.compile(r'^\s+?saved date:\s+?(.*)', re.MULTILINE)
if self.singlefile_path:
with self.singlefile_path.open('rt') as fh:
head = fh.read(500)
if 'Page saved with SingleFile' not in head:
return
try:
if match := match_url.findall(head):
self.url = match[0].strip()
except Exception as e:
logger.error(f'Could not get URL from singlefile {self.singlefile_path}', exc_info=e)
try:
if match := match_date.findall(head):
dt = match[0].strip()
dt = ' '.join(dt.split(' ')[:5])
# SingleFile uses GMT.
dt = datetime.datetime.strptime(
dt,
'%a %b %d %Y %H:%M:%S' # Fri Jun 17 2022 19:24:52
).replace(tzinfo=pytz.timezone('GMT'))
self.archive_datetime = dt
except Exception as e:
logger.error(f'Could not get archive date from singlefile {self.singlefile_path}', exc_info=e)
def read_readability_data(self):
"""Read the Readability JSON file, apply its contents to this record."""
from modules.archive.lib import get_or_create_domain, get_title_from_html
readability_json_file = self.readability_json_file
if not readability_json_file or not readability_json_file.path.is_file():
raise InvalidArchive(f'Archive must have a readability json file: {self.singlefile_file.path}')
try:
with readability_json_file.path.open() as fh:
json_contents = json.load(fh)
url = json_contents.get('url')
title = json_contents.get('title')
except Exception as e:
raise InvalidArchive() from e
domain = None
if url:
domain = get_or_create_domain(Session.object_session(self), url)
if not title:
title = get_title_from_html(self.singlefile_path.read_text())
self.url = url
self.title = title
self.domain_id = domain.id if domain else None
def validate(self):
try:
self.read_readability_data()
self.read_singlefile_data()
self.validated = True
except Exception as e:
logger.warning(f'Unable to validate {self}', exc_info=e)
@staticmethod
def find_by_paths(paths, session):
archives = list(session.query(Archive).filter(Archive.singlefile_path.in_(paths)))
return archives
@property
def primary_path(self):
return self.singlefile_path
class Domain(Base, ModelHelper):
@ -119,3 +227,11 @@ class Domain(Base, ModelHelper):
session = Session.object_session(self)
session.execute('DELETE FROM archive WHERE domain_id=:id', dict(id=self.id))
session.query(Domain).filter_by(id=self.id).delete()
@validates('domain')
def validate_domain(self, key, value: str):
if not isinstance(value, str):
raise ValueError('Domain must be a string')
if len(value.split('.')) < 2:
raise ValueError(f'Domain must contain at least one "." domain={repr(value)}')
return value

1
modules/archive/schema.py
View file

@ -29,6 +29,7 @@ class ArchiveSearchRequest:
domain: Optional[str] = None
offset: Optional[int] = None
limit: Optional[int] = None
order_by: Optional[str] = None
@dataclass

86
modules/archive/test/test_api.py
View file

@ -5,16 +5,22 @@ from http import HTTPStatus
def check_results(test_client, data, ids):
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
assert response.status_code == HTTPStatus.OK, response.json
assert [i['id'] for i in response.json['archives']] == ids, \
f'{response.json["archives"][0]["id"]}..{response.json["archives"][19]["id"]}'
if ids:
assert response.json['archives'], f'Expected {len(ids)} archives but did not receive any'
assert [i['archive']['id'] for i in response.json['archives']] == ids, \
f'{response.json["archives"][0]["archive"]["id"]}..{response.json["archives"]["archive"][19]["id"]}'
def test_archives_search(test_session, archive_directory, archive_factory, test_client):
"""Archives can be searched by their title and their contents."""
# Search with no archives.
check_results(test_client, {'search_str': 'foo'}, [])
archive_factory('example.com', 'https://example.com/one', 'my archive', 'foo bar qux')
archive_factory('example.com', 'https://example.com/one', 'other archive', 'foo baz qux qux')
archive_factory('example.org', title='archive third', contents='baz qux qux qux')
archive_factory('example.org') # has no contents
test_session.commit()
# 1 and 2 contain "foo".
data = {'search_str': 'foo'}
@ -44,15 +50,20 @@ def test_archives_search(test_session, archive_directory, archive_factory, test_
data = {'search_str': 'third'}
check_results(test_client, data, [3, ])
# All titles contain "qux", but they contain different amounts. They are ordered by the amount.
# All contents contain "qux", but they contain different amounts. They are ordered by the amount.
data = {'search_str': 'qux'}
check_results(test_client, data, [3, 2, 1])
data = {'search_str': 'qux', 'order_by': 'bad order_by'}
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
assert response.status_code == HTTPStatus.BAD_REQUEST
def test_search_offset(archive_factory, test_client):
def test_search_offset(test_session, archive_factory, test_client):
"""Archive search can be offset."""
for i in range(500):
archive_factory('example.com', f'https://example.com/{i}', contents='foo bar')
test_session.commit()
data = {'search_str': None, 'offset': 0}
check_results(test_client, data, list(range(500, 480, -1)))
@ -66,28 +77,87 @@ def test_search_offset(archive_factory, test_client):
check_results(test_client, data, [])
def test_archives_search_no_query(archive_factory, test_client):
def test_archives_search_no_query(test_session, archive_factory, test_client):
"""Archive Search API endpoint does not require data in the body."""
# Add 100 random archives.
for _ in range(100):
archive_factory('example.com')
test_session.commit()
# All archives are returned when no `search_str` is passed.
request, response = test_client.post('/api/archive/search', content='{}')
assert response.status_code == HTTPStatus.OK, response.json
assert [i['id'] for i in response.json['archives']] == list(range(100, 80, -1))
assert [i['archive']['id'] for i in response.json['archives']] == list(range(100, 80, -1))
assert response.json['totals']['archives'] == 100
# All archives are from "example.com".
data = dict(domain='example.com')
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
assert response.status_code == HTTPStatus.OK, response.json
assert [i['id'] for i in response.json['archives']] == list(range(100, 80, -1))
assert [i['archive']['id'] for i in response.json['archives']] == list(range(100, 80, -1))
assert response.json['totals']['archives'] == 100
# No archives are from "example.org".
data = dict(domain='example.org')
request, response = test_client.post('/api/archive/search', content=json.dumps(data))
assert response.status_code == HTTPStatus.OK, response.json
assert [i['id'] for i in response.json['archives']] == []
assert [i['archive']['id'] for i in response.json['archives']] == []
assert response.json['totals']['archives'] == 0
def test_archive_and_domain_crud(test_session, test_client, archive_factory):
"""
Getting an Archive returns it's File. Testing deleting Archives.
"""
# Can get empty results.
request, response = test_client.get(f'/api/archive/1')
assert response.status_code == HTTPStatus.NOT_FOUND
request, response = test_client.get(f'/api/archive/domains')
assert response.status_code == HTTPStatus.OK
assert response.json['domains'] == []
archive1 = archive_factory(domain='example.com')
archive2 = archive_factory(domain='example.com')
test_session.commit()
# Archive1 has Archive2 as alternative.
request, response = test_client.get(f'/api/archive/{archive1.id}')
assert response.status_code == HTTPStatus.OK
assert response.json['file']['archive']['id'] == archive1.id
assert response.json['alternatives'][0]['archive']['id'] == archive2.id
# Archive2 has Archive1 as alternative.
request, response = test_client.get(f'/api/archive/{archive2.id}')
assert response.status_code == HTTPStatus.OK
assert archive2.id == response.json['file']['archive']['id']
assert response.json['alternatives'][0]['archive']['id'] == archive1.id
# Only one domain.
request, response = test_client.get(f'/api/archive/domains')
assert response.status_code == HTTPStatus.OK
assert response.json['domains'][0]['domain'] == 'example.com'
# Deleting works.
request, response = test_client.delete(f'/api/archive/{archive1.id}')
assert response.status_code == HTTPStatus.NO_CONTENT
# Trying to delete again returns NOT_FOUND.
request, response = test_client.delete(f'/api/archive/{archive1.id}')
assert response.status_code == HTTPStatus.NOT_FOUND
# Can't get deleted Archive.
request, response = test_client.get(f'/api/archive/{archive1.id}')
assert response.status_code == HTTPStatus.NOT_FOUND
# Archive2 no longer has alternatives.
request, response = test_client.get(f'/api/archive/{archive2.id}')
assert response.status_code == HTTPStatus.OK
assert response.json['file']['archive']['id'] == archive2.id
assert response.json['alternatives'] == []
# No Archives, no Domains.
request, response = test_client.delete(f'/api/archive/{archive2.id}')
assert response.status_code == HTTPStatus.NO_CONTENT
request, response = test_client.get(f'/api/archive/domains')
assert response.status_code == HTTPStatus.OK
assert response.json['domains'] == []

328
modules/archive/test/test_lib.py
View file

@ -1,17 +1,18 @@
import json
import pathlib
from datetime import datetime
from datetime import datetime, timedelta
from http import HTTPStatus
import mock
import pytest
from modules.archive import lib
from modules.archive.lib import get_or_create_domain, _refresh_archives, \
get_new_archive_files, delete_archive, do_archive, get_domains, get_archive_directory, group_archive_files, \
ArchiveFiles
from modules.archive.lib import get_or_create_domain, get_new_archive_files, delete_archive, do_archive, get_domains, \
group_archive_files, ArchiveFiles
from modules.archive.models import Archive, Domain
from wrolpi.dates import local_timezone
from wrolpi.db import get_db_session
from wrolpi.files.models import File
from wrolpi.media_path import MediaPath
from wrolpi.root_api import CustomJSONEncoder
from wrolpi.test.common import skip_circleci
@ -35,8 +36,8 @@ def make_fake_request_archive(readability=True, screenshot=True, title=True):
async def test_no_screenshot(test_session):
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive(screenshot=False)):
archive = await do_archive('https://example.com')
assert isinstance(archive.singlefile_path, MediaPath)
assert isinstance(archive.readability_path, MediaPath)
assert isinstance(archive.singlefile_path, pathlib.Path)
assert isinstance(archive.readability_path, pathlib.Path)
# Screenshot was empty
assert archive.screenshot_path is None
@ -45,8 +46,8 @@ async def test_no_screenshot(test_session):
async def test_no_readability(test_session):
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive(readability=False)):
archive = await do_archive('https://example.com')
assert isinstance(archive.singlefile_path, MediaPath)
assert isinstance(archive.screenshot_path, MediaPath)
assert isinstance(archive.singlefile_path, pathlib.Path)
assert isinstance(archive.screenshot_path, pathlib.Path)
# Readability empty
assert archive.readability_path is None
assert archive.readability_txt_path is None
@ -66,7 +67,7 @@ async def test_relationships(test_session, test_directory):
url = 'https://wrolpi.org:443'
domain = get_or_create_domain(session, url)
archive = Archive(
singlefile_path=f'{test_directory.name}/wrolpi.org:443/foo',
singlefile_file=File(path=test_directory / 'wrolpi.org:443/foo'),
title='bar',
url=url,
domain_id=domain.id,
@ -77,18 +78,7 @@ async def test_relationships(test_session, test_directory):
assert archive.domain == domain
@pytest.mark.asyncio
async def test_validate_paths(test_session):
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
archive = await do_archive('https://example.com')
try:
with get_db_session(commit=True):
archive.singlefile_path = 'asdf'
except ValueError as e:
assert 'relative' in str(e), f'Relative path error was not raised'
def test_archive_refresh_deleted_archive(test_session, archive_directory, archive_factory):
def test_archive_refresh_deleted_archive(test_client, test_session, archive_directory, archive_factory):
"""
Archives/Domains should be deleted when archive files are deleted.
"""
@ -97,58 +87,58 @@ def test_archive_refresh_deleted_archive(test_session, archive_directory, archiv
archive3 = archive_factory('example.com')
archive4 = archive_factory('example.org')
archive5 = archive_factory()
test_session.commit()
# Empty directories should be ignored.
empty = archive_directory / 'empty'
empty.mkdir()
(archive_directory / 'empty').mkdir()
def check_counts(archive_count, domain_count):
assert test_session.query(Archive).count() == archive_count
assert test_session.query(Domain).count() == domain_count
assert test_session.query(Archive).count() == archive_count, 'Archive count does not match'
assert test_session.query(Domain).count() == domain_count, 'Domain count does not match'
# All 3 archives are already in the DB.
check_counts(archive_count=5, domain_count=3)
_refresh_archives()
check_counts(archive_count=5, domain_count=3)
# All 5 archives are already in the DB.
check_counts(archive_count=5, domain_count=2)
test_client.post('/api/files/refresh')
check_counts(archive_count=5, domain_count=1)
# Delete archive2's files, it's the latest for 'https://example.com/1'
for path in archive2.my_paths():
for path in archive2.my_files():
path.unlink()
_refresh_archives()
check_counts(archive_count=4, domain_count=3)
test_client.post('/api/files/refresh')
check_counts(archive_count=4, domain_count=1)
# Delete archive1's files, now the URL is empty.
for path in archive1.my_paths():
for path in archive1.my_files():
path.unlink()
_refresh_archives()
check_counts(archive_count=3, domain_count=3)
test_client.post('/api/files/refresh')
check_counts(archive_count=3, domain_count=0)
# Delete archive3, now there is now example.com domain
for path in archive3.my_paths():
for path in archive3.my_files():
path.unlink()
_refresh_archives()
check_counts(archive_count=2, domain_count=2)
test_client.post('/api/files/refresh')
check_counts(archive_count=2, domain_count=0)
# Delete all the rest of the archives
for path in archive4.my_paths():
for path in archive4.my_files():
path.unlink()
for path in archive5.my_paths():
for path in archive5.my_files():
path.unlink()
_refresh_archives()
test_client.post('/api/files/refresh')
check_counts(archive_count=0, domain_count=0)
def test_fills_contents_with_refresh(test_session, archive_factory, test_client):
def test_fills_contents_with_refresh(test_client, test_session, archive_factory):
"""Refreshing archives fills in any missing contents."""
archive1 = archive_factory('example.com', 'https://example.com/one')
archive2 = archive_factory('example.com', 'https://example.com/one')
archive3 = archive_factory('example.org')
# Title can be found in archive3's singlefile
archive3 = archive_factory('example.org',
singlefile_contents='<html> some stuff<title>last title</title> other stuff</html>')
archive4 = archive_factory('example.org') # has no contents
# Clear the title, this will be filled in.
archive1.title = archive2.title = archive3.title = archive4.title = None
# Title can be found in archive3's singlefile
archive3.singlefile_path.path.write_text('<html> some stuff<title>last title</title> other stuff</html>')
contents_title_archive = [
('foo bar qux', 'my archive', archive1),
@ -156,26 +146,26 @@ def test_fills_contents_with_refresh(test_session, archive_factory, test_client)
('baz qux qux qux', None, archive3),
]
for contents, json_title, archive in contents_title_archive:
with archive.readability_txt_path.path.open('wt') as fh:
with archive.readability_txt_path.open('wt') as fh:
fh.write(contents)
if json_title:
archive.readability_json_path.path.write_text(json.dumps({'title': json_title, 'url': archive.url}))
archive.readability_json_path.write_text(json.dumps({'title': json_title, 'url': archive.url}))
test_session.commit()
# Contents are empty. Archives are all missing the title.
assert not archive1.contents
assert not archive2.contents
assert not archive3.contents
assert not archive4.contents
assert not archive1.singlefile_file.d_text
assert not archive2.singlefile_file.d_text
assert not archive3.singlefile_file.d_text
assert not archive4.singlefile_file.d_text
# Fill the contents.
_refresh_archives()
test_client.post('/api/files/refresh')
# The archives will be renamed with their title.
archive1, archive2, archive3, archive4 = test_session.query(Archive).order_by(Archive.id)
assert archive1.contents
assert archive2.contents
assert archive3.contents
assert not archive4.contents
assert archive1.singlefile_file.title
assert archive2.singlefile_file.title
assert archive3.singlefile_file.title
assert not archive4.singlefile_file.d_text
# Missing title is filled.
assert archive1.title == 'my archive'
@ -207,6 +197,7 @@ def test_get_domains(test_session, archive_factory):
archive1 = archive_factory('example.com')
archive2 = archive_factory('example.com')
archive3 = archive_factory('example.org')
test_session.commit()
assert [i['domain'] for i in get_domains()] == ['example.com', 'example.org']
@ -231,22 +222,22 @@ async def test_new_archive(test_session, fake_now):
# Everything is filled out.
assert isinstance(archive1, Archive)
assert archive1.archive_datetime is not None
assert isinstance(archive1.singlefile_path, MediaPath)
assert isinstance(archive1.readability_path, MediaPath)
assert isinstance(archive1.readability_txt_path, MediaPath)
assert isinstance(archive1.screenshot_path, MediaPath)
assert isinstance(archive1.singlefile_path, pathlib.Path)
assert isinstance(archive1.readability_path, pathlib.Path)
assert isinstance(archive1.readability_txt_path, pathlib.Path)
assert isinstance(archive1.screenshot_path, pathlib.Path)
assert archive1.title == 'ジにてこちら'
assert archive1.url is not None
assert archive1.domain is not None
# The actual files were dumped and read correctly.
with open(archive1.singlefile_path.path) as fh:
with open(archive1.singlefile_path) as fh:
assert fh.read() == '<html>\ntest single-file\nジにてこちら\n<title>some title</title></html>'
with open(archive1.readability_path.path) as fh:
with open(archive1.readability_path) as fh:
assert fh.read() == '<html>test readability content</html>'
with open(archive1.readability_txt_path.path) as fh:
with open(archive1.readability_txt_path) as fh:
assert fh.read() == '<html>test readability textContent</html>'
with open(archive1.readability_json_path.path) as fh:
with open(archive1.readability_json_path) as fh:
assert json.load(fh) == {'title': 'ジにてこちら', 'url': 'https://example.com'}
fake_now(datetime(2000, 1, 2))
@ -259,7 +250,7 @@ async def test_new_archive(test_session, fake_now):
async def test_get_title_from_html(test_session, fake_now):
fake_now(datetime(2000, 1, 1))
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
archive = await do_archive('example.com')
archive = await do_archive('https://example.com')
assert archive.title == 'ジにてこちら'
async def fake_request_archive(_):
@ -273,7 +264,7 @@ async def test_get_title_from_html(test_session, fake_now):
fake_now(datetime(2000, 1, 2))
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
archive = await do_archive('example.com')
archive = await do_archive('https://example.com')
assert archive.title == 'some title'
async def fake_request_archive(_):
@ -287,7 +278,7 @@ async def test_get_title_from_html(test_session, fake_now):
fake_now(datetime(2000, 1, 3))
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
archive = await do_archive('example.com')
archive = await do_archive('https://example.com')
assert archive.title is None
@ -321,20 +312,20 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
fake_now(datetime(2000, 1, 1))
with mock.patch('modules.archive.lib.request_archive', make_fake_request_archive()):
archive1 = await do_archive('example.com')
archive1 = await do_archive('https://example.com')
assert archive1.title == 'ジにてこちら'
assert str(archive1.singlefile_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.html'
assert str(archive1.readability_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.readability.html'
assert str(archive1.readability_txt_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.readability.txt'
assert str(archive1.readability_json_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.readability.json'
assert str(archive1.screenshot_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_ジにてこちら.png'
assert str(archive1.singlefile_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.html'
assert str(archive1.readability_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.html'
assert str(archive1.readability_txt_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.txt'
assert str(archive1.readability_json_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.readability.json'
assert str(archive1.screenshot_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_ジにてこちら.png'
async def fake_request_archive(_):
singlefile = '<html>\ntest single-file\nジにてこちら\n</html>' # no title in HTML
@ -347,18 +338,18 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
return singlefile, r, s
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
archive2 = await do_archive('example.com')
archive2 = await do_archive('https://example.com')
assert str(archive2.singlefile_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.html'
assert str(archive2.readability_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.readability.html'
assert str(archive2.readability_txt_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.readability.txt'
assert str(archive2.readability_json_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.readability.json'
assert str(archive2.screenshot_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_NA.png'
assert str(archive2.singlefile_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.html'
assert str(archive2.readability_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.readability.html'
assert str(archive2.readability_txt_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.readability.txt'
assert str(archive2.readability_json_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.readability.json'
assert str(archive2.screenshot_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_NA.png'
async def fake_request_archive(_):
singlefile = '<html>\ntest single-file\nジにてこちら\n<title>dangerous ;\\//_title</html></html>'
@ -371,62 +362,101 @@ async def test_title_in_filename(test_session, fake_now, test_directory):
return singlefile, r, s
with mock.patch('modules.archive.lib.request_archive', fake_request_archive):
archive3 = await do_archive('example.com')
archive3 = await do_archive('https://example.com')
assert str(archive3.singlefile_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.html'
assert str(archive3.readability_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.html'
assert str(archive3.readability_txt_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.txt'
assert str(archive3.readability_json_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.readability.json'
assert str(archive3.screenshot_path.path.relative_to(test_directory)) == \
'archive/2000-01-01-00-00-00_dangerous ;_title.png'
assert str(archive3.singlefile_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.html'
assert str(archive3.readability_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.html'
assert str(archive3.readability_txt_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.txt'
assert str(archive3.readability_json_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.readability.json'
assert str(archive3.screenshot_path.relative_to(test_directory)) == \
'archive/example.com/2000-01-01-00-00-00_dangerous ;_title.png'
@skip_circleci
def test_refresh_archives(test_session, test_directory):
"""Archives can be found and put in the database. A single Archive will have multiple files."""
archive_directory = get_archive_directory()
def test_refresh_archives(test_session, test_directory, test_client, make_files_structure):
"""
Archives can be found and put in the database. A single Archive will have multiple files.
"""
# The start of a typical singlefile html file.
singlefile_contents = '''<!DOCTYPE html> <html lang="en"><!--
Page saved with SingleFile
url: https://example.com
saved date: Mon May 16 2022 23:51:35 GMT+0000 (Coordinated Universal Time)
--><head><meta charset="utf-8">'''
# Make some test files to refresh.
example_dir = archive_directory / 'example.com'
example_dir.mkdir()
(example_dir / '2021-10-05 16:20:10.346823.html').touch() # renames to 2021-10-05-16-20-10_NA.html
(example_dir / '2021-10-05 16:20:10.346823.png').touch()
(example_dir / '2021-10-05 16:20:10.346823-readability.txt').touch()
(example_dir / '2021-10-05 16:20:10.346823-readability.json').touch()
(example_dir / '2021-10-05 16:20:10.346823-readability.html').touch()
with (example_dir / '2021-10-05 16:20:10.346823-readability.json').open('wt') as fh:
json.dump({'url': 'foo'}, fh)
# These should log an error because they are missing the singlefile path.
(example_dir / '2021-10-05 16:20:10.readability.html').touch()
(example_dir / '2021-10-05 16:20:11.readability.json').touch()
# Bad files should also be ignored.
(example_dir / 'random file').touch()
(example_dir / '2021-10-05 16:20:10.346827-something.html').mkdir()
(example_dir / '2021-10-05 16:20:10 no extension').touch()
make_files_structure({
# These are all for an individual Archive.
'archive/example.com/2021-10-05 16:20:10.346823.html': singlefile_contents, # renames to 2021-10-05-16-20-10_NA
'archive/example.com/2021-10-05 16:20:10.346823.png': None,
'archive/example.com/2021-10-05 16:20:10.346823-readability.txt': 'article text contents',
'archive/example.com/2021-10-05 16:20:10.346823-readability.json': '{"url": "https://example.com"}',
'archive/example.com/2021-10-05 16:20:10.346823-readability.html': '<html></html>',
# These should log an error because they are missing the singlefile path.
'archive/example.com/2021-10-05 16:20:10.readability.html': '<html></html>',
'archive/example.com/2021-10-05 16:20:10.readability.json': None,
# Bad files should also be ignored.
'archive/example.com/random file': 'hello',
'archive/example.com/2021-10-05 16:20:10 no extension': None,
})
# The single archive is found.
_refresh_archives()
with get_db_session() as session:
assert session.query(Archive).count() == 1
test_client.post('/api/files/refresh')
assert test_session.query(Archive).count() == 1
# Running the refresh does not result in a new archive.
_refresh_archives()
with get_db_session() as session:
assert session.query(Archive).count() == 1
test_client.post('/api/files/refresh')
assert test_session.query(Archive).count() == 1
# Archives file format was changed, lets check the new formats are found.
(example_dir / '2021-10-05-16-20-11_The Title.html').touch()
(example_dir / '2021-10-05-16-20-11_The Title.readability.json').write_text(json.dumps({'url': 'bar'}))
_refresh_archives()
make_files_structure({
'archive/example.com/2021-10-05-16-20-11_The Title.html': '<html></html>',
'archive/example.com/2021-10-05-16-20-11_The Title.readability.json': '{"url": "https://example.com"}',
})
test_client.post('/api/files/refresh')
# The old formatted archive above is renamed.
assert (example_dir / '2021-10-05-16-20-10_NA.html').is_file()
with get_db_session() as session:
assert session.query(Archive).count() == 2
assert (test_directory / 'archive/example.com/2021-10-05-16-20-10_NA.html').is_file()
assert test_session.query(Archive).count() == 2
content = json.dumps({'search_str': 'text', 'model': 'archive'})
request, response = test_client.post('/api/files/search', content=content)
assert response.status_code == HTTPStatus.OK
assert response.json['files'], 'No files matched "text"'
assert response.json['files'][0]['model'] == 'archive', 'Returned file was not an archive'
assert response.json['files'][0].get('path') == 'archive/example.com/2021-10-05-16-20-10_NA.html', \
'Could not find readability text file containing "text"'
assert response.json['files'][0]['archive']['archive_datetime'], 'Archive has no datetime'
def test_refresh_archives_invalid_file(test_session, test_directory, test_client, make_files_structure):
"""
Invalid Archives should be removed during refresh.
"""
*_, bogus_path = make_files_structure({
'2022-09-04-16-20-11_The Title.html': '<html></html>',
'2022-09-04-16-20-11_The Title.readability.json': '{"url": "https://example.com"}',
'bogus file': None,
})
test_client.post('/api/files/refresh')
assert test_session.query(File).count() == 3 # Three files total.
assert test_session.query(Archive).count() == 1 # Only one real archive.
# Create an invalid Archive, it should be removed during refresh.
test_session.add(Archive(singlefile_path=bogus_path))
# Claim the bogus file as an Archive, this should be removed.
bogus_file: File = test_session.query(File).filter_by(path=bogus_path).one()
bogus_file.model = 'archive'
test_session.commit()
assert test_session.query(File).count() == 3
assert test_session.query(Archive).count() == 2
test_client.post('/api/files/refresh')
assert test_session.query(File).count() == 3, 'More files in DB than we created'
assert test_session.query(Archive).count() == 1, 'The bogus Archive was not removed'
assert bogus_file.model is None, 'Model was not removed'
def test_group_archive_files(test_directory):
@ -488,9 +518,10 @@ def test_migrate_archive_files(test_session, archive_directory, archive_factory)
archive1 = archive_factory('example.com', title='the title')
archive2 = archive_factory('example.com')
archive3 = archive_factory('example.org')
assert archive1.singlefile_path.path.is_file()
assert archive2.singlefile_path.path.is_file()
assert archive3.singlefile_path.path.is_file()
test_session.commit()
assert archive1.singlefile_path.is_file()
assert archive2.singlefile_path.is_file()
assert archive3.singlefile_path.is_file()
example_com_directory = archive_directory / 'example.com'
example_org_directory = archive_directory / 'example.org'
@ -532,6 +563,25 @@ def test_migrate_archive_files(test_session, archive_directory, archive_factory)
'example.org/2000-01-01-00-00-00_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.readability.json')),
]
# New Archive files still exist.
assert all(i.is_file() for i in archive1.my_paths())
assert all(i.is_file() for i in archive2.my_paths())
assert all(i.is_file() for i in archive3.my_paths())
assert all(i.is_file() for i in archive1.my_files())
assert all(i.is_file() for i in archive2.my_files())
assert all(i.is_file() for i in archive3.my_files())
def test_archive_files(test_directory):
af = lib.ArchiveFiles(
singlefile=test_directory / 'archive/singlefile.html',
)
assert repr(af) == \
"<ArchiveFiles singlefile='singlefile.html' readability=None readability_txt=None readability_json=None" \
" screenshot=None>"
def test_archive_order(test_session, test_directory, archive_factory):
archive1 = archive_factory()
archive2 = archive_factory()
archive2.archive_datetime += timedelta(seconds=10)
test_session.commit()
assert archive2 > archive1
assert archive1 < archive2

21
modules/files/conftest.py
View file

@ -1,21 +0,0 @@
import pathlib
from typing import List, Callable
import pytest
@pytest.fixture
def make_files_structure(test_directory) -> Callable[[List[str]], List[pathlib.Path]]:
def create_files(paths: List[str]) -> List[pathlib.Path]:
files = []
for name in paths:
path = test_directory / name
if name.endswith('/'):
path.mkdir()
else:
path.parent.mkdir(parents=True, exist_ok=True)
path.touch()
files.append(path)
return files
return create_files

185
modules/files/lib.py
View file

@ -1,185 +0,0 @@
import asyncio
import re
import subprocess
from functools import wraps
from pathlib import Path
from typing import List
from uuid import uuid4
import psycopg2
from sqlalchemy.orm import Session
from modules.files.models import File
from wrolpi.cmd import which
from wrolpi.common import get_media_directory, wrol_mode_check, walk, chunks, logger
from wrolpi.dates import from_timestamp
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models
from wrolpi.errors import InvalidFile
from wrolpi.vars import PYTEST
logger = logger.getChild(__name__)
__all__ = ['list_files', 'delete_file', 'split_file_name', 'upsert_file', 'refresh_files', 'search', 'get_mimetype']
def filter_parent_directories(directories: List[Path]) -> List[Path]:
"""
Remove parent directories if their children are in the list.
>>> filter_parent_directories([Path('foo'), Path('foo/bar'), Path('baz')])
[Path('foo/bar'), Path('baz')]
"""
unique_children = set()
for directory in sorted(directories):
for parent in directory.parents:
# Remove any parent of this child.
if parent in unique_children:
unique_children.remove(parent)
unique_children.add(directory)
# Restore the original order.
new_directories = [i for i in directories if i in unique_children]
return new_directories
def list_files(directories: List[str]) -> List[Path]:
"""
List all files down to the directories provided. This includes all parent directories of the directories.
"""
media_directory = get_media_directory()
# Always display the media_directory files.
paths = list(media_directory.iterdir())
if directories:
directories = [media_directory / i for i in directories if i]
directories = filter_parent_directories(directories)
for directory in directories:
directory = media_directory / directory
for parent in directory.parents:
is_relative_to = str(media_directory).startswith(str(parent))
if parent == Path('.') or is_relative_to:
continue
paths.extend(parent.iterdir())
paths.extend(directory.iterdir())
return paths
@wrol_mode_check
def delete_file(file: str):
"""Delete a file in the media directory."""
file = get_media_directory() / file
if file.is_dir() or not file.is_file():
raise InvalidFile(f'Invalid file {file}')
file.unlink()
FILE_NAME_REGEX = re.compile(r'[_ ]')
def split_file_name(path: Path) -> List[str]:
"""
Split a file name into words.
>>> split_file_name(Path('foo.mp4'))
['foo']
>>> split_file_name(Path('foo bar.mp4'))
['foo', 'bar']
"""
words = [str(i) for i in FILE_NAME_REGEX.split(path.stem)]
return words
FILE_BIN = which('file', '/usr/bin/file')
def get_mimetype(path: Path) -> str:
"""Get the mimetype using the builtin `file` command."""
cmd = (FILE_BIN, '--mime-type', str(path.absolute()))
output = subprocess.check_output(cmd)
output = output.decode()
mimetype = output.split(' ')[-1].strip()
return mimetype
def upsert_file(path: Path, session: Session) -> File:
"""Update/insert a File in the DB. Gather metadata about it."""
file = session.query(File).filter_by(path=path).one_or_none()
if not file:
file = File(path=path)
session.add(file)
if not file.mimetype:
file.mimetype = get_mimetype(path)
if not file.title:
file.title = split_file_name(path)
if not file.size or not file.modification_datetime:
stat = path.stat()
file.size = stat.st_size
file.modification_datetime = file.modification_datetime or from_timestamp(stat.st_mtime)
return file
def _refresh_files():
"""Find and index all files"""
logger.info('Refreshing Files')
# Mark all files as unverified. Any records that are unverified after refresh will be deleted.
with get_db_curs(commit=True) as curs:
curs.execute('UPDATE file SET idempotency=null') # noqa
paths = filter(lambda i: i.is_file(), walk(get_media_directory()))
idempotency = str(uuid4())
for chunk in chunks(paths, 20):
with get_db_session(commit=True) as session:
for path in chunk:
file = upsert_file(path, session)
file.idempotency = idempotency
# Remove any records where the file no longer exists.
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM file WHERE idempotency IS NULL')
logger.info('Done refreshing Files')
@wraps(_refresh_files)
def refresh_files():
"""Schedule a refresh task if not testing. If testing, do a synchronous refresh."""
if PYTEST:
return _refresh_files()
async def _():
return _refresh_files()
asyncio.create_task(_())
def search(search_str: str, limit: int, offset: int):
"""
Search the "title" of each file. The title is the words in the file's stem.
"""
with get_db_curs() as curs:
stmt = '''
SELECT id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total
FROM file
WHERE textsearch @@ websearch_to_tsquery(%(search_str)s)
OFFSET %(offset)s LIMIT %(limit)s
'''
params = dict(search_str=search_str, offset=offset, limit=limit)
curs.execute(stmt, params)
try:
results = list(map(dict, curs.fetchall()))
except psycopg2.ProgrammingError:
# No files.
return [], 0
total = results[0]['total'] if results else 0
ranked_ids = [i['id'] for i in results]
with get_db_session() as session:
results = get_ranked_models(ranked_ids, File, session)
results = [i.__json__() for i in results]
return results, total

36
modules/files/models.py
View file

@ -1,36 +0,0 @@
from sqlalchemy import Integer, Column, String, Computed
from sqlalchemy.orm import deferred
from wrolpi.common import Base, ModelHelper, get_media_directory, tsvector
from wrolpi.dates import TZDateTime
from wrolpi.media_path import MediaPathType
class File(ModelHelper, Base):
__tablename__ = 'file'
id = Column(Integer, primary_key=True)
path = Column(MediaPathType)
idempotency = Column(String)
mimetype = Column(String)
modification_datetime = Column(TZDateTime)
size = Column(Integer)
title = deferred(Column(String)) # used by textsearch column
textsearch = deferred(
Column(tsvector, Computed('''setweight(to_tsvector('english'::regconfig, title), 'A'::"char")''')))
def __repr__(self):
return f'<File id={self.id} path={self.path.relative_to(get_media_directory())} mime={self.mimetype}>'
def __json__(self):
path = self.path.path.relative_to(get_media_directory())
d = dict(
id=self.id,
mimetype=self.mimetype,
modified=self.modification_datetime, # React browser expects this.
path=path,
size=self.size,
key=path,
)
return d

207
modules/files/test/test_lib.py
View file

@ -1,207 +0,0 @@
import shutil
from pathlib import Path
from typing import List, Iterable
import pytest
from sqlalchemy.orm import Session
from modules.files import lib
from modules.files.models import File
from wrolpi.errors import InvalidFile
from wrolpi.media_path import MediaPathType
from wrolpi.vars import PROJECT_DIR
def assert_files(session: Session, expected):
files = {str(i.path.relative) for i in session.query(File).all()}
assert files == set(expected)
def test_list_files(make_files_structure, test_directory):
files = [
'archives/bar.txt',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory/',
'videos/other video.mp4',
'videos/some video.mp4',
]
make_files_structure(files)
def check_list_files(path, expected):
result = lib.list_files(path)
result = sorted(result)
assert [str(i.relative_to(test_directory)) for i in result] == expected
check_list_files([], ['archives', 'empty directory', 'videos'])
# Falsey directories are ignored.
check_list_files([None, ''], ['archives', 'empty directory', 'videos'])
check_list_files(['empty directory'], ['archives', 'empty directory', 'videos'])
check_list_files(['empty directory'], ['archives', 'empty directory', 'videos'])
# Trailing slash is ignored.
check_list_files(['empty directory/'], ['archives', 'empty directory', 'videos'])
check_list_files(['archives'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/foo.txt',
'empty directory',
'videos',
])
check_list_files(['archives/baz'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
])
# Including duplicate directories does not change results.
check_list_files(['archives/baz', 'archives'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
])
check_list_files(['videos'], [
'archives',
'empty directory',
'videos',
'videos/other video.mp4',
'videos/some video.mp4',
])
# All files and directories are listed.
check_list_files(['videos', 'archives/baz'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
'videos/other video.mp4',
'videos/some video.mp4',
])
@pytest.mark.parametrize(
'directories,expected', [
([Path('foo')], [Path('foo')]),
([Path('foo'), Path('bar')], [Path('foo'), Path('bar')]),
([Path('foo'), Path('foo/bar'), Path('baz')], [Path('foo/bar'), Path('baz')]),
([Path('foo/bar'), Path('foo'), Path('baz')], [Path('foo/bar'), Path('baz')]),
([Path('baz'), Path('foo/bar'), Path('foo')], [Path('baz'), Path('foo/bar')]),
([Path('foo/bar'), Path('foo'), Path('foo/bar/baz')], [Path('foo/bar/baz')]),
]
)
def test_filter_parent_directories(directories, expected):
assert lib.filter_parent_directories(directories) == expected
def test_delete_file(make_files_structure, test_directory):
"""
File in the media directory can be deleted.
"""
files = [
'archives/foo.txt',
'bar.txt',
'baz/',
]
make_files_structure(files)
lib.delete_file('bar.txt')
assert (test_directory / 'archives/foo.txt').is_file()
assert not (test_directory / 'bar.txt').is_file()
assert (test_directory / 'baz').is_dir()
lib.delete_file('archives/foo.txt')
assert not (test_directory / 'archives/foo.txt').is_file()
assert not (test_directory / 'bar.txt').is_file()
with pytest.raises(InvalidFile):
lib.delete_file('baz')
with pytest.raises(InvalidFile):
lib.delete_file('does not exist')
assert (test_directory / 'baz').is_dir()
@pytest.mark.parametrize(
'path,expected',
[
('foo.mp4', ['foo']),
('foo bar.txt', ['foo', 'bar']),
('foo_bar.mp4', ['foo', 'bar']),
('foo-bar', ['foo-bar']),
('123 foo bar', ['123', 'foo', 'bar']),
('123foo bar', ['123foo', 'bar']),
]
)
def test_split_file_name(path, expected):
assert lib.split_file_name(Path(path)) == expected
def test_refresh_files(test_session, make_files_structure, test_directory):
files = make_files_structure([
'foo.txt',
'bar.txt',
'baz.txt',
])
foo, bar, baz = files
def get_relative_strs(files_: Iterable[MediaPathType]) -> List[str]:
return sorted([str(i.path.path.relative_to(test_directory)) for i in files_])
lib.refresh_files()
results = test_session.query(File)
assert get_relative_strs(results) == ['bar.txt', 'baz.txt', 'foo.txt']
baz.unlink()
lib.refresh_files()
results = test_session.query(File)
assert get_relative_strs(results) == ['bar.txt', 'foo.txt']
foo.unlink()
bar.unlink()
lib.refresh_files()
results = test_session.query(File)
assert get_relative_strs(results) == []
def test_mime_type(test_session, make_files_structure, test_directory):
"""Files module uses the `file` command to get the mimetype of each file."""
from PIL import Image
foo, bar, baz, empty = make_files_structure([
'dir/foo text.txt',
'dir/bar.jpeg',
'dir/baz.mp4',
'dir/empty',
])
foo.write_text('some text')
Image.new('RGB', (25, 25), color='grey').save(bar)
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', baz)
lib.refresh_files()
assert_files(test_session, ['dir/foo text.txt', 'dir/bar.jpeg', 'dir/baz.mp4', 'dir/empty'])
foo = test_session.query(File).filter_by(path=f'{test_directory}/dir/foo text.txt').one()
bar = test_session.query(File).filter_by(path=f'{test_directory}/dir/bar.jpeg').one()
baz = test_session.query(File).filter_by(path=f'{test_directory}/dir/baz.mp4').one()
empty = test_session.query(File).filter_by(path=f'{test_directory}/dir/empty').one()
assert foo.mimetype == 'text/plain'
assert bar.mimetype == 'image/jpeg'
assert baz.mimetype == 'video/mp4'
assert empty.mimetype == 'inode/x-empty'

10
modules/inventory/api.py
View file

@ -8,7 +8,7 @@ from sanic_ext.extensions.openapi import openapi
from wrolpi.common import run_after, recursive_map
from wrolpi.errors import ValidationError
from wrolpi.root_api import get_blueprint, json_response
from . import common, inventory, schema
from modules.inventory import common, inventory, schema
NAME = 'inventory'
@ -16,25 +16,25 @@ bp = get_blueprint('Inventory', '/api/inventory')
@bp.get('/categories')
def _get_categories(_: Request):
def get_categories(_: Request):
categories = inventory.get_categories()
return json_response(dict(categories=categories))
@bp.get('/brands')
def _get_brands(_: Request):
def get_brands(_: Request):
brands = inventory.get_brands()
return json_response(dict(brands=brands))
@bp.get('/')
def inventories_get(_: Request):
def get_inventories(_: Request):
inventories = inventory.get_inventories()
return json_response(dict(inventories=inventories))
@bp.get('/<inventory_id:int>')
def inventory_get(_: Request, inventory_id: int):
def get_inventory(_: Request, inventory_id: int):
by_category = common.get_inventory_by_category(inventory_id)
by_subcategory = common.get_inventory_by_subcategory(inventory_id)
by_name = common.get_inventory_by_name(inventory_id)

2
modules/inventory/inventory.py
View file

@ -123,7 +123,7 @@ def get_categories() -> List[Tuple[str, str]]:
"""
categories = DEFAULT_CATEGORIES.copy()
with get_db_curs() as curs:
curs.execute('SELECT DISTINCT subcategory, category FROM item ORDER BY 1, 2')
curs.execute("SELECT DISTINCT COALESCE(subcategory, ''), COALESCE(category, '') FROM item ORDER BY 1, 2")
try:
db_categories = curs.fetchall()
db_categories = [tuple(i) for i in db_categories]

9
modules/inventory/test/test_api.py
View file

@ -84,6 +84,15 @@ def test_inventory_api(test_session, test_client):
inventory = response.json['inventories'][0]
inventory_id = inventory['id']
request, response = test_client.get(f'/api/inventory/{inventory_id}')
assert response.status_code == HTTPStatus.OK
request, response = test_client.get('/api/inventory/categories')
assert response.status_code == HTTPStatus.OK
request, response = test_client.get('/api/inventory/brands')
assert response.status_code == HTTPStatus.OK
inventory = dict(name='new name')
request, response = test_client.put(f'/api/inventory/{inventory_id}', content=json.dumps(inventory))
assert response.status_code == HTTPStatus.NO_CONTENT, response.status_code

2
modules/map/models.py
View file

@ -35,5 +35,5 @@ class MapFile(Base, ModelHelper):
'path': self.path,
'imported': self.imported,
'size': self.size,
'time_to_import': dates.seconds_to_timestamp(seconds_to_import(self.path.path, self.size)),
'time_to_import': dates.seconds_to_timestamp(seconds_to_import(self.path, self.size)),
}

21
modules/map/test/test_lib.py Normal file
View file

@ -0,0 +1,21 @@
from unittest import mock
from modules.map import lib
def test_get_map_paths(test_directory):
pbf_dir = test_directory / 'map/pbf'
pbf_dir.mkdir(parents=True)
pbf = pbf_dir / 'map.osm.pbf'
pbf.touch()
dump = pbf_dir / 'map.dump'
dump.touch()
with mock.patch('modules.map.lib.subprocess.check_output') as mock_check_output:
mock_check_output.return_value = b'OpenStreetMap Protocolbuffer Binary Format'
assert lib.get_map_paths() == [pbf, ]
with mock.patch('modules.map.lib.subprocess.check_output') as mock_check_output:
mock_check_output.return_value = b'PostgreSQL custom database dump'
assert lib.get_map_paths() == [dump, ]

124
modules/videos/__init__.py
View file

@ -1 +1,123 @@
from modules.videos import downloader
import json
import pathlib
from abc import ABC
from typing import List, Dict
from sqlalchemy.orm import Session
from modules.videos.models import Video, Channel
from wrolpi.captions import read_captions, extract_captions
from wrolpi.common import logger, register_modeler, register_after_refresh, limit_concurrent
from wrolpi.db import get_db_curs
from wrolpi.files.indexers import Indexer, register_indexer
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from .downloader import video_downloader # Import downloaders so they are registered.
logger = logger.getChild(__name__)
__all__ = ['video_modeler', 'VideoIndexer']
@register_modeler
def video_modeler(groups: Dict[str, List[File]], session: Session):
# Only process groups that contain a video file.
count = 0
local_groups = groups.copy()
for stem, group in local_groups.items():
video_file = next((i for i in group if i.mimetype.split('/')[0] == 'video' and i.path.suffix != '.part'), None)
if not video_file:
# Not a video group.
continue
session.flush(group)
poster_file = next((i for i in group if i.mimetype.split('/')[0] == 'image'), None)
caption_file = next((i for i in group if i.path.name.endswith('.en.vtt') or i.path.name.endswith('.en.srt')),
None)
info_json_file = next((i for i in group if i.path.name.endswith('.info.json')), None)
if poster_file:
poster_file.associated = True
poster_file.do_index()
if caption_file:
caption_file.associated = True
caption_file.do_index()
if info_json_file:
info_json_file.associated = True
info_json_file.do_index()
video: Video = Video.upsert(video_file, session)
video.poster_file = poster_file
video.caption_file = caption_file
video.info_json_file = info_json_file
video.video_file.do_index()
video.validate(session)
# Remove this group, it will not be processed by other modelers.
del groups[stem]
count += 1
if not count % 100:
# Commit occasionally.
session.commit()
@register_after_refresh
@limit_concurrent(1)
def video_cleanup():
# Claim all Videos in a Channel's directory for that Channel.
logger.info('Claiming Videos for their Channels')
with get_db_curs(commit=True) as curs:
curs.execute('''
UPDATE video v
SET channel_id = c.id
FROM channel c
WHERE
v.video_path LIKE c.directory || '/%'::VARCHAR
AND v.channel_id IS NULL
''')
EXTRACT_SUBTITLES = False
@register_indexer('video')
class VideoIndexer(Indexer, ABC):
"""Handles video files like mp4/ogg."""
@staticmethod
def get_description(file):
video_path: pathlib.Path = file.path.path if hasattr(file.path, 'path') else file.path
info_json_path = video_path.with_suffix('.info.json')
if info_json_path.is_file():
with info_json_path.open('rt') as fh:
try:
return json.load(fh)['description']
except Exception:
if not PYTEST:
logger.warning(f'Video info json file exists, but cannot get description. {file}')
return None
@classmethod
def create_index(cls, file):
"""
Index the video file and it's associated files.
a = title.
b = <empty>
c = description from info json.
d = captions from info json, or extracted from video file.
"""
a = cls.get_title(file)
c = cls.get_description(file)
# Detect the associated caption file.
caption_path = file.path.with_suffix('.en.vtt')
d = None
if caption_path.is_file():
d = read_captions(caption_path)
elif EXTRACT_SUBTITLES:
d = extract_captions(file.path) or ''
return a, None, c, d

48
modules/videos/api.py
View file

@ -1,16 +1,12 @@
import asyncio
from functools import wraps
from http import HTTPStatus
from typing import List
from sanic import Blueprint, response
from sanic.request import Request
from sanic_ext import validate
from sanic_ext.extensions.openapi import openapi
from wrolpi.common import create_websocket_feed, get_sanic_url, \
wrol_mode_check
from wrolpi.common import logger
from wrolpi.common import wrol_mode_check
from wrolpi.root_api import add_blueprint, json_response
from wrolpi.schema import JSONErrorResponse
from . import lib, schema
@ -29,43 +25,6 @@ add_blueprint(bp)
logger = logger.getChild(__name__)
refresh_queue, refresh_event = create_websocket_feed('refresh', '/feeds/refresh', content_bp)
@content_bp.post('/refresh')
@content_bp.post('/refresh/<channel_id:str>')
@openapi.description('Search for videos that have previously been downloaded and stored.')
@openapi.response(HTTPStatus.OK, schema.StreamResponse)
@openapi.response(HTTPStatus.BAD_REQUEST, JSONErrorResponse)
@wrol_mode_check
async def refresh(_, channel_id: int = None):
refresh_logger = logger.getChild('refresh')
stream_url = get_sanic_url(scheme='ws', path='/api/videos/feeds/refresh')
# Only one refresh can run at a time
if refresh_event.is_set():
return response.json({'error': 'Refresh already running', 'stream_url': stream_url}, HTTPStatus.CONFLICT)
refresh_event.set()
async def do_refresh():
try:
refresh_logger.info('refresh started')
channel_ids = [channel_id] if channel_id else None
await refresh_videos(channel_ids)
refresh_logger.info('refresh complete')
except Exception:
raise
finally:
refresh_event.clear()
coro = do_refresh()
asyncio.ensure_future(coro)
refresh_logger.debug('do_refresh scheduled')
return response.json({'code': 'stream-started', 'stream_url': stream_url})
@content_bp.post('/download/<channel_id:int>')
@openapi.description('Update a channel catalog, download any missing videos')
@ -77,11 +36,6 @@ def download(_, channel_id: int = None):
return response.empty()
@wraps(lib.refresh_videos)
async def refresh_videos(channel_ids: List[int] = None):
return lib.refresh_videos(channel_ids=channel_ids)
@content_bp.post('/favorite')
@openapi.definition(
description='Toggle the favorite flag on a video',

33
modules/videos/channel/api.py
View file

@ -1,4 +1,5 @@
import asyncio
import pathlib
from http import HTTPStatus
from sanic import response, Blueprint
@ -8,11 +9,12 @@ from sanic_ext.extensions.openapi import openapi
from wrolpi.common import logger, wrol_mode_check, make_media_directory, \
get_media_directory, run_after
from wrolpi.errors import UnknownDirectory, UnknownChannel
from wrolpi.root_api import json_response
from wrolpi.schema import JSONErrorResponse
from wrolpi.vars import PYTEST
from . import lib
from .. import schema
from .. import schema, Channel
from ..lib import save_channels_config
channel_bp = Blueprint('Channel', url_prefix='/api/videos/channels')
@ -32,7 +34,7 @@ async def get_channels(_: Request):
@openapi.description('Get a Channel')
@openapi.response(HTTPStatus.OK, schema.ChannelResponse)
def channel_get(_: Request, channel_id: int = None):
channel = lib.get_channel(channel_id=channel_id)
channel = lib.get_channel(channel_id=channel_id, return_dict=True)
channel.pop('info_json')
return json_response({'channel': channel})
@ -55,9 +57,7 @@ def channel_post(_: Request, body: schema.ChannelPostRequest):
# Refresh the videos asynchronously
if not PYTEST:
from ..api import refresh_videos
coro = refresh_videos([channel.id])
asyncio.ensure_future(coro)
asyncio.ensure_future(channel.refresh_files())
return response.json({'success': 'Channel created successfully'}, HTTPStatus.CREATED,
{'Location': f'/api/videos/channels/{channel.id}'})
@ -74,6 +74,7 @@ def channel_post(_: Request, body: schema.ChannelPostRequest):
@run_after(save_channels_config)
@wrol_mode_check
def channel_update(_: Request, channel_id: int, body: schema.ChannelPutRequest):
body.directory = pathlib.Path(body.directory) if body.directory else None
channel = lib.update_channel(data=body, channel_id=channel_id)
return response.raw('', HTTPStatus.NO_CONTENT,
headers={'Location': f'/api/videos/channels/{channel.id}'})
@ -86,3 +87,25 @@ def channel_update(_: Request, channel_id: int, body: schema.ChannelPutRequest):
def channel_delete(_: Request, channel_id: int):
lib.delete_channel(channel_id=channel_id)
return response.raw('', HTTPStatus.NO_CONTENT)
@channel_bp.post('/refresh/<channel_id:int>')
@openapi.description('Refresh all files in the Channel directory')
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@wrol_mode_check
def channel_refresh(_: Request, channel_id: int):
channel: Channel = lib.get_channel(channel_id=channel_id, return_dict=False)
if not channel:
raise UnknownChannel()
asyncio.ensure_future(channel.refresh_files())
return response.empty()
@channel_bp.post('/download/<channel_id:int>')
@openapi.description('Download the latest catalog for a Channel, download any missing videos.')
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
@wrol_mode_check
def channel_download(_: Request, channel_id: int):
lib.download_channel(channel_id)
return response.empty()

9
modules/videos/channel/lib.py
View file

@ -1,5 +1,6 @@
import functools
from pathlib import Path
from typing import List, Dict, Union
from typing import List, Dict, Union, Optional
from sqlalchemy.orm import Session
from sqlalchemy.orm.exc import NoResultFound
@ -7,7 +8,6 @@ from sqlalchemy.orm.exc import NoResultFound
from wrolpi.common import run_after, logger, \
get_media_directory
from wrolpi.db import get_db_curs, optional_session
from wrolpi.downloader import download_manager
from wrolpi.errors import UnknownChannel, UnknownDirectory, APIError, ValidationError, InvalidDownload
from .. import schema
from ..common import check_for_channel_conflicts
@ -125,6 +125,9 @@ def update_channel(session: Session, *, data: schema.ChannelPutRequest, channel_
data.directory.mkdir()
else:
raise UnknownDirectory()
elif data.directory:
# Keep the old directory because the user can't change a Channel's directory.
data.directory = channel.directory
# Verify that the URL/Name/directory aren't taken
check_for_channel_conflicts(
@ -165,7 +168,7 @@ def create_channel(session: Session, data: schema.ChannelPostRequest, return_dic
name=data.name,
url=data.url or None,
match_regex=data.match_regex,
directory=data.directory, # noqa
directory=get_media_directory() / data.directory,
download_frequency=data.download_frequency,
source_id=data.source_id,
)

171
modules/videos/channel/test/test_api.py
View file

@ -9,11 +9,10 @@ import mock
import pytest
from modules.videos.channel.lib import delete_channel
from modules.videos.common import get_no_channel_directory
from modules.videos.models import Channel, Video
from modules.videos.models import Channel
from wrolpi.dates import now
from wrolpi.db import get_db_session
from wrolpi.downloader import download_manager, Download
from wrolpi.downloader import download_manager, Download, DownloadResult
from wrolpi.test.common import assert_dict_contains
@ -26,31 +25,8 @@ def test_get_channels(test_directory, channel_factory, test_client):
assert len(response.json['channels']) == 3
@mock.patch('modules.videos.lib.validate_video')
def test_refresh_no_channel(_, test_session, test_client, test_directory, simple_channel):
# A regular user-defined channel
vid1 = simple_channel.directory / 'MyChannelName_20000101_abcdefghijk_title1.mp4'
vid1.touch()
vid2 = simple_channel.directory / 'MyChannelName_20000101_abcdefghijk_title2.mp4'
vid2.touch()
# The special NO CHANNEL directory.
no_channel_path = get_no_channel_directory()
vid3 = no_channel_path / 'some video.mp4'
vid3.touch()
no_channel_subdir = no_channel_path / 'subdir'
no_channel_subdir.mkdir()
vid4 = no_channel_subdir / 'other video.mp4'
vid4.touch()
test_client.post('/api/videos/refresh')
assert test_session.query(Video).count() == 4
def test_get_video(test_client, test_session, simple_channel, video_factory):
"""
Test that you get can information about a video. Test that video file can be gotten.
"""
"""Test that you get can information about a video. Test that video file can be gotten."""
now_ = now()
video1 = video_factory(channel_id=simple_channel.id)
video1.upload_date, video1.title = now_, 'vid1'
@ -66,11 +42,11 @@ def test_get_video(test_client, test_session, simple_channel, video_factory):
# Get the video info we inserted
_, response = test_client.get('/api/videos/video/1')
assert response.status_code == HTTPStatus.OK, response.json
assert_dict_contains(response.json['video'], {'title': 'vid1'})
assert_dict_contains(response.json['file']['video'], {'title': 'vid1'})
# The next video is included.
assert response.json['prev'] is None
assert_dict_contains(response.json['next'], {'title': 'vid2'})
assert_dict_contains(response.json['next']['video'], {'title': 'vid2'})
def test_channel_no_download_frequency(test_client, test_session, test_directory, simple_channel):
@ -118,7 +94,7 @@ def test_channel_frequency_update(download_channel, test_client, test_session, t
assert old_frequency
data = dict(
directory=str(download_channel.directory.path),
directory=str(download_channel.directory),
name=download_channel.name,
url=download_channel.url,
download_frequency=100,
@ -192,73 +168,12 @@ def test_channel_download_crud(test_session, simple_channel):
assert test_session.query(Download).count() == 0
def test_video_search(test_client, test_session):
"""
Test that videos can be searched and that their order is by their textsearch rank.
"""
# These captions have repeated letters, so they will be higher in the ranking.
videos = [
('1', 'b b b b e d d'),
('2', '2 b b b d'),
('3', 'b b'),
('4', 'b e e'),
('5', ''),
]
for title, caption in videos:
test_session.add(Video(title=title, caption=caption, video_path='foo'))
test_session.commit()
def do_search(search_str, limit=20):
d = json.dumps({'search_str': search_str, 'limit': limit})
_, resp = test_client.post('/api/videos/search', content=d)
return resp
def search_is_as_expected(resp, expected):
assert resp.status_code == HTTPStatus.OK
response_ids = [i['id'] for i in resp.json['videos']]
assert response_ids == expected
assert resp.json['totals']['videos'] == len(expected)
# Repeated runs should return the same result
for _ in range(2):
# Only videos with a b are returned, ordered by the amount of b's
response = do_search('b')
search_is_as_expected(response, [1, 2, 3, 4])
# Only two captions have e
response = do_search('e')
search_is_as_expected(response, [4, 1])
# Only two captions have d
response = do_search('d')
search_is_as_expected(response, [1, 2])
# 5 can be gotten by it's title
response = do_search('5')
search_is_as_expected(response, [5])
# only video 1 has e and d
response = do_search('e d')
search_is_as_expected(response, [1])
# video 1 and 4 have b and e, but 1 has more
response = do_search('b e')
search_is_as_expected(response, [1, 4])
# Check totals are correct even with a limit
response = do_search('b', 2)
assert [i['id'] for i in response.json['videos']] == [1, 2]
assert response.json['totals']['videos'] == 4
def test_video_file_name(test_session, simple_video, test_client):
"""
If a Video has no title, the front-end can use the file name as the title.
"""
"""If a Video has no title, the front-end can use the file name as the title."""
_, resp = test_client.get(f'/api/videos/video/{simple_video.id}')
assert resp.status_code == HTTPStatus.OK
assert resp.json['video']['video_path'] == 'simple_video.mp4'
assert resp.json['video'].get('stem') == 'simple_video'
assert resp.json['file']['video']['video_path'] == 'simple_video.mp4'
assert resp.json['file']['video'].get('stem') == 'simple_video'
def test_channel_conflicts(test_client, test_session, test_directory):
@ -339,26 +254,46 @@ def test_channel_empty_url_doesnt_conflict(test_client, test_session, test_direc
@pytest.mark.asyncio
async def test_download_channel_no_refresh(test_session, download_channel, video_download_manager):
"""A Channel cannot be downloaded until it has been refreshed."""
async def test_download_channel_no_refresh(test_session, download_channel, video_download_manager, video_factory):
"""A Channel cannot be downloaded until it has been refreshed.
Videos already downloaded are not downloaded again."""
vid = video_factory(channel_id=download_channel.id, with_video_file=True, with_poster_ext='jpg')
vid.url = 'https://example.com/1'
d = download_channel.get_download()
test_session.commit()
assert not d.next_download
def check_refreshed(expected: bool):
def assert_refreshed(expected: bool):
channel = test_session.query(Channel).one()
assert channel.refreshed == expected
check_refreshed(False)
assert_refreshed(False)
test_session.commit()
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info:
mock_extract_info.return_value = {'entries': [], 'url': 'foo', 'uploader': 'some uploader',
downloaded_urls = []
async def do_download(_, download):
# Track all downloaded URLs.
downloaded_urls.append(download.url)
return DownloadResult(success=True)
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
mock.patch('modules.videos.downloader.VideoDownloader.do_download', do_download):
entries = [
dict(id=vid.source_id, view_count=0, webpage_url='https://example.com/1'), # Already downloaded.
dict(id='not downloaded', view_count=0, webpage_url='https://example.com/2'),
]
mock_extract_info.return_value = {'entries': entries, 'url': 'foo', 'uploader': 'some uploader',
'channel_id': 'the id', 'id': 'the id'}
await video_download_manager.do_downloads()
await video_download_manager.wait_for_all_downloads()
check_refreshed(True)
# Channel was refreshed before downloading videos.
assert_refreshed(True)
assert d.next_download
# Only the missing video was downloaded.
assert downloaded_urls == ['https://example.com/2']
def test_channel_post_directory(test_session, test_client, test_directory):
@ -367,7 +302,7 @@ def test_channel_post_directory(test_session, test_client, test_directory):
data = dict(name='foo', directory='foo')
request, response = test_client.post('/api/videos/channels', content=json.dumps(data))
assert response.status_code == HTTPStatus.CREATED
directory = test_session.query(Channel).filter_by(id=1).one().directory.path
directory = test_session.query(Channel).filter_by(id=1).one().directory
assert (test_directory / 'foo') == directory
assert not directory.is_dir()
assert directory.is_absolute()
@ -376,7 +311,7 @@ def test_channel_post_directory(test_session, test_client, test_directory):
data = dict(name='bar', directory='bar', mkdir=True)
request, response = test_client.post('/api/videos/channels', content=json.dumps(data))
assert response.status_code == HTTPStatus.CREATED
directory = test_session.query(Channel).filter_by(id=2).one().directory.path
directory = test_session.query(Channel).filter_by(id=2).one().directory
assert (test_directory / 'bar') == directory
assert directory.is_dir()
assert directory.is_absolute()
@ -455,7 +390,7 @@ def test_channel_crud(test_session, test_client, test_directory):
assert response.status_code == HTTPStatus.NOT_FOUND
def test_search_channel_videos(test_client, test_session):
def test_search_videos_channel(test_client, test_session, video_factory):
with get_db_session(commit=True) as session:
channel1 = Channel(name='Foo')
channel2 = Channel(name='Bar')
@ -469,29 +404,29 @@ def test_search_channel_videos(test_client, test_session):
d = dict(channel_id=channel1.id)
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
assert response.status_code == HTTPStatus.OK
assert len(response.json['videos']) == 0
assert len(response.json['files']) == 0
with get_db_session(commit=True) as session:
vid1 = Video(title='vid1', channel_id=channel2.id, video_path='foo')
vid2 = Video(title='vid2', channel_id=channel1.id, video_path='foo')
vid1 = video_factory(title='vid1', channel_id=channel2.id)
vid2 = video_factory(title='vid2', channel_id=channel1.id)
session.add(vid1)
session.add(vid2)
# Videos are gotten by their respective channels
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
assert response.status_code == HTTPStatus.OK
assert len(response.json['videos']) == 1
assert response.json['totals']['videos'] == 1
assert_dict_contains(response.json['videos'][0], dict(id=2, title='vid2', channel_id=channel1.id))
assert len(response.json['files']) == 1
assert response.json['totals']['files'] == 1
assert_dict_contains(response.json['files'][0], dict(path='vid2.mp4', video=dict(channel_id=channel1.id)))
d = dict(channel_id=channel2.id)
request, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
assert response.status_code == HTTPStatus.OK
assert len(response.json['videos']) == 1
assert_dict_contains(response.json['videos'][0], dict(id=1, title='vid1', channel_id=channel2.id))
assert len(response.json['files']) == 1
assert_dict_contains(response.json['files'][0], dict(path='vid1.mp4', video=dict(channel_id=channel2.id)))
def test_get_channel_videos_pagination(test_client, test_session, simple_channel, video_factory):
def test_get_channel_videos_pagination(test_client, test_session, simple_channel, video_factory, assert_video_search):
for i in range(50):
video_factory(channel_id=simple_channel.id)
@ -500,7 +435,7 @@ def test_get_channel_videos_pagination(test_client, test_session, simple_channel
test_session.flush()
test_session.refresh(channel2)
video_factory(channel_id=channel2.id)
test_session.add(Video(title='vid2', channel_id=channel2.id, video_path='foo'))
video_factory(title='vid2', channel_id=channel2.id, with_video_file=True)
test_session.commit()
# Get first, second, third, and empty pages of videos.
@ -513,10 +448,8 @@ def test_get_channel_videos_pagination(test_client, test_session, simple_channel
]
last_ids = []
for offset, video_count in tests:
d = dict(channel_id=simple_channel.id, order_by='id', offset=offset)
_, response = test_client.post(f'/api/videos/search', content=json.dumps(d))
assert response.status_code == HTTPStatus.OK
assert len(response.json['videos']) == video_count
current_ids = [i['id'] for i in response.json['videos']]
_, response = assert_video_search(channel_id=simple_channel.id, order_by='id', offset=offset, limit=20)
assert len(response.json['files']) == video_count, 'Returned videos does not match'
current_ids = [i['video']['id'] for i in response.json['files']]
assert current_ids != last_ids, f'IDs are unchanged current_ids={current_ids}'
last_ids = current_ids

40
modules/videos/channel/test/test_lib.py
View file

@ -1,9 +1,11 @@
from unittest import mock
import pytest
from modules.videos import schema
from modules.videos.channel import lib
from modules.videos.lib import save_channels_config, import_channels_config
from modules.videos.models import Channel
from modules.videos.models import Channel, Video
from wrolpi.errors import UnknownChannel
@ -89,3 +91,39 @@ def test_channels_no_url(test_session, test_directory):
save_channels_config()
import_channels_config()
def test_refresh_videos_finds_channel(test_session, test_client, channel_factory, video_factory):
"""A Video will be assigned to a Channel by its info json first, then to its directory."""
channel1 = channel_factory(source_id='channel1')
channel2 = channel_factory(source_id='channel2')
# Put the video in the wrong directory. It should be matched to the info json channel first, then to it's directory
channel1.source_id = 'channel source id'
vid1 = video_factory(
with_video_file=(channel2.directory / 'video1.mp4'),
with_info_json={'channel_id': channel1.source_id},
)
test_session.commit()
assert not vid1.channel_id
with mock.patch('wrolpi.common.after_refresh') as mock_after_refresh:
# Channel can be found in `video_cleanup`, prevent that with mock.
mock_after_refresh.side_effect = []
test_client.post('/api/files/refresh')
# Channel in info_json was found during validation.
assert vid1.channel_id == channel1.id
# Refresh using cleanup, info json should be trusted.
test_client.post('/api/files/refresh')
assert vid1.channel_id == channel1.id
# Remove the info json, the Video's Channel will be its directory.
vid1.channel_id = None
vid1.info_json_path.unlink()
test_session.commit()
test_client.post('/api/files/refresh')
test_session.refresh(vid1)
vid1 = test_session.query(Video).one()
assert vid1.channel_id == channel2.id

117
modules/videos/common.py
View file

@ -1,12 +1,15 @@
import json
import json
import os
import pathlib
import re
import subprocess
import tempfile
from datetime import timedelta
from decimal import Decimal
from functools import partial
from pathlib import Path
from typing import Union, Tuple, List, Set, Iterable, Optional
from typing import Union, List, Set, Iterable, Optional, Tuple
import PIL
from PIL import Image
@ -14,13 +17,13 @@ from sqlalchemy.orm import Session
from wrolpi.cmd import which
from wrolpi.common import logger, iterify, get_media_directory, \
minimize_dict, any_extensions
minimize_dict, any_extensions, match_paths_to_suffixes
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.errors import UnknownFile, ChannelNameConflict, ChannelURLConflict, \
from wrolpi.errors import ChannelNameConflict, ChannelURLConflict, \
ChannelDirectoryConflict, ChannelSourceIdConflict
from wrolpi.media_path import MediaPath
from wrolpi.files.models import File
from wrolpi.vars import DEFAULT_FILE_PERMISSIONS
from .models import Channel, Video
from .models import Channel
logger = logger.getChild(__name__)
@ -32,6 +35,7 @@ MINIMUM_VIDEO_KEYS = {'id', 'title', 'upload_date', 'duration', 'channel', 'chan
'view_count'}
# These are the supported video formats. These are in order of their preference.
VIDEO_EXTENSIONS = ('mp4', 'ogg', 'webm', 'flv')
match_video_extensions = partial(any_extensions, extensions=VIDEO_EXTENSIONS)
FFMPEG_BIN = which('ffmpeg', '/usr/bin/ffmpeg', warn=True)
@ -49,40 +53,13 @@ def get_videos_directory() -> pathlib.Path:
def get_no_channel_directory() -> pathlib.Path:
"""Get the "NO CHANNEL" directory in the videos directory. Make it if it does not exist."""
"""Get the "NO CHANNEL" directory in the videos directory. Make it if it does not exist.""" # noqa
directory = get_videos_directory() / 'NO CHANNEL'
if not directory.is_dir():
directory.mkdir(parents=True)
return directory
VALID_VIDEO_KINDS = {'video', 'caption', 'poster', 'description', 'info_json'}
def get_absolute_video_path(video: Video, kind: str = 'video') -> MediaPath:
if kind not in VALID_VIDEO_KINDS:
raise Exception(f'Unknown video path kind {kind}')
path = getattr(video, f'{kind}_path')
if path:
return path
raise UnknownFile()
match_video_extensions = partial(any_extensions, extensions=VIDEO_EXTENSIONS)
def generate_video_paths(directory: Union[str, pathlib.Path]) -> Tuple[str, pathlib.Path]:
"""Generate a list of video paths in the provided directory."""
directory = pathlib.Path(directory)
for child in directory.iterdir():
if child.is_file() and match_video_extensions(child.name):
child = child.absolute()
yield child
elif child.is_dir():
yield from generate_video_paths(child)
@iterify(set)
def remove_duplicate_video_paths(paths: Iterable[Path]) -> Set[Path]:
"""
@ -190,13 +167,46 @@ def get_matching_directories(path: Union[str, Path]) -> List[str]:
return paths
def generate_video_poster(video_path: Path) -> Path:
"""Create a poster (aka thumbnail) next to the provided video_path."""
def duration_to_seconds(duration: str) -> int:
"""Convert ffmpeg duration to seconds"""
hours, minutes, seconds = duration.split(':')
seconds, microseconds = seconds.split('.')
return int(timedelta(
hours=int(hours),
minutes=int(minutes),
seconds=int(seconds),
microseconds=int(microseconds),
).total_seconds())
def seconds_to_duration(seconds: int) -> str:
"""Convert integer seconds to ffmpeg duration"""
hours, seconds = divmod(seconds, 3600)
minutes, seconds = divmod(seconds, 60)
return f'{hours:02}:{minutes:02}:{seconds:02}.000'
DURATION_REGEX = re.compile(r'Duration: ([\d:.]+),')
def get_duration_from_ffmpeg(stderr: str) -> int:
"""Extract the duration from FFPEG's stderr, return this duration as integer seconds."""
duration = DURATION_REGEX.findall(stderr)
duration = duration_to_seconds(duration[0]) if duration else None
return duration
def generate_video_poster(video_path: Path) -> Tuple[Path, Optional[int]]:
"""Create a poster (aka thumbnail) next to the provided video_path. Also returns the video duration."""
poster_path = video_path.with_suffix('.jpg')
cmd = (FFMPEG_BIN, '-n', '-i', str(video_path), '-f', 'mjpeg', '-vframes', '1', '-ss', '00:00:05.000',
str(poster_path))
if poster_path.exists():
# Poster already exists.
return poster_path, None
try:
subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stderr = proc.stderr.decode()
logger.info(f'Generated poster at {poster_path}')
except subprocess.CalledProcessError as e:
logger.warning(f'FFMPEG poster generation failed with stdout: {e.stdout.decode()}')
@ -205,7 +215,8 @@ def generate_video_poster(video_path: Path) -> Path:
if not poster_path.exists():
raise Exception(f'Failed to find generated poster! {poster_path}')
return poster_path
duration = get_duration_from_ffmpeg(stderr)
return poster_path, duration
def convert_image(existing_path: Path, destination_path: Path, ext: str = 'jpeg'):
@ -237,7 +248,7 @@ def is_valid_poster(poster_path: Path) -> bool:
FFPROBE_BIN = which('ffprobe', '/usr/bin/ffprobe', warn=True)
def get_video_duration(video_path: Path) -> Optional[int]:
def get_video_duration(video_path: Path) -> int:
"""Get the duration of a video in seconds. Do this using ffprobe."""
if not isinstance(video_path, Path):
video_path = Path(video_path)
@ -261,25 +272,29 @@ def get_video_duration(video_path: Path) -> Optional[int]:
def check_for_video_corruption(video_path: Path) -> bool:
"""Uses ffprobe to check for specific ways a video file can be corrupt."""
if not isinstance(video_path, Path):
video_path = Path(video_path)
"""Uses ffprobe to check for specific ways a video file can be corrupt.
Also attempts to screenshot the video near the end of it's duration, this check if the video is corrupted somewhere
in the middle."""
video_path = Path(video_path) if not isinstance(video_path, Path) else video_path
if not video_path.is_file():
raise FileNotFoundError(f'{video_path} does not exist!')
if not FFPROBE_BIN:
raise SystemError('ffprobe is not installed!')
cmd = [FFPROBE_BIN, str(video_path)]
# Just read the video using FFMPEG, this prints out useful information about the video.
cmd = (FFPROBE_BIN, str(video_path))
try:
proc = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
logger.warning(f'FFPROBE failed to check for corruption with stdout: {e.stdout.decode()}')
logger.warning(f'FFPROBE failed to check for corruption with stderr: {e.stderr.decode()}')
return True # video is corrupt.
messages = (
b'Invalid NAL unit size',
b'Error splitting the input into NAL units',
b'Invalid data found when processing input',
)
corrupt = False
for error in messages:
@ -347,3 +362,19 @@ def minimize_video(video: dict) -> dict:
video['info_json'] = minimize_video_info_json(video['info_json'])
return video
match_video_paths = partial(match_paths_to_suffixes, suffix_groups=(
tuple(f'.{i}' for i in VIDEO_EXTENSIONS),
('.jpg', '.jpeg', '.webp', '.png'),
('.description',),
('.en.vtt', '.en.srt'),
('.info.json',),
))
@iterify(tuple)
def match_video_files(files: List[File]) -> Tuple[File, File, File, File, File]:
video_paths = match_video_paths([i.path for i in files])
for path in video_paths:
yield next((i for i in files if i.path == path), None)

133
modules/videos/conftest.py
View file

@ -1,6 +1,7 @@
import json
import pathlib
import shutil
from http import HTTPStatus
from itertools import zip_longest
from typing import List
from uuid import uuid4
@ -13,6 +14,7 @@ from modules.videos.downloader import VideoDownloader, ChannelDownloader
from modules.videos.lib import set_test_channels_config, set_test_downloader_config
from modules.videos.models import Channel, Video
from wrolpi.downloader import DownloadFrequency, DownloadManager, Download
from wrolpi.files.models import File
from wrolpi.vars import PROJECT_DIR
@ -54,9 +56,7 @@ def channel_factory(test_session, test_directory):
@pytest.fixture
def download_channel(test_session, test_directory, video_download_manager) -> Channel:
"""
Get a test Channel that has a download frequency.
"""
"""Get a test Channel that has a download frequency."""
# Add a frequency to the test channel, then give it a download.
channel = Channel(
directory=test_directory,
@ -75,64 +75,75 @@ def simple_video(test_session, test_directory, simple_channel) -> Video:
"""A Video with an empty video file whose channel is the Simple Channel."""
video_path = test_directory / 'simple_video.mp4'
video_path.touch()
video = Video(video_path=video_path, channel_id=simple_channel.id)
video_file = File(path=video_path, model='video')
video = Video(video_file=video_file, channel_id=simple_channel.id)
test_session.add(video)
test_session.commit()
video = test_session.query(Video).filter_by(id=video.id).one()
return video
@pytest.fixture
def video_file(test_directory) -> pathlib.Path:
destination = test_directory / f'{uuid4()}.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
yield destination
destination.unlink()
@pytest.fixture
def video_factory(test_session, test_directory):
"""Creates Videos for testing."""
def factory(channel_id: int = None, title: str = None, upload_date=None, with_video_file: bool = False,
with_info_json: dict = None, with_poster_ext: str = None, with_caption_file: bool = False):
def factory(channel_id: int = None, title: str = None, upload_date=None, with_video_file=None,
with_info_json: dict = None, with_poster_ext: str = None, with_caption_file: bool = False,
source_id: str = None):
title = title or str(uuid4())
if channel_id:
if with_video_file and isinstance(with_video_file, (pathlib.Path, str)):
# Put the video exactly where specified for the test.
path = pathlib.Path(with_video_file) if not isinstance(with_video_file, pathlib.Path) else with_video_file
elif channel_id:
# Put the video in its Channel's directory.
channel = test_session.query(Channel).filter_by(id=channel_id).one()
path = (channel.directory or test_directory) / f'{title}.mp4'
else:
(test_directory / 'videos/NO CHANNEL').mkdir(parents=True, exist_ok=True)
# Put any video not in a Channel and without a specified file in the NO CHANNEL directory.
(test_directory / 'videos/NO CHANNEL').mkdir(exist_ok=True, parents=True)
path = test_directory / f'videos/NO CHANNEL/{title}.mp4'
video_file = File(path=path, model='video')
test_session.add(video_file)
if with_video_file:
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', path)
else:
path.touch()
info_json_path = None
if with_info_json is True:
# User requests a info json file, but does not provide one.
info_json_path = path.with_suffix('.info.json')
info_json_path.write_text(json.dumps({}))
elif with_info_json:
if with_info_json:
# Use the provided object as the json.
with_info_json = {'duration': 5} if with_info_json is True else with_info_json
info_json_path = path.with_suffix('.info.json')
info_json_path.write_text(json.dumps(with_info_json))
test_session.add(File(path=info_json_path))
poster_path = None
if with_poster_ext:
poster_path = path.with_suffix(f'.{with_poster_ext}')
Image.new('RGB', (25, 25)).save(poster_path)
test_session.add(File(path=poster_path))
caption_path = None
if with_caption_file:
caption_path = path.with_suffix('.en.vtt')
caption_path.touch()
shutil.copy(PROJECT_DIR / 'test/example1.en.vtt', caption_path)
test_session.add(File(path=caption_path))
video = Video(video_path=str(path), title=title, channel_id=channel_id, source_id=title,
info_json_path=info_json_path, poster_path=poster_path, caption_path=caption_path,
upload_date=upload_date)
video = Video(
video_path=path,
title=title,
channel_id=channel_id,
source_id=source_id or title,
info_json_path=info_json_path,
poster_path=poster_path,
caption_path=caption_path,
upload_date=upload_date,
)
test_session.add(video)
video_file.do_index()
return video
return factory
@ -199,3 +210,71 @@ def assert_video_ids(test_session):
assert id_ == expected, f'Video ids were not as expected: {expected_video_ids=} != {video_ids=}'
return checker
@pytest.fixture
def video_with_search_factory(test_session, test_directory):
"""A factory that creates a Video record with an associated video File record."""
def video_with_search(path: str = None, title: str = None, b_text: str = None, c_text: str = None,
d_text: str = None, mimetype: str = 'video/mp4'):
path = path or (test_directory / str(uuid4())).absolute()
video_file = File(path=path, a_text=title, d_text=d_text, mimetype=mimetype, model='video')
video = Video(title=title, video_file=video_file)
test_session.add(video)
return video
return video_with_search
@pytest.fixture
def assert_video_search(test_client):
"""A fixture which performs a video search request against the API.
If assert_* params are passed, the response is checked."""
def search_videos(
assert_total: int = None,
assert_ids: List[int] = None,
assert_paths: List[str] = None,
search_str: str = None,
filters: List[str] = None,
offset: int = None,
limit: int = None,
order_by: str = None,
channel_id: int = None,
):
content = dict()
if search_str is not None:
content['search_str'] = search_str
if filters is not None:
content['filters'] = filters
if offset is not None:
content['offset'] = offset
if limit is not None:
content['limit'] = limit
if order_by is not None:
content['order_by'] = order_by
if channel_id is not None:
content['channel_id'] = channel_id
request, response = test_client.post('/api/videos/search', content=json.dumps(content))
assert response.json
assert response.status_code == HTTPStatus.OK
if assert_ids is not None:
assert 'files' in response.json, 'No video files in response'
assert [i['video']['id'] for i in response.json['files']] == assert_ids, 'Video IDs do not match'
if assert_paths is not None:
assert 'files' in response.json, 'No video files in response'
assert [i['video']['video_path'] for i in response.json['files']] == assert_paths, \
'Video paths do not match'
if assert_total:
assert response.json['totals']['files'] == int(assert_total), 'Total video files does not match'
return request, response
return search_videos

83
modules/videos/downloader.py
View file

@ -13,19 +13,20 @@ from yt_dlp.extractor import YoutubeTabIE # noqa
from yt_dlp.utils import UnsupportedError, DownloadError
from wrolpi.cmd import which
from wrolpi.common import logger, extract_domain, get_media_directory
from wrolpi.common import logger, get_media_directory, escape_file_name
from wrolpi.dates import now
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.db import optional_session
from wrolpi.downloader import Downloader, Download, DownloadResult
from wrolpi.errors import UnknownChannel, ChannelURLEmpty, UnrecoverableDownloadError
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from .channel.lib import create_channel, get_channel
from .common import apply_info_json, get_no_channel_directory, get_videos_directory
from .lib import upsert_video, refresh_channel_videos, get_downloader_config
from .lib import get_downloader_config
from .models import Video, Channel
from .schema import ChannelPostRequest
from .video_url_resolver import video_url_resolver
from .video_url_resolver import normalize_youtube_shorts_url
logger = logger.getChild(__name__)
ydl_logger = logger.getChild('youtube-dl')
@ -84,16 +85,16 @@ class ChannelDownloader(Downloader, ABC):
async def do_download(self, download: Download) -> DownloadResult:
"""Update a Channel's catalog, then schedule downloads of every missing video."""
info = extract_info(download.url, process=False)
# Resolve the entries generator.
# Resolve the "entries" generator.
info['entries'] = list(info['entries'])
download.info_json = info
if session := Session.object_session(download):
# May not have a session during testing.
session.commit()
name = info['uploader']
kind = info['uploader']
channel_source_id = info['channel_id']
channel = get_or_create_channel(channel_source_id, download.url, name)
channel = get_or_create_channel(channel_source_id, download.url, kind)
channel.dict() # get all attributes while we have the session.
location = f'/videos/channel/{channel.id}/video' if channel and channel.id else None
@ -103,33 +104,49 @@ class ChannelDownloader(Downloader, ABC):
if is_a_playlist:
downloads = self.get_playlist_downloads(download)
else:
downloads = self.get_channel_downloads(download, info, channel)
downloads = await self.get_channel_downloads(download, info, channel)
return DownloadResult(success=True, location=location, downloads=downloads)
except Exception:
name = 'playlist' if is_a_playlist else 'channel'
logger.warning(f'Failed to update catalog of {name} {download.url}')
if PYTEST:
raise
kind = 'playlist' if is_a_playlist else 'channel'
logger.warning(f'Failed to update catalog of {kind} {download.url}')
return DownloadResult(success=False, location=location, error=str(traceback.format_exc()))
@staticmethod
def get_channel_downloads(download: Download, info: dict, channel: Channel) -> List[str]:
async def get_channel_downloads(download: Download, info: dict, channel: Channel) -> List[str]:
"""Get a list of all videos in a Channel, schedule downloads for any missing videos."""
channel_id = channel.id
update_channel_catalog(channel, info)
domain = extract_domain(download.url)
missing_videos = find_all_missing_videos(channel.id)
downloads = []
for _, _, missing_video in missing_videos:
url = missing_video.get('webpage_url') or video_url_resolver(domain, missing_video)
downloads.append(url)
with get_db_session(commit=True) as session:
channel = get_channel(session, channel_id=channel_id, return_dict=False)
if not channel.videos or not channel.refreshed:
logger.warning(f'Refreshing videos in {channel.directory} because {channel} has no video records!')
await channel.refresh_files()
session.commit() # Commit the refresh.
session.refresh(channel)
# Get all videos in the channel.
downloads = [i.get('webpage_url') or i.get('url') for i in download.info_json['entries']]
# YouTube Shorts are handled specially.
downloads = list(map(normalize_youtube_shorts_url, downloads))
# Only download those that have not yet been downloaded.
already_downloaded = [i.url for i in video_downloader.already_downloaded(*downloads)]
downloads = [i for i in downloads if i not in already_downloaded]
return downloads
@staticmethod
def get_playlist_downloads(download: Download) -> List[str]:
"""Get a list of all videos in a playlist, schedule downloads for any missing videos."""
# Get all videos in the playlist
downloads = [i['url'] for i in download.info_json['entries']]
downloads = [i.get('webpage_url') or i.get('url') for i in download.info_json['entries']]
# YouTube Shorts are handled specially.
downloads = [normalize_youtube_shorts_url(i) for i in downloads]
# Only download those that have not yet been downloaded.
downloads = [i for i in downloads if not video_downloader.already_downloaded(i)]
already_downloaded = [i.url for i in video_downloader.already_downloaded(*downloads)]
downloads = [i for i in downloads if i not in already_downloaded]
return downloads
@ -153,9 +170,13 @@ class VideoDownloader(Downloader, ABC):
return f'<VideoDownloader>'
@optional_session
def already_downloaded(self, url: str, session: Session = None) -> bool:
def already_downloaded(self, *urls: str, session: Session = None) -> List:
# We only consider a video record with a video file as "downloaded".
return bool(session.query(Video).filter(Video.url == url, Video.video_path != None, ).count()) # noqa
videos = list(session.query(Video).filter(
Video.url.in_(urls),
Video.video_path != None, # noqa
))
return videos
@classmethod
def valid_url(cls, url) -> Tuple[bool, Optional[dict]]:
@ -200,7 +221,7 @@ class VideoDownloader(Downloader, ABC):
# Use the default directory if this video has no channel.
out_dir = get_no_channel_directory()
if channel:
out_dir = channel.directory.path
out_dir = channel.directory
out_dir.mkdir(exist_ok=True, parents=True)
logs = None # noqa
@ -244,10 +265,14 @@ class VideoDownloader(Downloader, ABC):
)
with get_db_session(commit=True) as session:
# If the video is from a channel, it will already be in the database.
source_id = entry['id']
existing_video = session.query(Video).filter_by(source_id=source_id).one_or_none()
video = upsert_video(session, video_path, channel, id_=existing_video.id if existing_video else None)
video_file = File.upsert(video_path, session)
video: Video = Video.upsert(video_file, session)
video.source_id = entry['id']
if channel:
video.channel_id = channel.id
video.video_file.do_index()
video.validate(session)
session.flush([video])
video_id = video.id
except UnrecoverableDownloadError:
raise
@ -335,7 +360,7 @@ def get_or_create_channel(source_id: str = None, url: str = None, name: str = No
raise ValueError(f'Cannot create channel without a name')
# Channel does not exist. Create one in the video directory.
channel_directory = get_videos_directory() / name
channel_directory = get_videos_directory() / escape_file_name(name)
if not channel_directory.is_dir():
channel_directory.mkdir(parents=True)
data = ChannelPostRequest(
@ -409,7 +434,7 @@ def update_channel_catalog(channel: Channel, info: dict):
# Write the Channel's info to a JSON file.
if channel.directory:
info_json_path = channel.directory.path / f'{channel.name}.info.json'
info_json_path = channel.directory / f'{channel.name}.info.json'
with info_json_path.open('wt') as fh:
json.dump(info, fh, indent=2)
@ -440,7 +465,7 @@ def _find_all_missing_videos(channel_id: id) -> List[Tuple]:
return missing_videos
def find_all_missing_videos(channel_id: int = None) -> Tuple[dict, dict]:
async def find_all_missing_videos(channel_id: int = None):
"""
Find all videos that don't have a video file, but are found in the DB (taken from the channel's info_json).
@ -453,7 +478,7 @@ def find_all_missing_videos(channel_id: int = None) -> Tuple[dict, dict]:
# Check that the channel has some videos. We can't be sure what is missing if we don't know what we have.
if channel.refreshed is False:
# Refresh this channel's videos.
refresh_channel_videos(channel)
await channel.refresh_files()
match_regex = re.compile(channel.match_regex) if channel.match_regex else None

351
modules/videos/lib.py
View file

@ -1,24 +1,23 @@
import functools
import html
import pathlib
import re
from collections import defaultdict
from typing import Tuple, Optional, List, Union
from uuid import uuid1
from typing import Tuple, Optional, Union
from sqlalchemy import or_
from sqlalchemy.orm import Session
from yt_dlp import YoutubeDL
from wrolpi import before_startup
from wrolpi.common import chunks, ConfigFile, get_media_directory, sanitize_link
from wrolpi.common import ConfigFile, get_media_directory, sanitize_link
from wrolpi.dates import from_timestamp, Seconds
from wrolpi.db import get_db_curs, get_db_session, optional_session
from wrolpi.media_path import MediaPath
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from .captions import get_captions
from .common import generate_video_paths, remove_duplicate_video_paths, apply_info_json, get_video_duration, \
is_valid_poster, convert_image, generate_video_poster, logger, REQUIRED_OPTIONS, ConfigError, \
get_no_channel_directory, check_for_video_corruption
from .common import is_valid_poster, convert_image, \
generate_video_poster, logger, REQUIRED_OPTIONS, ConfigError, \
get_no_channel_directory, get_video_duration
from .models import Channel, Video
logger = logger.getChild(__name__)
@ -26,156 +25,66 @@ logger = logger.getChild(__name__)
DEFAULT_DOWNLOAD_FREQUENCY = Seconds.week
def refresh_channel_videos(channel: Channel):
"""
Find all video files in a channel's directory. Add any videos not in the DB to the DB.
"""
# Set the idempotency key, we can remove any videos not touched during this search.
with get_db_curs(commit=True) as curs:
curs.execute('UPDATE video SET idempotency=NULL WHERE channel_id=%s', (channel.id,))
idempotency = str(uuid1())
directory = channel.directory.path
# A set of absolute paths that exist in the file system
possible_new_paths = generate_video_paths(directory)
possible_new_paths = remove_duplicate_video_paths(possible_new_paths)
# Update all videos that match the current video paths
new_paths = [str(i) for i in possible_new_paths]
with get_db_curs(commit=True) as curs:
query = 'UPDATE video SET idempotency = %s WHERE channel_id = %s AND video_path = ANY(%s) RETURNING video_path'
curs.execute(query, (idempotency, channel.id, new_paths))
existing_paths = {i for (i,) in curs.fetchall()}
# Get the paths for any video not yet in the DB
# (paths in DB are relative, but we need to pass an absolute path)
new_videos = {i for i in possible_new_paths if str(i) not in existing_paths}
for chunk in chunks(new_videos, 20):
with get_db_session(commit=True) as session:
for video_path in chunk:
video_path = pathlib.Path(video_path)
upsert_video(session, video_path, channel, idempotency=idempotency)
logger.debug(f'{channel.name}: Added {video_path.name}')
with get_db_curs(commit=True) as curs:
stmt = 'DELETE FROM video WHERE channel_id=%s AND idempotency IS NULL AND video_path IS NOT NULL RETURNING id'
curs.execute(stmt, (channel.id,))
deleted_count = len(curs.fetchall())
if deleted_count:
deleted_status = f'Deleted {deleted_count} video records from channel {channel.name}'
logger.info(deleted_status)
logger.info(f'{channel.name}: {len(new_videos)} new videos, {len(existing_paths)} already existed. ')
with get_db_session(commit=True) as session:
channel = session.query(Channel).filter_by(id=channel.id).one()
channel.refreshed = True
apply_info_json(channel.id)
class VideoInfoJSON(object):
channel_source_id = None
title = None
duration = None
view_count = None
url = None
def refresh_no_channel_videos():
"""
Refresh the Videos in the NO CHANNEL directory.
"""
from modules.videos.downloader import get_no_channel_directory
directory = get_no_channel_directory()
if not directory.is_dir():
return
logger.info('Refreshing NO CHANNEL videos')
idempotency = str(uuid1())
# Get all WROLPi compatible videos, remove any duplicates (different formats).
possible_new_paths = generate_video_paths(directory)
possible_new_paths = remove_duplicate_video_paths(possible_new_paths)
new_paths = [str(i) for i in possible_new_paths]
with get_db_curs(commit=True) as curs:
query = 'UPDATE video SET idempotency = %s WHERE video_path = ANY(%s) RETURNING video_path'
curs.execute(query, (idempotency, new_paths))
existing_paths = {i for (i,) in curs.fetchall()}
new_videos = {i for i in possible_new_paths if str(i) not in existing_paths}
for chunk in chunks(new_videos, 20):
with get_db_session(commit=True) as session:
for video_path in chunk:
upsert_video(session, pathlib.Path(video_path), idempotency=idempotency)
logger.debug(f'Added NO CHANNEL video {video_path}')
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM video WHERE channel_id IS NULL AND idempotency IS NULL RETURNING id')
deleted_count = len(curs.fetchall())
if deleted_count:
deleted_status = f'Deleted {deleted_count} video records in NO CHANNEL.'
logger.info(deleted_status)
def process_video_info_json(video: Video):
def process_video_info_json(video: Video) -> VideoInfoJSON:
"""
Parse the Video's info json file, return the relevant data.
"""
title = duration = view_count = url = None
video_info_json = VideoInfoJSON()
if info_json := video.get_info_json():
title = info_json.get('fulltitle') or info_json.get('title')
title = html.unescape(title) if title else None
video_info_json.title = html.unescape(title) if title else None
duration = info_json.get('duration')
view_count = info_json.get('view_count')
url = info_json.get('webpage_url') or info_json.get('url')
video_info_json.duration = info_json.get('duration')
video_info_json.view_count = info_json.get('view_count')
video_info_json.url = info_json.get('webpage_url') or info_json.get('url')
video_info_json.channel_source_id = info_json.get('channel_id')
return title, duration, view_count, url
return video_info_json
def validate_videos():
@optional_session
@functools.lru_cache
def get_channel_id_by_source_id(session: Session, source_id: str) -> Optional[int]:
"""Return the id of the Channel which matches the provided `source_id`.
These are cached results."""
channel = session.query(Channel).filter_by(source_id=source_id).one_or_none()
return channel.id if channel else None
def validate_video(video: Video, channel_generate_poster: bool, session: Session):
"""
Validate all Videos not yet validated. A Video is validated when we have attempted to find its: title, duration,
view_count, url, caption, size. A Video is also valid when it has a JPEG poster, if any. If no poster can be
found, it will be generated from the video file.
Validate a single video.
This function marks the Video as validated, even if no data can be found so a Video will not be validated multiple
times.
"""
with get_db_curs() as curs:
curs.execute('SELECT id FROM video WHERE video_path IS NOT NULL AND validated IS FALSE')
video_ids = [i['id'] for i in curs.fetchall()]
curs.execute('SELECT id, generate_posters FROM channel')
channel_generate_posters = {i['id']: i['generate_posters'] for i in curs.fetchall()}
A Video is validated when we have attempted to find its: title, duration, view_count, url, caption, size, and
channel.
logger.info(f'Validating {len(video_ids)} videos.')
for chunk in chunks(video_ids, 20):
with get_db_session(commit=True) as session:
videos = session.query(Video).filter(Video.id.in_(chunk)).all()
for video in videos:
video.validate()
def validate_video(video: Video, channel_generate_poster: bool):
"""
Validate a single video. A Video is validated when we have attempted to find its: title, duration,
view_count, url, caption, size. A Video is also valid when it has a JPEG poster, if any. If no poster can be
A Video is also valid when it has a JPEG poster, if any. If no poster can be
found, it will be generated from the video file.
"""
if not video.title or not video.duration or not video.view_count or not video.url:
info_json_path = video.info_json_file.path if video.info_json_file else video.info_json_path
if info_json_path and (not video.title or not video.duration or not video.view_count or not video.url):
# These properties can be found in the info json.
title, duration, view_count, url = process_video_info_json(video)
video.title = title
video.duration = duration
video.url = url
video_info_json = process_video_info_json(video)
video.title = video_info_json.title
video.duration = video_info_json.duration
video.url = video_info_json.url
# View count will probably be overwritten by more recent data when this Video's Channel is
# updated.
video.view_count = video.view_count or view_count
video.view_count = video.view_count or video_info_json.view_count
video_path = video.video_path.path if isinstance(video.video_path, MediaPath) else video.video_path
if video_info_json.channel_source_id:
video.channel_id = get_channel_id_by_source_id(source_id=video_info_json.channel_source_id)
# Check for corruption, but don't throw away the video even if it is corrupt. Errors will be logged in this call.
check_for_video_corruption(video_path)
video_path = video.video_path
if not video.title or not video.upload_date or not video.source_id:
# Video is missing things that can be extracted from the video file name.
@ -184,41 +93,41 @@ def validate_video(video: Video, channel_generate_poster: bool):
video.title = video.title or html.unescape(title)
video.upload_date = video.upload_date or upload_date
video.source_id = video.source_id or source_id
if not video.duration:
# Video duration was not in the info json, use ffprobe.
video.duration = get_video_duration(video_path)
if not video.caption and video.caption_path:
video.caption = get_captions(video.caption_path.path)
if not video.size or not video.modification_datetime:
stat = video_path.stat()
video.size = video.size or stat.st_size
video.modification_datetime = video.modification_datetime or from_timestamp(stat.st_mtime)
if not video.poster_path:
# Video poster is not found, lets check near the video file.
for ext in ('.jpg', '.jpeg', '.webp', '.png'):
if (poster_path := video_path.with_suffix(ext)).is_file():
video.poster_path = poster_path
break
if channel_generate_poster:
# Try to convert/generate, but keep the old poster if those fail.
video.poster_path = convert_or_generate_poster(video) or video.poster_path
new_poster_path, duration = convert_or_generate_poster(video) or video.poster_path
if new_poster_path:
# Poster was created/updated.
poster_file = File.upsert(new_poster_path, session)
session.flush([poster_file])
video.poster_file = poster_file
video.poster_path = new_poster_path
if duration:
video.duration = duration
if not video.duration:
# Duration was not retrieved during poster generation.
video.duration = get_video_duration(video_path)
def convert_or_generate_poster(video: Video) -> Optional[pathlib.Path]:
def convert_or_generate_poster(video: Video) -> Tuple[Optional[pathlib.Path], Optional[int]]:
"""
If a Video has a poster, but the poster is invalid, convert it. If a Video has no poster, generate one from the
video file.
Returns None if the poster was not converted, and not generated.
"""
video_path = video.video_path.path
video_path = video.video_path
# Modification/generation of poster is enabled for this channel.
if video.poster_path:
if video.poster_path or video.poster_file:
# Check that the poster is a more universally supported JPEG.
old: pathlib.Path = video.poster_path.path if \
isinstance(video.poster_path, MediaPath) else video.poster_path
old: pathlib.Path = video.poster_path if video.poster_path else video.poster_file.path
new = old.with_suffix('.jpg')
if old != new and new.exists():
@ -231,63 +140,22 @@ def convert_or_generate_poster(video: Video) -> Optional[pathlib.Path]:
try:
convert_image(old, new)
old.unlink(missing_ok=True)
logger.info(f'Converted invalid poster {old} to {new}')
return new
logger.info(f'Converted invalid poster {repr(str(old))} to {repr(str(new))}')
return new, None
except Exception as e:
logger.error(f'Failed to convert invalid poster {old} to {new}', exc_info=e)
return
return None, None
if not video.poster_path:
if not video.poster_path or not video.poster_file:
# Video poster was not discovered, or converted. Let's generate it.
try:
poster_path = generate_video_poster(video_path)
poster_path, duration = generate_video_poster(video_path)
logger.debug(f'Generated poster for {video}')
return poster_path
return poster_path, duration
except Exception as e:
logger.error(f'Failed to generate poster for {video}', exc_info=e)
def refresh_videos(channel_ids: List[int] = None):
"""
Find any videos in the channel directories and add them to the DB. Delete DB records of any videos not in the
file system.
Yields status updates to be passed to the UI.
:return:
"""
logger.info('Refreshing video files')
with get_db_session() as session:
if channel_ids:
channels = session.query(Channel).filter(Channel.id.in_(channel_ids))
else:
channels = session.query(Channel).all()
channels = list(channels)
if not channels and channel_ids:
raise Exception(f'No channels match id(s): {channel_ids}')
elif not channels:
raise Exception(f'No channels in DB. Have you created any?')
for channel in channels:
try:
refresh_channel_videos(channel)
except Exception as e:
logger.fatal(f'Failed to refresh videos for channel {channel.name}!', exc_info=e)
pass
if not channel_ids:
# Refresh NO CHANNEL videos when not refreshing a specific channel.
refresh_no_channel_videos()
# Fill in any missing data for all videos.
if not PYTEST:
import_channels_config()
validate_videos()
logger.info('Refresh of video files complete')
return None, None
class ChannelsConfig(ConfigFile):
@ -470,9 +338,10 @@ def get_channels_config_from_db(session: Session) -> dict:
favorites = defaultdict(lambda: {})
for video in favorite_videos:
if video.channel:
favorites[str(video.channel.directory.relative)][video.video_path.path.name] = dict(favorite=video.favorite)
directory = video.channel.directory.relative_to(get_media_directory())
favorites[str(directory)][video.video_path.name] = dict(favorite=video.favorite)
else:
favorites['NO CHANNEL'][video.video_path.path.name] = dict(favorite=video.favorite)
favorites['NO CHANNEL'][video.video_path.name] = dict(favorite=video.favorite)
favorites = dict(favorites)
return dict(channels=channels, favorites=favorites)
@ -510,7 +379,6 @@ def import_channels_config():
"""Import channel settings to the DB. Existing channels will be updated."""
logger.info('Importing videos config')
try:
from .downloader import ChannelDownloader
config = get_channels_config()
channels, favorites = config.channels, config.favorites
@ -569,7 +437,7 @@ def import_channels_config():
if not channel:
logger.warning(f'Cannot find channel {directory=} for favorites!')
continue
channel_dir = channel.directory.path
channel_dir = channel.directory
else:
channel_dir = get_no_channel_directory()
@ -698,74 +566,3 @@ def parse_video_file_name(video_path: pathlib.Path) -> \
# Return the stem as a last resort
title = pathlib.Path(video_path).stem.strip()
return None, None, None, title
def upsert_video(session: Session, video_path: pathlib.Path, channel: Channel = None, idempotency: str = None,
id_: str = None) -> Video:
"""
Insert a video into the DB. Also, find any meta-files near the video file and store them on the video row.
If id_ is provided, update that entry.
"""
if not video_path.is_absolute():
raise ValueError(f'Video path is not absolute: {video_path}')
# This function can update or insert a Video.
video = None
if id_:
video = session.query(Video).filter_by(id=id_).one()
if not video:
# The video may already be in the database, match the source_id to any existing video.
_, _, source_id, _ = parse_video_file_name(video_path)
if source_id:
video = session.query(Video).filter_by(source_id=source_id).one_or_none()
if not video:
video = Video()
session.add(video)
# Set the file values, all other things can be found using these files.
poster_path, description_path, caption_path, info_json_path = find_meta_files(video_path)
video.caption_path = caption_path
video.description_path = description_path
video.info_json_path = info_json_path
video.poster_path = poster_path
video.video_path = video_path
if channel and not str(video_path).startswith(str(channel.directory.path)):
raise ValueError(f'Video path is not within its channel {video_path=} not in {channel.directory=}')
if channel:
video.channel_id = channel.id
video.idempotency = idempotency
session.flush()
session.refresh(video)
video.validate()
session.flush()
return video
def find_meta_files(path: pathlib.Path) -> Tuple[pathlib.Path, pathlib.Path, pathlib.Path, pathlib.Path]:
"""
Find all files that share a file's full path, except their extensions. It is assumed that file with the
same name, but different extension is related to that file. A None will be yielded if the meta file doesn't exist.
Example:
>>> foo = pathlib.Path('foo.bar')
>>> find_meta_files(foo)
(pathlib.Path('foo.jpg'), pathlib.Path('foo.description'),
pathlib.Path('foo.en.vtt'), pathlib.Path('foo.info.json'))
"""
suffix = path.suffix
name, suffix, _ = str(path.name).rpartition(suffix)
meta_file_exts = (('.jpg', '.webp', '.png'), ('.description',), ('.en.vtt', '.en.srt'), ('.info.json',))
for meta_exts in meta_file_exts:
for meta_ext in meta_exts:
meta_path = path.with_suffix(meta_ext)
if meta_path.exists():
yield meta_path
break
else:
yield None

192
modules/videos/models.py
View file

@ -1,18 +1,21 @@
import json
import pathlib
from datetime import datetime
from pathlib import Path
from typing import Optional, Generator
from typing import Optional, Generator, Dict, List
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, ARRAY, ForeignKey, Computed
from sqlalchemy.orm import relationship, Session, deferred
from sqlalchemy import Column, Integer, String, Boolean, JSON, Date, ARRAY, ForeignKey
from sqlalchemy.orm import relationship, Session
from sqlalchemy.orm.collections import InstrumentedList
from wrolpi.common import Base, tsvector, ModelHelper, logger, get_media_directory
from wrolpi.common import Base, ModelHelper, logger, get_media_directory
from wrolpi.dates import now, TZDateTime
from wrolpi.db import get_db_curs
from wrolpi.downloader import Download, download_manager
from wrolpi.errors import UnknownVideo, UnknownFile, UnknownDirectory
from wrolpi.media_path import MediaPathType, MediaPath
from wrolpi.errors import UnknownVideo
from wrolpi.files.lib import refresh_directory_files_recursively, glob_shared_stem
from wrolpi.files.models import File
from wrolpi.media_path import MediaPathType
logger = logger.getChild(__name__)
@ -22,18 +25,8 @@ class Video(ModelHelper, Base):
id = Column(Integer, primary_key=True)
channel_id = Column(Integer, ForeignKey('channel.id'))
channel = relationship('Channel', primaryjoin='Video.channel_id==Channel.id', back_populates='videos')
idempotency = Column(String)
validated = Column(Boolean, default=False)
# File paths
caption_path = Column(MediaPathType)
description_path = Column(MediaPathType)
ext = Column(String)
info_json_path = Column(MediaPathType)
poster_path = Column(MediaPathType)
video_path = Column(MediaPathType)
caption = deferred(Column(String)) # slow to fetch
censored = Column(Boolean, default=False)
duration = Column(Integer)
favorite = Column(TZDateTime)
@ -46,18 +39,42 @@ class Video(ModelHelper, Base):
view_count = Column(Integer)
viewed = Column(TZDateTime)
textsearch = deferred(
Column(tsvector, Computed('''to_tsvector('english'::regconfig,
((COALESCE(title, ''::text) || ' '::text) ||
COALESCE(caption, ''::text)))''')))
# Associated Files
caption_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
caption_file: File = relationship('File', primaryjoin='Video.caption_path==File.path')
info_json_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
info_json_file: File = relationship('File', primaryjoin='Video.info_json_path==File.path')
poster_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
poster_file: File = relationship('File', primaryjoin='Video.poster_path==File.path')
video_path: pathlib.Path = Column(MediaPathType, ForeignKey('file.path', ondelete='CASCADE'))
video_file: File = relationship('File', primaryjoin='Video.video_path==File.path')
def __repr__(self):
video_path = self.video_path
if self.video_path and isinstance(self.video_path, MediaPath):
video_path = self.video_path.path.relative_to(get_media_directory())
return f'<Video id={self.id} title={repr(self.title)} path={video_path} channel={self.channel_id} ' \
v = self.video_file
if self.video_path:
video_path = self.video_path.path if hasattr(self.video_path, 'path') else self.video_path
v = video_path.relative_to(get_media_directory())
return f'<Video id={self.id} title={repr(self.title)} path={v} channel={self.channel_id} ' \
f'source_id={repr(self.source_id)}>'
@classmethod
def upsert(cls, video_file: File, session: Session):
video = session.query(Video).filter_by(video_file=video_file).one_or_none()
if not video:
video = Video(video_file=video_file)
video: Video
video.video_path = video.video_path or video_file.path
video.video_file = video.video_file or video_file
video.size = video.video_path.stat().st_size
session.add(video)
video.find_my_files()
# Claim the video file for this model.
video.video_file.model = cls.__tablename__
return video
def dict(self) -> dict:
d = super().dict()
if self.channel_id:
@ -72,35 +89,34 @@ class Video(ModelHelper, Base):
from .common import minimize_video
return minimize_video(self.dict())
def _clear_paths(self):
self.caption_path = None
self.description_path = None
self.ext = None
self.info_json_path = None
self.poster_path = None
self.video_path = None
def _clear_files(self):
# Use setattr to avoid confusing PyCharm
setattr(self, 'caption_file', None)
setattr(self, 'description_file', None)
setattr(self, 'ext', None)
setattr(self, 'info_json_file', None)
setattr(self, 'poster_file', None)
setattr(self, 'video_file', None)
def my_paths(self) -> Generator[Path, None, None]:
def my_files(self) -> Generator[Path, None, None]:
"""Return all paths of this Video. Returns nothing if all paths are None.""" # noqa
if self.description_path: # noqa
yield self.description_path.path
if self.poster_path:
yield self.poster_path.path
if self.caption_path:
yield self.caption_path.path
if self.info_json_path:
yield self.info_json_path.path
if self.video_path:
yield self.video_path.path
if self.poster_file:
yield self.poster_file.path
if self.caption_file:
yield self.caption_file.path
if self.info_json_file:
yield self.info_json_file.path
if self.video_file:
yield self.video_file.path
def delete(self):
"""
Remove all files related to this video. Add it to it's Channel's skip list.
"""
for path in self.my_paths():
for path in self.my_files():
path.unlink(missing_ok=True)
self._clear_paths()
self._clear_files()
needs_save = bool(self.favorite)
@ -126,21 +142,15 @@ class Video(ModelHelper, Base):
def set_viewed(self):
self.viewed = now()
def get_info_json(self) -> Optional[JSON]:
def get_info_json(self) -> Optional[Dict]:
"""If this Video has an info_json file, return it's contents. Otherwise, return None."""
if not self.info_json_path:
info_json_path = self.info_json_file.path if self.info_json_file else self.info_json_path
if not info_json_path:
return
try:
info_json_path = self.info_json_path.path if \
isinstance(self.info_json_path, MediaPath) else self.info_json_path
with open(info_json_path, 'rb') as fh:
contents = json.load(fh)
return contents
except UnknownFile:
pass
except UnknownDirectory:
pass
with info_json_path.open('rb') as fh:
return json.load(fh)
except Exception as e:
logger.warning(f'Unable to parse info json {self.info_json_path}', exc_info=e)
return None
@ -230,7 +240,7 @@ class Video(ModelHelper, Base):
def __json__(self):
from modules.videos.common import minimize_video_info_json
info_json = minimize_video_info_json(self.get_info_json()) if self.info_json_path else None
stem = self.video_path.path.stem if self.video_path else None
stem = self.video_path.stem if self.video_path else None
d = dict(
caption_path=self.caption_path,
@ -249,44 +259,78 @@ class Video(ModelHelper, Base):
upload_date=self.upload_date,
url=self.url,
validated=self.validated,
video_path=self.video_path,
view_count=self.view_count,
video_path=self.video_path.relative_to(get_media_directory()),
viewed=self.viewed,
)
return d
def validate(self):
def validate(self, session: Session):
"""Perform a validation of this video and it's files. Mark this video as validated if no errors occur."""
if not self.video_path:
if not self.video_file:
# Can't validate if there is no video file.
return False
from .lib import validate_video
try:
validate_video(self, self.channel.generate_posters if self.channel else False)
validate_video(self, self.channel.generate_posters if self.channel else False, session)
self.validated = True
except Exception as e:
logger.warning(f'Failed to validate video {self}', exc_info=e)
return self.validated
def check_for_corruption(self):
"""Uses ffprobe to check if this video is corrupt."""
def find_my_files(self):
"""Search near the video file for it's associated files. (poster, caption, etc.)"""
from modules.videos.common import match_video_files
if not self.video_path:
return
raise ValueError("Can't find files when I don't have a video path!")
from .lib import check_for_video_corruption
return check_for_video_corruption(self.video_path.path)
# Associated files share the same stem, but different extensions.
associated_paths = glob_shared_stem(self.video_path)
associated_paths.pop(associated_paths.index(self.video_path))
# Get the session from the video first, fallback to the video file.
session: Session = Session.object_session(self) or Session.object_session(self.video_file)
files = [File.upsert(i, session) for i in associated_paths]
_, poster_file, description_file, caption_file, info_json_file = match_video_files(files)
if poster_file:
poster_file.do_index()
if caption_file:
caption_file.do_index()
if info_json_file:
info_json_file.do_index()
session.flush([i for i in files])
self.poster_file = poster_file
self.caption_file = caption_file
self.info_json_file = info_json_file
@staticmethod
def find_by_path(path, session):
video = session.query(Video).filter_by(video_path=path).one_or_none()
return video
@staticmethod
def find_by_paths(paths: List[pathlib.Path], session) -> List:
videos = list(session.query(Video).filter(Video.video_path.in_(paths)))
return videos
@property
def primary_path(self):
return self.video_path
class Channel(ModelHelper, Base):
__tablename__ = 'channel'
id = Column(Integer, primary_key=True)
name = Column(String)
idempotency = Column(String)
url = Column(String, unique=True)
match_regex = Column(String)
directory: MediaPath = Column(MediaPathType)
directory: pathlib.Path = Column(MediaPathType)
skip_download_videos = Column(ARRAY(String))
generate_posters = Column(Boolean, default=False) # generating posters may delete files, and can be slow.
calculate_duration = Column(Boolean, default=True)
@ -337,6 +381,12 @@ class Channel(ModelHelper, Base):
for key, value in data.items():
setattr(self, key, value)
# We need an absolute directory.
if isinstance(self.directory, pathlib.Path) and not self.directory.is_absolute():
self.directory = get_media_directory() / self.directory
elif isinstance(self.directory, str) and not pathlib.Path(self.directory).is_absolute():
self.directory = get_media_directory() / self.directory
# All channels with a URL and download_frequency should have a download.
session = Session.object_session(self)
if download and not self.download_frequency:
@ -360,7 +410,7 @@ class Channel(ModelHelper, Base):
"""
config = dict(
calculate_duration=self.calculate_duration,
directory=str(self.directory.path),
directory=str(self.directory),
download_frequency=self.download_frequency,
favorites={},
generate_posters=self.generate_posters,
@ -400,6 +450,7 @@ class Channel(ModelHelper, Base):
def dict(self, with_statistics: bool = False):
d = super(Channel, self).dict()
d['directory'] = self.directory.relative_to(get_media_directory())
if with_statistics:
d['statistics'] = self.get_statistics()
return d
@ -425,3 +476,8 @@ class Channel(ModelHelper, Base):
largest_video=largest_video,
)
return statistics
async def refresh_files(self):
"""Refresh all files within this Channel's directory. Mark this channel as refreshed."""
await refresh_directory_files_recursively(self.directory)
self.refreshed = True

4
modules/videos/schema.py
View file

@ -131,7 +131,9 @@ class ExtendedVideo(Video):
@dataclass
class VideoResponse:
video = ExtendedVideo
file = ExtendedVideo
prev = ExtendedVideo
next = ExtendedVideo
@dataclass

148
modules/videos/test/test_api.py
View file

@ -1,9 +1,12 @@
import json
import shutil
from http import HTTPStatus
from pathlib import Path
from modules.videos.models import Video
from modules.videos.models import Video, Channel
from modules.videos.video import lib as video_lib
from wrolpi.downloader import DownloadFrequency
from wrolpi.files.models import File
from wrolpi.vars import PROJECT_DIR
@ -11,18 +14,17 @@ def test_refresh_videos(test_client, test_session, test_directory, simple_channe
subdir = test_directory / 'subdir'
subdir.mkdir()
# video1 is in a subdirectory.
video1 = video_factory(channel_id=simple_channel.id, with_video_file=True, with_info_json=True,
# video1 is in a subdirectory, move its files into the subdirectory.
video1 = video_factory(channel_id=simple_channel.id, with_video_file=subdir / 'video1.mp4',
with_info_json=True,
with_poster_ext='jpg')
test_session.commit()
shutil.move(video1.video_path.path, subdir / video1.video_path.path.name)
shutil.move(video1.poster_path.path, subdir / video1.poster_path.path.name)
video1.video_path = subdir / video1.video_path.path.name
test_session.delete(video1.poster_file)
video1.poster_path = None
# video2 is in the test directory.
video2 = video_factory(channel_id=simple_channel.id, with_video_file=True, with_info_json=True,
with_poster_ext='jpg')
video2.poster_path = video1.poster_path = None
video2.poster_path = video2.poster_file = video1.poster_path = None
test_session.commit()
assert not video1.size, 'video1 should not have size during creation'
@ -36,27 +38,73 @@ def test_refresh_videos(test_client, test_session, test_directory, simple_channe
orphan_poster.touch()
# Create a bogus video in the channel.
bogus = Video(video_path='foo', channel_id=simple_channel.id)
video_file = File(path=test_directory / 'foo')
video_file.idempotency = None # File was not added in the current refresh.
bogus = Video(video_file=video_file, channel_id=simple_channel.id)
test_session.add(bogus)
test_session.commit()
test_client.post('/api/videos/refresh')
test_client.post('/api/files/refresh')
# Posters were found during refresh.
assert video1.poster_path
assert 'subdir' in str(video1.poster_path.path)
assert 'subdir' in str(video1.poster_path)
assert video2.poster_path
# Missing video3 was found
video3: Video = test_session.query(Video).filter_by(id=4).one()
assert video3.video_path.path == vid3
assert video3.video_path != video1.video_path
assert video3.video_path != video2.video_path
assert video3.video_path == vid3
assert len(set(test_session.query(Video.video_path))) == 3, 'Should have 3 distinct video files'
# Bogus video was removed.
assert not any('foo' in str(i.video_path.path) for i in test_session.query(Video).all())
assert not any('foo' in str(i.video_path) for i in test_session.query(Video).all())
# Orphan file was not deleted.
assert orphan_poster.is_file(), 'Orphan poster was removed!'
assert video1.size, 'video1 size was not found'
# Remove video1's poster, video1 should be updated.
video1_video_path = str(video1.video_path)
video1.poster_path.unlink()
test_client.post('/api/files/refresh')
video1 = test_session.query(Video).filter_by(video_path=video1_video_path).one()
assert not video1.poster_path
def test_channels_with_videos(test_session, test_client, test_directory, channel_factory, video_factory):
channel1 = channel_factory('channel1', name='channel1')
channel2 = channel_factory('channel2', name='channel2')
vid1 = video_factory(channel_id=channel1.id, with_video_file=True)
vid2 = video_factory(channel_id=channel2.id, with_video_file=True)
vid3 = video_factory(with_video_file=True)
test_session.commit()
vid1_path = vid1.video_path
vid2_path = vid2.video_path
vid3_path = vid3.video_path
test_session.delete(vid1)
test_session.delete(vid2)
test_session.delete(vid3)
test_session.commit()
assert test_session.query(Video).count() == 0, 'Videos were not deleted'
assert test_session.query(Channel).count() == 2, 'Channels were deleted!'
assert vid1_path.is_file(), 'Video file was deleted'
assert vid2_path.is_file(), 'Video file was deleted'
assert vid3_path.is_file(), 'Video file was deleted'
test_client.post('/api/files/refresh')
assert test_session.query(Video).count() == 3, 'Did not find correct amount of video files.'
assert {i[0] for i in test_session.query(Channel.name)} == {'channel1', 'channel2'}, 'Channels were changed.'
vid1: Video = test_session.query(Video).filter_by(video_path=vid1_path).one()
vid2: Video = test_session.query(Video).filter_by(video_path=vid2_path).one()
vid3: Video = test_session.query(Video).filter_by(video_path=vid3_path).one()
assert vid1.channel == channel1
assert vid2.channel == channel2
assert vid3.channel is None
def test_api_download_channel(test_session, test_client, simple_channel):
"""A Channel download (a catalog update) can be triggered via the API."""
@ -70,3 +118,75 @@ def test_api_download_channel(test_session, test_client, simple_channel):
test_session.commit()
request, response = test_client.post(f'/api/videos/download/{simple_channel.id}')
assert response.status_code == HTTPStatus.NO_CONTENT
def test_search_videos_file(test_client, test_session, test_directory, video_with_search_factory):
"""Test that videos can be searched and that their order is by their textsearch rank."""
# These captions have repeated letters, so they will be higher in the ranking.
videos = [
('1', 'b b b b e d d'),
('2', '2 b b b d'),
('3', 'b b'),
('4', 'b e e'),
('5', ''),
]
for title, caption in videos:
video_with_search_factory(title=title, d_text=caption)
test_session.commit()
def assert_search(search_str, expected, limit=20, total=None):
d = json.dumps({'search_str': search_str, 'limit': limit})
_, resp = test_client.post('/api/videos/search', content=d)
assert resp.status_code == HTTPStatus.OK
response_ids = [i['video']['id'] for i in resp.json['files']]
assert response_ids == expected
assert resp.json['totals']['files'] == (total or len(expected))
# Repeated runs should return the same result
for _ in range(2):
# Only videos with a b are returned, ordered by the amount of b's
assert_search('b', [1, 2, 3, 4])
# Only two captions have e
assert_search('e', [4, 1])
# Only two captions have d
assert_search('d', [1, 2])
# 5 can be gotten by it's title
assert_search('5', [5])
# only video 1 has e and d
assert_search('e d', [1])
# video 1 and 4 have b and e, but 1 has more
assert_search('b e', [1, 4])
# Check totals are correct even with a limit
assert_search('b', [1, 2], limit=2, total=4)
def test_search_videos(test_client, test_session, video_factory, assert_video_search, simple_channel):
"""Search the Video table. This does not need to use a join with the File table."""
vid1 = video_factory(upload_date='2022-09-16', with_video_file=True)
vid2 = video_factory(upload_date='2022-09-17', with_video_file=True, channel_id=simple_channel.id)
vid3 = video_factory(upload_date='2022-09-18', with_video_file=True)
vid1.set_favorite(True)
test_session.commit()
assert test_session.query(Video).count() == 3
assert test_session.query(File).count() == 3
assert_video_search(assert_total=3, assert_ids=[vid1.id, vid2.id, vid3.id], order_by='upload_date')
assert_video_search(assert_total=3, assert_ids=[vid3.id, vid2.id, vid1.id], order_by='-upload_date')
assert_video_search(assert_total=3, assert_ids=[vid3.id, vid2.id], order_by='-upload_date', limit=2)
assert_video_search(assert_total=3, assert_ids=[vid2.id, vid1.id], order_by='-upload_date', limit=2, offset=1)
assert_video_search(assert_total=1, assert_ids=[vid2.id], order_by='-upload_date', channel_id=simple_channel.id)
assert_video_search(assert_total=1, assert_ids=[vid1.id], filters=['favorite'])
assert_video_search(assert_total=1, assert_ids=[vid1.id], filters=['favorite'], limit=1)
# No results, no total is returned from the DB.
assert_video_search(assert_total=0, assert_ids=[], filters=['favorite'], offset=1)
# Check all order_by.
for order_by in video_lib.VIDEO_ORDERS:
assert_video_search(order_by=order_by)

104
modules/videos/test/test_captions.py
View file

@ -1,104 +0,0 @@
import shutil
from pathlib import Path
from modules.videos import captions
from modules.videos.models import Video
from wrolpi.db import get_db_curs, get_db_context
from wrolpi.test.common import wrap_test_db, TestAPI
from wrolpi.vars import PROJECT_DIR
class TestCaption(TestAPI):
def setUp(self) -> None:
super().setUp()
self.vtt_path1 = Path(self.tmp_dir.name) / 'example1.en.vtt'
self.vtt_path1.touch()
self.vtt_path1.write_text((PROJECT_DIR / 'test/example1.en.vtt').read_text())
self.vtt_path2 = Path(self.tmp_dir.name) / 'example2.en.vtt'
self.vtt_path2.touch()
self.vtt_path2.write_text((PROJECT_DIR / 'test/example2.en.vtt').read_text())
def test_example_en_vtt(self):
expected = ['okay welcome to this session this is',
'okay welcome to this session this is',
'okay welcome to this session this is\ncalled the kinetic bunny need to meet',
'called the kinetic bunny need to meet',
'called the kinetic bunny need to meet\nthe virtual bonnie emilio stop this is',
'the virtual bonnie emilio stop this is',
'the virtual bonnie emilio stop this is\njana car who just to give a little bit',
'jana car who just to give a little bit',
'jana car who just to give a little bit\nof background information we come from a',
'of background information we come from a',
'of background information we come from a\ncompany called le code from finland and']
self.assertEqual(list(captions.get_caption_text(self.vtt_path1)), expected)
expected = ['okay welcome to this session this is',
'called the kinetic bunny need to meet',
'the virtual bonnie emilio stop this is',
'jana car who just to give a little bit',
'of background information we come from a',
'company called le code from finland and']
self.assertEqual(list(captions.get_unique_caption_lines(self.vtt_path1)), expected)
@wrap_test_db
def test_process_captions(self):
_, session = get_db_context()
video1 = Video(title='scream', caption_path=str(self.vtt_path1))
session.add(video1)
session.commit()
video1.caption = captions.get_captions(video1.caption_path.path)
video2 = Video(title='bar', caption_path=str(self.vtt_path2))
session.add(video2)
session.commit()
video2.caption = captions.get_captions(video2.caption_path.path)
session.commit()
session.flush()
session.refresh(video1)
session.refresh(video2)
# Get the video from the DB
video1 = session.query(Video).filter_by(id=video1.id).one()
self.assertIsNotNone(video1.caption)
video2 = session.query(Video).filter_by(id=video2.id).one()
self.assertIsNotNone(video2.caption)
# Search using the tsvector, "sessions" never actually appears in the text, but "session" does
with get_db_curs() as curs:
def select_textsearch(*args):
curs.execute('SELECT id FROM video WHERE textsearch @@ to_tsquery(%s) ORDER BY id', args)
select_textsearch('sessions')
self.assertEqual(curs.fetchall(), [[1, ]])
# Matches video1.title and video2.caption
select_textsearch('scream')
self.assertEqual(curs.fetchall(), [[1, ], [2, ]])
# Matches video1.title and video2.caption
select_textsearch('scream | sessions')
self.assertEqual(curs.fetchall(), [[1, ], [2, ]])
# Only matches video1.title
select_textsearch('scream & sessions')
self.assertEqual(curs.fetchall(), [[1, ]])
# Matches neither
select_textsearch('scream & sess')
self.assertEqual(curs.fetchall(), [])
# Matches video2.caption
select_textsearch('yawn | sess')
self.assertEqual(curs.fetchall(), [[2, ]])
# Matches video2.caption
select_textsearch('yawn')
self.assertEqual(curs.fetchall(), [[2, ]])
# Matches video2.title
select_textsearch('bar')
self.assertEqual(curs.fetchall(), [[2, ]])
def test_bad_caption(test_directory):
"""A caption file that cannot be parsed is ignored."""
bad_caption_file = PROJECT_DIR / 'test/bad_caption.en.vtt'
test_file = test_directory / 'bad_caption.en.vtt'
shutil.copy(bad_caption_file, test_file)
result = captions.get_captions(test_file)
assert result is None

31
modules/videos/test/test_common.py
View file

@ -244,7 +244,7 @@ def test_update_censored_videos(test_session, video_factory, simple_channel):
check_censored([(vid1.id, True), (vid2.id, True), (vid3.id, True), (vid4.id, False)])
def test_import_favorites(test_session, simple_channel, video_factory, test_channels_config):
def test_import_favorites(test_session, test_directory, simple_channel, video_factory, test_channels_config):
"""
A favorited Video will be preserved through everything (channel deletion, DB wipe) and can be imported and will be
favorited again.
@ -254,14 +254,13 @@ def test_import_favorites(test_session, simple_channel, video_factory, test_chan
video_factory(channel_id=simple_channel.id) # never favorited
vid2 = video_factory(channel_id=None) # has no channel
vid3 = video_factory(channel_id=simple_channel.id)
favorite = vid2.favorite = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
vid3.favorite = favorite
favorite = vid3.favorite = vid2.favorite = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
test_session.commit()
vid2_video_path = vid2.video_path.path
vid2_video_path = vid2.video_path
favorites = {
str(simple_channel.directory.relative): {vid3.video_path.path.name: {'favorite': favorite}},
'NO CHANNEL': {vid2.video_path.path.name: {'favorite': favorite}},
str(simple_channel.directory.relative_to(test_directory)): {vid3.video_path.name: {'favorite': favorite}},
'NO CHANNEL': {vid2.video_path.name: {'favorite': favorite}},
}
# Save config, verify that favorite is set.
@ -299,7 +298,7 @@ def test_import_favorites(test_session, simple_channel, video_factory, test_chan
# Deleting the Video in the model really removes the favorite status.
vid3.delete()
config = get_channels_config()
assert config.favorites == {'NO CHANNEL': {vid2.video_path.path.name: {'favorite': favorite}}}
assert config.favorites == {'NO CHANNEL': {vid2.video_path.name: {'favorite': favorite}}}
import_and_verify([vid2.id])
@ -369,11 +368,11 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
vid4 = video_factory(title='vid4')
test_session.commit()
channel1_name, channel_1_directory = channel1.name, channel1.directory.path
channel2_name, channel_2_directory = channel2.name, channel2.directory.path
channel1_name, channel_1_directory = channel1.name, channel1.directory
channel2_name, channel_2_directory = channel2.name, channel2.directory
assert channel1.directory.path.exists() and channel1.directory.path.is_absolute()
assert channel2.directory.path.exists() and channel2.directory.path.is_absolute()
assert channel1.directory.exists() and channel1.directory.is_absolute()
assert channel2.directory.exists() and channel2.directory.is_absolute()
# Channel's used to have a relative directory in the DB.
test_session.execute('UPDATE channel SET directory=:directory WHERE id=2',
@ -390,9 +389,9 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
channel['download_frequency'] = DownloadFrequency.biweekly
# Change the favorites to the old "link" method as well.
channels_config.favorites = {
sanitize_link(channel1.name): {str(vid1.video_path.path): {'favorite': now()}},
sanitize_link(channel2.name): {str(vid2.video_path.path): {'favorite': now()}},
'NO CHANNEL': {str(vid3.video_path.path): {'favorite': now()}},
sanitize_link(channel1.name): {str(vid1.video_path): {'favorite': now()}},
sanitize_link(channel2.name): {str(vid2.video_path): {'favorite': now()}},
'NO CHANNEL': {str(vid3.video_path): {'favorite': now()}},
}
channels_config.save()
@ -407,9 +406,9 @@ def test_import_channels_config_outdated(test_session, test_directory, channel_f
# No new channels were created. Existing channels were updated.
assert test_session.query(Channel).count() == 2
channel1, channel2 = test_session.query(Channel).order_by(Channel.id)
assert channel1.name == channel1_name and str(channel1.directory.path) == str(channel_1_directory)
assert channel1.name == channel1_name and str(channel1.directory) == str(channel_1_directory)
# Channel2's directory is now absolute.
assert channel2.name == channel2_name and str(channel2.directory.path) == str(channel_2_directory)
assert channel2.name == channel2_name and str(channel2.directory) == str(channel_2_directory)
assert channel1.download_frequency == DownloadFrequency.biweekly
assert channel2.download_frequency == DownloadFrequency.biweekly
# Videos were favorited by their channel "link".

52
modules/videos/test/test_downloader.py
View file

@ -14,12 +14,14 @@ from modules.videos.models import Channel, Video
from wrolpi.db import get_db_context
from wrolpi.downloader import DownloadManager, Download, DownloadResult
from wrolpi.errors import InvalidDownload
from wrolpi.files.models import File
from wrolpi.test.common import TestAPI
from wrolpi.test.test_downloader import HTTPDownloader
from wrolpi.vars import PROJECT_DIR
def test_find_all_missing_videos(test_session, channel_factory, video_factory):
@pytest.mark.asyncio
async def test_find_all_missing_videos(test_session, channel_factory, video_factory):
channel1 = channel_factory(url='some url')
channel1.info_json = {'entries': [{'id': 'foo', 'view_count': 0}]}
test_session.commit()
@ -29,14 +31,14 @@ def test_find_all_missing_videos(test_session, channel_factory, video_factory):
video_factory()
# No missing videos
assert [] == list(find_all_missing_videos(channel1.id))
assert [] == [i async for i in find_all_missing_videos(channel1.id)]
# Create a video that has no video file.
video = Video(title='needs to be downloaded', channel_id=channel1.id, source_id='foo')
test_session.add(video)
test_session.commit()
missing = list(find_all_missing_videos(channel1.id))
missing = [i async for i in find_all_missing_videos(channel1.id)]
assert len(missing) == 1
# The video returned is the one we faked.
@ -175,7 +177,7 @@ async def test_video_download_no_channel(test_session, video_download_manager, v
mock_video_process_runner.assert_called_once()
video: Video = test_session.query(Video).one()
assert str(video.video_path.path) == f'{channel_dir}/video.mp4'
assert str(video.video_path) == f'{channel_dir}/video.mp4'
@pytest.mark.asyncio
@ -364,19 +366,23 @@ async def test_invalid_download_url(test_session, test_download_manager, mock_vi
@pytest.mark.asyncio
async def test_video_download_1(test_session, test_directory, simple_channel, video_download_manager,
mock_video_process_runner):
mock_video_process_runner, image_file):
"""A video download is performed, files are associated."""
simple_channel.source_id = example_video_json['channel_id']
simple_channel.directory = test_directory / 'videos/channel name'
simple_channel.directory.mkdir(parents=True)
video_file = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_file)
video_path = simple_channel.directory / 'a video.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
# Create a poster file which was downloaded.
poster_path = video_path.with_suffix('.png')
image_file.rename(poster_path)
url = 'https://www.youtube.com/watch?v=31jPEBiAC3c'
with mock.patch('modules.videos.downloader.YDL.extract_info') as mock_extract_info, \
mock.patch('modules.videos.downloader.VideoDownloader.prepare_filename') as mock_prepare_filename:
mock_extract_info.return_value = example_video_json
mock_prepare_filename.return_value = (video_file, {'id': 'foo'})
mock_prepare_filename.return_value = (video_path, {'id': 'foo'})
video_download_manager.create_download(url)
await video_download_manager.wait_for_all_downloads()
@ -389,7 +395,10 @@ async def test_video_download_1(test_session, test_directory, simple_channel, vi
assert test_session.query(Channel).one()
video: Video = test_session.query(Video).one()
assert video.video_path.path.is_absolute()
assert video.video_path.is_absolute(), 'Video path is not absolute'
assert video.poster_path == poster_path, 'Video poster was not discovered'
assert video.validated, 'Video was not validated'
assert video.video_file and video.video_file.indexed, 'Video was not indexed'
@pytest.mark.asyncio
@ -428,9 +437,9 @@ example_playlist_json = {
'ie_key': 'Youtube',
'title': 'video 2 title',
'uploader': None,
'url': 'https://youtube.com/watch?v=video_2_url',
'url': 'https://www.youtube.com/shorts/video_2_url',
'view_count': 1413,
'webpage_url': 'https://youtube.com/watch?v=video_2_url'},
'webpage_url': 'https://www.youtube.com/shorts/video_2_url'},
{'_type': 'url',
'description': None,
'duration': None,
@ -438,9 +447,9 @@ example_playlist_json = {
'ie_key': 'Youtube',
'title': 'video 1 title',
'uploader': None,
'url': 'https://youtube.com/watch?v=video_1_url',
'url': 'https://www.youtube.com/watch?v=video_1_url',
'view_count': 58504,
'webpage_url': 'https://youtube.com/watch?v=video_1_url'},
'webpage_url': 'https://www.youtube.com/watch?v=video_1_url'},
{'_type': 'url',
'description': None,
'duration': None,
@ -473,15 +482,14 @@ example_playlist_json = {
@pytest.mark.asyncio
async def test_download_playlist(test_session, test_directory, mock_video_extract_info, video_download_manager):
"""
All videos in a playlist can be downloaded for it's Channel.
"""
"""All videos in a playlist can be downloaded for it's Channel."""
download = Download(url='playlist url')
test_session.add(download)
channel = get_or_create_channel(example_playlist_json['channel_id'], download.url, example_channel_json['uploader'])
test_session.add(Video(url='https://youtube.com/watch?v=video_1_url',
source_id='video 1 id', channel_id=channel.id, video_path='video exists',
poster_path='poster exists'))
video_file = File(path=test_directory / 'video file.mp4')
test_session.add(video_file)
test_session.add(Video(url='https://www.youtube.com/watch?v=video_1_url',
source_id='video 1 id', channel_id=channel.id, video_file=video_file))
test_session.commit()
mock_video_extract_info.return_value = example_playlist_json # Playlist info is fetched first.
@ -491,6 +499,10 @@ async def test_download_playlist(test_session, test_directory, mock_video_extrac
result = await channel_downloader.do_download(download)
assert result.success is True, 'Download was not successful'
assert set(result.downloads) == {
'https://youtube.com/watch?v=video_2_url',
'https://www.youtube.com/watch?v=video_2_url', # Shorts is converted to regular URL.
'https://youtube.com/watch?v=video_3_url',
}
def test_channel_download_match_title(test_session, test_directory, simple_channel, video_download_manager):
pass

314
modules/videos/test/test_lib.py
View file

@ -1,25 +1,20 @@
import json
import pathlib
import shutil
from datetime import datetime
from unittest import mock
import pytest
import sqlalchemy
from modules.videos import lib
from modules.videos.common import apply_info_json
from modules.videos.lib import validate_videos, parse_video_file_name, upsert_video
from modules.videos.lib import parse_video_file_name, validate_video, get_statistics
from modules.videos.models import Video
from modules.videos.video.lib import video_search
from wrolpi.dates import local_timezone
from wrolpi.vars import PROJECT_DIR
from wrolpi.files import lib as files_lib
from wrolpi.files.models import File
def test_search_censored_videos(test_session, simple_channel):
def test_search_censored_videos(test_session, simple_channel, video_factory):
for i in map(str, range(50)):
test_session.add(Video(source_id=i, channel=simple_channel, video_path='foo'))
vid = Video(source_id='51', video_path='bar') # this should never be modified because it has no channel
video_factory(channel_id=simple_channel.id, source_id=i)
vid = video_factory(source_id=51) # this should never be modified because it has no channel
test_session.add(vid)
test_session.commit()
@ -32,135 +27,23 @@ def test_search_censored_videos(test_session, simple_channel):
# All source_ids are in the entries.
set_entries(map(str, range(50)))
videos, total = video_search(filters=['censored'], order_by='id')
assert [i['source_id'] for i in videos] == []
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
assert [i['video']['source_id'] for i in videos] == []
assert total == 0
# First 5 are censored.
set_entries(map(str, range(5, 50)))
videos, total = video_search(filters=['censored'], order_by='id')
assert [i['source_id'] for i in videos] == [str(i) for i in range(5)]
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
assert [i['video']['source_id'] for i in videos] == [str(i) for i in range(5)]
assert total == 5
# First 25 are censored.
set_entries(map(str, range(25, 50)))
videos, total = video_search(filters=['censored'], order_by='id')
assert [i['source_id'] for i in videos] == [str(i) for i in range(20)]
videos, total = video_search(filters=['censored'], order_by='id', limit=20)
assert [i['video']['source_id'] for i in videos] == [str(i) for i in range(20)]
assert total == 25
def test_validate_videos(test_session, simple_channel, video_factory):
"""
Videos that aren't validated should have their data filled in while being validated.
"""
simple_channel.generate_posters = True
vid1_json = {'title': 'info&apos;s title'}
vid3_json = {'duration': 100}
vid5_json = {'view_count': 42}
vid6_json = {'webpage_url': 'https://example.com/webpage', 'url': 'https://example.com/url'}
vid1 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid1_json)
vid2 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=True)
vid3 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid3_json)
vid4 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=True)
vid5 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid5_json)
vid6 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg', with_info_json=vid6_json)
vid7 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='jpg')
vid8 = video_factory(simple_channel.id, with_video_file=True, with_poster_ext='webp')
vid9 = video_factory(simple_channel.id, with_video_file=True)
# These videos are missing this data, and a refresh should fill them in.
vid1.title = None
vid2.caption = None
vid2.title = None
vid3.duration = None
vid4.size = None
vid5.view_count = None
vid6.url = None
vid7.poster_path = None
test_session.commit()
# Write the associated files. This data should be processed.
vid2.caption_path = vid2.video_path.path.with_suffix('.en.vtt')
shutil.copy(PROJECT_DIR / 'test/example1.en.vtt', vid2.caption_path)
test_session.commit()
# All Videos are validated, all data is filled out. The info json data is trusted first.
validate_videos()
assert all([i.validated for i in test_session.query(Video)])
assert vid1.title == "info's title"
assert vid2.caption.startswith('okay welcome')
assert 'mp4' not in vid2.title
assert vid3.duration == 100
assert vid4.size == 1055736
assert vid5.view_count == 42
assert vid6.url == 'https://example.com/webpage'
assert str(vid7.poster_path).endswith('.jpg')
assert str(vid8.poster_path).endswith('.jpg')
assert str(vid9.poster_path).endswith('.jpg') # this was generated from the video file.
vid1.title = None
vid3.duration = None
vid5.view_count = None
vid6.url = None
vid9.poster_path = None
test_session.commit()
# Validation does not happen because all are still validated.
validate_videos()
# All videos need to be validated again.
for i in test_session.query(Video):
i.validated = False
# Use backup methods to fetch data.
vid1.info_json_path = None
vid3.info_json_path = None
vid5.info_json_path = None
vid6.info_json_path.path.write_text(json.dumps({'url': 'https://example.com/url'}))
test_session.commit()
validate_videos()
assert all([i.validated for i in test_session.query(Video)])
assert vid1.title != "info's title" # file name is used.
assert vid3.duration == 5 # video file duration is used.
assert vid5.view_count is None
assert vid6.url == 'https://example.com/url' # url is used as backup url.
# Posters are both JPEG format. Webp file is removed.
assert vid7.poster_path.path.is_file()
assert vid8.poster_path.path.is_file()
assert not vid8.poster_path.path.with_suffix('.webp').is_file()
assert str(vid9.poster_path).endswith('.jpg') # the generated file was rediscovered.
def test_validate_video_exception(test_session, simple_channel, video_factory):
"""
Test that even if a Video cannot be validated, the other Videos will still be validated.
"""
vid1 = video_factory(simple_channel.id, with_video_file=True)
vid2 = video_factory(simple_channel.id, with_video_file=True)
# The validator should not even check this Video because it does not have a video file.
vid3 = video_factory(simple_channel.id)
vid3.video_path = None
assert not all([i.validated for i in test_session.query(Video)])
test_session.commit()
with mock.patch('modules.videos.lib.process_video_info_json') as mock_process_video_info_json:
mock_process_video_info_json.side_effect = [
Exception('skip this video'),
(None, None, None, None),
('should not be set to video 3', None, None, None), # This should not be called!
]
validate_videos()
# Only first two videos are validated.
assert mock_process_video_info_json.call_count == 2, mock_process_video_info_json.call_count
test_session.commit()
assert not vid1.validated
assert vid2.validated
assert not vid3.validated
@pytest.mark.parametrize('file_name,expected', [
('channel_20000101_12345678910_ some title.mp4', ('channel', '20000101', '12345678910', 'some title')),
('channel name_NA_12345678910_ some title.mp4', ('channel name', None, '12345678910', 'some title')),
@ -186,67 +69,120 @@ def test_parse_video_file_name(file_name, expected):
assert parse_video_file_name(video_path) == expected, f'{file_name} != {expected}'
def test_validate_does_not_generate_when_disabled(test_session, channel_factory, video_factory):
@pytest.mark.asyncio
async def test_associated_files(test_session, test_directory, video_file, image_file, vtt_file1):
"""A Video file has files related to it. They should be attached to the Video record."""
caption_file = video_file.with_suffix('.en.vtt')
vtt_file1.rename(caption_file)
poster_file = video_file.with_suffix('.jpeg')
image_file.rename(poster_file)
await files_lib._refresh_all_files()
test_session.commit()
# All files were found.
file_paths = {i.path for i in test_session.query(File)}
assert len(file_paths) == 3
video: Video = test_session.query(Video).one()
video_id = video.id
assert video.video_file == video_file
assert video.poster_file == poster_file
assert video.caption_file == caption_file
assert video.video_file.modification_datetime
assert video.video_file.size
assert video.video_file.indexed
await files_lib._refresh_all_files()
test_session.commit()
# No new files were added.
assert {i.path for i in test_session.query(File)} == file_paths
# Same video exists.
video: Video = test_session.query(Video).one()
assert video.id == video_id, 'Video id changed. Was the video recreated?'
@pytest.mark.asyncio
async def test_unassociated_video(test_session, test_directory, video_file):
"""A Video file without associated files is still a Video."""
# Move the video file into a new subdirectory. The video file should still be found.
sub_dir = test_directory / 'subdir'
sub_dir.mkdir()
new_video_file_path = sub_dir / video_file.name
video_file.rename(new_video_file_path)
video_file = new_video_file_path
await files_lib._refresh_all_files()
test_session.commit()
video: Video = test_session.query(Video).one()
assert video.video_file == video_file
@pytest.mark.asyncio
async def test_video_factory(test_session, video_factory, channel_factory):
"""
A poster should not be generated if the Video's Channel forbids it.
The `video_factory` pytest fixture is used in many video tests. Test all it's functionality.
"""
channel1 = channel_factory() # posters should not be generated by default.
channel = channel_factory()
video = video_factory(
channel_id=channel.id,
with_video_file=True,
with_info_json={'description': 'hello'},
with_caption_file=True,
with_poster_ext='png',
source_id='some id',
)
test_session.commit()
assert video.video_path and video.video_file and video.video_path.is_file()
assert video.caption_path and video.caption_file and video.caption_path.is_file()
assert video.poster_path and video.poster_file and video.poster_path.is_file()
assert video.info_json_path and video.info_json_file and video.info_json_path.is_file()
assert video.video_file.a_text # title
assert video.video_file.b_text is None
assert video.video_file.c_text and 'hello' in video.video_file.c_text # description
assert video.video_file.d_text # captions
assert video.source_id == 'some id'
def test_validate_video(test_session, test_directory, video_factory):
"""A video poster will be generated only if the channel permits."""
vid1 = video_factory(with_video_file=True)
assert not vid1.poster_path
validate_video(vid1, True, test_session)
assert vid1.poster_path, 'Poster was not created'
assert vid1.poster_path.is_file(), 'Poster path does not exist'
assert vid1.video_path.stem == vid1.poster_path.stem
assert vid1.poster_path.suffix == '.jpg'
# A PNG is replaced.
vid2 = video_factory(with_video_file=True, with_poster_ext='.png')
vid2.poster_path.with_suffix('.jpg').touch() # New poster exists, replace it.
assert vid2.poster_path and vid2.poster_path.is_file() and vid2.poster_path.suffix == '.png', \
'Poster was not initialized'
validate_video(vid2, True, test_session)
assert vid2.poster_path.is_file(), 'New poster was not generated'
assert vid2.poster_path.suffix == '.jpg'
@pytest.mark.asyncio
async def test_get_statistics(test_session, video_factory, channel_factory):
# Can get statistics in empty DB.
await get_statistics()
channel1 = channel_factory()
channel2 = channel_factory()
channel2.generate_posters = True
vid1 = video_factory(channel1.id, with_video_file=True)
vid2 = video_factory(channel2.id, with_video_file=True)
# webp will not be converted unless generate_posters is enabled.
vid3 = video_factory(channel1.id, with_video_file=True, with_poster_ext='webp')
test_session.commit()
assert not vid1.poster_path
assert not vid2.poster_path
assert str(vid3.poster_path).endswith('webp')
video_factory(channel_id=channel1.id)
video_factory(channel_id=channel1.id)
video_factory(channel_id=channel2.id)
video_factory()
validate_videos()
assert not vid1.poster_path
assert vid2.poster_path
assert str(vid3.poster_path).endswith('webp')
def test_video_modification_datetime(test_session, simple_channel, video_file):
"""
A Video has a modification_datetime which represents the video file's mtime.
"""
expected = local_timezone(datetime(2000, 1, 1, 0, 0, 0))
with mock.patch('modules.videos.lib.from_timestamp') as mock_from_timestamp:
mock_from_timestamp.return_value = expected
video = upsert_video(test_session, video_file, simple_channel)
test_session.commit()
assert video.modification_datetime == expected
def test_refresh_channel_videos(test_session, channel_factory, video_factory, video_file, assert_video_ids):
assert_video_ids([])
# Simulate a channel's catalog being updated.
channel1 = channel_factory(name='channel1')
test_session.add(Video(source_id='source1', channel_id=channel1.id))
test_session.add(Video(source_id='source2', channel_id=channel1.id))
# Add some real files to be found.
destination = channel1.directory / f'channel1_20220510_source1_the title.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
destination = channel1.directory / f'channel1_20220511_source2_the title.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
lib.refresh_channel_videos(channel1)
assert_video_ids([1, 2])
# These video files were deleted.
test_session.add(Video(source_id='this should be removed', channel_id=channel1.id, video_path='foo'))
test_session.add(Video(source_id='this should also be removed', video_path='bar'))
test_session.commit()
assert_video_ids([1, 2, 3, 4])
lib.refresh_channel_videos(channel1)
assert_video_ids([1, 2, 4])
lib.refresh_videos()
assert_video_ids([1, 2])
result = await get_statistics()
assert 'statistics' in result
assert 'videos' in result['statistics']
assert 'channels' in result['statistics']
assert 'historical' in result['statistics']

6
modules/videos/video/api.py
View file

@ -25,7 +25,7 @@ logger = logger.getChild(__name__)
@openapi.response(HTTPStatus.NOT_FOUND, JSONErrorResponse)
def video_get(_: Request, video_id: int):
video, previous_video, next_video = lib.get_video_for_app(video_id)
return json_response({'video': video, 'prev': previous_video, 'next': next_video})
return json_response({'file': video, 'prev': previous_video, 'next': next_video})
@video_bp.post('/search')
@ -40,7 +40,7 @@ async def search(_: Request, body: schema.VideoSearchRequest):
if body.order_by not in lib.VIDEO_ORDERS:
raise InvalidOrderBy('Invalid order by')
videos, videos_total = lib.video_search(
files, videos_total = lib.video_search(
body.search_str,
body.offset,
body.limit,
@ -49,7 +49,7 @@ async def search(_: Request, body: schema.VideoSearchRequest):
body.filters,
)
ret = {'videos': list(videos), 'totals': {'videos': videos_total}}
ret = {'files': list(files), 'totals': {'files': videos_total}}
return json_response(ret)

174
modules/videos/video/lib.py
View file

@ -1,3 +1,4 @@
import pathlib
from datetime import datetime
from typing import Tuple, Optional, List
@ -8,6 +9,9 @@ from sqlalchemy.orm.exc import NoResultFound
from wrolpi.common import run_after, logger
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models
from wrolpi.errors import UnknownVideo
from wrolpi.files.lib import assign_file_prefetched_models, handle_search_results
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
from ..lib import save_channels_config
from ..models import Video
@ -31,49 +35,49 @@ def get_video_for_app(video_id: int) -> Tuple[dict, Optional[dict], Optional[dic
video.set_viewed()
previous_video, next_video = video.get_surrounding_videos()
caption = video.caption
video = video.__json__()
video['caption'] = caption
previous_video = previous_video.__json__() if previous_video else None
next_video = next_video.__json__() if next_video else None
caption = video.video_file.d_text
video = video.video_file.__json__()
video['video']['caption'] = caption
previous_video = previous_video.video_file.__json__() if previous_video else None
next_video = next_video.video_file.__json__() if next_video else None
return video, previous_video, next_video
VIDEO_ORDERS = {
'upload_date': 'upload_date ASC, LOWER(video_path) ASC',
'-upload_date': 'upload_date DESC NULLS LAST, LOWER(video_path) DESC',
'rank': '2 DESC, LOWER(video_path) DESC',
'-rank': '2 ASC, LOWER(video_path) ASC',
'id': 'id ASC',
'-id': 'id DESC',
'size': 'size ASC, LOWER(video_path) ASC',
'-size': 'size DESC, LOWER(video_path) DESC',
'duration': 'duration ASC, LOWER(video_path) ASC',
'-duration': 'duration DESC, LOWER(video_path) DESC',
'favorite': 'favorite ASC, LOWER(video_path) ASC',
'-favorite': 'favorite DESC, LOWER(video_path) DESC',
'viewed': 'viewed ASC',
'-viewed': 'viewed DESC',
'view_count': 'view_count ASC',
'-view_count': 'view_count DESC',
'modification_datetime': 'modification_datetime ASC',
'-modification_datetime': 'modification_datetime DESC',
'upload_date': 'v.upload_date ASC, LOWER(v.video_path) ASC',
'-upload_date': 'v.upload_date DESC NULLS LAST, LOWER(v.video_path) DESC',
'rank': '2 DESC, LOWER(v.video_path) DESC',
'-rank': '2 ASC, LOWER(v.video_path) ASC',
'id': 'v.id ASC',
'-id': 'v.id DESC',
'size': 'f.size ASC, LOWER(v.video_path) ASC',
'-size': 'f.size DESC, LOWER(v.video_path) DESC',
'duration': 'duration ASC, LOWER(v.video_path) ASC',
'-duration': 'duration DESC, LOWER(v.video_path) DESC',
'favorite': 'favorite ASC, LOWER(v.video_path) ASC',
'-favorite': 'favorite DESC, LOWER(v.video_path) DESC',
'viewed': 'v.viewed ASC',
'-viewed': 'v.viewed DESC',
'view_count': 'v.view_count ASC',
'-view_count': 'v.view_count DESC',
'modification_datetime': 'f.modification_datetime ASC',
'-modification_datetime': 'f.modification_datetime DESC',
}
NO_NULL_ORDERS = {
'viewed': '\nAND viewed IS NOT NULL',
'-viewed': '\nAND viewed IS NOT NULL',
'duration': '\nAND duration IS NOT NULL',
'-duration': '\nAND duration IS NOT NULL',
'size': '\nAND size IS NOT NULL',
'-size': '\nAND size IS NOT NULL',
'view_count': '\nAND view_count IS NOT NULL',
'-view_count': '\nAND view_count IS NOT NULL',
'modification_datetime': '\nAND modification_datetime IS NOT NULL',
'-modification_datetime': '\nAND modification_datetime IS NOT NULL',
'viewed': 'v.viewed IS NOT NULL',
'-viewed': 'v.viewed IS NOT NULL',
'duration': 'v.duration IS NOT NULL',
'-duration': 'v.duration IS NOT NULL',
'size': 'f.size IS NOT NULL',
'-size': 'f.size IS NOT NULL',
'view_count': 'v.view_count IS NOT NULL',
'-view_count': 'v.view_count IS NOT NULL',
'modification_datetime': 'f.modification_datetime IS NOT NULL',
'-modification_datetime': 'f.modification_datetime IS NOT NULL',
}
DEFAULT_VIDEO_ORDER = 'rank'
VIDEO_QUERY_LIMIT = 20
VIDEO_QUERY_LIMIT = 24
def video_search(
@ -84,71 +88,61 @@ def video_search(
order_by: str = None,
filters: List[str] = None,
) -> Tuple[List[dict], int]:
with get_db_curs() as curs:
params = dict(search_str=search_str, offset=offset)
channel_where = ''
if channel_id:
channel_where = 'AND channel_id = %(channel_id)s'
params['channel_id'] = channel_id
wheres = ['v.video_path IS NOT NULL']
# Filter for/against favorites, if it was provided
favorites_where = ''
if isinstance(filters, list) and 'favorite' in filters:
favorites_where = 'AND favorite IS NOT NULL'
params = dict(search_str=search_str, offset=offset)
if channel_id:
wheres.append('v.channel_id = %(channel_id)s')
params['channel_id'] = channel_id
censored_where = ''
if isinstance(filters, list) and 'censored' in filters:
censored_where = 'AND censored = true'
# Filter for/against favorites, if it was provided
if isinstance(filters, list) and 'favorite' in filters:
wheres.append('v.favorite IS NOT NULL')
where = ''
if search_str:
# A search_str was provided by the user, modify the query to filter by it.
columns = 'id, ts_rank_cd(textsearch, websearch_to_tsquery(%(search_str)s)), COUNT(*) OVER() AS total'
where = 'AND textsearch @@ websearch_to_tsquery(%(search_str)s)'
params['search_str'] = search_str
else:
# No search_str provided. Get id and total only.
columns = 'id, COUNT(*) OVER() AS total'
if isinstance(filters, list) and 'censored' in filters:
wheres.append('v.censored = true')
# Convert the user-friendly order by into a real order by, restrict what can be interpolated by using the
# whitelist.
order = VIDEO_ORDERS[DEFAULT_VIDEO_ORDER]
if order_by:
try:
order = VIDEO_ORDERS[order_by]
except KeyError:
raise
if order_by in NO_NULL_ORDERS:
where += NO_NULL_ORDERS[order_by]
if search_str:
# A search_str was provided by the user, modify the query to filter by it.
select_columns = 'f.path, ts_rank(f.textsearch, websearch_to_tsquery(%(search_str)s)), ' \
'COUNT(*) OVER() AS total'
wheres.append('f.textsearch @@ websearch_to_tsquery(%(search_str)s)')
params['search_str'] = search_str
join = 'LEFT JOIN file f on f.path = v.video_path'
else:
# No search_str provided. Get path and total only.
select_columns = 'v.video_path AS path, COUNT(*) OVER() AS total'
join = ''
query = f'''
SELECT
{columns}
FROM video
WHERE
video_path IS NOT NULL
{where}
{channel_where}
{favorites_where}
{censored_where}
ORDER BY {order}
OFFSET %(offset)s LIMIT {int(limit)}
'''.strip()
logger.debug(query)
if order_by in {'size', '-size', 'modification_datetime', '-modification_datetime'}:
# Size and modification_datetime are from the video file.
join = 'LEFT JOIN file f on f.path = v.video_path'
curs.execute(query, params)
# Convert the user-friendly order by into a real order by, restrict what can be interpolated by using the
# whitelist.
order = VIDEO_ORDERS[DEFAULT_VIDEO_ORDER]
if order_by:
try:
results = [dict(i) for i in curs.fetchall()]
except psycopg2.ProgrammingError:
# No videos
return [], 0
total = results[0]['total'] if results else 0
ranked_ids = [i['id'] for i in results]
order = VIDEO_ORDERS[order_by]
except KeyError:
raise
if order_by in NO_NULL_ORDERS:
wheres.append(NO_NULL_ORDERS[order_by])
with get_db_session() as session:
results = get_ranked_models(ranked_ids, Video, session=session)
results = [i.__json__() for i in results]
wheres = '\n AND '.join(wheres)
where = f'WHERE\n{wheres}' if wheres else ''
stmt = f'''
SELECT
{select_columns}
FROM video v
{join}
{where}
ORDER BY {order}
OFFSET %(offset)s LIMIT {int(limit)}
'''.strip()
logger.debug(stmt, params)
results, total = handle_search_results(stmt, params)
return results, total

65
modules/videos/video/test/test_api.py
View file

@ -1,7 +1,6 @@
from datetime import timedelta
from http import HTTPStatus
from json import dumps
from unittest import mock
from modules.videos.models import Video
from modules.videos.video.lib import get_video_for_app
@ -71,7 +70,7 @@ def test_get_video_prev_next(test_session, channel_factory, video_factory):
def test_get_video_for_app(test_session, simple_channel, simple_video):
vid, prev, next_ = get_video_for_app(simple_video.id)
assert vid['id'] == simple_video.id
assert vid['video']['id'] == simple_video.id
def test_video_delete(test_session, test_directory, channel_factory, video_factory):
@ -81,8 +80,8 @@ def test_video_delete(test_session, test_directory, channel_factory, video_facto
vid2 = video_factory(channel_id=channel2.id, with_video_file=True, with_info_json=True)
test_session.commit()
vid1_video_path, vid1_caption_path = vid1.video_path.path, vid1.caption_path.path
vid2_video_path, vid2_info_json_path = vid2.video_path.path, vid2.info_json_path.path
vid1_video_path, vid1_caption_path = vid1.video_path, vid1.caption_path
vid2_video_path, vid2_info_json_path = vid2.video_path, vid2.info_json_path
# No videos have been deleted yet.
assert vid1_video_path.is_file() and vid1_caption_path.is_file()
@ -106,36 +105,42 @@ def test_video_delete(test_session, test_directory, channel_factory, video_facto
assert vid2_video_path.is_file() is False and vid2_info_json_path.is_file() is False
def test_wrol_mode(test_directory, simple_channel, simple_video):
def test_wrol_mode(test_directory, simple_channel, simple_video, wrol_mode_fixture, test_download_manager):
"""Many methods are blocked when WROL Mode is enabled."""
channel = dumps(dict(name=simple_channel.name, directory=str(simple_channel.directory)))
favorite = dumps(dict(video_id=simple_video.id, favorite=True))
with mock.patch('wrolpi.common.wrol_mode_enabled', lambda: True):
# Can't create, update, or delete a channel.
_, resp = api_app.test_client.post('/api/videos/channels', content=channel)
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.put(f'/api/videos/channels/{simple_channel.id}', content=channel)
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.delete(f'/api/videos/channels/{simple_channel.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
wrol_mode_fixture(True)
# Can't delete a video
_, resp = api_app.test_client.delete(f'/api/videos/video/{simple_video.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# Can't create, update, or delete a channel.
_, resp = api_app.test_client.post('/api/videos/channels', content=channel)
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.put(f'/api/videos/channels/{simple_channel.id}', content=channel)
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.delete(f'/api/videos/channels/{simple_channel.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# Can't refresh or download
_, resp = api_app.test_client.post('/api/videos/refresh')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.post(f'/api/videos/download/{simple_channel.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# Can't delete a video
_, resp = api_app.test_client.delete(f'/api/videos/video/{simple_video.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# THE REST OF THESE METHODS ARE ALLOWED
_, resp = api_app.test_client.post('/api/videos/favorite', content=favorite)
assert resp.status_code == HTTPStatus.OK
# Can't refresh or download
_, resp = api_app.test_client.post('/api/files/refresh')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
_, resp = api_app.test_client.post(f'/api/videos/download/{simple_channel.id}')
assert resp.status_code == HTTPStatus.FORBIDDEN
assert resp.json['code'] == API_ERRORS[WROLModeEnabled]['code']
# THE REST OF THESE METHODS ARE ALLOWED
_, resp = api_app.test_client.post('/api/videos/favorite', content=favorite)
assert resp.status_code == HTTPStatus.OK
assert test_download_manager.stopped.is_set()
wrol_mode_fixture(False)
assert not test_download_manager.stopped.is_set()

19
modules/videos/video_url_resolver.py
View file

@ -1,3 +1,9 @@
import re
from wrolpi.common import logger
logger = logger.getChild(__name__)
formatters = {
'youtube.com': lambda i: f'https://youtube.com/watch?v={i["url"]}',
'www.youtube.com': lambda i: f'https://youtube.com/watch?v={i["url"]}',
@ -22,3 +28,16 @@ def video_url_resolver(domain: str, entry: dict) -> str:
except KeyError:
# Unable to resolve, try the URL as is.
return entry['url']
YOUTUBE_SHORT_REGEX = re.compile(r'https?:\/\/w{3}?\.?youtube.com\/shorts\/(.{5,15})')
def normalize_youtube_shorts_url(url: str) -> str:
"""Convert a YouTube "shorts" URL to a regular video URL.
This is to avoid downloading shorts twice."""
if match := YOUTUBE_SHORT_REGEX.match(url.strip()):
source_id = match.group(1)
return f'https://www.youtube.com/watch?v={source_id}'
return url

2
pytest.ini Normal file
View file

@ -0,0 +1,2 @@
[pytest]
asyncio_mode=auto

3
requirements.txt
View file

@ -20,4 +20,5 @@ selenium==4.1.0
srt==3.5.1
websockets==10.1
webvtt-py==0.4.6
yt-dlp==2022.7.18
yt-dlp==2022.9.1
python-magic~=0.4.27

BIN
test/big_buck_bunny_720p_1mb.mp4
View file

Binary file not shown.

33
wrolpi/__init__.py
View file

@ -1,6 +1,4 @@
import inspect
import multiprocessing
from functools import wraps
import wrolpi.files.api # noqa Import from files so the blueprint is attached.
BEFORE_STARTUP_FUNCTIONS = []
@ -20,32 +18,3 @@ def after_startup(func: callable):
from .root_api import api_app
api_app.after_server_start(func)
return func
def limit_concurrent(limit: int):
"""
Wrapper that limits the amount of concurrently running functions.
"""
sema = multiprocessing.Semaphore(value=limit)
def wrapper(func: callable):
if inspect.iscoroutinefunction(func):
@wraps(func)
async def wrapped(*a, **kw):
acquired = sema.acquire(block=False)
if not acquired:
return
return await func(*a, **kw)
return wrapped
else:
@wraps(func)
def wrapped(*a, **kw):
acquired = sema.acquire(block=False)
if not acquired:
return
return func(*a, **kw)
return wrapped
return wrapper

43
modules/videos/captions.py → wrolpi/captions.py
View file

@ -1,13 +1,17 @@
#! /usr/bin/env python3
import pathlib
import subprocess
import tempfile
from pathlib import Path
from typing import Generator, Union
from typing import Generator, Union, Optional
import srt
import webvtt
from wrolpi.cmd import which
from wrolpi.common import logger
__all__ = ['get_captions']
__all__ = ['read_captions', 'extract_captions']
def get_caption_text(caption_path: Union[str, Path]) -> Generator:
@ -24,9 +28,7 @@ def get_caption_text(caption_path: Union[str, Path]) -> Generator:
def get_unique_caption_lines(caption_path: Union[str, Path]) -> Generator:
"""
Return all unique lines from each caption of a caption file.
"""
"""Return all unique lines from each caption of a caption file."""
last_line = None
for text in get_caption_text(caption_path):
for line in text.split('\n'):
@ -35,21 +37,44 @@ def get_unique_caption_lines(caption_path: Union[str, Path]) -> Generator:
yield line
def get_captions(caption_path: Path):
def read_captions(caption_path: Path):
"""Parse video captions from the video's captions file."""
try:
lines = get_unique_caption_lines(str(caption_path))
block = '\n'.join(lines)
return block
except UnicodeDecodeError:
except UnicodeDecodeError as e:
# Failed to decode the caption file
# TODO handle this error
pass
except webvtt.errors.MalformedFileError:
except webvtt.errors.MalformedFileError as e:
# File format is broken
pass
except webvtt.errors.MalformedCaptionError:
except webvtt.errors.MalformedCaptionError as e:
# Captions form is broken somehow
pass
logger.debug(f'Failed to parse caption file {caption_path}')
FFMPEG_BIN = which('ffmpeg', '/usr/bin/ffmpeg')
def extract_captions(path: pathlib.Path) -> Optional[str]:
"""Extract captions that are embedded in a video file."""
with tempfile.TemporaryDirectory() as directory:
directory = pathlib.Path(directory)
file_path = directory / 'captions.vtt'
cmd = (FFMPEG_BIN, '-i', str(path.absolute()), file_path)
assert not file_path.exists()
try:
subprocess.check_call(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except subprocess.SubprocessError:
logger.debug(f'Unable to extract subtitles {path}')
return ''
if file_path.stat().st_size > 0:
captions = read_captions(file_path)
return captions
# No captions could be extracted.
return

298
wrolpi/common.py
View file

@ -1,14 +1,15 @@
import asyncio
import collections
import contextlib
import inspect
import json
import logging
import multiprocessing
import os
import pathlib
import queue
import re
import string
import sys
import tempfile
from copy import deepcopy
from datetime import datetime, date
@ -18,7 +19,7 @@ from itertools import islice, filterfalse, tee
from multiprocessing import Event, Queue, Lock, Manager
from pathlib import Path
from typing import Union, Callable, Tuple, Dict, List, Iterable, Optional, Generator, Any
from urllib.parse import urlunsplit, urlparse
from urllib.parse import urlparse
import aiohttp
import yaml
@ -27,10 +28,11 @@ from sanic.request import Request
from sqlalchemy import types
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
from wrolpi.dates import DEFAULT_TIMEZONE
from wrolpi.errors import WROLModeEnabled, NativeOnly
from wrolpi.vars import PUBLIC_HOST, PUBLIC_PORT, PYTEST, MODULES_DIR, \
from wrolpi.vars import PYTEST, MODULES_DIR, \
DOCKERIZED, CONFIG_DIR, MEDIA_DIRECTORY
logger = logging.getLogger()
@ -49,19 +51,28 @@ class ModelHelper:
d = {i.name: getattr(self, i.name) for i in self.__table__.columns} # noqa
return d
@classmethod
def upsert(cls, file, session: Session) -> Base:
raise NotImplementedError('This model has not defined this method.')
class PathColumn(types.TypeDecorator):
impl = types.String
@staticmethod
def find_by_path(path, session) -> Base:
raise NotImplementedError('This model has not defined this method.')
def process_bind_param(self, value, dialect):
if isinstance(value, Path):
return value
elif value:
return str(value)
@staticmethod
def find_by_paths(paths, session) -> List:
raise NotImplementedError('This model has not defined this method.')
def process_result_value(self, value, dialect):
if value:
return Path(value)
@property
def primary_path(self):
raise NotImplementedError('This model has not defined this method.')
def get_model_by_table_name(table_name):
"""Find a model by its table name."""
for klass in Base._decl_class_registry.values():
if hasattr(klass, '__tablename__') and klass.__tablename__ == table_name:
return klass
class tsvector(types.TypeDecorator):
@ -82,10 +93,6 @@ def sanitize_link(link: str) -> str:
return new_link
def string_to_boolean(s: str) -> bool:
return str(s).lower() in {'true', 't', '1', 'on'}
DEFAULT_QUEUE_SIZE = 1000
QUEUE_TIMEOUT = 10
@ -140,35 +147,6 @@ def create_websocket_feed(name: str, uri: str, blueprint: Blueprint, maxsize: in
return q, event
# The following code is used to consistently construct URLs that will reference this service.
SANIC_HOST = None
SANIC_PORT = None
URL_COMPONENTS = collections.namedtuple('Components', ['scheme', 'netloc', 'path', 'query', 'fragment'])
def set_sanic_url_parts(host, port):
"""
Set the global parts of this service's URL. This is used to consistently construct URLs that will reference this
service.
"""
global SANIC_HOST
global SANIC_PORT
SANIC_HOST = host
SANIC_PORT = port
def get_sanic_url(scheme: str = 'http', path: str = None, query: list = None, fragment: str = None):
"""
Build a URL with the provided parts that references this running service.
"""
host = PUBLIC_HOST or SANIC_HOST
port = PUBLIC_PORT or SANIC_PORT
components = URL_COMPONENTS(scheme=scheme, netloc=f'{host}:{port}', path=path,
query=query, fragment=fragment)
unparsed = str(urlunsplit(components))
return unparsed
class ConfigFile:
"""
This class keeps track of the contents of a config file. You can update the config by calling
@ -365,11 +343,6 @@ def wrol_mode_enabled() -> bool:
return get_config().wrol_mode
def set_wrol_mode(enable: bool):
"""Enable or disable WROL Mode for all processes. This also updates the config file."""
get_config().wrol_mode = enable
def wrol_mode_check(func):
"""Wraps a function so that it cannot be called when WROL Mode is enabled."""
@ -385,10 +358,40 @@ def wrol_mode_check(func):
return check
def enable_wrol_mode(download_manager=None):
"""
Modify config to enable WROL Mode.
Stop downloads and Download Manager.
"""
logger.warning('ENABLING WROL MODE')
get_config().wrol_mode = True
if not download_manager:
from wrolpi.downloader import download_manager
download_manager.stop()
else:
# Testing.
download_manager.stop()
def disable_wrol_mode(download_manager=None):
"""
Modify config to disable WROL Mode.
Start downloads and Download Manager.
"""
logger.warning('DISABLING WROL MODE')
get_config().wrol_mode = False
if not download_manager:
from wrolpi.downloader import download_manager
download_manager.enable()
else:
# Testing.
download_manager.enable()
def insert_parameter(func: Callable, parameter_name: str, item, args: Tuple, kwargs: Dict) -> Tuple[Tuple, Dict]:
"""
Insert a parameter wherever it fits in the Callable's signature.
"""
"""Insert a parameter wherever it fits in the Callable's signature."""
sig = inspect.signature(func)
if parameter_name not in sig.parameters:
raise TypeError(f'Function {func} MUST have a {parameter_name} parameter!')
@ -403,9 +406,7 @@ def insert_parameter(func: Callable, parameter_name: str, item, args: Tuple, kwa
def iterify(kind: type = list):
"""
Convenience function to convert the output of the wrapped function to the type provided.
"""
"""Convenience function to convert the output of the wrapped function to the type provided."""
def wrapper(func):
@wraps(func)
@ -575,9 +576,7 @@ def minimize_dict(d: dict, keys: Iterable) -> Optional[dict]:
def make_media_directory(path: Union[str, Path]):
"""
Make a directory relative within the media directory.
"""
"""Make a directory relative within the media directory."""
media_dir = get_media_directory()
path = media_dir / str(path)
path.mkdir(parents=True)
@ -592,6 +591,8 @@ def extract_domain(url):
"""
parsed = urlparse(url)
domain = parsed.netloc
if not domain:
raise ValueError(f'URL does not have a domain: {url=}')
domain = domain.decode() if hasattr(domain, 'decode') else domain
if domain.startswith('www.'):
# Remove leading www.
@ -601,17 +602,18 @@ def extract_domain(url):
def import_modules():
"""Import all WROLPi Modules in the modules directory. Raise an ImportError if there are no modules."""
try:
modules = [i.name for i in MODULES_DIR.iterdir() if
i.is_dir() and not (i.name.startswith('_') or i.name.startswith('.'))]
for module in modules:
module = f'modules.{module}.api'
logger.debug(f'Importing {module}')
modules = [i.name for i in MODULES_DIR.iterdir() if
i.is_dir() and not (i.name.startswith('_') or i.name.startswith('.'))]
imported = []
for module_name in modules:
module = f'modules.{module_name}.api'
logger.debug(f'Importing {module}')
try:
__import__(module, globals(), locals(), [], 0)
except ImportError as e:
logger.fatal('No modules could be found!', exc_info=e)
raise
return modules
imported.append(module_name)
except ImportError as e:
logger.fatal(f'Unable to import {module}', exc_info=e)
return imported
def api_param_limiter(maximum: int, default: int = 20) -> callable:
@ -636,16 +638,9 @@ def api_param_limiter(maximum: int, default: int = 20) -> callable:
return limiter_
@contextlib.contextmanager
def temporary_directory_path(*a, **kw):
with tempfile.TemporaryDirectory(*a, **kw) as d:
yield pathlib.Path(d)
def partition(pred, iterable):
"""
Use a predicate to partition entries into false entries
and true entries
"""Use a predicate to partition entries into false entries and true entries
>>> is_even = lambda i: not i % 2
>>> partition(is_even, range(10))
([0, 2, 4, 6, 8], [1, 3, 5, 7, 9])
@ -731,13 +726,25 @@ def zig_zag(low: ZIG_TYPE, high: ZIG_TYPE) -> Generator[ZIG_TYPE, None, None]:
def walk(path: Path) -> Generator[Path, None, None]:
"""Recursively Walk a directory structure yielding all files and directories."""
"""Recursively walk a directory structure yielding all files and directories."""
if not path.is_dir():
raise ValueError('Can only walk a directory.')
for path in path.iterdir():
yield path
if path.is_dir():
yield from walk(path)
def get_files_and_directories(path: Path):
"""Walk a directory, return a tuple of all files and all directories within. (Not recursive)."""
if not path.is_dir():
raise ValueError('Can only walk a directory.')
directories, files = partition(lambda i: i.is_dir(), path.iterdir())
return files, directories
# These characters are invalid in Windows or Linux.
INVALID_FILE_CHARS = re.compile(r'[/<>:\|"\\\?\*%!]')
@ -793,3 +800,134 @@ async def aiohttp_post(url: str, json_, timeout: int = None) -> Tuple[Dict, int]
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.post(url, json=json_) as response:
return await response.json(), response.status
modelers = []
def register_modeler(modeler: callable):
modelers.append(modeler)
return modeler
def apply_modelers(files, session: Session):
from wrolpi.files.lib import split_path_stem_and_suffix, get_mimetype
if not files:
return
# Group all files by their common name (without the suffix).
groups = {}
for file in files:
if not file.mimetype:
file.mimetype = get_mimetype(file.path)
stem, suffix = split_path_stem_and_suffix(file.path)
try:
groups[stem].append(file)
except KeyError:
groups[stem] = [file, ]
for modeler in modelers:
modeler(groups, session)
session.commit()
for stem, group in groups.items():
# Index any files that were not claimed by modelers.
for file in group:
file.do_index()
after_refresh = []
def register_after_refresh(func: callable):
after_refresh.append(func)
def apply_after_refresh():
for func in after_refresh:
logger.info(f'Applying after-refresh {func.__name__}')
func()
@iterify(tuple)
def match_paths_to_suffixes(paths: List[pathlib.Path], suffix_groups: List[Tuple[str]]):
paths = paths.copy()
for group in suffix_groups:
match = None
for suffix in group:
# Compare all suffixes in the group to the path's name. Yield the first path that shares the first
# suffix. If no suffix matches any path, yield None.
match = next(filter(lambda i: i.name.endswith(suffix), paths), None)
if match:
paths.pop(paths.index(match))
break
yield match
@contextlib.contextmanager
def timer(name):
"""Prints out the time elapsed during the call of some block."""
before = datetime.now()
try:
yield
finally:
logger.warning(f'{name} elapsed {(datetime.now() - before).total_seconds()} seconds')
def limit_concurrent(limit: int):
"""
Wrapper that limits the amount of concurrently running functions.
"""
sema = multiprocessing.Semaphore(value=limit)
def wrapper(func: callable):
if inspect.iscoroutinefunction(func):
@wraps(func)
async def wrapped(*a, **kw):
acquired = sema.acquire(block=False)
if not acquired:
return
try:
return await func(*a, **kw)
finally:
sema.release()
return wrapped
else:
@wraps(func)
def wrapped(*a, **kw):
acquired = sema.acquire(block=False)
if not acquired:
return
try:
return func(*a, **kw)
finally:
sema.release()
return wrapped
return wrapper
async def count_files(path: pathlib.Path) -> int:
return len([i for i in walk(path) if i.is_file()])
def truncate_object_bytes(obj: Union[List[str], str, None], maximum_bytes: int) -> List[str]:
"""
Shorten an object. This is useful when inserting something into a tsvector.
>>> i = ['foo', 'foo', 'foo', 'foo', 'foo', 'foo', 'foo', 'foo', 'foo', 'foo']
>>> truncate_object_bytes(i, 100)
['foo', 'foo', 'foo', 'foo', 'foo']
"""
if not obj:
# Can't decrease an empty object.
return obj
size = sys.getsizeof(obj)
if size < maximum_bytes:
return obj
index = -1 * round(len(obj) * .2)
obj = obj[:index or -1]
return truncate_object_bytes(obj, maximum_bytes)

88
wrolpi/conftest.py
View file

@ -3,21 +3,26 @@ Fixtures for Pytest tests.
"""
import asyncio
import pathlib
import shutil
import tempfile
from typing import List, Callable, Union, Dict
from typing import Tuple, Set
from unittest import mock
from uuid import uuid1
from uuid import uuid1, uuid4
import pytest
from PIL import Image
from sanic_testing.testing import SanicTestClient
from sqlalchemy.engine import Engine, create_engine
from sqlalchemy.orm import Session, sessionmaker
from wrolpi.common import iterify
from wrolpi.common import set_test_media_directory, Base, set_test_config
from wrolpi.dates import set_test_now
from wrolpi.db import postgres_engine, get_db_args
from wrolpi.downloader import DownloadManager, DownloadResult, set_test_download_manager_config, Download
from wrolpi.root_api import BLUEPRINTS, api_app
from wrolpi.vars import PROJECT_DIR
def get_test_db_engine():
@ -131,11 +136,11 @@ async def test_download_manager(
loop = asyncio.get_running_loop()
except RuntimeError:
loop = asyncio.get_event_loop()
manager.start_workers(loop=loop)
manager.enable(loop)
yield manager
manager.cancel_workers()
manager.stop()
@pytest.fixture
@ -164,3 +169,80 @@ def assert_download_urls(test_session):
assert {i.url for i in downloads} == set(urls)
return asserter
@pytest.fixture
def video_file(test_directory) -> pathlib.Path:
"""Return a copy of the example Big Buck Bunny video in the `test_directory`."""
destination = test_directory / f'{uuid4()}.mp4'
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', destination)
yield destination
@pytest.fixture
def image_file(test_directory) -> pathlib.Path:
"""Create a small image file in the `test_directory`."""
destination = test_directory / f'{uuid4()}.jpeg'
Image.new('RGB', (25, 25), color='grey').save(destination)
yield destination
@pytest.fixture
def vtt_file1(test_directory) -> pathlib.Path:
"""Return a copy of the example1 VTT file in the `test_directory`."""
destination = test_directory / f'{uuid4()}.jpeg'
shutil.copy(PROJECT_DIR / 'test/example1.en.vtt', destination)
yield destination
@pytest.fixture
def vtt_file2(test_directory) -> pathlib.Path:
"""Return a copy of the example2 VTT file in the `test_directory`."""
destination = test_directory / f'{uuid4()}.jpeg'
shutil.copy(PROJECT_DIR / 'test/example2.en.vtt', destination)
yield destination
@pytest.fixture
def make_files_structure(test_directory) -> Callable[[Union[List, Dict]], List[pathlib.Path]]:
def create_files(paths: List) -> List[pathlib.Path]:
files = []
@iterify(list)
def touch_paths(paths_):
for name in paths_:
path = test_directory / name
if name.endswith('/'):
path.mkdir()
else:
path.parent.mkdir(parents=True, exist_ok=True)
path.touch()
yield path
if isinstance(paths, list):
files = touch_paths(paths)
elif isinstance(paths, Dict):
files = touch_paths(paths.keys())
for path, content in paths.items():
path = test_directory / path
if isinstance(content, str):
path.write_text(content)
elif isinstance(content, bytes):
path.write_bytes(content)
return files
return create_files
@pytest.fixture
def wrol_mode_fixture(test_config, test_download_manager):
from wrolpi.common import enable_wrol_mode, disable_wrol_mode
def set_wrol_mode(enabled: bool):
if enabled:
enable_wrol_mode(test_download_manager)
else:
disable_wrol_mode(test_download_manager)
return set_wrol_mode

16
wrolpi/db.py
View file

@ -1,10 +1,10 @@
from contextlib import contextmanager
from functools import wraps
from typing import ContextManager, Tuple, List, Union
from typing import ContextManager, Tuple, List, Union, Type
import psycopg2
import sqlalchemy.exc
from sqlalchemy import create_engine
from sqlalchemy import create_engine, inspect
from sqlalchemy.orm import sessionmaker, Session
from sqlalchemy.pool import NullPool
@ -156,10 +156,10 @@ def optional_session(commit: Union[callable, bool] = False):
@optional_session
def get_ranked_models(ranked_ids: List[int], model: Base, session: Session = None) -> List[Base]:
"""
Get all objects whose ids are in the `ranked_ids`, order them by their position in `ranked_ids`.
"""
results = session.query(model).filter(model.id.in_(ranked_ids)).all()
results = sorted(results, key=lambda i: ranked_ids.index(i.id))
def get_ranked_models(ranked_ids: List, model: Type[Base], session: Session = None) -> List[Base]:
"""Get all objects whose ids are in the `ranked_ids`, order them by their position in `ranked_ids`."""
pkey = inspect(model).primary_key[0]
pkey_name = pkey.name
results = list(session.query(model).filter(pkey.in_(ranked_ids)).all())
results = sorted(results, key=lambda i: ranked_ids.index(getattr(i, pkey_name)))
return results

127
wrolpi/downloader.py
View file

@ -1,14 +1,16 @@
import asyncio
import multiprocessing
import os
import pathlib
import traceback
from abc import ABC
from asyncio import Queue, Task, QueueEmpty
from asyncio import Task
from dataclasses import dataclass, field
from datetime import timedelta, datetime
from enum import Enum
from functools import partial
from operator import attrgetter
from queue import Empty
from typing import List, Dict, Generator
from typing import Tuple, Optional
from urllib.parse import urlparse
@ -48,7 +50,7 @@ class DownloadResult:
success: bool = False
class Download(ModelHelper, Base):
class Download(ModelHelper, Base): # noqa
"""Model that is used to schedule downloads."""
__tablename__ = 'download' # noqa
id = Column(Integer, primary_key=True)
@ -178,7 +180,7 @@ class Downloader:
raise NotImplementedError()
@optional_session
def already_downloaded(self, url: str, session: Session = None):
def already_downloaded(self, urls: List[str], session: Session = None):
raise NotImplementedError()
@property
@ -275,10 +277,10 @@ class DownloadManager:
self.instances: Tuple[Downloader] = tuple()
self._instances = dict()
self.disabled = multiprocessing.Event()
self.stopped: bool = False
self.download_queue: Queue = Queue()
self.stopped = multiprocessing.Event()
self.download_queue: multiprocessing.Queue = multiprocessing.Queue()
self.workers: List[Task] = []
self.worker_count: int = 4
self.worker_count: int = 1
self.data = multiprocessing.Manager().dict()
# We haven't started downloads yet, so no domains are downloading.
self.data['processing_domains'] = []
@ -295,32 +297,47 @@ class DownloadManager:
self.instances = tuple(self.priority_sorter(i))
self._instances[instance.name] = instance
async def download_worker(self, num: int, queue: Queue):
async def download_worker(self, num: int):
"""Fetch a download from the queue, perform the download then store the results.
Calls DownloadManger.start_downloads() after a download completes.
"""
logger.debug(f'DownloadManager.download_worker {num} starting up')
from wrolpi.db import get_db_session
pid = os.getpid()
worker_logger = logger.getChild(f'download_worker.{pid}.{num}')
status = 'disabled' if self.disabled.is_set() else 'enabled'
worker_logger.info(f'Starting up. DownloadManager is {status}.')
while True:
if self.stopped:
if self.stopped.is_set():
# Service is restarting, close the worker.
return
try:
download: Download = queue.get_nowait()
download_id, url = download.id, download.url
logger.debug(f'Download worker {num} got download {download}')
disabled = self.disabled.is_set()
downloader: Downloader = download.get_downloader()
if not downloader:
logger.warning(f'Could not find downloader for {download.downloader=}')
if disabled:
# Downloading is disabled, wait for it to enable.
await asyncio.sleep(1)
continue
try:
download_id, url = self.download_queue.get_nowait()
worker_logger.debug(f'Got download {download_id}')
with get_db_session(commit=True) as session:
# Mark the download as started in new session so the change is committed.
download = session.query(Download).filter_by(id=download_id).one()
download.started()
# Set the Download's manager. Testing will not use the global manager.
download.manager = self
downloader: Downloader = download.get_downloader()
if not downloader:
worker_logger.warning(f'Could not find downloader for {download.downloader=}')
self.data['processing_domains'].append(download.domain)
try_again = True
@ -331,15 +348,15 @@ class DownloadManager:
result = downloader.do_download(download)
except UnrecoverableDownloadError as e:
# Download failed and should not be retried.
logger.warning(f'UnrecoverableDownloadError for {url}', exc_info=e)
worker_logger.warning(f'UnrecoverableDownloadError for {url}', exc_info=e)
result = DownloadResult(success=False, error=str(traceback.format_exc()))
try_again = False
except Exception as e:
logger.warning(f'Failed to download {url}. Will be tried again later.', exc_info=e)
worker_logger.warning(f'Failed to download {url}. Will be tried again later.', exc_info=e)
result = DownloadResult(success=False, error=str(traceback.format_exc()))
error_len = len(result.error) if result.error else 0
logger.debug(f'Got success={result.success} from {downloader} with {error_len=}')
worker_logger.debug(f'Got success={result.success} from {downloader} with {error_len=}')
with get_db_session(commit=True) as session:
# Modify the download in a new session because downloads may take a long time.
@ -352,7 +369,7 @@ class DownloadManager:
download.next_download = self.calculate_next_download(download, session)
if result.downloads:
logger.info(f'Adding {len(result.downloads)} downloads from result of {download.url}')
worker_logger.info(f'Adding {len(result.downloads)} downloads from result of {download.url}')
self.create_downloads(result.downloads, session,
downloader=download.sub_downloader)
@ -364,21 +381,20 @@ class DownloadManager:
else:
download.defer()
queue.task_done()
# Remove this domain from the running list.
self._remove_domain(download.domain)
# Request any new downloads be added to the queue.
asyncio.create_task(self.queue_downloads())
except asyncio.CancelledError:
logger.warning(f'download_worker canceled!')
queue.task_done()
worker_logger.warning('Canceled!')
self.download_queue.task_done()
return
except QueueEmpty:
except Empty:
# No work yet.
await asyncio.sleep(0.1)
continue
except Exception as e:
logger.warning(f'Download worker had unexpected error', exc_info=e)
worker_logger.warning(f'Unexpected error', exc_info=e)
def _add_domain(self, domain: str):
"""Add a domain to the processing list.
@ -404,9 +420,10 @@ class DownloadManager:
# Loop isn't running, start one. Probably testing.
loop = asyncio.get_event_loop()
logger.debug(f'Starting {self.worker_count} download workers')
pid = os.getpid()
logger.warning(f'Starting {self.worker_count} download workers. {pid=}')
for i in range(self.worker_count):
coro = self.download_worker(i, self.download_queue)
coro = self.download_worker(i)
task = loop.create_task(coro)
self.workers.append(task)
@ -522,8 +539,9 @@ class DownloadManager:
async def queue_downloads(self, session: Session = None):
"""Put all downloads in queue. Will only queue downloads if there are workers to take them. Each worker
only receives one domain, this is to prevent downloading from one domain many times at once."""
if self.disabled.is_set():
raise InvalidDownload('DownloadManager is disabled')
if self.disabled.is_set() or self.stopped.is_set():
# Don't queue downloads when disabled.
return
if len(self.data['processing_domains']) >= len(self.workers):
return
@ -532,14 +550,17 @@ class DownloadManager:
new_downloads = session.query(Download).filter(
Download.status == 'new',
Download.domain not in self.data['processing_domains'],
).order_by(Download.id) # noqa
).order_by(
Download.frequency.is_(None),
Download.frequency,
Download.id) # noqa
count = 0
for download in new_downloads:
download.manager = self # Assign this Download to this manager.
domain = download.domain
if domain not in self.data['processing_domains']:
self._add_domain(domain)
await self.download_queue.put(download)
self.download_queue.put((download.id, download.url))
count += 1
if count:
logger.debug(f'Added {count} downloads to queue.')
@ -586,7 +607,6 @@ class DownloadManager:
if self.download_queue.empty():
# Queue is empty.
break
await self.download_queue.join()
@staticmethod
def reset_downloads():
@ -698,32 +718,37 @@ class DownloadManager:
downloader.kill()
download.fail()
def stop(self):
"""Stop all Downloaders, defer all pending downloads."""
logger.warning('Stopping all downloads')
self.stopped = True
def disable(self):
"""Stop all downloads and downloaders. Workers will stay idle."""
logger.info('Disabling downloads and downloaders.')
self.disabled.set()
for downloader in self.instances:
downloader.kill()
for download in self.get_pending_downloads():
download.defer()
self.cancel_workers()
def kill(self):
"""Kill all downloads. Do not start new downloads."""
self.disabled.set()
try:
self.stop()
except Exception as e:
logger.critical(f'Failed to kill downloads!', exc_info=e)
def stop(self):
"""Stop all downloads, downloaders and workers, defer all pending downloads."""
logger.warning('Stopping all workers')
self.stopped.set()
self.disable()
def enable(self):
"""Enable downloading. Start downloading."""
def enable(self, loop=None):
"""Enable downloading. Start downloading. Start workers."""
logger.info('Enabling downloading')
for downloader in self.instances:
downloader.clear()
self.stopped.clear()
self.disabled.clear()
self.start_workers()
asyncio.create_task(self.do_downloads())
self.start_workers(loop)
try:
asyncio.create_task(self.do_downloads())
except RuntimeError:
# This may not work while testing.
if not PYTEST:
raise
FINISHED_STATUSES = ('complete', 'failed')
@ -807,6 +832,14 @@ class DownloadManager:
)
return data
@staticmethod
def get_summary():
with get_db_curs() as curs:
stmt = """SELECT COUNT(*) AS pending_downloads FROM download WHERE status='pending'"""
curs.execute(stmt)
result = [dict(i) for i in curs.fetchall()][0]
return result
@optional_session
def get_pending_downloads(self, session: Session) -> List[Download]:
downloads = session.query(Download).filter_by(status='pending').all()

9
wrolpi/errors.py
View file

@ -148,6 +148,10 @@ class HotspotError(APIError):
pass
class UnknownArchive(APIError):
pass
error_codes = iter(range(1, 1000))
API_ERRORS = {
@ -325,5 +329,10 @@ API_ERRORS = {
'code': next(error_codes),
'message': 'Updating/accessing Hotspot encountered an error',
'status': HTTPStatus.INTERNAL_SERVER_ERROR,
},
UnknownArchive: {
'code': next(error_codes),
'message': 'Unable to find the archive',
'status': HTTPStatus.NOT_FOUND,
}
}

0
modules/files/__init__.py → wrolpi/files/__init__.py
View file

24
modules/files/api.py → wrolpi/files/api.py
View file

@ -1,3 +1,4 @@
import asyncio
import pathlib
from typing import List
from urllib.request import Request
@ -6,10 +7,11 @@ from sanic import response
from sanic_ext import validate
from sanic_ext.extensions.openapi import openapi
from wrolpi.common import get_media_directory, api_param_limiter
from wrolpi.common import get_media_directory, api_param_limiter, wrol_mode_check
from wrolpi.errors import InvalidFile
from wrolpi.root_api import get_blueprint, json_response
from . import lib, schema
from ..vars import PYTEST
bp = get_blueprint('Files', '/api/files')
@ -74,9 +76,23 @@ files_offset_limiter = api_param_limiter(100, 0)
@bp.post('/refresh')
@openapi.description('Find and index all files')
@openapi.description('Find and index all files in the media directory.')
@wrol_mode_check
async def refresh(_: Request):
lib.refresh_files()
await lib.refresh_files()
return response.empty()
@bp.post('/refresh/directory')
@openapi.description('Find and index all files in the provided directory.')
@validate(schema.DirectoryRefreshRequest)
@wrol_mode_check
async def refresh(_: Request, body: schema.DirectoryRefreshRequest):
directory = get_media_directory() / body.directory
if PYTEST:
await lib.refresh_directory_files_recursively(directory)
else:
asyncio.create_task(lib.refresh_directory_files_recursively(directory))
return response.empty()
@ -87,5 +103,5 @@ async def refresh(_: Request):
)
@validate(schema.FilesSearchRequest)
async def search_files(_: Request, body: schema.FilesSearchRequest):
files, total = lib.search(body.search_str, body.limit, body.offset)
files, total = lib.file_search(body.search_str, body.limit, body.offset, body.mimetype, body.model)
return json_response(dict(files=files, totals=dict(files=total)))

129
wrolpi/files/indexers.py Normal file
View file

@ -0,0 +1,129 @@
import pathlib
import re
from abc import ABC
from collections import defaultdict
from typing import List, Type, Tuple
from zipfile import ZipFile
from wrolpi.common import logger
logger = logger.getChild(__name__)
__all__ = ['Indexer', 'DefaultIndexer', 'ZipIndexer', 'TextIndexer', 'find_indexer', 'register_indexer',
'MAX_TEXT_FILE_BYTES']
class Indexer(object):
@staticmethod
def get_title(file):
from wrolpi.files.lib import split_path_stem_and_suffix
path = file.path.path if hasattr(file.path, 'path') else file.path
return split_path_stem_and_suffix(path)
@classmethod
def create_index(cls, file) -> Tuple:
# All files can have their title as the highest priority search.
a = cls.get_title(file)
return a, None, None, None
@classmethod
def detect_special_indexer(cls, file):
return cls
class DefaultIndexer(Indexer, ABC):
"""If this Indexer is used, it is because we could not match the file to a more specific Indexer."""
pass
indexer_map = defaultdict(lambda: DefaultIndexer)
def find_indexer(file) -> Type[Indexer]:
"""Find the Indexer for a given File model."""
if not file.mimetype:
return DefaultIndexer
if file.mimetype in indexer_map:
# Use the specific indexer.
indexer = indexer_map[file.mimetype]
else:
# Use the broad indexer.
indexer = indexer_map[file.mimetype.split('/')[0]]
indexer = indexer.detect_special_indexer(file)
return indexer
def register_indexer(*mimetypes: str):
"""Register an Indexer for a file mimetype."""
def wrapper(indexer: Indexer):
for mimetype in mimetypes:
if mimetype in indexer_map:
raise KeyError(f'Cannot register indexer. {mimetype} is already registered.')
indexer_map[mimetype] = indexer
return indexer
return wrapper
@register_indexer('application/gzip', 'application/x-bzip2', 'application/zip')
class ZipIndexer(Indexer, ABC):
"""Handles archive files lik zip/gzip/bzip."""
@classmethod
def create_index(cls, file):
a = cls.get_title(file)
file_names = cls.get_file_names(file)
return a, None, None, file_names
@classmethod
def get_file_names(cls, file) -> List[str]:
"""Return all the names of the files in the zip file."""
path = file.path.path if hasattr(file.path, 'path') else file.path
try:
with ZipFile(path, 'r') as zip_:
file_names = [pathlib.Path(name).name for name in zip_.namelist()]
return file_names
except Exception as e:
logger.error(f'Unable to get information from zip: {path}', exc_info=e)
WORD_SPLITTER = re.compile(r"([\w’'-]+)")
MAX_TEXT_FILE_BYTES = 100_000
@register_indexer('text/plain')
class TextIndexer(Indexer, ABC):
"""Handles plain text files.
Detects VTT (caption) files and forwards them on."""
@classmethod
def create_index(cls, file):
path = file.path.path if hasattr(file.path, 'path') else file.path
a = cls.get_title(file)
words = cls.get_words(path)
return a, None, None, words
@staticmethod
def get_words(path: pathlib.Path) -> List[str]:
"""Read the words from a file."""
# TODO this only supports English.
contents = path.read_text()
words = WORD_SPLITTER.findall(contents)
return words
@classmethod
def detect_special_indexer(cls, file):
if file.path.name.endswith('.json'):
# JSON files are detected as "text/plain" but we don't want them to be indexed.
return DefaultIndexer
if file.path.name.endswith('.vtt') or file.path.name.endswith('.srt'):
# Don't process caption files.
return DefaultIndexer
return cls

404
wrolpi/files/lib.py Normal file
View file

@ -0,0 +1,404 @@
import asyncio
import datetime
import glob
import pathlib
import re
import subprocess
from functools import wraps
from pathlib import Path
from typing import List, Tuple, Union
import psycopg2
from sqlalchemy.orm import Session
from wrolpi.cmd import which
from wrolpi.common import get_media_directory, wrol_mode_check, logger, limit_concurrent, \
get_files_and_directories, apply_modelers, apply_after_refresh, get_model_by_table_name
from wrolpi.dates import now
from wrolpi.db import get_db_session, get_db_curs, get_ranked_models
from wrolpi.errors import InvalidFile
from wrolpi.files.models import File
from wrolpi.vars import PYTEST
try:
import magic
mime = magic.Magic(mime=True)
no_magic = False
except ImportError:
# Magic is not installed
no_magic = True
logger = logger.getChild(__name__)
__all__ = ['list_files', 'delete_file', 'split_path_stem_and_suffix', 'refresh_files', 'file_search', 'get_mimetype']
def filter_parent_directories(directories: List[Path]) -> List[Path]:
"""
Remove parent directories if their children are in the list.
>>> filter_parent_directories([Path('foo'), Path('foo/bar'), Path('baz')])
[Path('foo/bar'), Path('baz')]
"""
unique_children = set()
for directory in sorted(directories):
for parent in directory.parents:
# Remove any parent of this child.
if parent in unique_children:
unique_children.remove(parent)
unique_children.add(directory)
# Restore the original order.
new_directories = [i for i in directories if i in unique_children]
return new_directories
def list_files(directories: List[str]) -> List[Path]:
"""List all files down to the directories provided. This includes all parent directories of the directories."""
media_directory = get_media_directory()
# Always display the media_directory files.
paths = list(media_directory.iterdir())
if directories:
directories = [media_directory / i for i in directories if i]
directories = filter_parent_directories(directories)
for directory in directories:
directory = media_directory / directory
for parent in directory.parents:
is_relative_to = str(media_directory).startswith(str(parent))
if parent == Path('') or is_relative_to:
continue
paths.extend(parent.iterdir())
paths.extend(directory.iterdir())
return paths
@wrol_mode_check
def delete_file(file: str):
"""Delete a file in the media directory."""
file = get_media_directory() / file
if file.is_dir() or not file.is_file():
raise InvalidFile(f'Invalid file {file}')
file.unlink()
FILE_NAME_REGEX = re.compile(r'[_ .]')
def split_file_name_into_words(path: Path) -> List[str]:
"""
Split a file name into words.
>>> split_path_stem_and_suffix(Path('foo.mp4'))
['foo']
>>> split_path_stem_and_suffix(Path('foo bar.mp4'))
['foo', 'bar']
"""
words = [str(i) for i in FILE_NAME_REGEX.split(path.stem)]
return words
FILE_BIN = which('file', '/usr/bin/file')
def get_mimetype(path: Path) -> str:
"""Get the mimetype of a file, prefer using `magic`, fallback to builtin `file` command."""
if no_magic:
# This method is slow, prefer the speedier `magic` module.
cmd = (FILE_BIN, '--mime-type', str(path.absolute()))
output = subprocess.check_output(cmd)
output = output.decode()
mimetype = output.split(' ')[-1].strip()
return mimetype
else:
return mime.from_file(path)
# Special suffixes within WROLPi.
SUFFIXES = {
'.info.json',
'.en.vtt',
'.en.srt',
'.readability.json',
'.readability.txt',
'.readability.html',
}
def split_path_stem_and_suffix(path: Union[pathlib.Path, str]) -> Tuple[str, str]:
"""Get the path's stem and suffix.
This function handles WROLPi suffixes like .info.json."""
if isinstance(path, str):
path = pathlib.Path(path)
full = str(path) # May or may not be absolute.
suffix = next(filter(lambda i: full.endswith(i), SUFFIXES), path.suffix)
if suffix:
return path.name[:-1 * len(suffix)], suffix
# Path has no suffix.
return path.name, ''
@wrol_mode_check
def _refresh_files_list(paths: List[pathlib.Path], idempotency: datetime.datetime):
"""Upsert, index, and model all provided paths."""
# Update the idempotency if any files already in the DB.
if not paths:
return
paths = set(paths)
with get_db_session(commit=True) as session:
existing_files = list(session.query(File).filter(File.path.in_(paths)))
for existing_file in existing_files:
existing_file.idempotency = idempotency
session.commit()
existing_paths = {i.path for i in existing_files}
# Add any new files into the database.
new_files = []
if new_paths := (paths - existing_paths):
new_files = [File(path=i, idempotency=idempotency) for i in new_paths]
session.add_all(new_files)
# Commit before modeling. Just in case modeling fails.
session.commit()
# Apply models to all files. Many files will have no model.
apply_modelers(existing_files + new_files, session)
refresh_logger = logger.getChild('refresh')
async def _refresh_directory_files_recursively(directory, idempotency):
"""
Refresh all files in a directory, recursively refresh all files in subdirectories.
"""
directories = [directory, ]
while directories:
directory = directories.pop(0)
try:
files, new_directories = get_files_and_directories(directory)
refresh_logger.debug(f'Refreshing {len(files)} files in {directory}')
directories.extend(new_directories)
_refresh_files_list(files, idempotency)
except Exception as e:
logger.error(f'Failed to refresh files in {directory}', exc_info=e)
@limit_concurrent(1) # Only one refresh at a time.
async def _refresh_all_files():
"""Find, model, and index all files in the media directory."""
refresh_logger.warning('Refreshing all files')
# TODO remove this later when everyone has migrated their files.
from modules.archive.lib import migrate_archive_files
migrate_archive_files()
idempotency = now()
# Add all files in the media directory to the DB.
await _refresh_directory_files_recursively(get_media_directory(), idempotency)
# Remove any records where the file no longer exists.
with get_db_curs(commit=True) as curs:
curs.execute('DELETE FROM file WHERE idempotency < %s OR idempotency is null RETURNING path', (idempotency,))
deleted = list(curs.fetchall())
logger.warning(f'{deleted=}')
logger.warning(f'Removed {len(deleted)} missing files')
apply_after_refresh()
refresh_logger.warning('Done refreshing Files')
@wraps(_refresh_all_files)
@wrol_mode_check
async def refresh_files():
"""Schedule a refresh task if not testing. If testing, do a synchronous refresh."""
if PYTEST:
return await _refresh_all_files()
asyncio.create_task(_refresh_all_files())
@limit_concurrent(1)
@wrol_mode_check
async def refresh_directory_files_recursively(directory: Union[pathlib.Path, str]):
"""Upsert and index all files within a directory (recursively).
Any records of the files that are no longer in the directory will be removed."""
if isinstance(directory, str):
directory = pathlib.Path(directory)
if not directory.is_dir():
raise ValueError(f'Cannot refresh files of a file: {directory=}')
# All Files older than this will be removed.
idempotency = now()
refresh_logger.warning(f'Recursively refreshing all files in {directory}')
await _refresh_directory_files_recursively(directory, idempotency)
# Remove any records where the file no longer exists.
with get_db_curs(commit=True) as curs:
refresh_logger.debug(f'Deleting files no longer in {directory}')
params = dict(
directory=f'{directory}/%', # trailing / is important!
idempotency=idempotency,
)
stmt = '''
DELETE FROM file
WHERE
(idempotency < %(idempotency)s OR idempotency IS NULL)
AND path LIKE %(directory)s'''
curs.execute(stmt, params)
apply_after_refresh()
refresh_logger.info(f'Done refreshing files in {directory}')
def file_search(search_str: str, limit: int, offset: int, mimetype: str = None, model: str = None) -> Tuple[List[dict], int]:
"""
Search the Files table. Order the returned Files by their rank if `search_str` is provided. Return all files if
`search_str` is empty.
Parameters:
search_str: Search the ts_vector of the file. Returns all files if this is empty.
limit: Return only this many files.
offset: Offset the query.
mimetype: Only return files that match this mimetype.
model: Only return files that match this model.
"""
params = dict(offset=offset, limit=limit)
wheres = []
selects = []
order_by = 'associated, 1 ASC'
if search_str:
params['search_str'] = search_str
wheres.append('textsearch @@ websearch_to_tsquery(%(search_str)s)')
selects.append('ts_rank(textsearch, websearch_to_tsquery(%(search_str)s))')
order_by = 'associated, 2 DESC'
if mimetype and len(mimetype.split('/')) == 1:
params['mimetype'] = f'{mimetype}/%'
wheres.append('mimetype LIKE %(mimetype)s')
elif mimetype:
wheres.append('mimetype = %(mimetype)s')
params['mimetype'] = mimetype
if model:
params['model'] = model
wheres.append('model = %(model)s')
wheres = '\n AND '.join(wheres)
selects = f"{', '.join(selects)}, " if selects else ""
stmt = f'''
SELECT path, {selects} COUNT(*) OVER() AS total
FROM file
{f"WHERE {wheres}" if wheres else ""}
ORDER BY {order_by}
OFFSET %(offset)s LIMIT %(limit)s
'''
results, total = handle_search_results(stmt, params)
return results, total
def handle_search_results(statement: str, params):
"""
Execute the provided SQL statement and fetch the Files returned.
WARNING: This expects specific queries to be executed and shouldn't be used for things not related to search.
See: `search`
"""
with get_db_curs() as curs:
curs.execute(statement, params)
try:
results = [dict(i) for i in curs.fetchall()]
except psycopg2.ProgrammingError:
# No videos
return [], 0
total = results[0]['total'] if results else 0
ranked_paths = [pathlib.Path(i['path']) for i in results]
with get_db_session() as session:
results = get_ranked_models(ranked_paths, File, session=session)
try:
results = assign_file_prefetched_models(results)
except Exception as e:
logger.error(f'Failed to assign file prefetched models', exc_info=e)
if PYTEST:
raise
results = [i.__json__() for i in results]
return results, total
def glob_shared_stem(path: pathlib.Path) -> List[pathlib.Path]:
"""Return all paths that share the same stem and parent path as the provided path.
Example paths: foo.mp4, foo.png, foo.info.json, foobar.txt
>>> glob_shared_stem(pathlib.Path('foo.mp4'))
['foo.mp4', 'foo.png', 'foo.info.json']
"""
if isinstance(path, str):
path = pathlib.Path(path)
stem, suffix = split_path_stem_and_suffix(path)
escaped_stem = glob.escape(stem)
paths = [pathlib.Path(i) for i in path.parent.glob(f'{escaped_stem}*') if
split_path_stem_and_suffix(i)[0] == stem]
return paths
def assign_file_prefetched_models(files: List[File]) -> List[File]:
"""
Prefetch the sub-models (matching `File.model`) in bulk to avoid issuing a query for every File.
>>> f = File(path='something', model='video')
>>> v = Video(video_path='something')
>>> results = assign_file_prefetched_models([f])
>>> results[0]
File(path='something', model='video')
>>> results[0].prefetched_model == v
True
"""
if not files:
return files
files = files.copy()
files_index = {i.path: files.index(i) for i in files}
session = Session.object_session(files[0])
files_by_model = dict()
for file in files:
if not file.model:
continue
try:
files_by_model[file.model].append(file)
except KeyError:
files_by_model[file.model] = [file, ]
logger.debug(f'Prefetching file models: {list(files_by_model.keys())}')
table_map = {i: get_model_by_table_name(i) for i in {j.model for j in files}}
for model, model_files in files_by_model.items():
table = table_map[model]
try:
sub_models = table.find_by_paths([i.path for i in model_files], session)
for sub_model in sub_models:
file: File = files[files_index[sub_model.primary_path]]
file.prefetched_model = sub_model
except NotImplementedError:
# Model has not defined the necessary methods.
pass
return files

137
wrolpi/files/models.py Normal file
View file

@ -0,0 +1,137 @@
import pathlib
from sqlalchemy import Integer, Column, String, Computed, Boolean
from sqlalchemy.orm import deferred, Session, validates
import wrolpi.files.indexers
from wrolpi.common import Base, ModelHelper, get_media_directory, tsvector, logger, get_model_by_table_name, \
truncate_object_bytes
from wrolpi.dates import TZDateTime, from_timestamp, now
from wrolpi.files.indexers import MAX_TEXT_FILE_BYTES
from wrolpi.media_path import MediaPathType
logger = logger.getChild(__name__)
class File(ModelHelper, Base):
"""A representation of a file on disk.
Can be searched using `textsearch` once the data has been populated by the respective Indexer.
"""
__tablename__ = 'file'
path: pathlib.Path = Column(MediaPathType, primary_key=True)
associated = Column(Boolean, default=False)
idempotency = Column(TZDateTime, default=lambda: now())
indexed = Column(Boolean, default=False)
mimetype = Column(String)
model = Column(String)
modification_datetime = Column(TZDateTime)
size = Column(Integer)
suffix = Column(String)
title = Column(String)
a_text = deferred(Column(String))
b_text = deferred(Column(String))
c_text = deferred(Column(String))
d_text = deferred(Column(String))
textsearch = deferred(
Column(tsvector, Computed('''
setweight(to_tsvector('english'::regconfig, COALESCE(a_text, '')), 'A'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(b_text, '')), 'B'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(c_text, '')), 'C'::"char") ||
setweight(to_tsvector('english'::regconfig, COALESCE(d_text, '')), 'D'::"char")
''')))
prefetched_model = None # This may be filled with record matching `model.
def __repr__(self):
path = str(self.path.relative_to(get_media_directory()))
return f'<File {path=} mime={self.mimetype} model={self.model}>'
def __json__(self):
path = self.path.relative_to(get_media_directory())
d = dict(
associated=self.associated,
key=path, # React browser expects this.
mimetype=self.mimetype,
model=self.model,
modified=self.modification_datetime, # React browser expects this.
path=path,
size=self.size,
suffix=self.suffix,
title=self.title,
)
if self.model and self.prefetched_model:
d[self.model] = self.prefetched_model.__json__()
elif self.model:
model: ModelHelper = get_model_by_table_name(self.model)
session = Session.object_session(self)
instance = model.find_by_path(self.path, session)
if not instance:
logger.warning(f'Could not find instance of {model} for {self.path}')
return d
d[self.model] = instance.__json__()
return d
def __eq__(self, other):
if isinstance(other, File):
return other.path == self.path
if isinstance(other, pathlib.Path):
return other == self.path
if isinstance(other, str):
return other == str(self.path)
return False
@property
def indexer(self):
from wrolpi.files import lib
return wrolpi.files.indexers.find_indexer(self)
def do_index(self, force_index: bool = False):
"""Gather any missing information about this file. Index the contents of this file using an Indexer."""
from wrolpi.files.lib import split_path_stem_and_suffix
if not self.mimetype:
from wrolpi.files.lib import get_mimetype
self.mimetype = get_mimetype(self.path)
self.title = self.title or self.path.name
_, self.suffix = split_path_stem_and_suffix(self.path)
stat = self.path.stat()
self.size = stat.st_size
self.modification_datetime = from_timestamp(stat.st_mtime)
try:
if force_index or self.indexed is not True:
# Only read the contents of the file once.
self.a_text, self.b_text, self.c_text, self.d_text = self.indexer.create_index(self)
self.a_text = truncate_object_bytes(self.a_text, MAX_TEXT_FILE_BYTES)
self.b_text = truncate_object_bytes(self.b_text, MAX_TEXT_FILE_BYTES)
self.c_text = truncate_object_bytes(self.c_text, MAX_TEXT_FILE_BYTES)
self.d_text = truncate_object_bytes(self.d_text, MAX_TEXT_FILE_BYTES)
self.indexed = True
except Exception as e:
logger.error(f'Failed to index {self.path}', exc_info=e)
@validates('path')
def validate_path(self, key, value):
value = pathlib.Path(value) if not isinstance(value, pathlib.Path) else value
if not value.is_absolute():
raise ValueError(f'File path must always be absolute! {value}')
if not str(value).startswith(str(get_media_directory())):
raise ValueError(f'File path {value} not in {get_media_directory()}')
return value
@classmethod
def upsert(cls, path, session: Session) -> Base:
if file := session.query(File).filter_by(path=path).one_or_none():
return file
file = File(path=path)
session.add(file)
return file

9
modules/files/schema.py → wrolpi/files/schema.py
View file

@ -14,6 +14,13 @@ class DeleteRequest:
@dataclass
class FilesSearchRequest:
search_str: str
search_str: Optional[str] = None
limit: Optional[int] = 20
offset: Optional[int] = 0
mimetype: Optional[str] = None
model: Optional[str] = None
@dataclass
class DirectoryRefreshRequest:
directory: str = None

0
modules/files/test/__init__.py → wrolpi/files/test/__init__.py
View file

77
modules/files/test/test_api.py → wrolpi/files/test/test_api.py
View file

@ -6,6 +6,7 @@ from unittest import mock
import pytest
from wrolpi.errors import API_ERRORS, WROLModeEnabled
from wrolpi.files.models import File
from wrolpi.test.common import assert_dict_contains
from wrolpi.vars import PROJECT_DIR
@ -102,20 +103,19 @@ def test_delete_file(test_client, make_files_structure, test_directory):
]
)
def test_delete_invalid_file(test_client, file):
with mock.patch('modules.files.api.lib.delete_file') as mock_delete_file:
with mock.patch('wrolpi.files.api.lib.delete_file') as mock_delete_file:
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': file}))
assert response.status_code == HTTPStatus.BAD_REQUEST
mock_delete_file.assert_not_called()
def test_delete_wrol_mode(test_client):
"""
Can't delete a file when WROL Mode is enabled.
"""
with mock.patch('wrolpi.common.wrol_mode_enabled', lambda: True):
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'foo'}))
assert response.status_code == HTTPStatus.FORBIDDEN
assert response.json['code'] == API_ERRORS[WROLModeEnabled]['code']
def test_delete_wrol_mode(test_client, wrol_mode_fixture):
"""Can't delete a file when WROL Mode is enabled."""
wrol_mode_fixture(True)
request, response = test_client.post('/api/files/delete', content=json.dumps({'file': 'foo'}))
assert response.status_code == HTTPStatus.FORBIDDEN
assert response.json['code'] == API_ERRORS[WROLModeEnabled]['code']
def do_search(test_client, search_str, total, expected):
@ -152,8 +152,61 @@ def test_files_search(test_session, test_client, make_files_structure):
do_search(test_client, 'foo', 1, [dict(path='foo_is_the_name.txt', mimetype='text/plain', size=12)])
do_search(test_client, 'bar', 1, [dict(path='archives/bar.txt', mimetype='text/plain', size=16)])
do_search(test_client, 'baz', 2, [
dict(path='baz.mp4', mimetype='video/mp4', size=1055736),
dict(path='baz baz two.mp4', mimetype='video/mp4', size=1055736),
dict(path='baz baz two.mp4', mimetype='video/mp4', size=1056318),
dict(path='baz.mp4', mimetype='video/mp4', size=1056318),
])
do_search(test_client, 'two', 1, [dict(path='baz baz two.mp4', mimetype='video/mp4', size=1055736)])
do_search(test_client, 'two', 1, [dict(path='baz baz two.mp4', mimetype='video/mp4', size=1056318)])
do_search(test_client, 'nothing', 0, [])
# No search string returns all Files.
do_search(test_client, None, 4, [
dict(path='archives/bar.txt'),
dict(path='baz baz two.mp4'),
dict(path='baz.mp4'),
dict(path='foo_is_the_name.txt'),
])
def test_associated_files(test_session, test_client, make_files_structure):
mp4, png, j, txt = make_files_structure([
'video.mp4',
'video.png',
'video.info.json',
'not a video.txt',
])
test_client.post('/api/files/refresh')
mp4_file = test_session.query(File).filter_by(path=mp4).one()
png_file = test_session.query(File).filter_by(path=png).one()
j_file = test_session.query(File).filter_by(path=j).one()
txt_file = test_session.query(File).filter_by(path=txt).one()
def check_files_search(content, expected):
request, response = test_client.post('/api/files/search', content=json.dumps(content))
assert response.status_code == HTTPStatus.OK
assert [{'path': i['path']} for i in response.json['files']] == expected
check_files_search(
{'search_str': 'video'},
[
{'path': 'not a video.txt'},
{'path': 'video.info.json'},
{'path': 'video.mp4'},
{'path': 'video.png'},
]
)
mp4_file.associated = False # the modeled file
png_file.associated = True
j_file.associated = True
test_session.commit()
# Associated files are last.
check_files_search(
{'search_str': 'video'},
[
{'path': 'not a video.txt'},
{'path': 'video.mp4'},
{'path': 'video.info.json'},
{'path': 'video.png'},
]
)

366
wrolpi/files/test/test_lib.py Normal file
View file

@ -0,0 +1,366 @@
import json
import shutil
import zipfile
from pathlib import Path
from typing import List
import mock
import pytest
from PIL import Image
from sqlalchemy.orm import Session
from modules import videos
from wrolpi.common import get_media_directory
from wrolpi.errors import InvalidFile
from wrolpi.files import lib, indexers
from wrolpi.files.models import File
from wrolpi.vars import PROJECT_DIR
def assert_files(session: Session, expected):
files = {str(i.path.relative_to(get_media_directory())) for i in session.query(File).all()}
assert files == set(expected)
def test_list_files(make_files_structure, test_directory):
files = [
'archives/bar.txt',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory/',
'videos/other video.mp4',
'videos/some video.mp4',
]
make_files_structure(files)
def check_list_files(path, expected):
result = lib.list_files(path)
result = sorted(result)
assert [str(i.relative_to(test_directory)) for i in result] == expected
check_list_files([], ['archives', 'empty directory', 'videos'])
# Falsey directories are ignored.
check_list_files([None, ''], ['archives', 'empty directory', 'videos'])
check_list_files(['empty directory'], ['archives', 'empty directory', 'videos'])
check_list_files(['empty directory'], ['archives', 'empty directory', 'videos'])
# Trailing slash is ignored.
check_list_files(['empty directory/'], ['archives', 'empty directory', 'videos'])
check_list_files(['archives'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/foo.txt',
'empty directory',
'videos',
])
check_list_files(['archives/baz'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
])
# Including duplicate directories does not change results.
check_list_files(['archives/baz', 'archives'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
])
check_list_files(['videos'], [
'archives',
'empty directory',
'videos',
'videos/other video.mp4',
'videos/some video.mp4',
])
# All files and directories are listed.
check_list_files(['videos', 'archives/baz'], [
'archives',
'archives/bar.txt',
'archives/baz',
'archives/baz/bar.txt',
'archives/baz/foo.txt',
'archives/foo.txt',
'empty directory',
'videos',
'videos/other video.mp4',
'videos/some video.mp4',
])
@pytest.mark.parametrize(
'directories,expected', [
([Path('foo')], [Path('foo')]),
([Path('foo'), Path('bar')], [Path('foo'), Path('bar')]),
([Path('foo'), Path('foo/bar'), Path('baz')], [Path('foo/bar'), Path('baz')]),
([Path('foo/bar'), Path('foo'), Path('baz')], [Path('foo/bar'), Path('baz')]),
([Path('baz'), Path('foo/bar'), Path('foo')], [Path('baz'), Path('foo/bar')]),
([Path('foo/bar'), Path('foo'), Path('foo/bar/baz')], [Path('foo/bar/baz')]),
]
)
def test_filter_parent_directories(directories, expected):
assert lib.filter_parent_directories(directories) == expected
def test_delete_file(make_files_structure, test_directory):
"""
File in the media directory can be deleted.
"""
files = [
'archives/foo.txt',
'bar.txt',
'baz/',
]
make_files_structure(files)
lib.delete_file('bar.txt')
assert (test_directory / 'archives/foo.txt').is_file()
assert not (test_directory / 'bar.txt').is_file()
assert (test_directory / 'baz').is_dir()
lib.delete_file('archives/foo.txt')
assert not (test_directory / 'archives/foo.txt').is_file()
assert not (test_directory / 'bar.txt').is_file()
with pytest.raises(InvalidFile):
lib.delete_file('baz')
with pytest.raises(InvalidFile):
lib.delete_file('does not exist')
assert (test_directory / 'baz').is_dir()
@pytest.mark.parametrize(
'path,expected',
[
('foo.mp4', ['foo']),
('foo bar.txt', ['foo', 'bar']),
('foo_bar.mp4', ['foo', 'bar']),
('foo-bar', ['foo-bar']),
('123 foo bar', ['123', 'foo', 'bar']),
('123foo bar', ['123foo', 'bar']),
]
)
def split_file_name_into_words(path, expected):
assert lib.split_file_name_into_words(Path(path)) == expected
@pytest.mark.parametrize(
'path,expected',
[
('foo', ('foo', '')),
('foo.mp4', ('foo', '.mp4')),
('foo.info.json', ('foo', '.info.json')),
('foo.something.info.json', ('foo.something', '.info.json')),
('foo-something.info.json', ('foo-something', '.info.json')),
('/absolute/foo-something.info.json', ('foo-something', '.info.json')),
('/absolute/foo', ('foo', '')),
('/absolute/foo.bar', ('foo', '.bar')),
]
)
def test_split_path_stem_and_suffix(path, expected):
assert lib.split_path_stem_and_suffix(Path(path)) == expected
@pytest.mark.asyncio
async def test_refresh_files(test_session, make_files_structure, test_directory):
"""All files in the media directory should be found when calling `refresh_files`"""
files = make_files_structure([
'foo.txt',
'bar.txt',
'baz.txt',
])
foo, bar, baz = files
await lib.refresh_files()
assert_files(test_session, ['bar.txt', 'baz.txt', 'foo.txt'])
baz.unlink()
await lib.refresh_files()
assert_files(test_session, ['bar.txt', 'foo.txt'])
foo.unlink()
bar.unlink()
await lib.refresh_files()
assert_files(test_session, [])
@pytest.mark.asyncio
async def test_refresh_files_in_directory(test_session, make_files_structure, test_directory):
"""A subdirectory can be refreshed, files above it can be ignored."""
ignored, foo, similar = make_files_structure([
'ignored.txt',
'subdir/foo.txt',
'subdir-similarly-named.mp4',
])
await lib.refresh_directory_files_recursively(test_directory / 'subdir')
assert_files(test_session, ['subdir/foo.txt'])
await lib.refresh_directory_files_recursively(test_directory)
assert_files(test_session, ['ignored.txt', 'subdir/foo.txt', 'subdir-similarly-named.mp4'])
# The similarly named file is not deleted when refreshing the directory which shares the name.
foo.unlink()
await lib.refresh_directory_files_recursively(test_directory / 'subdir')
assert_files(test_session, ['ignored.txt', 'subdir-similarly-named.mp4'])
@pytest.mark.asyncio
async def test_mime_type(test_session, make_files_structure, test_directory):
"""Files module uses the `file` command to get the mimetype of each file."""
from PIL import Image
foo, bar, baz, empty = make_files_structure([
'dir/foo text.txt',
'dir/bar.jpeg',
'dir/baz.mp4',
'dir/empty',
])
foo.write_text('some text')
Image.new('RGB', (25, 25), color='grey').save(bar)
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', baz)
await lib.refresh_files()
assert_files(test_session, ['dir/foo text.txt', 'dir/bar.jpeg', 'dir/baz.mp4', 'dir/empty'])
foo = test_session.query(File).filter_by(path=f'{test_directory}/dir/foo text.txt').one()
bar = test_session.query(File).filter_by(path=f'{test_directory}/dir/bar.jpeg').one()
baz = test_session.query(File).filter_by(path=f'{test_directory}/dir/baz.mp4').one()
empty = test_session.query(File).filter_by(path=f'{test_directory}/dir/empty').one()
assert foo.mimetype == 'text/plain'
assert bar.mimetype == 'image/jpeg'
assert baz.mimetype == 'video/mp4'
assert empty.mimetype == 'inode/x-empty'
@pytest.mark.asyncio
async def test_files_indexer(test_session, make_files_structure):
"""An Indexer is provided for each file based on it's mimetype or contents."""
source_files: List[str] = [
'a bzip file.bzip',
'a gzip file.gzip',
'a text file.txt',
'a zip file.zip',
'images/an image file.jpeg',
'unknown file',
'videos/a video file.info.json',
'videos/a video file.mp4',
]
bzip_path, gzip_path, text_path, zip_path, image_path, unknown_path, info_json_path, video_path \
= make_files_structure(source_files)
text_path.write_text('text file contents')
Image.new('RGB', (25, 25), color='grey').save(image_path)
shutil.copy(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4', video_path)
with zipfile.ZipFile(zip_path, 'w') as zip_file:
zip_file.write(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4')
with zipfile.ZipFile(bzip_path, 'w') as bzip_file:
bzip_file.write(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4')
with zipfile.ZipFile(gzip_path, 'w') as gzip_file:
gzip_file.write(PROJECT_DIR / 'test/big_buck_bunny_720p_1mb.mp4')
info_json_path.write_text(json.dumps({'description': 'the video description'}))
# Enable slow feature for testing.
# TODO can this be sped up to always be included?
with mock.patch('modules.videos.EXTRACT_SUBTITLES', True):
await lib._refresh_all_files()
bzip_file, gzip_file, text_file, zip_file, image_file, unknown_file, info_json_file, video_file \
= test_session.query(File).order_by(File.path)
# Indexers are detected correctly.
assert bzip_file.path.suffix == '.bzip' and bzip_file.indexer == indexers.ZipIndexer
assert gzip_file.path.suffix == '.gzip' and gzip_file.indexer == indexers.ZipIndexer
assert text_file.path.suffix == '.txt' and text_file.indexer == indexers.TextIndexer
assert zip_file.path.suffix == '.zip' and zip_file.indexer == indexers.ZipIndexer
assert image_file.path.suffix == '.jpeg' and image_file.indexer == indexers.DefaultIndexer
assert unknown_file.path.suffix == '' and unknown_file.indexer == indexers.DefaultIndexer
assert info_json_file.path.suffix == '.json' and info_json_file.indexer == indexers.DefaultIndexer
assert video_file.path.suffix == '.mp4' and video_file.indexer == videos.VideoIndexer
# Path suffix is copied to File.suffix.
assert bzip_file.suffix == '.bzip'
assert gzip_file.suffix == '.gzip'
assert text_file.suffix == '.txt'
assert zip_file.suffix == '.zip'
assert image_file.suffix == '.jpeg'
assert unknown_file.suffix == ''
assert info_json_file.suffix == '.info.json'
assert video_file.suffix == '.mp4'
# File are indexed by their titles and contents.
files, total = lib.file_search('file', 10, 0)
assert total == len(source_files), 'All files contain "file" in their file name.'
files, total = lib.file_search('image', 10, 0)
assert total == 1 and files[0]['title'] == 'an image file.jpeg', 'The image file title contains "image".'
files, total = lib.file_search('contents', 10, 0)
assert total == 1 and files[0]['title'] == 'a text file.txt', 'The text file contains "contents".'
files, total = lib.file_search('video', 10, 0)
assert total == 2 and {i['title'] for i in files} == {'a video file.info.json', 'a video file.mp4'}, \
'The video files contains "video".'
files, total = lib.file_search('yawn', 10, 0)
assert total == 1 and files[0]['title'] == 'a video file.mp4', 'The video file captions contain "yawn".'
files, total = lib.file_search('bunny', 10, 0)
assert total == 3 and {i['title'] for i in files} == {'a zip file.zip', 'a bzip file.bzip', 'a gzip file.gzip'}, \
'The zip files contain a file with "bunny" in the title.'
with mock.patch('modules.videos.VideoIndexer.create_index') as mock_create_index:
mock_create_index.side_effect = Exception('This should not be called twice')
await lib.refresh_files()
@pytest.mark.asyncio
async def test_large_text_indexer(test_session, make_files_structure):
"""
Large files have their indexes truncated.
"""
large, = make_files_structure({
'large_file.txt': 'foo ' * 1_000_000,
})
await lib._refresh_all_files()
assert test_session.query(File).count() == 1
assert large.is_file() and large.stat().st_size == 4_000_000
large_file: File = test_session.query(File).one()
assert len(large_file.d_text) < large.stat().st_size
assert len(large_file.d_text) == 46_117
def test_glob_shared_stem(make_files_structure):
mp4, png, j, name, video, something, vid2, vid2j = make_files_structure([
'video.mp4',
'video.png',
'video.info.json',
'video-name.txt',
'video/',
'something',
'videos/video2 [name].mp4',
'videos/video2 [name].info.json',
])
def check(path, expected):
assert sorted([i.name for i in lib.glob_shared_stem(path)]) == sorted(expected)
check(mp4, ['video.mp4', 'video.png', 'video.info.json', 'video'])
check(png, ['video.mp4', 'video.png', 'video.info.json', 'video'])
check(j, ['video.mp4', 'video.png', 'video.info.json', 'video'])
check(video, ['video.mp4', 'video.png', 'video.info.json', 'video'])
check(something, ['something'])
check(vid2, ['video2 [name].mp4', 'video2 [name].info.json'])
check(vid2j, ['video2 [name].mp4', 'video2 [name].info.json'])

26
wrolpi/media_path.py
View file

@ -9,9 +9,11 @@ PATH_TYPE = Union[str, pathlib.Path]
class MediaPath:
"""Enforce that a given path is within the Media Directory.
Provide access to methods and properties on the underlying pathlib.Path.
"""
Enforce that a given path is within the Media Directory.
"""
impl = types.String
def __init__(self, path: PATH_TYPE):
path = self._validate_path(path)
@ -23,6 +25,9 @@ class MediaPath:
def __str__(self):
return str(self.relative)
def __getitem__(self, item):
return self._path
@classmethod
def _validate_path(cls, path: PATH_TYPE):
if not path:
@ -60,25 +65,38 @@ class MediaPath:
def startswith(self, other):
return str(self._path).startswith(other)
def endswith(self, other):
return str(self._path).endswith(other)
@property
def name(self):
return self.path.name
@property
def relative(self):
"""Get the path relative to the media directory."""
return self._path.relative_to(get_media_directory())
def __hash__(self):
return hash(str(self._path)) if self._path else hash(None)
class MediaPathType(types.TypeDecorator): # noqa
impl = types.String
def process_bind_param(self, value, dialect):
if isinstance(value, str) and len(value) == 0:
"""Convert paths into what the DB expects. (pathlib.Path -> str)"""
if value == '':
raise ValueError('MediaPath cannot be empty')
if isinstance(value, pathlib.Path):
return str(value.absolute())
elif isinstance(value, str) or value is None:
return value
elif isinstance(value, MediaPath):
return str(value.path)
raise ValueError(f'Invalid MediaPath ({type(value)}): {value}')
def process_result_value(self, value, dialect):
if value:
return MediaPath(value)
return pathlib.Path(value)

31
wrolpi/root_api.py
View file

@ -18,12 +18,11 @@ from sanic_ext.extensions.openapi import openapi
from wrolpi import admin, status
from wrolpi.admin import HotspotStatus
from wrolpi.common import set_sanic_url_parts, logger, get_config, wrol_mode_enabled, Base, get_media_directory, \
wrol_mode_check, native_only, set_wrol_mode
from wrolpi.common import logger, get_config, wrol_mode_enabled, Base, get_media_directory, \
wrol_mode_check, native_only, disable_wrol_mode, enable_wrol_mode
from wrolpi.dates import set_timezone
from wrolpi.downloader import download_manager
from wrolpi.errors import WROLModeEnabled, InvalidTimezone, API_ERRORS, APIError, ValidationError, HotspotError
from wrolpi.media_path import MediaPath
from wrolpi.schema import RegexRequest, RegexResponse, SettingsRequest, SettingsResponse, DownloadRequest, EchoResponse
from wrolpi.vars import DOCKERIZED
from wrolpi.version import __version__
@ -53,8 +52,6 @@ def add_blueprint(bp: Union[Blueprint, BlueprintGroup]):
def run_webserver(loop, host: str, port: int, workers: int = 8):
set_sanic_url_parts(host, port)
# Attach all blueprints after they have been defined.
for bp in BLUEPRINTS:
api_app.blueprint(bp)
@ -118,6 +115,8 @@ def get_settings(_: Request):
config = get_config()
settings = {
'download_manager_disabled': download_manager.disabled.is_set(),
'download_manager_stopped': download_manager.stopped.is_set(),
'download_on_startup': config.download_on_startup,
'download_timeout': config.download_timeout,
'hotspot_device': config.hotspot_device,
@ -125,7 +124,7 @@ def get_settings(_: Request):
'hotspot_password': config.hotspot_password,
'hotspot_ssid': config.hotspot_ssid,
'hotspot_status': admin.hotspot_status().name,
'media_directory': get_media_directory(),
'media_directory': str(get_media_directory()), # Convert to string to avoid conversion to relative.
'throttle_on_startup': config.throttle_on_startup,
'throttle_status': admin.throttle_status().name,
'timezone': config.timezone,
@ -145,11 +144,11 @@ def update_settings(_: Request, body: SettingsRequest):
if body.wrol_mode is False:
# Disable WROL Mode
set_wrol_mode(False)
disable_wrol_mode()
return response.empty()
elif body.wrol_mode is True:
# Enable WROL Mode
set_wrol_mode(True)
enable_wrol_mode()
return response.empty()
try:
@ -164,9 +163,6 @@ def update_settings(_: Request, body: SettingsRequest):
old_password = wrolpi_config.hotspot_password
wrolpi_config.update(config)
if body.wrol_mode:
download_manager.kill()
# If the password was changed, we need to restart the hotspot.
password_changed = (new_password := config.get('hotspot_password')) and old_password != new_password
@ -231,7 +227,7 @@ async def kill_download(_: Request, download_id: int):
@root_api.post('/download/kill')
@openapi.description('Kill all downloads. Disable downloading.')
async def kill_downloads(_: Request):
download_manager.kill()
download_manager.disable()
return response.empty()
@ -319,7 +315,8 @@ async def get_status(_: Request):
cpu_info = await status.get_cpu_info()
load = await status.get_load()
drives = await status.get_drives_info()
ret = dict(cpu_info=cpu_info, load=load, drives=drives)
downloads = download_manager.get_summary()
ret = dict(cpu_info=cpu_info, load=load, drives=drives, downloads=downloads)
return json_response(ret)
@ -340,10 +337,12 @@ class CustomJSONEncoder(json.JSONEncoder):
if hasattr(obj, 'dict'):
return obj.dict()
elif isinstance(obj, Path):
return str(obj)
elif isinstance(obj, MediaPath):
media_directory = get_media_directory()
path = obj.path.relative_to(media_directory)
try:
path = obj.relative_to(media_directory)
except ValueError:
# Path may not be absolute.
path = obj
if str(path) == '.':
return ''
return str(path)

3
wrolpi/status.py Normal file → Executable file
View file

@ -8,9 +8,8 @@ from decimal import Decimal
from pathlib import Path
from typing import List
from wrolpi.common import logger
from wrolpi.cmd import which
from wrolpi.common import logger
logger = logger.getChild(__name__)

2
wrolpi/test/common.py
View file

@ -183,4 +183,4 @@ def assert_dict_contains(d1: dict, d2: dict):
if isinstance(d1[k2], dict):
assert_dict_contains(d1[k2], d2[k2])
else:
assert d1[k2] == d2[k2], f'{k2} of value {d1[k2]} does not equal {d2[k2]} in dict 1'
assert d1[k2] == d2[k2], f'{repr(k2)} of value {repr(d1[k2])} does not equal {repr(d2[k2])} in dict 1'

50
wrolpi/test/test_captions.py Normal file
View file

@ -0,0 +1,50 @@
import shutil
from pathlib import Path
from wrolpi import captions
from wrolpi.test.common import TestAPI
from wrolpi.vars import PROJECT_DIR
class TestCaption(TestAPI):
def setUp(self) -> None:
super().setUp()
self.vtt_path1 = Path(self.tmp_dir.name) / 'example1.en.vtt'
self.vtt_path1.touch()
self.vtt_path1.write_text((PROJECT_DIR / 'test/example1.en.vtt').read_text())
self.vtt_path2 = Path(self.tmp_dir.name) / 'example2.en.vtt'
self.vtt_path2.touch()
self.vtt_path2.write_text((PROJECT_DIR / 'test/example2.en.vtt').read_text())
def test_example_en_vtt(self):
expected = ['okay welcome to this session this is',
'okay welcome to this session this is',
'okay welcome to this session this is\ncalled the kinetic bunny need to meet',
'called the kinetic bunny need to meet',
'called the kinetic bunny need to meet\nthe virtual bonnie emilio stop this is',
'the virtual bonnie emilio stop this is',
'the virtual bonnie emilio stop this is\njana car who just to give a little bit',
'jana car who just to give a little bit',
'jana car who just to give a little bit\nof background information we come from a',
'of background information we come from a',
'of background information we come from a\ncompany called le code from finland and']
self.assertEqual(list(captions.get_caption_text(self.vtt_path1)), expected)
expected = ['okay welcome to this session this is',
'called the kinetic bunny need to meet',
'the virtual bonnie emilio stop this is',
'jana car who just to give a little bit',
'of background information we come from a',
'company called le code from finland and']
self.assertEqual(list(captions.get_unique_caption_lines(self.vtt_path1)), expected)
def test_bad_caption(test_directory):
"""A caption file that cannot be parsed is ignored."""
bad_caption_file = PROJECT_DIR / 'test/bad_caption.en.vtt'
test_file = test_directory / 'bad_caption.en.vtt'
shutil.copy(bad_caption_file, test_file)
result = captions.read_captions(test_file)
assert result is None

1
wrolpi/test/test_cmd.py
View file

@ -2,7 +2,6 @@ import os
from pathlib import Path
from wrolpi import cmd
from wrolpi.test.common import skip_circleci
def test_which(test_directory):

45
wrolpi/test/test_common.py
View file

@ -8,7 +8,7 @@ from decimal import Decimal
import pytest
from wrolpi.common import insert_parameter, date_range, api_param_limiter, chdir, zig_zag, \
escape_file_name
escape_file_name, match_paths_to_suffixes, truncate_object_bytes
from wrolpi.dates import set_timezone, now
from wrolpi.errors import InvalidTimezone
from wrolpi.test.common import build_test_directories
@ -297,3 +297,46 @@ def test_zig_zag(low, high, expected):
)
def test_escape_file_name(name, expected):
assert escape_file_name(name) == expected
@pytest.mark.parametrize(
'paths,suffix_groups,expected', [
([], [], ()),
(['foo.mp4', 'foo.info.json'], [('.mp4',)], ('foo.mp4',)),
(['foo.mp4', 'foo.info.json'], [('.mp4',), ('.info.json',)], ('foo.mp4', 'foo.info.json')),
(['foo.mp4', 'foo.info.json'], [('.mp4',), ('.nope',), ('.info.json',)], ('foo.mp4', None, 'foo.info.json')),
(['foo.mp4', 'foo.info.json', 'extra.txt'], [('.mp4',)], ('foo.mp4',)),
(['foo.mp4'], [('.mp4', '.flv'), ('.info.json',)], ('foo.mp4', None)),
(
# Two files are matched to the closest suffix.
['foo.info.json', 'bar.json'],
[('.info.json',), ('.json',)], # TODO longest suffix must be first
('foo.info.json', 'bar.json'),
),
]
)
def test_match_paths_to_suffixes(paths, suffix_groups, expected):
paths = [pathlib.Path(i) for i in paths]
expected = tuple(pathlib.Path(i) if i else None for i in expected)
assert (i := match_paths_to_suffixes(paths, suffix_groups)) == expected, f'{i} != {expected}'
def test_truncate_object_bytes():
"""
Objects can be truncated (lists will be shortened) so they will fit in tsvector columns.
"""
assert truncate_object_bytes(['foo'] * 10, 100) == ['foo'] * 5
assert truncate_object_bytes(['foo'] * 1_000, 100) == ['foo'] * 5
assert truncate_object_bytes(['foo'] * 1_000_000, 100) == ['foo'] * 5
assert truncate_object_bytes(['foo'] * 1_000_000, 200) == ['foo'] * 14
assert truncate_object_bytes([], 200) == []
assert truncate_object_bytes(None, 100) is None
assert truncate_object_bytes('', 100) is ''
assert truncate_object_bytes('foo' * 100, 99) == 'foofoofoofoofoofoofoofoofoofoofoofoofoof'
assert truncate_object_bytes('foo' * 100, 80) == 'foofoofoofoofoofoofoofoofo'
assert truncate_object_bytes('foo' * 100, 55) == 'foofo'
assert truncate_object_bytes('foo' * 100, 51) == 'f'
assert truncate_object_bytes('foo' * 100, 50) == ''
assert truncate_object_bytes('foo' * 100, 0) == ''

3
wrolpi/test/test_db.py
View file

@ -1,6 +1,6 @@
from sqlalchemy.orm import Session
from wrolpi.db import optional_session, get_db_session
from wrolpi.db import optional_session
def test_optional_session(test_session):
@ -34,4 +34,3 @@ def test_optional_session(test_session):
func()
func(session=test_session)
func(test_session)

40
wrolpi/test/test_downloader.py
View file

@ -1,6 +1,9 @@
import asyncio
import json
import time
from abc import ABC
from datetime import datetime
from http import HTTPStatus
from itertools import zip_longest
from typing import Tuple, Optional
from unittest import mock
@ -365,40 +368,3 @@ async def test_process_runner_timeout(test_directory):
elapsed = datetime.now() - start
assert 2 < elapsed.total_seconds() < 4
@pytest.mark.asyncio
async def test_multi_domain_download(test_session, test_directory, test_download_manager):
class TestDownloader(Downloader, ABC):
name = 'test_downloader'
@classmethod
def valid_url(cls, url: str) -> Tuple[bool, Optional[dict]]:
return True, {}
async def do_download(self, download: Download) -> DownloadResult:
await asyncio.sleep(1)
return DownloadResult(success=True)
async def wait_and_assert(urls_, max_seconds):
downloads = test_download_manager.create_downloads(urls_, downloader=test_downloader.name)
assert downloads and all(isinstance(i, Download) for i in downloads), 'Did not get Downloads'
start = now()
await test_download_manager.wait_for_all_downloads()
assert (elapsed := now() - start).total_seconds() < max_seconds, \
f'Downloads took {elapsed} seconds. Were they in parallel?'
downloads = test_download_manager.get_downloads(test_session)
incomplete = [i.url for i in downloads if i.status != 'complete']
assert not incomplete, f'Some downloads did not complete: {incomplete}'
test_downloader = TestDownloader()
test_download_manager.register_downloader(test_downloader)
# Each domain's downloads will be processed one at a time.
# example.com and example.org will both download in parallel.
urls = ['https://example.com/1', 'https://example.com/2', 'https://example.com/3', 'https://example.org/1',
'https://example.org/2']
await wait_and_assert(urls, 4)
# Downloading more domains that workers is possible.
urls = [f'https://example.{i}' for i in range(test_download_manager.worker_count + 2)]
await wait_and_assert(urls, 3)

40
wrolpi/test/test_root_api.py
View file

@ -301,10 +301,48 @@ def test_clear_downloads(test_session, test_client, test_config, test_download_m
assert DOWNLOAD_MANAGER_CONFIG.skip_urls == ['https://example.com/5', ]
def test_get_status(test_client):
def test_get_status(test_client, test_session):
"""Get the server status information."""
request, response = test_client.get('/api/status')
assert response.status_code == HTTPStatus.OK
assert 'cpu_info' in response.json and isinstance(response.json['cpu_info'], dict)
assert 'load' in response.json and isinstance(response.json['load'], dict)
assert 'drives' in response.json and isinstance(response.json['drives'], list)
def test_post_download(test_session, test_client, test_download_manager_config):
"""Test creating once-downlaods and recurring downloads."""
async def queue_downloads(*a, **kw):
pass
with mock.patch('wrolpi.downloader.DownloadManager.queue_downloads', queue_downloads):
# Create a single recurring Download.
content = dict(
urls='https://example.com/1\n',
frequency=1_000,
downloader='archive',
)
request, response = test_client.post('/api/download', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert {i.url for i in test_session.query(Download)} == {'https://example.com/1', }
# Create to once-downloads.
content = dict(
urls='https://example.com/2\nhttps://example.com/3\n',
downloader='archive',
)
request, response = test_client.post('/api/download', content=json.dumps(content))
assert response.status_code == HTTPStatus.NO_CONTENT
assert {i.url for i in test_session.query(Download)} == {f'https://example.com/{i}' for i in range(1, 4)}
def test_get_downloaders(test_client):
"""A list of Downloaders the user can use can be gotten."""
request, response = test_client.get('/api/downloaders')
assert response.status_code == HTTPStatus.OK
assert 'downloaders' in response.json, 'Downloaders not returned'
assert isinstance(response.json['downloaders'], list) and len(response.json['downloaders']), \
'No downloaders returned'

2
wrolpi/test/test_rss.py
View file

@ -1,5 +1,3 @@
import json
from http import HTTPStatus
from itertools import zip_longest
from typing import List

10
wrolpi/vars.py
View file

@ -23,11 +23,8 @@ if not MEDIA_DIRECTORY.is_dir() and not PYTEST:
CONFIG_DIR: Path = MEDIA_DIRECTORY / 'config'
MODULES_DIR: Path = PROJECT_DIR / 'modules'
PUBLIC_HOST = os.environ.get('PUBLIC_HOST')
PUBLIC_PORT = os.environ.get('PUBLIC_PORT')
DEFAULT_TIMEZONE_STR = 'America/Boise'
LAST_MODIFIED_DATE_FORMAT = '%a, %d %b %Y %H:%M:%S GMT'
DATE_FORMAT = '%Y-%M-%d'
DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S'
DATETIME_FORMAT_MS = '%Y-%m-%d %H:%M:%S.%f'
@ -40,10 +37,3 @@ DB_PORT = int(os.environ.get('DB_PORT', 5432))
DB_NAME = os.environ.get('DB_NAME', 'wrolpi')
DB_USER = os.environ.get('DB_USER', 'wrolpi')
DB_PASSWORD = os.environ.get('DB_PASSWORD', 'wrolpi')
EXAMPLE_CONFIG = {
'hotspot_on_startup': True,
'throttle_on_startup': False,
'timezone': 'America/Boise',
'wrol_mode': False,
}

2
wrolpi/version.py
View file

@ -1,4 +1,4 @@
__version__ = '0.6.1-beta'
__version__ = '0.7.0-beta'
import subprocess