wrolpi/modules/videos/channel/lib.py
lrnselfreliance 46796f783a
Arbitrary videos can be downloaded. (#63)
* Squashed commit of the following:

commit c527475fcb4d4906457be5c10f0e9c6dee7fedc1
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:16:36 2021 -0700

    Only saving a favorite video to the config if it has a source_id.

commit ae8b712ebeaef467b8d5a5b87ca0f4f276373a0d
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:14:26 2021 -0700

    Ading description and changing Save button to Download.

commit 03a72be6dfdfa383915950a80e3f1ee9febf6705
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:09:44 2021 -0700

    Removing uploader for now.

commit 5d0231284bf84e459687f6ab903b06bf4e3762bd
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:07:17 2021 -0700

    Removing unique data from example info.

commit 9cc721a32d4715d9d6433e51db3b35e039c78d68
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 22:02:33 2021 -0700

    Updating downloader tests.

commit 8e193b03349ae4613f105418cd3e8176c7101304
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:53:31 2021 -0700

    Toasting error when WROL Mode is enabled.  Disabling new downloads during WROL Mode.

commit 48c4311c31db8f2d2c8dd0a65f979a5f0a49a911
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:45:06 2021 -0700

    Raising error if download exits without a 0.  Clearing kill event when starting a new download.

commit 54ba3a890d59facc837410d34bf7afb40db3b685
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Dec 14 21:16:46 2021 -0700

    Downloads can be restarted and killed.

commit bd60751262247259c3e9ceb5116e017d6314c219
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Dec 13 16:28:18 2021 -0700

    Downloads can be started again.  Downloads are sorted on Downloads page.

commit ea7f39231b92982ae0b263b76c5bd1902cc8169d
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 21:24:16 2021 -0700

    Building Downloads tab.

commit d3321872ac3743ff6df781b07ef0fb537f1d4c98
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 20:41:46 2021 -0700

    Not scheduling channels withtout a URL/frequency.

commit 84f268a0d4f58c85f86b6377900492e0766f6619
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 19:56:08 2021 -0700

    Channel downloads will be spread out when each channel is created.

commit 26fa6beea38ef3f8d6b7bf218f18222a51a87a60
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Dec 11 13:08:11 2021 -0700

    Recurring downloads are renewed on startup.

commit fc2ab850533f1c3ad073f4692c7dea2be43e85d6
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 17:03:57 2021 -0700

    Fetching download again after do_download.

commit fdadad327fc09c17a49eb63b11e632fb72cc3195
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 16:20:51 2021 -0700

    Fixing error with test.  Mock was not being reset.

commit 070dee238e53377acd22097c63cf3794e68ed423
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Dec 10 15:36:39 2021 -0700

    Fixed bug in date handling (big surprise).

commit e53a8992268d68620c679455adaec3a590f20602
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 16:48:36 2021 -0700

    Using chunks for regular channel refresh.

commit 3bd60c58d1bef0226325f321978c0434b36b3be5
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 15:22:02 2021 -0700

    Reformatting.

commit 4fdc325f25a32c311ca6543fe321624b2bd564b2
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 15:13:34 2021 -0700

    Creating function that refreshes NO CHANNEL videos.

commit d67feafb6c2994f5a4a1295920a70db369cc6816
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 14:35:54 2021 -0700

    Removing unused functions.

commit 6b3918e4e78fbac64920f85c4de0df873ec366a1
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Dec 1 14:31:48 2021 -0700

    Removing reporter and events.  This will be replaced in the future with some sort of SSE.

commit 54fa1b0f5e93ca25c80b780dcdc3876b700273b1
Author: Roland <roland@learningselfreliance.com>
Date:   Sun Nov 28 17:55:43 2021 -0700

    Better error handling for archive button.

commit c31d124e511b27f626ed2df71486d47302b396f3
Author: Roland <roland@learningselfreliance.com>
Date:   Sun Nov 28 17:29:43 2021 -0700

    Displaying View Count in a human readable format.

commit 0ee2bd65588fd93b475cad6144e76f1b45191b39
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Nov 27 09:29:41 2021 -0700

    Fixing tests.

commit e64aff32bab95c311ea3747e1a21b58b99750d5e
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Nov 27 08:59:51 2021 -0700

    Handling when a Video has no Channel in UI.

commit b9549ecd5aac03c002adca7b7667f0a7189b662a
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Nov 26 08:35:36 2021 -0700

    Downloaders can raise UnrecoverableDownloadError to signal a downlaod should not be retried.

commit da88798e2584d3e84900a70e34166c8c208c64fb
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 20:40:29 2021 -0700

    Downloaders can refuse a download, which will then be deleted.

commit b865d856121db5ec8a3660c5d3c13b9846fde688
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 19:35:43 2021 -0700

    Removing relative path handling.

commit 6d7757870b02ba030d78b3ef7e43f28b906eea61
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 19:28:27 2021 -0700

    Using __json__ for Video and Channel models.

commit 649e4d6a61608272759d1e7ee02b490bb5c20b3e
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 18:23:38 2021 -0700

    Fixing video links in UI.

commit 9d081af12742070b8519bd752812130cb5dbe393
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:42:38 2021 -0700

    Fixing docs.

commit 1202cbe0f9ae744d7801b53a5071cded8ea39924
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:40:36 2021 -0700

    Fixing wrolpi tests.

commit 3f6af8cc1123f2f63a3fc0f21ec4d994c642cf13
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 24 14:13:35 2021 -0700

    Fixing tests to use MediaPath.

commit faec04f2a5e4c267a502c03884940a9c5bcfe387
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Nov 19 10:21:20 2021 -0700

    Testing when a video has no channel.

commit cbf22aad7a0b72e61685e0236e843554b57c4af1
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Nov 18 16:24:03 2021 -0700

    Using MediaPath for Video and Channel models.  Testing VideoDownloader.do_download.

commit 7916ed61e43439e6cd60f23ce3fdf6c1a75bf02d
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Nov 18 12:43:44 2021 -0700

    Channels/Videos downloaders can get gotten.

commit 808ef796cec01db6e7009065f9e26d25d9500ce2
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 21:45:19 2021 -0700

    Upgrading to Sanic 21.9.1.

commit ea32813be28e78393ac456ee782a7245c098188f
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 20:29:01 2021 -0700

    Creating Downloader and DownloaderManager classes which handle downloading URLs.

commit e859a77a77277ddf8b6e221ec9c9383c8ab91d66
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Nov 17 12:41:39 2021 -0700

    Creating UI segement which allows the download/upload of media.

* Displaying all recurring downloads.

* Not downloading channels with empty url.

* Showing positive row for recurring downloads.

* Removing default last_successful_download.

* Only displaying Last Successful Download when there is one.

* Refreshing videos before finding missing videos.

* Using short video path name for log.

* Removing download history feature in favor of new Downloads page.

* Only showing info date if it exists.

* Making get_or_create_channel more reliable.

* Allowing null for channel download frequency.

* Logging when searching for channel.

* Storing channel's source_id in config.

* Importing using link instead of name.

* Getting channel id when importing channels.

* Removing redunant parentheses from repr.

* Preserving database channel values if there are not values in the config.

* Only highlighting pending downloads.

* Deleting old downloads periodically.

* Sorting recurring by next_download/frequency.

* Creating fixtures for channels, media directory and db session.

* Removing unused function.  Wrapping config for tests.

* Deleting channel download when deleting a channel.

* Testing that a channel's download is deleted when the channel is deleted.

* Updating youtube-dl

* Switching from youtube-dl in favor of yt-dlp.

* Sorting once-downlaods by last_successful_download then id.

* Adding special handling to video_url_resolver for full urls.

* Consolidating downloader methods and functions.  Reversing once-download sort order so the most recently downloaded are at the top.

* Getting session if one is not provided.

* Adding test for create_downloads.

* Not passing format option when getting filename.

* Linting

Signed-off-by: Roland <roland@learningselfreliance.com>
2021-12-20 16:09:11 -07:00

242 lines
8.2 KiB
Python

from collections import defaultdict
from datetime import timedelta
from typing import List, Dict, Union
from sqlalchemy.orm.exc import NoResultFound
from wrolpi.common import sanitize_link, run_after, get_relative_to_media_directory, make_media_directory, logger
from wrolpi.dates import today
from wrolpi.db import get_db_session, get_db_curs
from wrolpi.downloader import download_manager, Download
from wrolpi.errors import UnknownChannel, UnknownDirectory, APIError, ValidationError
from ..common import check_for_channel_conflicts
from ..lib import save_channels_config
from ..models import Channel
logger = logger.getChild(__name__)
async def get_minimal_channels() -> List[dict]:
"""
Get the minimum amount of information necessary about all channels.
"""
with get_db_curs() as curs:
# Get all channels, even if they don't have videos.
stmt = '''
SELECT
c.id, name, link, directory, url, download_frequency, info_date
FROM
channel AS c
ORDER BY LOWER(name)
'''
curs.execute(stmt)
channels = list(map(dict, curs.fetchall()))
video_counts = await get_channels_video_count()
for channel in channels:
channel_id = channel['id']
channel['video_count'] = video_counts[channel_id]
return channels
async def get_channels_video_count() -> Dict[int, int]:
"""
Add video counts to all channels
"""
with get_db_curs() as curs:
stmt = 'SELECT id FROM channel'
curs.execute(stmt)
# Get all channel IDs, start them with a count of 0.
video_counts = {int(i['id']): 0 for i in curs.fetchall()}
stmt = '''
SELECT
c.id, COUNT(v.id) AS video_count
FROM
channel AS c
LEFT JOIN video AS v ON v.channel_id = c.id
WHERE
v.video_path IS NOT NULL
GROUP BY 1
'''
curs.execute(stmt)
# Replace all the counts of those channels with videos.
video_counts.update({int(i['id']): int(i['video_count']) for i in curs.fetchall()})
return video_counts
def get_channel(source_id: str = None, link: str = None, url: str = None, return_dict: bool = True
) -> Union[dict, Channel]:
"""
Attempt to find a Channel using the provided params. The params are in order of reliability.
Raises UnknownChannel if no channel is found.
"""
with get_db_session() as session:
# Try by source_id first.
channel = None
if source_id:
channel = session.query(Channel).filter_by(source_id=source_id).one_or_none()
if not channel and link:
channel = session.query(Channel).filter_by(link=link).one_or_none()
if not channel and url:
channel = session.query(Channel).filter_by(url=url).one_or_none()
if not channel:
raise UnknownChannel(f'No channel matches {link=} {source_id=} {url=}')
logger.debug(f'Found {channel=} using {source_id=} {link=} {url=}')
return channel.dict() if return_dict else channel
def _spread_by_frequency(channels: List[Channel]):
channels_by_frequency = defaultdict(lambda: [])
for channel in channels:
channels_by_frequency[channel.download_frequency].append(channel)
for frequency, channels in channels_by_frequency.items():
# The seconds between each download.
chunk = frequency // len(channels)
for channel in channels:
# My position in the channel group.
index = channels.index(channel)
# My next download will be distributed by my frequency and my position.
position = chunk * index
next_download = today() + timedelta(seconds=frequency + position)
yield dict(url=channel.url, frequency=frequency, next_download=next_download)
def spread_channel_downloads():
"""
Channels should be downloaded in a manner that is spread out over their frequency. For example, three channels
with a frequency of a week should be downloaded on different days that week.
"""
with get_db_session(commit=True) as session:
channels = list(session.query(Channel).filter(
Channel.url != None,
Channel.url != '',
Channel.download_frequency != None
).order_by(Channel.link).all()) # noqa
url_next_download = _spread_by_frequency(channels)
for info in url_next_download:
download = download_manager.get_download(session, url=info['url'])
if not download:
download = Download(url=info['url'])
session.add(download)
download.frequency = info['frequency']
download.next_download = info['next_download']
@run_after(save_channels_config)
def update_channel(data, link):
with get_db_session(commit=True) as session:
try:
channel = session.query(Channel).filter_by(link=link).one()
except NoResultFound:
raise UnknownChannel()
# Only update directory if it was empty
if data.get('directory') and not channel.directory:
try:
data['directory'] = get_relative_to_media_directory(data['directory'])
except UnknownDirectory:
if data['mkdir']:
make_media_directory(data['directory'])
data['directory'] = get_relative_to_media_directory(data['directory'])
else:
raise
if 'directory' in data:
data['directory'] = str(data['directory'])
if 'download_frequency' in data:
try:
data['download_frequency'] = int(data['download_frequency'])
except ValueError:
if data['download_frequency'] in ('null', 'None', ''):
data['download_frequency'] = None
else:
raise APIError(f'Invalid download frequency {data["download_frequency"]}')
if data.get('match_regex') in ('None', 'null'):
data['match_regex'] = None
# Verify that the URL/Name/Link aren't taken
check_for_channel_conflicts(
session,
id_=channel.id,
url=data.get('url'),
name=data.get('name'),
link=data.get('link'),
directory=data.get('directory'),
)
# Apply the changes now that we've OK'd them
channel.update(data)
spread_channel_downloads()
return channel
@run_after(save_channels_config)
@run_after(spread_channel_downloads)
def create_channel(data: dict, return_dict: bool = True) -> Union[Channel, dict]:
"""
Create a new Channel. Check for conflicts with existing Channels.
"""
with get_db_session(commit=True) as session:
link = sanitize_link(data['name'])
try:
# Verify that the URL/Name/Link aren't taken
check_for_channel_conflicts(
session,
url=data.get('url'),
name=data['name'],
link=link,
directory=str(data['directory']),
source_id=data.get('source_id'),
)
except APIError as e:
raise ValidationError() from e
channel = Channel(
name=data['name'],
url=data.get('url'),
match_regex=data.get('match_regex'),
link=link,
directory=str(data['directory']),
download_frequency=data.get('download_frequency'),
source_id=data.get('source_id'),
)
session.add(channel)
session.commit()
session.flush()
session.refresh(channel)
if return_dict:
return channel.dict()
else:
return channel
@run_after(save_channels_config)
def delete_channel(link):
with get_db_session(commit=True) as session:
try:
channel = session.query(Channel).filter_by(link=link).one()
except NoResultFound:
raise UnknownChannel()
channel.delete_with_videos()
def download_channel(link: str):
channel = get_channel(link=link, return_dict=False)
with get_db_session(commit=True) as session:
download_manager.create_download(channel.url, session)