Commit graph

143 commits

Author SHA1 Message Date
Roland
4058f22428
Warning when validation fails on video upsert. 2022-04-29 19:30:53 -06:00
Roland
8318224db6
Removing troubleshooting print. 2022-04-29 19:19:12 -06:00
Roland
d75ddf097d
Removing unused function. 2022-04-29 19:09:49 -06:00
Roland
aa203cc102
Moving non-video functions to wrolpi.common. Handling Channel/Downlaod relationship in Channel.update.
Removing default download freqeuncy so channels will not be downloaded without a url and frequency.

Adding channel unique url constraint.

Handling when a Channel's directory may still be relative in DB.

Adding TODOs to remove old favorites.

Removing outdated favorites from config after favorites have been created.

Testing that old-style favorited videos in the config are imported.

Removing channels_by_link.

Using list to store channels in config.

Moving channel import function into lib.

Supporting old channel import with links.

Bumping version for channel.link removal.

Removing all reference to Channel.link in app and api.

Adding relative property to MediaPathType.

Testing that a channel can be downloaded after it has a frequency added.

All tests pass.

Video and Channel tests pass without channel.link.

Removing channel.link usage in channel api.

Changing video Statistics page to About.
2022-04-29 19:09:25 -06:00
lrnselfreliance
fda8636c92
Fixing bug in archives search where offset could not be greater than 100. (#117) 2022-04-14 09:21:26 -06:00
Roland
372056eefd
Removing troubleshooting print. 2022-04-08 09:34:18 -06:00
Roland
7656890835
Testing that video size is gathered during validation. 2022-04-07 13:42:01 -06:00
lrnselfreliance
35f5662571
Bugfix/channel download on import (#116)
* A Download record is created for every Channel imported.

* Downloads must have a URL.

* Handling when channel has download frequency but no url.

* Fetching downloads after clearing.

* Setting next download when importing channels.

* Not deleting recurring completed downloads.

* Fixing comments.

* Skipping some service tests in CircleCI because they shouldn't be tested off a local machine.

Signed-off-by: Roland <roland@learningselfreliance.com>
2022-04-07 11:18:15 -06:00
Roland
43709ac9e7
Creatind download manager config which will store skipped urls. 2022-04-02 16:01:43 -06:00
Roland
5178f72b66
Reworking map import to handle postgres data dumps. 2022-03-31 07:31:15 -06:00
Roland
33fa1444c6
Allowing multiple names for which function. 2022-03-19 17:08:44 -06:00
Roland
a02d1c4527
Using which for chromium-browser. 2022-03-19 16:58:49 -06:00
Roland
5c3c400dd1
Creating cmd.which which will find executables. 2022-03-19 15:18:40 -06:00
Roland
849a3fd43f
Creating debian install scripts.
Modifying Ubuntu 20.04 install script to work with Debian 11.

Should not need to compile mod_tile.

Uncommenting branch checkout.

Not removing nodejs.

Not installing nodejs.

Installing nodejs instead of npm??

Using debian 11 npm sources.

Installing archive tools using git+https

Updating carto location.

Using global renderd.conf in apache config.

Using -I to get debian ip.

Updating more file locations for debian 11.

Checking alternative locations for single-file and readability-extractor
2022-03-16 21:10:49 -06:00
Roland
c815aa68c6
Calculating time-to-import for map pbf files in the backend. These times were created using a test RPi4. Encoding URL of archive files to prevent error which blocked displaying of archives. 2022-03-12 09:50:16 -07:00
Roland
6a22090e54
Logging nicer timestamp for logging import elapsed time. 2022-03-10 13:17:12 -07:00
Roland
de5ec99954
Logging after process is complete. 2022-03-10 09:28:34 -07:00
Roland
f08292de56
Clearing map cache files if any import was successful. 2022-03-09 20:22:09 -07:00
Roland
b1d526fb0b
Adding __init__ to map/test. 2022-03-09 20:10:58 -07:00
Roland
aa345b6931
Logging map import duration. 2022-03-09 16:09:25 -07:00
Roland
79e1c8c1df
Merge branch 'master' of ssh://github.com-lrnselfreliance/lrnselfreliance/wrolpi 2022-03-09 16:09:06 -07:00
lrnselfreliance
a78263cb68
Merge pull request #106 from lrnselfreliance/circleci-project-setup
Add .circleci/config.yml
2022-03-09 16:07:03 -07:00
Roland
f91ce0bc09
Skipping failing test for now. 2022-03-09 16:03:56 -07:00
Roland
9b3f527f96
Using two baz to force order of test result. 2022-03-09 15:56:23 -07:00
Roland
e25c92a30b
Reversing order for circleci. 2022-03-09 15:54:51 -07:00
Roland
984aa04f0a
Ordering results by columns. 2022-03-09 15:51:24 -07:00
Roland
d0de92ef7e
Skipping date tests on circleci. 2022-03-09 15:44:02 -07:00
Roland
888352c984
Squashed commit of the following:
commit 26bde0fb48d59a86c2fb9f53a7218d49f22d4860
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 8 13:25:56 2022 -0700

    Using osm2pgsql append to preserve other imported maps.

commit c6c52623eaf02e691b1e75dcf39c3a8334ead55a
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 8 08:20:38 2022 -0700

    Using placeholder instead of loader when table is loading.

commit 44fec6490e26603c88a98f6bc70fc18686e17f09
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:58:34 2022 -0700

    Simplifying tableRow function.

commit 11b8b57f1fa59556397620f3cad9a33a0237501d
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:53:51 2022 -0700

    Removing scientific humanFileSize.

commit 0a09d0a6963afebed7f388aad71bbdd3151d2e9e
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:46:22 2022 -0700

    Fixing active loader.

commit 07f4deeac861351d7483ef57278eb2e0941e31fb
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:40:44 2022 -0700

    Increasing recommended map file size.

commit af3ac492c81f4c8f84563a448bc272ae9334d32d
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:35:13 2022 -0700

    Reworking logic order for simplicity.  Fetching downloads on interval.

commit 3d9d31fcec87ed02ced7f67c34907faa5a3b54e5
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 20:24:39 2022 -0700

    Handling when pbf import status are loading.

commit 3798961a48e7c9eb165ca4e4051f239dbb088d0c
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:53:50 2022 -0700

    Fetching import status every 30 seconds.

commit f7a8193b0955d14ead8090aa74ce3c5a681b57ec
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:33:17 2022 -0700

    Stopping renderd during installation.

commit 5bf050b59e0a6e391184596f5e6f0737dea1221d
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:26:59 2022 -0700

    Removing duplciate serve.

commit 892e3315eeeb5d2516996b901a32a73a66dae078
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:22:14 2022 -0700

    Setting bash flags in sub-install script.

commit d5d2acd2cd85e089c78aa85a27c5d966222a1347
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 19:20:19 2022 -0700

    Moving bulk of installation script into separate script so the latest code will be used during initial installation.

commit ab030d8af834a4a8772edd06df360c51e72fbb02
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 18:40:34 2022 -0700

    Removing mod_tile/ajt directory instead of entire contents of mod_tile.  Showing import as relative to media directory.

commit c05199db4ecf56f4d096051b58689457882af12a
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 18:39:59 2022 -0700

    Remvoing marker from map.

commit ba5df4cf62d61b55281277a6254836e973c766ae
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 11:57:41 2022 -0700

    Creating import_map.sh script.

commit 1a16e46130e775efe375adb22917ba12e44855cd
Author: Roland <roland@learningselfreliance.com>
Date:   Mon Mar 7 11:11:01 2022 -0700

    Increasing ram usage during import to avoid hanging import.

commit 0f5ce1a9f806f1c0878dd804b04587b6340ffd16
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 19:19:47 2022 -0700

    Increasing map_file.size size.

commit eb6e01512af59fb4a18ab15643c7bc61f3d48e53
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:55:55 2022 -0700

    Copying leaflet js/css files into docker container and to Apache directory so they will be available offline.

commit 5b687324e119f021673f42e8d079b83034b70d53
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:36:06 2022 -0700

    Matching tab height on Map page.

commit 8031786d672e18366e9a2eb2d7bdfcd8611d27f3
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:34:20 2022 -0700

    Shrinking HelpPopup Icon.

commit 5ea94488e43bbc5c414e61c9173a3cfa908c7104
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:31:30 2022 -0700

    Testing if no PBFs are requested to be imported.

commit c4e86be1faed78bd3af1f3487471c01a1958a433
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:18:48 2022 -0700

    Using Icon instead of Button for HelpPopup.

commit 51d94f6d85885803813544fea16a56372babaa82
Author: Roland <roland@learningselfreliance.com>
Date:   Sat Mar 5 18:07:03 2022 -0700

    PBF files are sorted by the api.

commit 40f3a7ff04a83db0153f64a88a37f1d56ce043aa
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Mar 4 18:35:51 2022 -0700

    Adding help popup.

commit 1da8d9fa403ec2a3b3c16c11a3937de37c895a35
Author: Roland <roland@learningselfreliance.com>
Date:   Fri Mar 4 09:41:01 2022 -0700

    Moving message up.  Clearing cached tiles after import.

commit 3192adf95192554c194c9e0a8aeb149822462682
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Mar 3 21:37:13 2022 -0700

    Removing osm2pgsql from api docker because it seems to be unsupported.

commit 0e10522f3cdc7edd374c14f230bb2dbd68ab1a94
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Mar 3 21:36:39 2022 -0700

    Disabling import button for docker.  Adding download instructions.

commit e1d8f55d159fb4d4ef2ece859ef7cc1059885791
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Mar 3 21:14:55 2022 -0700

    Map files can be imported on RPi.

commit 3128530e09968430a71b27cd2e75a567f85a708b
Author: Roland <roland@learningselfreliance.com>
Date:   Thu Mar 3 15:31:54 2022 -0700

    Adding shm_size to increase size to render large map tiles.

commit f3cf80fae0ce46f44ea9ac5033f68a32d0197ae3
Author: Roland <roland@learningselfreliance.com>
Date:   Wed Mar 2 10:39:18 2022 -0700

    Can get status of map imports.

commit 3b7bbce004cd55a5d89980c2c09124b72c450bc6
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 1 20:51:13 2022 -0700

    Adding map link to 50x.html.  Swapping links using hostname to make it easier on the user.

commit a8b6b60fe182233f40c2f1ecd825636c3d37fd23
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 1 12:01:50 2022 -0700

    Moving Inventory link to end.

commit c5346b8dc2ff83262d6fd153130912809dfd7574
Author: Roland <roland@learningselfreliance.com>
Date:   Tue Mar 1 12:00:47 2022 -0700

    Squashed commit of the following:

    commit 39f0b8f3fc049fffd140d551cd91d7d9dba6d416
    Author: Roland <roland@learningselfreliance.com>
    Date:   Tue Mar 1 12:00:13 2022 -0700

        Starting map refresh function.

    commit 2fac9ddf07e378389ddd85ca972f33b3323d30ac
    Author: Roland <roland@learningselfreliance.com>
    Date:   Mon Feb 28 16:43:47 2022 -0700

        Updating readme with map requirements.

    commit 8325e1ea89537128329e313ab4b5588f8809d1c9
    Author: Roland <roland@learningselfreliance.com>
    Date:   Mon Feb 28 16:05:35 2022 -0700

        Fixing order of map db init scripts.  Creating index.html to display map.  Using :8084 index file to display the map.

    commit 8f81539dd9d25fc26607f472c156d41ddcfc11f5
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 19:17:14 2022 -0700

        Installing apache2.

    commit c7aaef07f8291070aeaf5e7df553e34aca0719ec
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 18:41:07 2022 -0700

        Always configure postgres extensions.  Initialize gis with dc.

    commit aa725ed4b8ec46d17accebd96dd41508ea2d9caf
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 15:48:20 2022 -0700

        Map is displayed as iframe.

    commit de9bb92dcaf44996e7727adce7774f01895ddf52
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 12:16:56 2022 -0700

        Running script in openstreetmap directory.

    commit f277388095d2221aa6407643ab655f26dc349eb7
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 12:06:41 2022 -0700

        Bumping version for map.

    commit afd4aaeffb2aced2dc5c1a8f94038885c8287265
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 11:59:31 2022 -0700

        Checking that the installation script is on an rpi.

    commit a799382ceac3c201fc01d41fd63e37607282831a
    Author: Roland <roland@learningselfreliance.com>
    Date:   Sun Feb 27 11:49:21 2022 -0700

        Only running mod_tile install once.

    commit 0b98bb19cd4d86c9630a863cab4eb46410d1c9d1
    Author: Roland <roland@learningselfreliance.com>
    Date:   Fri Feb 25 11:31:21 2022 -0700

        Building out map install script.

    commit 7c799775c11df573fa6cd65451adea28c0832cb0
    Author: Roland <roland@learningselfreliance.com>
    Date:   Thu Feb 24 12:16:53 2022 -0700

        Creating OSM installation script.

    commit d3b93a3ee97d1ae4a00b6a0423ed59ea8b897ccd
    Author: Roland <roland@learningselfreliance.com>
    Date:   Tue Feb 22 15:39:04 2022 -0700

        Removing unused map module.
2022-03-08 13:26:03 -07:00
Roland
aabfe6a695
Downloading bestvideo+bestaudio as last resort. 2022-03-02 15:48:06 -07:00
Roland
5342d573c2
Using yt-dlp's match-filter to skip live videos. 2022-02-20 10:45:53 -07:00
Roland
f2af5e048b
Configs are initialized on import so Manager is started for all processes. 2022-02-16 23:05:43 -07:00
Roland
08065155cf
Squashing config rework. 2022-02-16 21:57:27 -07:00
StruckCroissant
9a1fe7323a Added channel whitespace test 2022-02-13 00:06:42 -06:00
StruckCroissant
d04a9c05d0
Merge branch 'lrnselfreliance:master' into master 2022-02-10 20:29:39 -06:00
Roland
59bd675afd
Adding video format 18 for 360p format. 2022-02-08 10:00:32 -07:00
StruckCroissant
8bfa2cdba9
Update api.py 2022-02-07 13:47:17 -06:00
StruckCroissant
d26ddd2ee0 Stripping whitespace from posted channel names 2022-02-07 12:04:38 -06:00
Roland
a550b7d27b
Using lib in otp tests. 2022-02-03 13:16:10 -07:00
Roland
f2e4fcfa9c
Importing modules so its easier to see the relation between an api and its dependencies. 2022-02-03 12:39:14 -07:00
Roland
b4c1565fbb
Using new openapi.definition to show validation in docs. 2022-02-03 11:36:26 -07:00
Roland
56f31639db
Adding link to /docs for new Sanic version. 2022-02-03 10:07:04 -07:00
Roland
72cb9ce4ff
Missing Archive titles are filled in during refresh. 2022-02-03 08:50:40 -07:00
Roland
01bfe4de97
Limiting maximum title length in Archive files. 2022-02-02 20:48:43 -07:00
Roland
7b1c07f076
Archive files are migrated to new format. 2022-02-02 20:43:11 -07:00
Roland
8d3f65425a
Creating method that returns all Video files. Using this new method to delete the video files. 2022-02-02 16:30:58 -07:00
Roland
f655924c82
Fixing bug with DirectoriesRequest. 2022-02-02 16:18:08 -07:00
Roland
829f5eea08
Fixing archive counting. 2022-02-02 16:10:59 -07:00
Roland
a6a3284b49
Removing accidental print. 2022-02-02 16:09:39 -07:00
Roland
2946255fc6
Default to false to avoid non-archive files. 2022-02-02 16:08:18 -07:00
Roland
c1baeabe1f
Archive files with title in the name can be found. 2022-02-02 15:58:22 -07:00