commit f54ac72e5ebffbf709df127d7692ae01ba60b603 Author: Roland Date: Sun Oct 15 16:31:49 2023 -0600 Slowly building all WROLPi help docs. diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..9f21b54 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +/venv/ diff --git a/.idea/.gitignore b/.idea/.gitignore new file mode 100644 index 0000000..13566b8 --- /dev/null +++ b/.idea/.gitignore @@ -0,0 +1,8 @@ +# Default ignored files +/shelf/ +/workspace.xml +# Editor-based HTTP Client requests +/httpRequests/ +# Datasource local storage ignored files +/dataSources/ +/dataSources.local.xml diff --git a/.idea/inspectionProfiles/profiles_settings.xml b/.idea/inspectionProfiles/profiles_settings.xml new file mode 100644 index 0000000..105ce2d --- /dev/null +++ b/.idea/inspectionProfiles/profiles_settings.xml @@ -0,0 +1,6 @@ + + + + \ No newline at end of file diff --git a/.idea/misc.xml b/.idea/misc.xml new file mode 100644 index 0000000..ed7d70c --- /dev/null +++ b/.idea/misc.xml @@ -0,0 +1,7 @@ + + + + + + \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml new file mode 100644 index 0000000..2d91b9b --- /dev/null +++ b/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/.idea/runConfigurations/mkdocs.xml b/.idea/runConfigurations/mkdocs.xml new file mode 100644 index 0000000..c5d3a98 --- /dev/null +++ b/.idea/runConfigurations/mkdocs.xml @@ -0,0 +1,24 @@ + + + + + \ No newline at end of file diff --git a/.idea/vcs.xml b/.idea/vcs.xml new file mode 100644 index 0000000..35eb1dd --- /dev/null +++ b/.idea/vcs.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/.idea/wrolpi-help.iml b/.idea/wrolpi-help.iml new file mode 100644 index 0000000..2d5e38d --- /dev/null +++ b/.idea/wrolpi-help.iml @@ -0,0 +1,10 @@ + + + + + + + + + + \ No newline at end of file diff --git a/docs/archives/archive-page.png b/docs/archives/archive-page.png new file mode 100644 index 0000000..a665cb5 Binary files /dev/null and b/docs/archives/archive-page.png differ diff --git a/docs/archives/files.md b/docs/archives/files.md new file mode 100644 index 0000000..9198739 --- /dev/null +++ b/docs/archives/files.md @@ -0,0 +1,39 @@ +# Archive Files + +WROLPi creates several files when archiving a webpage: + +1. A [Singlefile](#singlefile) which stores an exact copy of the web page. +2. A [Readability](#readability) file. +3. A Readability text file. +4. A Readability JSON file. +5. A screenshot file of the web page. + +These files are all share the same file name, but different suffixes. + +| File Suffix | Purpose | +|-------------------|--------------------------------------------| +| .html | The Singlefile. | +| .readability.html | The Readability extract of the Singlefile. | +| .readability.txt | The article text extracted by Readability. | +| .readability.json | Data from Readability (URL/author/etc). | +| .png | The screenshot of the web page. | + +Archive files are named using three parts: + +1. The domain directory +2. The archive date time (YYYY-mm-dd-HH-MM-SS). +3. The title of the web page. + +If archiving [https://wrolpi.org/getting-started](https://wrolpi.org/getting-started), the resulting files would be +named like so: + +``` +wrolpi.org/2023-01-01-12-00-00_Getting Started.html +wrolpi.org/2023-01-01-12-00-00_Getting Started.readability.html +wrolpi.org/2023-01-01-12-00-00_Getting Started.readability.txt +wrolpi.org/2023-01-01-12-00-00_Getting Started.readability.json +wrolpi.org/2023-01-01-12-00-00_Getting Started.png +``` + +WROLPi will attempt to extract the Readability and create a Screenshot from the Singlefile, if this is not possible +WROLPi will still keep the lone Singlefile. diff --git a/docs/archives/index.md b/docs/archives/index.md new file mode 100644 index 0000000..e750db8 --- /dev/null +++ b/docs/archives/index.md @@ -0,0 +1,32 @@ +# Archives + +The Archives module allows viewing, organizing, and downloading of webpages as SingleFile files. Each Archive's URL +is retained, they are organized by their domain in the Domains tab. + +![An Archive page](archive-page.png) + +## Singlefile + +A Singlefile is an HTML file which contains encoded copies of all files necessary to render a webpage. Normally a +webpage will link to many resources outside the page you are viewing, this causes problems when you attempt to save the +webpage because you will need to download and organize all of these files. Singlefile solves this problem by encoding +and embedding all the related files within a single HTML file. + +The Singlefile project has many advantages. + +* A single HTML file that can be easily shared with others. +* All that is necessary to view the archive is a compatible browser. No external files or services. +* An exact copy that handles rendering of Javascript that the page will contain. +* Images and style is preserved. + +## Readability + +[Readability](https://github.com/mozilla/readability) is a Mozilla project which extracts the article from a web page. +Because a web page is made up of many HTML, CSS, Javascript words that we do not want to search, WROLPi uses Readability +to extract the words that will be indexed. + +![A preview of the Readability extract of wrolpi.org](readability-preview.png) + +## Domain + +A domain is the host of the archived URL. For example, the host of https://wrolpi.org/getting-started is wrolpi.org. \ No newline at end of file diff --git a/docs/archives/readability-preview.png b/docs/archives/readability-preview.png new file mode 100644 index 0000000..a219a3b Binary files /dev/null and b/docs/archives/readability-preview.png differ diff --git a/docs/files/3d-benchy-preview.png b/docs/files/3d-benchy-preview.png new file mode 100644 index 0000000..50fb009 Binary files /dev/null and b/docs/files/3d-benchy-preview.png differ diff --git a/docs/files/files-page.png b/docs/files/files-page.png new file mode 100644 index 0000000..494f90c Binary files /dev/null and b/docs/files/files-page.png differ diff --git a/docs/files/index.md b/docs/files/index.md new file mode 100644 index 0000000..8ca9b34 --- /dev/null +++ b/docs/files/index.md @@ -0,0 +1,54 @@ +# Files + +The Files module allows you to view the files found in your media directory. No files outside the media directory +can be viewed. + +The Files module is different from other modules in WROLPi (Videos/Archives/etc.) because the files displayed are not +fetched from the database, they are read from the media directory live. This is necessary to allow refreshing of files +that have recently been created, and are not yet in the database. + +![Files Page](files-page.png) + +## Refreshing + +WROLPi can search any file in your media directory, this can only be accomplished after a full files refresh has been +completed. + +> To refresh all your files, click the **Files** link in the top navigation bar, then click **Refresh All** found at the +> bottom of the files page. + +## Files Tools + +The contents of the media directory can be manipulated in the Files module using the buttons on the bottom of the files +table. + +**It is recommended to manipulate files using the provided tools. WROLPi can lose track of your files if you manipulate +them outside of these tools.** + +The tool buttons are as displayed: + +1. **Refresh All** or **Refresh** button + * This button will refresh all files in your media directory, or any files/directories you have selected. +2. **Delete** button + * This button will delete any files or directories selected. +3. **Rename** button + * Allows you to rename a single file or directory. +4. **Move** button + * Allows you to move files/directories into another directory. +5. **New Directory** button + * Allows you to create a new directory in the directory selected (or, the top directory if no directory is + selected.) + +## File Preview +
hello
+ +When clicking on any file, WROLPi will attempt to provide you with a preview of the file. If no previewer has been +created for the particular file type has been created, then the generic file preview will be used. + +A preview of a 3D STL model file (which as been tagged with Tools): + +![A preview of a 3D STL file](3d-benchy-preview.png) + +A preview of a PDF file (which has been tagged with First Aid and Primitive): + +![A preview of a PDF file](pdf-preview.png) diff --git a/docs/files/pdf-preview.png b/docs/files/pdf-preview.png new file mode 100644 index 0000000..5fd7168 Binary files /dev/null and b/docs/files/pdf-preview.png differ diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..5124bff --- /dev/null +++ b/docs/index.md @@ -0,0 +1,22 @@ +# Home + +This is the WROLPi help documentation. + +## Modules + +* [Files](files/index.md) +* [Videos](videos/index.md) +* [Archives](archives/index.md) +* [Map](map/index.md) +* [Zim](zim/index.md) + +## WROLPi + +* [Systems](system/index.md) +* [Recommended Hardware](system/recommended-hardware.md) + +## Guiding Principals + +1. Storage is cheaper than power. +2. Two is one, one is none. +3. Run silently. diff --git a/docs/map/index.md b/docs/map/index.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/system/index.md b/docs/system/index.md new file mode 100644 index 0000000..5d80773 --- /dev/null +++ b/docs/system/index.md @@ -0,0 +1,53 @@ +# Overview of WROLPi Systems + +* WROLPi is based off [Debian OS](https://debian.org) and [Raspberry Pi OS](https://www.raspberrypi.com/software/) +* WROLPi has 4 user-facing services: + * [API](#api) + * [App](#app) + * [Map](#map) + * [Zim](#zim) + +## Services + +### API + +| Software Project | Purpose | +|------------------------------------------------------------|-------------------------------------------------------------------------| +| [Python](https://python.org) | The language of the WROLPi API and scripts. | +| [Sanic](https://sanic.dev) | An async web app framework, used for the API. | +| [SQLAlchemy](https://www.sqlalchemy.org/) | The database ORM. | +| [yt-dlp](https://github.com/yt-dlp/yt-dlp) | Used to download videos. | +| [SingleFile](https://github.com/gildas-lormeau/SingleFile) | Used to create Archives. | +| [Readability](https://github.com/mozilla/readability) | Used to extract the article from an Archive. | +| [Chromium](https://chromium.org) | Used to create Archives. | +| [FFMPEG](https://ffmpeg.org/) | Video processing software which is used by the API to manipulate videos | +| [libzim](https://pypi.org/project/libzim/) | Used to search the ZIM files | + +### DB + +| Software Project | Purpose | +|-------------------------------------------|----------------------------------------------------------------------| +| [Postgresql](https://www.postgresql.org/) | Facilitates searching of files, scheduling downloads, and much more. | + +### App + +| Software Project | Purpose | +|----------------------------------------------|--------------------------------------------------------------------| +| [ReactJS](https://react.dev/) | The framework which controls the App. Built on-top of Javascript. | +| [SemanticUI](https://react.semantic-ui.com/) | The framework and style of the App's buttons, tables, etc. | +| [NodeJS](https://nodejs.org) | The run-time environment which runs the ReactJS app. | + +### Map + +| Software Project | Purpose | +|-------------------------------------------------|------------------------------------------| +| [OpenStreetMap](https://www.openstreetmap.org/) | Provides the Map data | +| [osmium](https://osmcode.org/osmium-tool/) | Used to merge map PBF files. | +| [osm2pgsql](https://osm2pgsql.org/) | Used to import PBF files into Postgresql | + +### Zim + +| Software Project | Purpose | +|-------------------------------------------------------|-------------------------------| +| [Kiwix](https://kiwix.org) | Provides the ZIM files | +| [Kiwix Server](https://github.com/kiwix/kiwix-tools/) | Hosts the ZIM files interface | diff --git a/docs/system/recommended-hardware.md b/docs/system/recommended-hardware.md new file mode 100644 index 0000000..ab6e9f7 --- /dev/null +++ b/docs/system/recommended-hardware.md @@ -0,0 +1,18 @@ +## Recommended Hardware + +WROLPis is expected to run in one of two systems, on a Raspberry Pi, or on a computer running Debian OS. + +### Recommended Raspberry Pi Hardware + +1. Raspberry Pi 4 (4GB+ RAM). +2. Micro SD Card (32GB+). +3. External USB Drive. +4. Keyboard, mouse, monitor. +5. Power supplies, USB cables, etc. + +### Recommended Debian Hardware + +1. A laptop is recommended, but any other AMD64 compatible system is acceptable. + 1. 4GB RAM +2. External USB Drive. +3. Power supplies, USB cables, etc. diff --git a/docs/system/special-directories.md b/docs/system/special-directories.md new file mode 100644 index 0000000..80b43d5 --- /dev/null +++ b/docs/system/special-directories.md @@ -0,0 +1,44 @@ +# Special Directories + +| Path | Purpose | +|-------------------------|----------------------------------------------------------| +| `/media/wrolpi/` | The [Media Directory](#media-directory). | +| `/opt/wrolpi/` | The [source code](#wrolpi-source-directory) of WROLPi. | +| `/media/wrolpi/config/` | The [configuration files](#wrolpi-config) of the WROLPi. | +| `/opt/wrolpi-blobs/` | Contains files necessary to the repair of a WROLPi. | +| `/opt/wrolpi-help/` | These help files. | + +## Media Directory + +`/media/wrolpi/` + +The media directory is a ubiquitous part of WROLPi. It is expected that an external drive will be mounted here. All +files that are downloaded by WROLPi will be saved within this directory. + +The normal media directory is `/media/wrolpi` + +## WROLPi Source Directory + +`/opt/wrolpi/` + +This directory contains the source code of the WROLPi API, app, etc. + +## WROLPi Config + +`/media/wrolpi/config/` + +This directory contains the configuration files of the WROLPi. These files are considered the "source of truth" and +their contents will alter the behavior of the WROLPi. + +## WROLPi Blobs + +`/opt/wrolpi-blobs/` + +This directory contains the files necessary to repair a WROLPi. For example, the `gis-map.dump.gz` file contains a +Postgresql dump of the map database. **It is best to leave this directory as-is.** + +## WROLPi Help + +`/opt/wrolpi-help/` + +This help file you are reading is contained in this directory. diff --git a/docs/videos/index.md b/docs/videos/index.md new file mode 100644 index 0000000..1517563 --- /dev/null +++ b/docs/videos/index.md @@ -0,0 +1,6 @@ +# Videos + +## Overview + +The Videos module facilitates the viewing, organizing, and downloading of videos files. The main page of the Videos +Module (/videos) allows the searching of all video files found during a file refresh of the media directory. \ No newline at end of file diff --git a/docs/zim/index.md b/docs/zim/index.md new file mode 100644 index 0000000..e69de29 diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..645d187 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1 @@ +site_name: WROLPi Help diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..bc67705 --- /dev/null +++ b/requirements.txt @@ -0,0 +1 @@ +mkdocs>=1.5 \ No newline at end of file