rt/wrolpi-help
1
0
Fork 0
mirror of https://github.com/wrolpi/wrolpi-help.git synced 2026-10-25 18:40:11 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Slowly building all WROLPi help docs.

Browse source
This commit is contained in:
Roland 2023-10-15 16:31:49 -06:00
commit f54ac72e5e
No known key found for this signature in database
GPG key ID: 1BA620CE6C9A4680
25 changed files with 340 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored Normal file
View file

@ -0,0 +1 @@
/venv/

8
.idea/.gitignore generated vendored Normal file
View file

@ -0,0 +1,8 @@
# Default ignored files
/shelf/
/workspace.xml
# Editor-based HTTP Client requests
/httpRequests/
# Datasource local storage ignored files
/dataSources/
/dataSources.local.xml

6
.idea/inspectionProfiles/profiles_settings.xml generated Normal file
View file

@ -0,0 +1,6 @@
<component name="InspectionProjectProfileManager">
<settings>
<option name="USE_PROJECT_PROFILE" value="false" />
<version value="1.0" />
</settings>
</component>

7
.idea/misc.xml generated Normal file
View file

@ -0,0 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="Black">
<option name="sdkName" value="Python 3.11 (wrolpi-help)" />
</component>
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.11 (wrolpi-help)" project-jdk-type="Python SDK" />
</project>

8
.idea/modules.xml generated Normal file
View file

@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ProjectModuleManager">
<modules>
<module fileurl="file://$PROJECT_DIR$/.idea/wrolpi-help.iml" filepath="$PROJECT_DIR$/.idea/wrolpi-help.iml" />
</modules>
</component>
</project>

24
.idea/runConfigurations/mkdocs.xml generated Normal file
View file

@ -0,0 +1,24 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="mkdocs" type="PythonConfigurationType" factoryName="Python" nameIsGenerated="true">
<module name="wrolpi-help" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="PARENT_ENVS" value="true" />
<envs>
<env name="PYTHONUNBUFFERED" value="1" />
</envs>
<option name="SDK_HOME" value="" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="IS_MODULE_SDK" value="true" />
<option name="ADD_CONTENT_ROOTS" value="true" />
<option name="ADD_SOURCE_ROOTS" value="true" />
<EXTENSION ID="PythonCoverageRunConfigurationExtension" runner="coverage.py" />
<option name="SCRIPT_NAME" value="$PROJECT_DIR$/venv/bin/mkdocs" />
<option name="PARAMETERS" value="serve -a 0.0.0.0:8085" />
<option name="SHOW_COMMAND_LINE" value="false" />
<option name="EMULATE_TERMINAL" value="false" />
<option name="MODULE_MODE" value="false" />
<option name="REDIRECT_INPUT" value="false" />
<option name="INPUT_FILE" value="" />
<method v="2" />
</configuration>
</component>

6
.idea/vcs.xml generated Normal file
View file

@ -0,0 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="VcsDirectoryMappings">
<mapping directory="" vcs="Git" />
</component>
</project>

10
.idea/wrolpi-help.iml generated Normal file
View file

@ -0,0 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="PYTHON_MODULE" version="4">
<component name="NewModuleRootManager">
<content url="file://$MODULE_DIR$">
<excludeFolder url="file://$MODULE_DIR$/venv" />
</content>
<orderEntry type="jdk" jdkName="Python 3.11 (wrolpi-help)" jdkType="Python SDK" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
</module>

BIN
docs/archives/archive-page.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 112 KiB

39
docs/archives/files.md Normal file
View file

@ -0,0 +1,39 @@
# Archive Files
WROLPi creates several files when archiving a webpage:
1. A [Singlefile](#singlefile) which stores an exact copy of the web page.
2. A [Readability](#readability) file.
3. A Readability text file.
4. A Readability JSON file.
5. A screenshot file of the web page.
These files are all share the same file name, but different suffixes.
| File Suffix | Purpose |
|-------------------|--------------------------------------------|
| .html | The Singlefile. |
| .readability.html | The Readability extract of the Singlefile. |
| .readability.txt | The article text extracted by Readability. |
| .readability.json | Data from Readability (URL/author/etc). |
| .png | The screenshot of the web page. |
Archive files are named using three parts:
1. The domain directory
2. The archive date time (YYYY-mm-dd-HH-MM-SS).
3. The title of the web page.
If archiving [https://wrolpi.org/getting-started](https://wrolpi.org/getting-started), the resulting files would be
named like so:
```
wrolpi.org/2023-01-01-12-00-00_Getting Started.html
wrolpi.org/2023-01-01-12-00-00_Getting Started.readability.html
wrolpi.org/2023-01-01-12-00-00_Getting Started.readability.txt
wrolpi.org/2023-01-01-12-00-00_Getting Started.readability.json
wrolpi.org/2023-01-01-12-00-00_Getting Started.png
```
WROLPi will attempt to extract the Readability and create a Screenshot from the Singlefile, if this is not possible
WROLPi will still keep the lone Singlefile.

32
docs/archives/index.md Normal file
View file

@ -0,0 +1,32 @@
# Archives
The Archives module allows viewing, organizing, and downloading of webpages as SingleFile files. Each Archive's URL
is retained, they are organized by their domain in the Domains tab.
![An Archive page](archive-page.png)
## Singlefile
A Singlefile is an HTML file which contains encoded copies of all files necessary to render a webpage. Normally a
webpage will link to many resources outside the page you are viewing, this causes problems when you attempt to save the
webpage because you will need to download and organize all of these files. Singlefile solves this problem by encoding
and embedding all the related files within a single HTML file.
The Singlefile project has many advantages.
* A single HTML file that can be easily shared with others.
* All that is necessary to view the archive is a compatible browser. No external files or services.
* An exact copy that handles rendering of Javascript that the page will contain.
* Images and style is preserved.
## Readability
[Readability](https://github.com/mozilla/readability) is a Mozilla project which extracts the article from a web page.
Because a web page is made up of many HTML, CSS, Javascript words that we do not want to search, WROLPi uses Readability
to extract the words that will be indexed.
![A preview of the Readability extract of wrolpi.org](readability-preview.png)
## Domain
A domain is the host of the archived URL. For example, the host of https://wrolpi.org/getting-started is wrolpi.org.

BIN
docs/archives/readability-preview.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 82 KiB

BIN
docs/files/3d-benchy-preview.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 19 KiB

BIN
docs/files/files-page.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 24 KiB

54
docs/files/index.md Normal file
View file

@ -0,0 +1,54 @@
# Files
The Files module allows you to view the files found in your media directory. No files outside the media directory
can be viewed.
The Files module is different from other modules in WROLPi (Videos/Archives/etc.) because the files displayed are not
fetched from the database, they are read from the media directory live. This is necessary to allow refreshing of files
that have recently been created, and are not yet in the database.
![Files Page](files-page.png)
## Refreshing
WROLPi can search any file in your media directory, this can only be accomplished after a full files refresh has been
completed.
> To refresh all your files, click the **Files** link in the top navigation bar, then click **Refresh All** found at the
> bottom of the files page.
## Files Tools
The contents of the media directory can be manipulated in the Files module using the buttons on the bottom of the files
table.
**It is recommended to manipulate files using the provided tools. WROLPi can lose track of your files if you manipulate
them outside of these tools.**
The tool buttons are as displayed:
1. **Refresh All** or **Refresh** button
* This button will refresh all files in your media directory, or any files/directories you have selected.
2. **Delete** button
* This button will delete any files or directories selected.
3. **Rename** button
* Allows you to rename a single file or directory.
4. **Move** button
* Allows you to move files/directories into another directory.
5. **New Directory** button
* Allows you to create a new directory in the directory selected (or, the top directory if no directory is
selected.)
## File Preview
<details>hello</details>
When clicking on any file, WROLPi will attempt to provide you with a preview of the file. If no previewer has been
created for the particular file type has been created, then the generic file preview will be used.
A preview of a 3D STL model file (which as been tagged with Tools):
![A preview of a 3D STL file](3d-benchy-preview.png)
A preview of a PDF file (which has been tagged with First Aid and Primitive):
![A preview of a PDF file](pdf-preview.png)

BIN
docs/files/pdf-preview.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 295 KiB

22
docs/index.md Normal file
View file

@ -0,0 +1,22 @@
# Home
This is the WROLPi help documentation.
## Modules
* [Files](files/index.md)
* [Videos](videos/index.md)
* [Archives](archives/index.md)
* [Map](map/index.md)
* [Zim](zim/index.md)
## WROLPi
* [Systems](system/index.md)
* [Recommended Hardware](system/recommended-hardware.md)
## Guiding Principals
1. Storage is cheaper than power.
2. Two is one, one is none.
3. Run silently.

0
docs/map/index.md Normal file
View file

53
docs/system/index.md Normal file
View file

@ -0,0 +1,53 @@
# Overview of WROLPi Systems
* WROLPi is based off [Debian OS](https://debian.org) and [Raspberry Pi OS](https://www.raspberrypi.com/software/)
* WROLPi has 4 user-facing services:
* [API](#api)
* [App](#app)
* [Map](#map)
* [Zim](#zim)
## Services
### API
| Software Project | Purpose |
|------------------------------------------------------------|-------------------------------------------------------------------------|
| [Python](https://python.org) | The language of the WROLPi API and scripts. |
| [Sanic](https://sanic.dev) | An async web app framework, used for the API. |
| [SQLAlchemy](https://www.sqlalchemy.org/) | The database ORM. |
| [yt-dlp](https://github.com/yt-dlp/yt-dlp) | Used to download videos. |
| [SingleFile](https://github.com/gildas-lormeau/SingleFile) | Used to create Archives. |
| [Readability](https://github.com/mozilla/readability) | Used to extract the article from an Archive. |
| [Chromium](https://chromium.org) | Used to create Archives. |
| [FFMPEG](https://ffmpeg.org/) | Video processing software which is used by the API to manipulate videos |
| [libzim](https://pypi.org/project/libzim/) | Used to search the ZIM files |
### DB
| Software Project | Purpose |
|-------------------------------------------|----------------------------------------------------------------------|
| [Postgresql](https://www.postgresql.org/) | Facilitates searching of files, scheduling downloads, and much more. |
### App
| Software Project | Purpose |
|----------------------------------------------|--------------------------------------------------------------------|
| [ReactJS](https://react.dev/) | The framework which controls the App. Built on-top of Javascript. |
| [SemanticUI](https://react.semantic-ui.com/) | The framework and style of the App's buttons, tables, etc. |
| [NodeJS](https://nodejs.org) | The run-time environment which runs the ReactJS app. |
### Map
| Software Project | Purpose |
|-------------------------------------------------|------------------------------------------|
| [OpenStreetMap](https://www.openstreetmap.org/) | Provides the Map data |
| [osmium](https://osmcode.org/osmium-tool/) | Used to merge map PBF files. |
| [osm2pgsql](https://osm2pgsql.org/) | Used to import PBF files into Postgresql |
### Zim
| Software Project | Purpose |
|-------------------------------------------------------|-------------------------------|
| [Kiwix](https://kiwix.org) | Provides the ZIM files |
| [Kiwix Server](https://github.com/kiwix/kiwix-tools/) | Hosts the ZIM files interface |

18
docs/system/recommended-hardware.md Normal file
View file

@ -0,0 +1,18 @@
## Recommended Hardware
WROLPis is expected to run in one of two systems, on a Raspberry Pi, or on a computer running Debian OS.
### Recommended Raspberry Pi Hardware
1. Raspberry Pi 4 (4GB+ RAM).
2. Micro SD Card (32GB+).
3. External USB Drive.
4. Keyboard, mouse, monitor.
5. Power supplies, USB cables, etc.
### Recommended Debian Hardware
1. A laptop is recommended, but any other AMD64 compatible system is acceptable.
1. 4GB RAM
2. External USB Drive.
3. Power supplies, USB cables, etc.

44
docs/system/special-directories.md Normal file
View file

@ -0,0 +1,44 @@
# Special Directories
| Path | Purpose |
|-------------------------|----------------------------------------------------------|
| `/media/wrolpi/` | The [Media Directory](#media-directory). |
| `/opt/wrolpi/` | The [source code](#wrolpi-source-directory) of WROLPi. |
| `/media/wrolpi/config/` | The [configuration files](#wrolpi-config) of the WROLPi. |
| `/opt/wrolpi-blobs/` | Contains files necessary to the repair of a WROLPi. |
| `/opt/wrolpi-help/` | These help files. |
## Media Directory
`/media/wrolpi/`
The media directory is a ubiquitous part of WROLPi. It is expected that an external drive will be mounted here. All
files that are downloaded by WROLPi will be saved within this directory.
The normal media directory is `/media/wrolpi`
## WROLPi Source Directory
`/opt/wrolpi/`
This directory contains the source code of the WROLPi API, app, etc.
## WROLPi Config
`/media/wrolpi/config/`
This directory contains the configuration files of the WROLPi. These files are considered the "source of truth" and
their contents will alter the behavior of the WROLPi.
## WROLPi Blobs
`/opt/wrolpi-blobs/`
This directory contains the files necessary to repair a WROLPi. For example, the `gis-map.dump.gz` file contains a
Postgresql dump of the map database. **It is best to leave this directory as-is.**
## WROLPi Help
`/opt/wrolpi-help/`
This help file you are reading is contained in this directory.

6
docs/videos/index.md Normal file
View file

@ -0,0 +1,6 @@
# Videos
## Overview
The Videos module facilitates the viewing, organizing, and downloading of videos files. The main page of the Videos
Module (/videos) allows the searching of all video files found during a file refresh of the media directory.

0
docs/zim/index.md Normal file
View file

1
mkdocs.yml Normal file
View file

@ -0,0 +1 @@
site_name: WROLPi Help

1
requirements.txt Normal file
View file

@ -0,0 +1 @@
mkdocs>=1.5