diff --git a/README.md b/README.md index 472e3ec4..9bd9ca23 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ network so that any user with a laptop/tablet/phone can connect and use the libr * [Features](#features) * [Demo](#demo) +* [Download](#download) * [Try WROLPi](#try-wrolpi) * [Debian 11 Install](#debian-11-install) * [Raspberry Pi Install](#raspberrypi-install) @@ -46,7 +47,7 @@ network so that any user with a laptop/tablet/phone can connect and use the libr - [ ] Synchronizer and/or duplicator - [ ] Food storage calculator -## Feature Matrix +## Module features matrix | **Module** | **Search** | **View** | **Download** | |--------------------|:-----------|----------|--------------| @@ -61,6 +62,10 @@ network so that any user with a laptop/tablet/phone can connect and use the libr ![YouTube Demo Video](https://raw.githubusercontent.com/lrnselfreliance/wrolpi/master/demo_link.jpg) +# Download + +Images are available on [wrolpi.org](https://wrolpi.org) + # Try WROLPi! You can try out WROLPi by running the docker containers. @@ -99,21 +104,14 @@ Install WROLPi onto a fresh Debian 11 system. 4. Start WROLPi * `sudo systemctl start wrolpi.target` 5. Browse to the IP address of your WROLPi! -6. Run the map installation script (Optional) - * `sudo /bin/bash /opt/wrolpi/scripts/install_map_debian_11.sh` # RaspberryPi Install -## Download - -* Download the latest image from [wrolpi.org](https://wrolpi.org) - -## Manual install Install WROLPi onto a fresh Raspberry Pi. **Warning: this will clobber anything else on the Raspberry Pi! Do not run this installation script elsewhere!** -1. Install [Ubuntu Server 20.04](https://cdimage.ubuntu.com/releases/20.04/release/) onto a RaspberryPi. +1. Install [Raspberry Pi OS](https://www.raspberrypi.com/software/operating-systems/) onto a RaspberryPi. 2. Get the WROLPi installation script * `wget https://raw.githubusercontent.com/lrnselfreliance/wrolpi/release/install.sh -O /tmp/install.sh` 3. Run the installation script @@ -132,7 +130,7 @@ Install WROLPi onto a fresh Raspberry Pi. * External USB hard drive * Power supply and cables * WiFi USER device such as a phone, tablet, or laptop. -1. A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be +2. A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be scarce when WROLPi is needed most. ## User expectations @@ -143,13 +141,13 @@ Install WROLPi onto a fresh Raspberry Pi. ## Software properties and capabilities 1. A WROLPi instance should be able to "factory-reset" itself without any outside services. -1. WROLPi should function completely without any internet services. -1. A user should have easy access to their data if WROLPi fails: +2. WROLPi should function completely without any internet services. +3. A user should have easy access to their data if WROLPi fails: * For example, a user can watch their videos by navigating a short and intuitive directory structure and opening the video in their preferred video player. -1. WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER. +4. WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER. * If the UI isn't functional, the code should be documented such that a user can restore functionality. -1. WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or +5. WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or processing captions. This is to ensure that when a user adds content during non-WROL time, the processing for optimum performance is already done for a WROL event. diff --git a/UPGRADE.md b/UPGRADE.md index 239486ef..45723700 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -14,8 +14,8 @@ 3. Build all docker containers * `docker-compose build --parallel` 4. Create the map volumes - * `docker volume create --name=openstreetmap-data` - * `docker volume create --name=openstreetmap-rendered-tiles` + * `docker volume create --name=openstreetmap-data` + * `docker volume create --name=openstreetmap-rendered-tiles` 5. Turn on the database * `docker-compose up -d db` 6. Upgrade the database @@ -25,9 +25,5 @@ ## Upgrading Raspberry Pi or Debian 11 Installation -1. Get the latest release install script - * `wget https://raw.githubusercontent.com/lrnselfreliance/wrolpi/release/install.sh -O /tmp/install.sh` -2. Run the installation script - * `sudo /bin/bash /tmp/install.sh` -3. Start WROLPi - * `sudo systemctl start wrolpi.target` +1. Run the upgrade script + * `sudo /bin/bash /opt/wrolpi/upgrade.sh` diff --git a/etc/debian11/README.md b/etc/README.md similarity index 100% rename from etc/debian11/README.md rename to etc/README.md diff --git a/etc/ubuntu20.04/000-default.conf b/etc/raspberrypios/000-default.conf similarity index 93% rename from etc/ubuntu20.04/000-default.conf rename to etc/raspberrypios/000-default.conf index 06b39e80..4a7495d0 100644 --- a/etc/ubuntu20.04/000-default.conf +++ b/etc/raspberrypios/000-default.conf @@ -10,8 +10,8 @@ ServerAdmin webmaster@localhost - LoadTileConfigFile /usr/local/etc/renderd.conf - ModTileRenderdSocketName /var/run/renderd/renderd.sock + LoadTileConfigFile /etc/renderd.conf + ModTileRenderdSocketName /run/renderd/renderd.sock # Timeout before giving up for a tile to be rendered ModTileRequestTimeout 0 # Timeout before giving up for a tile to be rendered that is otherwise missing diff --git a/etc/debian11/index.html b/etc/raspberrypios/index.html similarity index 100% rename from etc/debian11/index.html rename to etc/raspberrypios/index.html diff --git a/etc/ubuntu20.04/mod_tile.conf b/etc/raspberrypios/mod_tile.conf similarity index 100% rename from etc/ubuntu20.04/mod_tile.conf rename to etc/raspberrypios/mod_tile.conf diff --git a/etc/debian11/ports.conf b/etc/raspberrypios/ports.conf similarity index 100% rename from etc/debian11/ports.conf rename to etc/raspberrypios/ports.conf diff --git a/etc/raspberrypios/renderd b/etc/raspberrypios/renderd new file mode 100644 index 00000000..133ef16f --- /dev/null +++ b/etc/raspberrypios/renderd @@ -0,0 +1,153 @@ +#! /bin/sh +### BEGIN INIT INFO +# Provides: renderd +# Required-Start: $remote_fs +# Required-Stop: $remote_fs +# Should-Start: postgresql +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: Mapnik rendering daemon +# Description: Mapnik rendering daemon. +### END INIT INFO + +# Do NOT "set -e" + +# PATH should only include /usr/* if it runs after the mountnfs.sh script +PATH=/sbin:/usr/sbin:/bin:/usr/bin +DESC="Mapnik rendering daemon" +NAME=renderd +DAEMON=/usr/bin/$NAME +DAEMON_ARGS="-c /etc/renderd.conf" +PIDSOCKDIR=/var/run/$NAME +PIDFILE=$PIDSOCKDIR/$NAME.pid +SCRIPTNAME=/etc/init.d/$NAME +RUNASUSER=_renderd + +# Exit if the package is not installed +[ -x "$DAEMON" ] || exit 0 + +# Read configuration variable file if it is present +[ -r /etc/default/$NAME ] && . /etc/default/$NAME + +# Load the VERBOSE setting and other rcS variables +. /lib/init/vars.sh + +# Define LSB log_* functions. +# Depend on lsb-base (>= 3.0-6) to ensure that this file is present. +. /lib/lsb/init-functions + +# +# Function that starts the daemon/service +# +do_start() +{ + [ -d "$PIDSOCKDIR" ] || mkdir -p $PIDSOCKDIR && chown $RUNASUSER $PIDSOCKDIR + # Return + # 0 if daemon has been started + # 1 if daemon was already running + # 2 if daemon could not be started + start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $RUNASUSER --exec $DAEMON --test > /dev/null \ + || return 1 + start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $RUNASUSER --exec $DAEMON -- \ + $DAEMON_ARGS \ + || return 2 + # Add code here, if necessary, that waits for the process to be ready + # to handle requests from services started subsequently which depend + # on this one. As a last resort, sleep for some time. +} + +# +# Function that stops the daemon/service +# +do_stop() +{ + # Return + # 0 if daemon has been stopped + # 1 if daemon was already stopped + # 2 if daemon could not be stopped + # other if a failure occurred + start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME + RETVAL="$?" + [ "$RETVAL" = 2 ] && return 2 + # Wait for children to finish too if this is a daemon that forks + # and if the daemon is only ever run from this initscript. + # If the above conditions are not satisfied then add some other code + # that waits for the process to drop all resources that could be + # needed by services started subsequently. A last resort is to + # sleep for some time. + start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON + [ "$?" = 2 ] && return 2 + # Many daemons don't delete their pidfiles when they exit. + rm -f $PIDFILE + return "$RETVAL" +} + +# +# Function that sends a SIGHUP to the daemon/service +# +do_reload() { + # + # If the daemon can reload its configuration without + # restarting (for example, when it is sent a SIGHUP), + # then implement that here. + # + start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME + return 0 +} + +case "$1" in + start) + log_daemon_msg "Starting $DESC" "$NAME" + do_start + case "$?" in + 0|1) log_end_msg 0 ;; + 2) log_end_msg 1 ;; + esac + ;; + stop) + log_daemon_msg "Stopping $DESC" "$NAME" + do_stop + case "$?" in + 0|1) log_end_msg 0 ;; + 2) log_end_msg 1 ;; + esac + ;; + #reload|force-reload) + # + # If do_reload() is not implemented then leave this commented out + # and leave 'force-reload' as an alias for 'restart'. + # + #log_daemon_msg "Reloading $DESC" "$NAME" + #do_reload + #log_end_msg $? + #;; + restart|force-reload) + # + # If the "reload" option is implemented then remove the + # 'force-reload' alias + # + log_daemon_msg "Restarting $DESC" "$NAME" + do_stop + case "$?" in + 0|1) + do_start + case "$?" in + 0) log_end_msg 0 ;; + 1) log_end_msg 1 ;; # Old process is still running + *) log_end_msg 1 ;; # Failed to start + esac + ;; + *) + # Failed to stop + log_end_msg 1 + ;; + esac + ;; + *) + #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2 + echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2 + exit 3 + ;; +esac + +: diff --git a/etc/debian11/renderd.conf b/etc/raspberrypios/renderd.conf similarity index 91% rename from etc/debian11/renderd.conf rename to etc/raspberrypios/renderd.conf index 21c6e68a..81fae66e 100644 --- a/etc/debian11/renderd.conf +++ b/etc/raspberrypios/renderd.conf @@ -11,7 +11,7 @@ font_dir_recurse=true [ajt] URI=/hot/ -TILEDIR=/var/lib/mod_tile +TILEDIR=/var/cache/renderd/tiles XML=/opt/openstreetmap-carto/mapnik.xml HOST=localhost TILESIZE=256 diff --git a/etc/ubuntu20.04/wrolpi-api.service b/etc/raspberrypios/wrolpi-api.service similarity index 76% rename from etc/ubuntu20.04/wrolpi-api.service rename to etc/raspberrypios/wrolpi-api.service index 069696c1..3533fe9a 100644 --- a/etc/ubuntu20.04/wrolpi-api.service +++ b/etc/raspberrypios/wrolpi-api.service @@ -5,7 +5,7 @@ PartOf=wrolpi.target [Service] Environment=PYTHONUNBUFFERED=1 -ExecStart=/usr/bin/python3 /opt/wrolpi/main.py -v api +ExecStart=/opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py -v api Group=wrolpi User=wrolpi WorkingDirectory=/opt/wrolpi/ diff --git a/etc/raspberrypios/wrolpi-app.service b/etc/raspberrypios/wrolpi-app.service new file mode 100644 index 00000000..11e59b43 --- /dev/null +++ b/etc/raspberrypios/wrolpi-app.service @@ -0,0 +1,13 @@ +[Unit] +Description=WROLPi React APP +Wants=wrolpi-api.service +PartOf=wrolpi.target + +[Service] +ExecStart=/usr/bin/serve -s build -l 5000 +Group=wrolpi +User=wrolpi +WorkingDirectory=/opt/wrolpi/app/ + +[Install] +WantedBy=default.target diff --git a/etc/ubuntu20.04/wrolpi.target b/etc/raspberrypios/wrolpi.target similarity index 100% rename from etc/ubuntu20.04/wrolpi.target rename to etc/raspberrypios/wrolpi.target diff --git a/etc/ubuntu20.04/index.html b/etc/ubuntu20.04/index.html deleted file mode 100644 index 2ab8047f..00000000 --- a/etc/ubuntu20.04/index.html +++ /dev/null @@ -1,37 +0,0 @@ - - - - - - - - - - -
- - - diff --git a/etc/ubuntu20.04/ports.conf b/etc/ubuntu20.04/ports.conf deleted file mode 100644 index ddcbc031..00000000 --- a/etc/ubuntu20.04/ports.conf +++ /dev/null @@ -1,15 +0,0 @@ -# If you just change the port or add more ports here, you will likely also -# have to change the VirtualHost statement in -# /etc/apache2/sites-enabled/000-default.conf - -Listen 8084 - - - Listen 443 - - - - Listen 443 - - -# vim: syntax=apache ts=4 sw=4 sts=4 sr noet diff --git a/etc/ubuntu20.04/renderd.conf b/etc/ubuntu20.04/renderd.conf deleted file mode 100644 index 7ad24e2e..00000000 --- a/etc/ubuntu20.04/renderd.conf +++ /dev/null @@ -1,17 +0,0 @@ -[renderd] -num_threads=4 -tile_dir=/var/lib/mod_tile -stats_file=/var/run/renderd/renderd.stats - -[mapnik] -plugins_dir=/usr/lib/mapnik/3.0/input -font_dir=/usr/share/fonts/truetype -font_dir_recurse=1 - -[ajt] -URI=/hot/ -TILEDIR=/var/lib/mod_tile -XML=/opt/openstreetmap-carto/mapnik.xml -HOST=localhost -TILESIZE=256 -MAXZOOM=20 diff --git a/install.sh b/install.sh index 729f9ec8..fde07466 100755 --- a/install.sh +++ b/install.sh @@ -59,7 +59,7 @@ git clone https://github.com/lrnselfreliance/wrolpi.git /opt/wrolpi || : (cd /opt/wrolpi && git fetch && git checkout "${BRANCH}" && git reset --hard origin/"${BRANCH}") || exit 4 if [ ${rpi} == true ]; then - /opt/wrolpi/scripts/install_ubuntu_20.04.sh 2>&1 | tee /opt/wrolpi/install.log + /opt/wrolpi/scripts/install_raspberrypios.sh 2>&1 | tee /opt/wrolpi/install.log install_code=${PIPESTATUS[0]} elif [ ${debian11} == true ]; then /opt/wrolpi/scripts/install_debian_11.sh 2>&1 | tee /opt/wrolpi/install.log diff --git a/pi-gen/stage2/04-wrolpi/01-packages b/pi-gen/stage2/04-wrolpi/01-packages index 758b2756..c31d0a59 100644 --- a/pi-gen/stage2/04-wrolpi/01-packages +++ b/pi-gen/stage2/04-wrolpi/01-packages @@ -62,10 +62,10 @@ postgresql-contrib protobuf-c-compiler python3-mapnik python3-pip -python3.9-dev -python3.9-doc -python3.9-minimal -python3.9-venv +python3-dev +python3-doc +python3-minimal +python3-venv renderd software-properties-common tar diff --git a/pi-gen/stage2/04-wrolpi/02-run.sh b/pi-gen/stage2/04-wrolpi/02-run.sh index 8946c269..bdb56714 100755 --- a/pi-gen/stage2/04-wrolpi/02-run.sh +++ b/pi-gen/stage2/04-wrolpi/02-run.sh @@ -1,4 +1,12 @@ #! /bin/bash -e mkdir -p "${ROOTFS_DIR}/opt/wrolpi-blobs" -cp files/district-of-columbia-230116.dump.gz "${ROOTFS_DIR}/opt/wrolpi-blobs/gis-map.dump.gz" +cp files/district-of-columbia-*.dump.gz "${ROOTFS_DIR}/opt/wrolpi-blobs/gis-map.dump.gz" + +set +x + +echo +echo "======================================================================" +echo "02-run.sh completed" +echo "======================================================================" +echo diff --git a/pi-gen/stage2/04-wrolpi/03-run-chroot.sh b/pi-gen/stage2/04-wrolpi/03-run-chroot.sh index 9aa17123..a160c049 100755 --- a/pi-gen/stage2/04-wrolpi/03-run-chroot.sh +++ b/pi-gen/stage2/04-wrolpi/03-run-chroot.sh @@ -6,7 +6,7 @@ set -x apt purge -y hostapd apt autoremove -y -cat >> /etc/profile << 'EOF' +cat >>/etc/profile <<'EOF' alias ll='ls -lh' alias la='ll -a' EOF @@ -15,22 +15,24 @@ EOF sed -i 's/port = 5433/port = 5432/' /etc/postgresql/13/main/postgresql.conf # Install Node console commands. -single-file --version || npm i -g serve@12.0.1 single-file-cli@1.0.15 'git+https://github.com/lrnselfreliance/readability-extractor' carto@1.2.0 +single-file --version || npm i -g serve@12.0.1 single-file-cli@1.0.15 readability-extractor@0.0.6 carto@1.2.0 # Put the latest WROLPi release in /opt/wrolpi. rm -rf /opt/wrolpi -git clone -b master https://github.com/lrnselfreliance/wrolpi.git /opt/wrolpi -chown -R ${FIRST_USER_NAME}:${FIRST_USER_NAME} /opt/wrolpi +git clone -b release https://github.com/lrnselfreliance/wrolpi.git /opt/wrolpi # Install Python requirements. pip3 install --upgrade pip -pip3 install -r /opt/wrolpi/requirements.txt +python3 -m venv /opt/wrolpi/venv +/opt/wrolpi/venv/bin/pip3 install -r /opt/wrolpi/requirements.txt # Install webapp. cd /opt/wrolpi/app npm install npm run build +chown -R ${FIRST_USER_NAME}:${FIRST_USER_NAME} /opt/wrolpi + # Configure nginx. cp /opt/wrolpi/nginx.conf /etc/nginx/nginx.conf cp /opt/wrolpi/50x.html /var/www/50x.html @@ -53,9 +55,9 @@ chmod 660 /etc/sudoers.d/90-wrolpi [ -d /media/wrolpi ] || mkdir /media/wrolpi chown ${FIRST_USER_NAME}:${FIRST_USER_NAME} /media/wrolpi -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi-api.service /etc/systemd/system/ -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi-app.service /etc/systemd/system/ -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi.target /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-api.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-app.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi.target /etc/systemd/system/ systemctl enable wrolpi-api.service systemctl enable wrolpi-app.service @@ -63,3 +65,10 @@ systemctl enable wrolpi-app.service # NetworkManager for the hotspot. systemctl enable NetworkManager +set +x + +echo +echo "======================================================================" +echo "03-run-chroot.sh completed" +echo "======================================================================" +echo diff --git a/pi-gen/stage2/04-wrolpi/04-run-chroot.sh b/pi-gen/stage2/04-wrolpi/04-run-chroot.sh index 98cf0258..f070eda2 100755 --- a/pi-gen/stage2/04-wrolpi/04-run-chroot.sh +++ b/pi-gen/stage2/04-wrolpi/04-run-chroot.sh @@ -5,7 +5,7 @@ set -x # Create mapnik config. if [ ! -d /opt/openstreetmap-carto ]; then - git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto + git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto fi (cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff) chown -R ${FIRST_USER_NAME}:${FIRST_USER_NAME} /opt/openstreetmap-carto @@ -15,7 +15,7 @@ if [[ ! -f /opt/openstreetmap-carto/mapnik.xml || ! -s /opt/openstreetmap-carto/ fi # WROLPi user can access WROLPi and Map database. -cat > /home/wrolpi/.pgpass << 'EOF' +cat >/home/wrolpi/.pgpass <<'EOF' 127.0.0.1:5432:gis:_renderd:wrolpi 127.0.0.1:5432:wrolpi:wrolpi:wrolpi EOF @@ -23,19 +23,27 @@ chown -R ${FIRST_USER_NAME}:${FIRST_USER_NAME} /home/wrolpi chmod 0600 /home/wrolpi/.pgpass # Configure renderd. -cp /opt/wrolpi/etc/debian11/renderd.conf /etc/renderd.conf +cp /opt/wrolpi/etc/raspberrypios/renderd.conf /etc/renderd.conf # Configure Apache2 to listen on 8084. Import renderd into apache. -cp /opt/wrolpi/etc/debian11/ports.conf /etc/apache2/ports.conf -cp /opt/wrolpi/etc/debian11/000-default.conf /etc/apache2/sites-available/000-default.conf -cp /opt/wrolpi/etc/ubuntu20.04/mod_tile.conf /etc/apache2/conf-enabled/mod_tile.conf +cp /opt/wrolpi/etc/raspberrypios/ports.conf /etc/apache2/ports.conf +cp /opt/wrolpi/etc/raspberrypios/000-default.conf /etc/apache2/sites-available/000-default.conf +cp /opt/wrolpi/etc/raspberrypios/mod_tile.conf /etc/apache2/conf-enabled/mod_tile.conf # Copy Leaflet files to Apache's directory so they can be used offline. -cp /opt/wrolpi/etc/debian11/index.html \ +cp /opt/wrolpi/etc/raspberrypios/index.html \ /opt/wrolpi/modules/map/leaflet.js \ /opt/wrolpi/modules/map/leaflet.css /var/www/html/ chmod 644 /var/www/html/* -mkdir /var/lib/mod_tile -chown -R _renderd:_renderd /var/lib/mod_tile - # Allow wrolpi user to delete map tiles. usermod -aG _renderd wrolpi + +[ -d /var/cache/renderd/tiles ] || mkdir /var/cache/renderd/tiles +chown -R _renderd:_renderd /var/cache/renderd/tiles + +set +x + +echo +echo "======================================================================" +echo "04-run-chroot.sh completed" +echo "======================================================================" +echo diff --git a/requirements.txt b/requirements.txt index d310cfc0..f5b35d7c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,7 @@ Pillow==9.3.0 Pint==0.18 PyPDF2[crypto]==2.11.2 -PyYAML==5.3.1 # this is the only version available on RPi +PyYAML==6.0 SQLAlchemy==1.3.22 aiohttp==3.8.3 alembic==1.7.5 diff --git a/scripts/build_api_and_app.sh b/scripts/build_api_and_app.sh new file mode 100755 index 00000000..e2693270 --- /dev/null +++ b/scripts/build_api_and_app.sh @@ -0,0 +1,21 @@ +#! /usr/bin/env bash +# This script will rebuild and install dependencies of the API and App. + +set -x +set -e + +# Install any App dependencies. +cd /opt/wrolpi/app || exit 1 +npm install || npm install || npm install || npm install # try install multiple times :( +# Build app in the background. +npm run build & + +# Install python requirements in background job. +/opt/wrolpi/venv/bin/pip3 install -r /opt/wrolpi/requirements.txt & + +wait $(jobs -p) + +# Upgrade the WROLPi database. +(cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py db upgrade) + +systemctl start wrolpi.target diff --git a/scripts/create_db.sh b/scripts/create_db.sh index 82cd4624..75f84956 100755 --- a/scripts/create_db.sh +++ b/scripts/create_db.sh @@ -2,13 +2,13 @@ # Create WROLPi API database. sudo -u postgres psql -c '\l' | grep wrolpi || ( - sudo -u postgres createuser wrolpi + sudo -u postgres createuser -s wrolpi # superuser so maps can be imported sudo -u postgres psql -c "alter user postgres password 'wrolpi'" sudo -u postgres psql -c "alter user wrolpi password 'wrolpi'" sudo -u postgres createdb -E UTF8 -O wrolpi wrolpi echo "Created wrolpi database" ) -sudo -u wrolpi /bin/bash -c 'cd /opt/wrolpi && ./main.py db upgrade' +sudo -u wrolpi /bin/bash -c 'cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 main.py db upgrade' # Create gis (map) database. sudo -u postgres psql -c '\l' | grep gis || ( @@ -25,4 +25,4 @@ sudo -u postgres psql -d gis -c 'ALTER TABLE geography_columns OWNER TO _renderd sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO _renderd' sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO _renderd" # Clear map tile cache files. -[ -d /var/lib/mod_tile/ajt ] && sudo rm -r /var/lib/mod_tile/ajt +[ -d /var/cache/renderd/tiles/ajt ] && sudo rm -r /var/cache/renderd/tiles/ajt diff --git a/scripts/import_map.sh b/scripts/import_map.sh index 9cc4a3f3..5e352d9a 100755 --- a/scripts/import_map.sh +++ b/scripts/import_map.sh @@ -18,7 +18,7 @@ if [ "${1}" == "" ]; then fi HAS_ICE_SHEET=false -psql gis wrolpi -c '\d' | grep icesheet_polygons && HAS_ICE_SHEET=true +psql gis -c '\d' | grep icesheet_polygons && HAS_ICE_SHEET=true if [ ${HAS_ICE_SHEET} = false ]; then echo "WROLPi: Missing icesheet_polygons. Run reset_map.sh" exit 4 @@ -58,12 +58,30 @@ if [[ ${1} == *.osm.pbf ]]; then pbf_path=${MERGED_TMP_FILE} fi - nice -n 18 osm2pgsql -d gis --create --slim -G --hstore -U _renderd -H 127.0.0.1 \ + # Import as wrolpi user (the user running this script), then change ownership to _renderd. + nice -n 18 osm2pgsql -d gis --create --slim -G --hstore \ --tag-transform-script /opt/openstreetmap-carto/openstreetmap-carto.lua \ -C ${MAX_CACHE} \ --number-processes 3 \ -S /opt/openstreetmap-carto/openstreetmap-carto.style \ "${pbf_path}" + # Assign all data to _renderd so the map can be rendered. + psql -d gis -c 'ALTER TABLE external_data OWNER TO _renderd' + psql -d gis -c 'ALTER TABLE geography_columns OWNER TO _renderd' + psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO _renderd' + psql -d gis -c "ALTER TABLE icesheet_outlines OWNER TO _renderd" + psql -d gis -c "ALTER TABLE icesheet_polygons OWNER TO _renderd" + psql -d gis -c "ALTER TABLE ne_100m_admin_0_boundary_lines_land OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_line OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_nodes OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_point OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_polygon OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_rels OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_roads OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_ways OWNER TO _renderd" + psql -d gis -c "ALTER TABLE simplified_water_polygons OWNER TO _renderd" + psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO _renderd" + psql -d gis -c "ALTER TABLE water_polygons OWNER TO _renderd" elif [[ ${1} == *.dump ]]; then if [ ! -f "${1}" ]; then echo "WROLPi: File does not exist! ${1}" diff --git a/scripts/initialize_map.sh b/scripts/initialize_map.sh index 54dc8dbd..c2e0db6b 100755 --- a/scripts/initialize_map.sh +++ b/scripts/initialize_map.sh @@ -8,9 +8,9 @@ fi set -x # Use D.C. to initialized DB because it is so small. -DC_MAP=district-of-columbia-latest.osm.pbf +DC_MAP=district-of-columbia-230222.osm.pbf DC_MAP_PATH=/tmp/${DC_MAP} -wget --continue https://download.geofabrik.de/north-america/us/${DC_MAP} -O ${DC_MAP_PATH} || : +[ -f "${DC_MAP_PATH}" ] || wget --continue https://wrolpi.org/downloads/${DC_MAP} -O ${DC_MAP_PATH} if [ ! -f "${DC_MAP_PATH}" ]; then echo "Could not download D.C. map" @@ -28,15 +28,15 @@ dc_import=false indexes=false while [ "${COUNT}" -lt "${MAX_TRIES}" ]; do [[ "${external_data}" = false ]] && - sudo -u wrolpi nice -n 18 /opt/openstreetmap-carto/scripts/get-external-data.py \ + sudo -u _renderd nice -n 18 /opt/openstreetmap-carto/scripts/get-external-data.py \ -C \ - -d gis -U wrolpi && + -d gis -U _renderd && external_data=true [[ "${dc_import}" = false ]] && - sudo -u wrolpi nice -n 18 /opt/wrolpi/scripts/import_map.sh ${DC_MAP_PATH} && + sudo -u _renderd nice -n 18 /opt/wrolpi/scripts/import_map.sh ${DC_MAP_PATH} && dc_import=true [[ "${indexes}" = false ]] && - sudo -u wrolpi psql -d gis -f /opt/openstreetmap-carto/indexes.sql && + sudo -u _renderd psql -d gis -f /opt/openstreetmap-carto/indexes.sql && indexes=true if [[ "${external_data}" == true && "${dc_import}" == true && "${indexes}" == true ]]; then SUCCESS=true diff --git a/scripts/install_debian_11.sh b/scripts/install_debian_11.sh index d02b7e1a..b262020e 100755 --- a/scripts/install_debian_11.sh +++ b/scripts/install_debian_11.sh @@ -4,28 +4,42 @@ echo "Debian 11 install start: $(date '+%Y-%m-%d %H:%M:%S')" set -x set -e -# Install npm repos. -npm --version || curl -fsSL https://deb.nodesource.com/setup_18.x | bash - - # Update if we haven't updated in the last day. [ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update # Install dependencies -apt install -y apt-transport-https ca-certificates curl gnupg-agent gcc libpq-dev software-properties-common \ - postgresql-13 nginx-full nginx-doc python3-full python3-dev python3-doc python3-venv htop vim ffmpeg hostapd \ - chromium chromium-driver cpufrequtils network-manager nodejs +apt install -y \ + apt-transport-https \ + ca-certificates \ + chromium \ + chromium-driver \ + cpufrequtils \ + curl \ + ffmpeg \ + gcc \ + gnupg-agent \ + hostapd \ + htop \ + libpq-dev \ + network-manager \ + nginx-doc \ + nginx-full \ + postgresql-13 \ + python3-dev \ + python3-doc \ + python3-full \ + python3-pip \ + python3-venv \ + software-properties-common \ + vim + +# Install npm. +npm --version || curl -fsSL https://deb.nodesource.com/setup_18.x | bash - +apt install -y nodejs +node -v +npm -v # Install serve, and archiving tools. -sudo npm i -g serve single-file-cli@1.0.33 'git+https://github.com/lrnselfreliance/readability-extractor' & - -# Setup the virtual environment that main.py expects -pip3 --version || ( - # If no pip, install pip - curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py && - python3 /tmp/get-pip.py & -) - -# Wait for npm and pip3. -wait $(jobs -p) +sudo npm i -g serve@12.0.1 single-file-cli@1.0.33 readability-extractor@0.0.6 # Build React app in background job. cd /opt/wrolpi/app || exit 5 @@ -33,7 +47,8 @@ npm install || npm install || npm install || npm install # try install multiple npm run build & # Install python requirements in background job. -pip3 install -r /opt/wrolpi/requirements.txt & +python3 -m venv /opt/wrolpi/venv +/opt/wrolpi/venv/bin/pip3 install -r /opt/wrolpi/requirements.txt & wait $(jobs -p) @@ -44,7 +59,12 @@ cp /opt/wrolpi/50x.html /var/www/50x.html # Create the WROLPi user grep wrolpi /etc/passwd || useradd -md /home/wrolpi wrolpi -s "$(command -v bash)" -chown -R wrolpi:wrolpi /opt/wrolpi +cat >/home/wrolpi/.pgpass <<'EOF' +127.0.0.1:5432:gis:_renderd:wrolpi +127.0.0.1:5432:wrolpi:wrolpi:wrolpi +EOF +chown -R wrolpi:wrolpi /home/wrolpi /opt/wrolpi +chmod 0600 /home/wrolpi/.pgpass # Give WROLPi group a few privileged commands via sudo without password. cat >/etc/sudoers.d/90-wrolpi <<'EOF' @@ -62,9 +82,9 @@ visudo -c -f /etc/sudoers.d/90-wrolpi chown wrolpi:wrolpi /media/wrolpi # Install the systemd services -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi-api.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-api.service /etc/systemd/system/ cp /opt/wrolpi/etc/debian11/wrolpi-app.service /etc/systemd/system/ -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi.target /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi.target /etc/systemd/system/ /usr/bin/systemctl daemon-reload systemctl enable wrolpi-api.service systemctl enable wrolpi-app.service @@ -74,13 +94,16 @@ systemctl stop wrolpi-app.service # Configure Postgresql. Do this after the API is stopped. sudo -u postgres psql -c '\l' | grep wrolpi || ( - sudo -u postgres createuser wrolpi && + sudo -u postgres createuser -s wrolpi && sudo -u postgres psql -c "alter user postgres password 'wrolpi'" && sudo -u postgres psql -c "alter user wrolpi password 'wrolpi'" && sudo -u postgres createdb -E UTF8 -O wrolpi wrolpi ) # Initialize/upgrade the WROLPi database. -(cd /opt/wrolpi && /usr/bin/python3 /opt/wrolpi/main.py db upgrade) +(cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py db upgrade) + +# Install map only if that script hasn't finished. +[ -f /var/www/html/leaflet.css ] || /opt/wrolpi/scripts/install_map_debian_11.sh set +x diff --git a/scripts/install_map_debian_11.sh b/scripts/install_map_debian_11.sh index c7baa5f0..0bd1ff54 100755 --- a/scripts/install_map_debian_11.sh +++ b/scripts/install_map_debian_11.sh @@ -9,48 +9,52 @@ set -e [ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update # Install map dependencies. apt install -y libapache2-mod-tile renderd tar unzip wget bzip2 apache2 lua5.1 mapnik-utils python3-mapnik gdal-bin \ - fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont osmium-tool + fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont osmium-tool python3-psycopg2 python3-yaml -/usr/bin/npm install -g carto -/usr/bin/carto -v +/usr/bin/npm install -g carto@1.2.0 +carto -v # Create `gis` database for map. -sudo -u postgres psql -c '\l' | grep gis || sudo -u postgres createdb -E UTF8 -O wrolpi gis -sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis;" || : -sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;" || : -sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO wrolpi;" -sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO wrolpi;" +sudo -u postgres psql -c '\l' | grep gis || ( + sudo -u postgres createuser _renderd || : + sudo -u postgres psql -c "alter user _renderd password 'wrolpi'" || : + sudo -u postgres createdb -E UTF8 -O _renderd gis + sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis" || : + sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore" || : + echo "Created gis database" +) +sudo -u postgres psql -d gis -c 'ALTER TABLE geography_columns OWNER TO _renderd' +sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO _renderd' +sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO _renderd" # Install Debian 11 compatible mod_tile. git clone -b switch2osm https://github.com/lrnselfreliance/mod_tile.git /tmp/mod_tile || : # Install init/systemd config to start map service on startup. -cp /tmp/mod_tile/debian/renderd.init /etc/init.d/renderd +cp /opt/wrolpi/etc/raspberrypios/renderd /etc/init.d/renderd sudo chmod +x /etc/init.d/renderd -sed -i -e 's/^RUNASUSER.*/RUNASUSER=wrolpi/' /etc/init.d/renderd -sed -i -e 's/^DAEMON=.*/DAEMON=\/usr\/bin\/$NAME/' /etc/init.d/renderd -sed -i -e 's/^DAEMON_ARGS=.*/DAEMON_ARGS="-c \/etc\/renderd.conf"/' /etc/init.d/renderd cp /tmp/mod_tile/debian/renderd.service /lib/systemd/system/ chmod +x /etc/init.d/renderd # Initialize gis database. [ -d /opt/openstreetmap-carto ] && chown -R wrolpi:wrolpi /opt/openstreetmap-carto git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto || : -chown -R wrolpi:wrolpi /opt/openstreetmap-carto -sudo -u wrolpi /bin/bash -c '(cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff)' +chown -R _renderd:_renderd /opt/openstreetmap-carto +sudo -u _renderd /bin/bash -c '(cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff)' cd /opt/openstreetmap-carto if [[ ! -f /opt/openstreetmap-carto/mapnik.xml || ! -s /opt/openstreetmap-carto/mapnik.xml ]]; then - /usr/bin/carto project.mml >/opt/openstreetmap-carto/mapnik.xml + carto project.mml >/opt/openstreetmap-carto/mapnik.xml fi +chown -R _renderd:_renderd /opt/openstreetmap-carto # Reset map. Force it. /opt/wrolpi/scripts/reset_map.sh -f -cp /opt/wrolpi/etc/debian11/renderd.conf /etc/renderd.conf +cp /opt/wrolpi/etc/raspberrypios/renderd.conf /etc/renderd.conf # Configure Apache2 to listen on 8084. -cp /opt/wrolpi/etc/debian11/ports.conf /etc/apache2/ports.conf +cp /opt/wrolpi/etc/raspberrypios/ports.conf /etc/apache2/ports.conf cp /opt/wrolpi/etc/debian11/000-default.conf /etc/apache2/sites-available/000-default.conf # Copy Leaflet files to Apache's directory so they can be used offline. -cp /opt/wrolpi/etc/debian11/index.html \ +cp /opt/wrolpi/etc/raspberrypios/index.html \ /opt/wrolpi/modules/map/leaflet.js \ /opt/wrolpi/modules/map/leaflet.css /var/www/html/ chmod 644 /var/www/html/* diff --git a/scripts/install_map_raspberrypios.sh b/scripts/install_map_raspberrypios.sh new file mode 100755 index 00000000..ae429a76 --- /dev/null +++ b/scripts/install_map_raspberrypios.sh @@ -0,0 +1,76 @@ +#! /usr/bin/env bash +# This script installs OpenStreetMap tile server. +# This server uses Apache2, renderd, and mod_tile to render and display map tiles. + +set -x +set -e + +# Update if we haven't updated in the last day. +[ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update +# Install map dependencies. +apt install -y libapache2-mod-tile libboost-all-dev git tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev \ + libgeos++-dev libpq-dev libbz2-dev libproj-dev protobuf-c-compiler libfreetype6-dev libtiff5-dev \ + libicu-dev libgdal-dev libcairo2-dev libcairomm-1.0-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-0-dev \ + postgis postgresql-13-postgis-3 postgresql-13-postgis-3-scripts osm2pgsql gdal-bin libmapnik-dev mapnik-utils \ + python3-mapnik apache2 apache2-dev libcurl4-gnutls-dev libiniparser-dev libmemcached-dev librados-dev \ + fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted postgresql-contrib osmium-tool python3-psycopg2 python3-yaml + +/usr/bin/npm install -g carto@1.2.0 +carto -v + +# Create `gis` database for map. +sudo -u postgres psql -c '\l' | grep gis || ( + sudo -u postgres createuser _renderd || : + sudo -u postgres psql -c "alter user _renderd password 'wrolpi'" || : + sudo -u postgres createdb -E UTF8 -O _renderd gis + sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis" || : + sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore" || : + echo "Created gis database" +) +sudo -u postgres psql -d gis -c 'ALTER TABLE geography_columns OWNER TO _renderd' +sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO _renderd' +sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO _renderd" + +# Install Raspberry Pi OS compatible mod_tile. +git clone -b switch2osm https://github.com/lrnselfreliance/mod_tile.git /tmp/mod_tile || : +# Install init/systemd config to start map service on startup. +cp /opt/wrolpi/etc/raspberrypios/renderd /etc/init.d/renderd +sudo chmod +x /etc/init.d/renderd +cp /tmp/mod_tile/debian/renderd.service /lib/systemd/system/ +chmod +x /etc/init.d/renderd + +# Initialize gis database. +[ -d /opt/openstreetmap-carto ] && chown -R wrolpi:wrolpi /opt/openstreetmap-carto +git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto || : +chown -R _renderd:_renderd /opt/openstreetmap-carto +sudo -u _renderd /bin/bash -c '(cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff)' +cd /opt/openstreetmap-carto +if [[ ! -f /opt/openstreetmap-carto/mapnik.xml || ! -s /opt/openstreetmap-carto/mapnik.xml ]]; then + carto project.mml >/opt/openstreetmap-carto/mapnik.xml +fi +chown -R _renderd:_renderd /opt/openstreetmap-carto + +# Reset map. Force it. +/opt/wrolpi/scripts/reset_map.sh -f + +cp /opt/wrolpi/etc/raspberrypios/renderd.conf /etc/renderd.conf +# Enable mod_tile. +cp /opt/wrolpi/etc/raspberrypios/mod_tile.conf /etc/apache2/conf-available/mod_tile.conf + +/usr/sbin/a2enconf mod_tile + +# Configure Apache2 to listen on 8084. +cp /opt/wrolpi/etc/raspberrypios/ports.conf /etc/apache2/ports.conf +cp /opt/wrolpi/etc/raspberrypios/000-default.conf /etc/apache2/sites-available/000-default.conf +# Copy Leaflet files to Apache's directory so they can be used offline. +cp /opt/wrolpi/etc/raspberrypios/index.html \ + /opt/wrolpi/modules/map/leaflet.js \ + /opt/wrolpi/modules/map/leaflet.css /var/www/html/ +chmod 644 /var/www/html/* + +systemctl enable renderd +systemctl stop renderd +systemctl start renderd + +systemctl stop apache2 +systemctl start apache2 diff --git a/scripts/install_map_ubuntu_20.04.sh b/scripts/install_map_ubuntu_20.04.sh deleted file mode 100755 index 315adffd..00000000 --- a/scripts/install_map_ubuntu_20.04.sh +++ /dev/null @@ -1,84 +0,0 @@ -#! /usr/bin/env bash -# This script installs OpenStreetMap tile server. -# This server uses Apache2, renderd, and mod_tile to render and display map tiles. - -function yes_or_no { - while true; do - read -p "$* [y/n]:" yn - case $yn in - [Yy]*) return 0 ;; - [Nn]*) return 1 ;; - esac - done -} - -set -x -set -e - -# Update if we haven't updated in the last day. -[ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update -# Install map dependencies. -apt install -y libboost-all-dev git tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev \ - libgeos++-dev libpq-dev libbz2-dev libproj-dev protobuf-c-compiler libfreetype6-dev libtiff5-dev \ - libicu-dev libgdal-dev libcairo2-dev libcairomm-1.0-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-0-dev \ - postgis postgresql-12-postgis-3 postgresql-12-postgis-3-scripts osm2pgsql gdal-bin libmapnik-dev mapnik-utils \ - python3-mapnik apache2 apache2-dev libcurl4-gnutls-dev libiniparser-dev libmemcached-dev librados-dev \ - fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted postgresql-contrib osmium-tool - -/usr/bin/npm install -g carto -/usr/bin/carto -v - -# Create `gis` database for map. -sudo -u postgres psql -c '\l' | grep gis || sudo -u postgres createdb -E UTF8 -O wrolpi gis -sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis;" || : -sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;" || : -sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO wrolpi;" -sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO wrolpi;" - -# Install Ubuntu 20.04 compatible mod_tile. -git clone -b switch2osm https://github.com/lrnselfreliance/mod_tile.git /tmp/mod_tile || : -if [ ! -f /usr/lib/apache2/modules/mod_tile.so ]; then - (cd /tmp/mod_tile && ./autogen.sh && ./configure && make -j4 && make install && make install-mod_tile && ldconfig) -fi -# Install init/systemd config to start map service on startup. -cp /tmp/mod_tile/debian/renderd.init /etc/init.d/renderd -sudo chmod +x /etc/init.d/renderd -cp /tmp/mod_tile/debian/renderd.service /lib/systemd/system/ -sed -ie 's/^RUNASUSER.*/RUNASUSER=wrolpi/' /etc/init.d/renderd -chmod +x /etc/init.d/renderd -[ ! -d /var/lib/mod_tile ] && mkdir /var/lib/mod_tile -chown wrolpi /var/lib/mod_tile - -# Initialize gis database. -[ -d /opt/openstreetmap-carto ] && chown -R wrolpi:wrolpi /opt/openstreetmap-carto -git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto || : -chown -R wrolpi:wrolpi /opt/openstreetmap-carto -sudo -u wrolpi /bin/bash -c '(cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff)' -cd /opt/openstreetmap-carto -if [[ ! -f /opt/openstreetmap-carto/mapnik.xml || ! -s /opt/openstreetmap-carto/mapnik.xml ]]; then - /usr/bin/carto project.mml >/opt/openstreetmap-carto/mapnik.xml -fi - -# Reset map. Force it. -/opt/wrolpi/scripts/reset_map.sh -f - -cp /opt/wrolpi/etc/ubuntu20.04/renderd.conf /usr/local/etc/renderd.conf -# Enable mod_tile. -cp /opt/wrolpi/etc/ubuntu20.04/mod_tile.conf /etc/apache2/conf-available/mod_tile.conf - -/usr/sbin/a2enconf mod_tile - -# Configure Apache2 to listen on 8084. -cp /opt/wrolpi/etc/ubuntu20.04/ports.conf /etc/apache2/ports.conf -cp /opt/wrolpi/etc/ubuntu20.04/000-default.conf /etc/apache2/sites-available/000-default.conf -cp /opt/wrolpi/etc/ubuntu20.04/index.html /var/www/html/index.html -# Copy Leaflet files to Apache's directory so they can be used offline. -cp /opt/wrolpi/modules/map/leaflet.js /var/www/html/leaflet.js -cp /opt/wrolpi/modules/map/leaflet.css /var/www/html/leaflet.css - -systemctl enable renderd -systemctl stop renderd -systemctl start renderd - -systemctl stop apache2 -systemctl start apache2 diff --git a/scripts/install_ubuntu_20.04.sh b/scripts/install_raspberrypios.sh similarity index 70% rename from scripts/install_ubuntu_20.04.sh rename to scripts/install_raspberrypios.sh index a62fcc4d..af723f94 100755 --- a/scripts/install_ubuntu_20.04.sh +++ b/scripts/install_raspberrypios.sh @@ -1,31 +1,45 @@ #!/usr/bin/env bash -echo "Ubuntu 20.04 install start: $(date '+%Y-%m-%d %H:%M:%S')" +echo "Raspberry Pi OS install start: $(date '+%Y-%m-%d %H:%M:%S')" set -x set -e -# Install npm repos. -npm --version || curl -fsSL https://deb.nodesource.com/setup_18.x | bash - - # Update if we haven't updated in the last day. [ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update # Install dependencies -apt install -y apt-transport-https ca-certificates curl gnupg-agent gcc libpq-dev software-properties-common \ - postgresql-12 nginx-full nginx-doc python3.8-minimal python3.8-dev python3.8-doc python3.8-venv \ - ffmpeg hostapd nodejs chromium-browser chromium-chromedriver cpufrequtils network-manager +apt install -y \ + apt-transport-https \ + ca-certificates \ + chromium \ + chromium-driver \ + cpufrequtils \ + curl \ + ffmpeg \ + gcc \ + gnupg-agent \ + hostapd \ + htop \ + libpq-dev \ + network-manager \ + nginx-doc \ + nginx-full \ + postgresql-13 \ + python3-dev \ + python3-doc \ + python3-full \ + python3-pip \ + python3-venv \ + software-properties-common \ + vim + +# Install npm. +npm --version || curl -fsSL https://deb.nodesource.com/setup_18.x | bash - +apt install -y nodejs +node -v +npm -v # Install serve, and archiving tools. -sudo npm i -g serve single-file-cli@1.0.33 'git+https://github.com/lrnselfreliance/readability-extractor' & - -# Setup the virtual environment that main.py expects -pip3 --version || ( - # If no pip, install pip - curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py && - python3 /tmp/get-pip.py & -) - -# Wait for npm and pip3. -wait $(jobs -p) +sudo npm i -g serve@12.0.1 single-file-cli@1.0.33 readability-extractor@0.0.6 # Build React app in background job. cd /opt/wrolpi/app || exit 5 @@ -33,7 +47,8 @@ npm install || npm install || npm install || npm install # try install multiple npm run build & # Install python requirements in background job. -pip3 install -r /opt/wrolpi/requirements.txt & +python3 -m venv /opt/wrolpi/venv +/opt/wrolpi/venv/bin/pip3 install -r /opt/wrolpi/requirements.txt & wait $(jobs -p) @@ -62,9 +77,9 @@ visudo -c -f /etc/sudoers.d/90-wrolpi chown wrolpi:wrolpi /media/wrolpi # Install the systemd services -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi-api.service /etc/systemd/system/ -cp /opt/wrolpi/etc/debian11/wrolpi-app.service /etc/systemd/system/ -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi.target /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-api.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-app.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi.target /etc/systemd/system/ /usr/bin/systemctl daemon-reload systemctl enable wrolpi-api.service systemctl enable wrolpi-app.service @@ -74,13 +89,16 @@ systemctl stop wrolpi-app.service # Configure Postgresql. Do this after the API is stopped. sudo -u postgres psql -c '\l' | grep wrolpi || ( - sudo -u postgres createuser wrolpi && + sudo -u postgres createuser -s wrolpi && sudo -u postgres psql -c "alter user postgres password 'wrolpi'" && sudo -u postgres psql -c "alter user wrolpi password 'wrolpi'" && sudo -u postgres createdb -E UTF8 -O wrolpi wrolpi ) # Initialize/upgrade the WROLPi database. -(cd /opt/wrolpi && /usr/bin/python3 /opt/wrolpi/main.py db upgrade) +(cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py db upgrade) + +# Install map only if that script hasn't finished. +[ -f /var/www/html/leaflet.css ] || /opt/wrolpi/scripts/install_map_raspberrypios.sh set +x diff --git a/scripts/reset_map.sh b/scripts/reset_map.sh index f5a5a62f..cdfb4bb3 100755 --- a/scripts/reset_map.sh +++ b/scripts/reset_map.sh @@ -66,11 +66,11 @@ if [ -d /var/lib/mod_tile ]; then else mkdir /var/lib/mod_tile fi -chown -R wrolpi:wrolpi /var/lib/mod_tile +chown -R _renderd:_renderd /var/lib/mod_tile sudo -u postgres dropdb gis || : -sudo -u postgres createdb -E UTF8 -O wrolpi gis +sudo -u postgres createdb -E UTF8 -O _renderd gis sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis;" || : sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;" || : sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO wrolpi;" diff --git a/upgrade.sh b/upgrade.sh new file mode 100755 index 00000000..66ec98a8 --- /dev/null +++ b/upgrade.sh @@ -0,0 +1,52 @@ +#! /usr/bin/env bash +# This script will upgrade the WROLPi API and App. It is required that you install WROLPi first. + +Help() { + # Display Help + echo "Upgrade WROLPi API and App on this machine." + echo + echo "Syntax: upgrade.sh [-h] [-b BRANCH]" + echo "options:" + echo "h Print this help." + echo "b Upgrade from this git BRANCH." + echo +} + +BRANCH="release" +while getopts ":hb:" option; do + case $option in + h) # display Help + Help + exit + ;; + b) + BRANCH="${OPTARG}" + ;; + *) # invalid argument(s) + echo "Error: Invalid option" + exit 1 + ;; + esac +done + +if [ ! -d /opt/wrolpi ] || [ ! -d /opt/wrolpi/wrolpi ]; then + echo "You must install WROLPi first. Try: /opt/wrolpi/install.sh" + exit 2 +fi + +set -x +set -e + +systemctl stop wrolpi-api +systemctl stop wrolpi-app + +# Pull the latest commit of the requested branch. +(cd /opt/wrolpi && git fetch && git checkout "${BRANCH}" && git reset --hard origin/"${BRANCH}") || exit 4 + +/opt/wrolpi/scripts/build_api_and_app.sh 2>&1 | tee /opt/wrolpi/upgrade.log + +set +x + +echo "Upgrade end $(date '+%Y-%m-%d %H:%M:%S')" >>/opt/wrolpi/upgrade.log + +echo "WROLPi upgrade has completed" diff --git a/wrolpi/cmd.py b/wrolpi/cmd.py index 310d81f8..561ab4f5 100644 --- a/wrolpi/cmd.py +++ b/wrolpi/cmd.py @@ -60,19 +60,19 @@ BASH_BIN = which('bash', '/bin/bash', warn=True) # Archive SINGLE_FILE_BIN = which('single-file', - '/usr/bin/single-file', # rpi ubuntu + '/usr/bin/single-file', # rpi os '/usr/local/bin/single-file', # debian warn=True, flag='singlefile_installed', ) CHROMIUM = which('chromium-browser', 'chromium', - '/usr/bin/chromium-browser', # rpi ubuntu + '/usr/bin/chromium-browser', # rpi os '/usr/bin/chromium', # debian warn=True, flag='chromium_installed', ) READABILITY_BIN = which('readability-extractor', - '/usr/bin/readability-extractor', # rpi ubuntu + '/usr/bin/readability-extractor', # rpi os '/usr/local/bin/readability-extractor', # debian warn=True, flag='readability_installed',