From abb0b49315fe85b3b0432e59638cdf49957b90f1 Mon Sep 17 00:00:00 2001 From: Roland Date: Thu, 23 Feb 2023 12:09:19 -0700 Subject: [PATCH] Checking for icesheet table as current user. Importing maps as wrolpi user, then changing ownership to _renderd. Changing ownership of /opt/wrolpi as well. Creating pgpass for wrolpi user in debian 11. Install specific version of serve. Not echoing the echo. Updating cache directory in create_db.sh. Use python binary to run db upgrade. Install renderd.conf to /etc on RPi OS. Fetching DC map from wrolpi.org to avoid broken daily PBFs. Only get latest dc map if it doesn't exist. carto switched again?? Changing mod_tile directory to /var/cache/renderd/tiles for RPi OS. Copying renderd configs for RPi OS. Installing postgres 13 for RPi OS map. Changing title of feature matrix. Moving download link up in readme. Changing mod_tile to _renderd. Using unix socket rather than host connection. Getting openstreetmap-carto as _renderd user. Assigning ownership of gis db to _renderd in reset_map.sh. Creating _renderd db user. Using _renderd to run renderd on debian 11. Installing python modules that map requires globally. Upgrading db using venv. Installing nodejs which installs npm. Install npm/node after other dependencies. Updating apt so nodejs can be installed. Upgrading PyYAML in new venv. Installing pip via apt. Including bash binary for portability. Moving images link out of list. Moving download up on Readme. Removing map installation from readme. Installing specific version of carto. Creating venv for wrolpi project. Using venv to install requirements and run wrolpi's main.py. Logging upgrade script. Including upgrade script in Upgrade readme. Moving upgrade logic into separate script. This is necessary so and repo update will take immediate effect on the upgrade. Modifying and unifying Debian installation with RPi OS installation. --- README.md | 26 ++- UPGRADE.md | 12 +- etc/{debian11 => }/README.md | 0 .../000-default.conf | 4 +- etc/{debian11 => raspberrypios}/index.html | 0 .../mod_tile.conf | 0 etc/{debian11 => raspberrypios}/ports.conf | 0 etc/raspberrypios/renderd | 153 ++++++++++++++++++ etc/{debian11 => raspberrypios}/renderd.conf | 2 +- .../wrolpi-api.service | 2 +- etc/raspberrypios/wrolpi-app.service | 13 ++ .../wrolpi.target | 0 etc/ubuntu20.04/index.html | 37 ----- etc/ubuntu20.04/ports.conf | 15 -- etc/ubuntu20.04/renderd.conf | 17 -- install.sh | 2 +- pi-gen/stage2/04-wrolpi/01-packages | 8 +- pi-gen/stage2/04-wrolpi/02-run.sh | 10 +- pi-gen/stage2/04-wrolpi/03-run-chroot.sh | 25 ++- pi-gen/stage2/04-wrolpi/04-run-chroot.sh | 28 ++-- requirements.txt | 2 +- scripts/build_api_and_app.sh | 21 +++ scripts/create_db.sh | 6 +- scripts/import_map.sh | 22 ++- scripts/initialize_map.sh | 12 +- scripts/install_debian_11.sh | 69 +++++--- scripts/install_map_debian_11.sh | 40 ++--- scripts/install_map_raspberrypios.sh | 76 +++++++++ scripts/install_map_ubuntu_20.04.sh | 84 ---------- ...untu_20.04.sh => install_raspberrypios.sh} | 66 +++++--- scripts/reset_map.sh | 4 +- upgrade.sh | 52 ++++++ wrolpi/cmd.py | 6 +- 33 files changed, 529 insertions(+), 285 deletions(-) rename etc/{debian11 => }/README.md (100%) rename etc/{ubuntu20.04 => raspberrypios}/000-default.conf (93%) rename etc/{debian11 => raspberrypios}/index.html (100%) rename etc/{ubuntu20.04 => raspberrypios}/mod_tile.conf (100%) rename etc/{debian11 => raspberrypios}/ports.conf (100%) create mode 100644 etc/raspberrypios/renderd rename etc/{debian11 => raspberrypios}/renderd.conf (91%) rename etc/{ubuntu20.04 => raspberrypios}/wrolpi-api.service (76%) create mode 100644 etc/raspberrypios/wrolpi-app.service rename etc/{ubuntu20.04 => raspberrypios}/wrolpi.target (100%) delete mode 100644 etc/ubuntu20.04/index.html delete mode 100644 etc/ubuntu20.04/ports.conf delete mode 100644 etc/ubuntu20.04/renderd.conf create mode 100755 scripts/build_api_and_app.sh create mode 100755 scripts/install_map_raspberrypios.sh delete mode 100755 scripts/install_map_ubuntu_20.04.sh rename scripts/{install_ubuntu_20.04.sh => install_raspberrypios.sh} (70%) create mode 100755 upgrade.sh diff --git a/README.md b/README.md index 472e3ec4..9bd9ca23 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ network so that any user with a laptop/tablet/phone can connect and use the libr * [Features](#features) * [Demo](#demo) +* [Download](#download) * [Try WROLPi](#try-wrolpi) * [Debian 11 Install](#debian-11-install) * [Raspberry Pi Install](#raspberrypi-install) @@ -46,7 +47,7 @@ network so that any user with a laptop/tablet/phone can connect and use the libr - [ ] Synchronizer and/or duplicator - [ ] Food storage calculator -## Feature Matrix +## Module features matrix | **Module** | **Search** | **View** | **Download** | |--------------------|:-----------|----------|--------------| @@ -61,6 +62,10 @@ network so that any user with a laptop/tablet/phone can connect and use the libr ![YouTube Demo Video](https://raw.githubusercontent.com/lrnselfreliance/wrolpi/master/demo_link.jpg) +# Download + +Images are available on [wrolpi.org](https://wrolpi.org) + # Try WROLPi! You can try out WROLPi by running the docker containers. @@ -99,21 +104,14 @@ Install WROLPi onto a fresh Debian 11 system. 4. Start WROLPi * `sudo systemctl start wrolpi.target` 5. Browse to the IP address of your WROLPi! -6. Run the map installation script (Optional) - * `sudo /bin/bash /opt/wrolpi/scripts/install_map_debian_11.sh` # RaspberryPi Install -## Download - -* Download the latest image from [wrolpi.org](https://wrolpi.org) - -## Manual install Install WROLPi onto a fresh Raspberry Pi. **Warning: this will clobber anything else on the Raspberry Pi! Do not run this installation script elsewhere!** -1. Install [Ubuntu Server 20.04](https://cdimage.ubuntu.com/releases/20.04/release/) onto a RaspberryPi. +1. Install [Raspberry Pi OS](https://www.raspberrypi.com/software/operating-systems/) onto a RaspberryPi. 2. Get the WROLPi installation script * `wget https://raw.githubusercontent.com/lrnselfreliance/wrolpi/release/install.sh -O /tmp/install.sh` 3. Run the installation script @@ -132,7 +130,7 @@ Install WROLPi onto a fresh Raspberry Pi. * External USB hard drive * Power supply and cables * WiFi USER device such as a phone, tablet, or laptop. -1. A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be +2. A WROLPi instance should consume a minimal amount of electricity during WROL event. It is expected power will be scarce when WROLPi is needed most. ## User expectations @@ -143,13 +141,13 @@ Install WROLPi onto a fresh Raspberry Pi. ## Software properties and capabilities 1. A WROLPi instance should be able to "factory-reset" itself without any outside services. -1. WROLPi should function completely without any internet services. -1. A user should have easy access to their data if WROLPi fails: +2. WROLPi should function completely without any internet services. +3. A user should have easy access to their data if WROLPi fails: * For example, a user can watch their videos by navigating a short and intuitive directory structure and opening the video in their preferred video player. -1. WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER. +4. WROLPi should be self-documented. The UI should contain a tutorial for USERS as well as the MAINTAINER. * If the UI isn't functional, the code should be documented such that a user can restore functionality. -1. WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or +5. WROLPi should favor pre-processing, rather than processing during a WROL event. Such as re-encoding a video, or processing captions. This is to ensure that when a user adds content during non-WROL time, the processing for optimum performance is already done for a WROL event. diff --git a/UPGRADE.md b/UPGRADE.md index 239486ef..45723700 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -14,8 +14,8 @@ 3. Build all docker containers * `docker-compose build --parallel` 4. Create the map volumes - * `docker volume create --name=openstreetmap-data` - * `docker volume create --name=openstreetmap-rendered-tiles` + * `docker volume create --name=openstreetmap-data` + * `docker volume create --name=openstreetmap-rendered-tiles` 5. Turn on the database * `docker-compose up -d db` 6. Upgrade the database @@ -25,9 +25,5 @@ ## Upgrading Raspberry Pi or Debian 11 Installation -1. Get the latest release install script - * `wget https://raw.githubusercontent.com/lrnselfreliance/wrolpi/release/install.sh -O /tmp/install.sh` -2. Run the installation script - * `sudo /bin/bash /tmp/install.sh` -3. Start WROLPi - * `sudo systemctl start wrolpi.target` +1. Run the upgrade script + * `sudo /bin/bash /opt/wrolpi/upgrade.sh` diff --git a/etc/debian11/README.md b/etc/README.md similarity index 100% rename from etc/debian11/README.md rename to etc/README.md diff --git a/etc/ubuntu20.04/000-default.conf b/etc/raspberrypios/000-default.conf similarity index 93% rename from etc/ubuntu20.04/000-default.conf rename to etc/raspberrypios/000-default.conf index 06b39e80..4a7495d0 100644 --- a/etc/ubuntu20.04/000-default.conf +++ b/etc/raspberrypios/000-default.conf @@ -10,8 +10,8 @@ ServerAdmin webmaster@localhost - LoadTileConfigFile /usr/local/etc/renderd.conf - ModTileRenderdSocketName /var/run/renderd/renderd.sock + LoadTileConfigFile /etc/renderd.conf + ModTileRenderdSocketName /run/renderd/renderd.sock # Timeout before giving up for a tile to be rendered ModTileRequestTimeout 0 # Timeout before giving up for a tile to be rendered that is otherwise missing diff --git a/etc/debian11/index.html b/etc/raspberrypios/index.html similarity index 100% rename from etc/debian11/index.html rename to etc/raspberrypios/index.html diff --git a/etc/ubuntu20.04/mod_tile.conf b/etc/raspberrypios/mod_tile.conf similarity index 100% rename from etc/ubuntu20.04/mod_tile.conf rename to etc/raspberrypios/mod_tile.conf diff --git a/etc/debian11/ports.conf b/etc/raspberrypios/ports.conf similarity index 100% rename from etc/debian11/ports.conf rename to etc/raspberrypios/ports.conf diff --git a/etc/raspberrypios/renderd b/etc/raspberrypios/renderd new file mode 100644 index 00000000..133ef16f --- /dev/null +++ b/etc/raspberrypios/renderd @@ -0,0 +1,153 @@ +#! /bin/sh +### BEGIN INIT INFO +# Provides: renderd +# Required-Start: $remote_fs +# Required-Stop: $remote_fs +# Should-Start: postgresql +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: Mapnik rendering daemon +# Description: Mapnik rendering daemon. +### END INIT INFO + +# Do NOT "set -e" + +# PATH should only include /usr/* if it runs after the mountnfs.sh script +PATH=/sbin:/usr/sbin:/bin:/usr/bin +DESC="Mapnik rendering daemon" +NAME=renderd +DAEMON=/usr/bin/$NAME +DAEMON_ARGS="-c /etc/renderd.conf" +PIDSOCKDIR=/var/run/$NAME +PIDFILE=$PIDSOCKDIR/$NAME.pid +SCRIPTNAME=/etc/init.d/$NAME +RUNASUSER=_renderd + +# Exit if the package is not installed +[ -x "$DAEMON" ] || exit 0 + +# Read configuration variable file if it is present +[ -r /etc/default/$NAME ] && . /etc/default/$NAME + +# Load the VERBOSE setting and other rcS variables +. /lib/init/vars.sh + +# Define LSB log_* functions. +# Depend on lsb-base (>= 3.0-6) to ensure that this file is present. +. /lib/lsb/init-functions + +# +# Function that starts the daemon/service +# +do_start() +{ + [ -d "$PIDSOCKDIR" ] || mkdir -p $PIDSOCKDIR && chown $RUNASUSER $PIDSOCKDIR + # Return + # 0 if daemon has been started + # 1 if daemon was already running + # 2 if daemon could not be started + start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $RUNASUSER --exec $DAEMON --test > /dev/null \ + || return 1 + start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $RUNASUSER --exec $DAEMON -- \ + $DAEMON_ARGS \ + || return 2 + # Add code here, if necessary, that waits for the process to be ready + # to handle requests from services started subsequently which depend + # on this one. As a last resort, sleep for some time. +} + +# +# Function that stops the daemon/service +# +do_stop() +{ + # Return + # 0 if daemon has been stopped + # 1 if daemon was already stopped + # 2 if daemon could not be stopped + # other if a failure occurred + start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME + RETVAL="$?" + [ "$RETVAL" = 2 ] && return 2 + # Wait for children to finish too if this is a daemon that forks + # and if the daemon is only ever run from this initscript. + # If the above conditions are not satisfied then add some other code + # that waits for the process to drop all resources that could be + # needed by services started subsequently. A last resort is to + # sleep for some time. + start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON + [ "$?" = 2 ] && return 2 + # Many daemons don't delete their pidfiles when they exit. + rm -f $PIDFILE + return "$RETVAL" +} + +# +# Function that sends a SIGHUP to the daemon/service +# +do_reload() { + # + # If the daemon can reload its configuration without + # restarting (for example, when it is sent a SIGHUP), + # then implement that here. + # + start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME + return 0 +} + +case "$1" in + start) + log_daemon_msg "Starting $DESC" "$NAME" + do_start + case "$?" in + 0|1) log_end_msg 0 ;; + 2) log_end_msg 1 ;; + esac + ;; + stop) + log_daemon_msg "Stopping $DESC" "$NAME" + do_stop + case "$?" in + 0|1) log_end_msg 0 ;; + 2) log_end_msg 1 ;; + esac + ;; + #reload|force-reload) + # + # If do_reload() is not implemented then leave this commented out + # and leave 'force-reload' as an alias for 'restart'. + # + #log_daemon_msg "Reloading $DESC" "$NAME" + #do_reload + #log_end_msg $? + #;; + restart|force-reload) + # + # If the "reload" option is implemented then remove the + # 'force-reload' alias + # + log_daemon_msg "Restarting $DESC" "$NAME" + do_stop + case "$?" in + 0|1) + do_start + case "$?" in + 0) log_end_msg 0 ;; + 1) log_end_msg 1 ;; # Old process is still running + *) log_end_msg 1 ;; # Failed to start + esac + ;; + *) + # Failed to stop + log_end_msg 1 + ;; + esac + ;; + *) + #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2 + echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2 + exit 3 + ;; +esac + +: diff --git a/etc/debian11/renderd.conf b/etc/raspberrypios/renderd.conf similarity index 91% rename from etc/debian11/renderd.conf rename to etc/raspberrypios/renderd.conf index 21c6e68a..81fae66e 100644 --- a/etc/debian11/renderd.conf +++ b/etc/raspberrypios/renderd.conf @@ -11,7 +11,7 @@ font_dir_recurse=true [ajt] URI=/hot/ -TILEDIR=/var/lib/mod_tile +TILEDIR=/var/cache/renderd/tiles XML=/opt/openstreetmap-carto/mapnik.xml HOST=localhost TILESIZE=256 diff --git a/etc/ubuntu20.04/wrolpi-api.service b/etc/raspberrypios/wrolpi-api.service similarity index 76% rename from etc/ubuntu20.04/wrolpi-api.service rename to etc/raspberrypios/wrolpi-api.service index 069696c1..3533fe9a 100644 --- a/etc/ubuntu20.04/wrolpi-api.service +++ b/etc/raspberrypios/wrolpi-api.service @@ -5,7 +5,7 @@ PartOf=wrolpi.target [Service] Environment=PYTHONUNBUFFERED=1 -ExecStart=/usr/bin/python3 /opt/wrolpi/main.py -v api +ExecStart=/opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py -v api Group=wrolpi User=wrolpi WorkingDirectory=/opt/wrolpi/ diff --git a/etc/raspberrypios/wrolpi-app.service b/etc/raspberrypios/wrolpi-app.service new file mode 100644 index 00000000..11e59b43 --- /dev/null +++ b/etc/raspberrypios/wrolpi-app.service @@ -0,0 +1,13 @@ +[Unit] +Description=WROLPi React APP +Wants=wrolpi-api.service +PartOf=wrolpi.target + +[Service] +ExecStart=/usr/bin/serve -s build -l 5000 +Group=wrolpi +User=wrolpi +WorkingDirectory=/opt/wrolpi/app/ + +[Install] +WantedBy=default.target diff --git a/etc/ubuntu20.04/wrolpi.target b/etc/raspberrypios/wrolpi.target similarity index 100% rename from etc/ubuntu20.04/wrolpi.target rename to etc/raspberrypios/wrolpi.target diff --git a/etc/ubuntu20.04/index.html b/etc/ubuntu20.04/index.html deleted file mode 100644 index 2ab8047f..00000000 --- a/etc/ubuntu20.04/index.html +++ /dev/null @@ -1,37 +0,0 @@ - - - - - - - - - - -
- - - diff --git a/etc/ubuntu20.04/ports.conf b/etc/ubuntu20.04/ports.conf deleted file mode 100644 index ddcbc031..00000000 --- a/etc/ubuntu20.04/ports.conf +++ /dev/null @@ -1,15 +0,0 @@ -# If you just change the port or add more ports here, you will likely also -# have to change the VirtualHost statement in -# /etc/apache2/sites-enabled/000-default.conf - -Listen 8084 - - - Listen 443 - - - - Listen 443 - - -# vim: syntax=apache ts=4 sw=4 sts=4 sr noet diff --git a/etc/ubuntu20.04/renderd.conf b/etc/ubuntu20.04/renderd.conf deleted file mode 100644 index 7ad24e2e..00000000 --- a/etc/ubuntu20.04/renderd.conf +++ /dev/null @@ -1,17 +0,0 @@ -[renderd] -num_threads=4 -tile_dir=/var/lib/mod_tile -stats_file=/var/run/renderd/renderd.stats - -[mapnik] -plugins_dir=/usr/lib/mapnik/3.0/input -font_dir=/usr/share/fonts/truetype -font_dir_recurse=1 - -[ajt] -URI=/hot/ -TILEDIR=/var/lib/mod_tile -XML=/opt/openstreetmap-carto/mapnik.xml -HOST=localhost -TILESIZE=256 -MAXZOOM=20 diff --git a/install.sh b/install.sh index 729f9ec8..fde07466 100755 --- a/install.sh +++ b/install.sh @@ -59,7 +59,7 @@ git clone https://github.com/lrnselfreliance/wrolpi.git /opt/wrolpi || : (cd /opt/wrolpi && git fetch && git checkout "${BRANCH}" && git reset --hard origin/"${BRANCH}") || exit 4 if [ ${rpi} == true ]; then - /opt/wrolpi/scripts/install_ubuntu_20.04.sh 2>&1 | tee /opt/wrolpi/install.log + /opt/wrolpi/scripts/install_raspberrypios.sh 2>&1 | tee /opt/wrolpi/install.log install_code=${PIPESTATUS[0]} elif [ ${debian11} == true ]; then /opt/wrolpi/scripts/install_debian_11.sh 2>&1 | tee /opt/wrolpi/install.log diff --git a/pi-gen/stage2/04-wrolpi/01-packages b/pi-gen/stage2/04-wrolpi/01-packages index 758b2756..c31d0a59 100644 --- a/pi-gen/stage2/04-wrolpi/01-packages +++ b/pi-gen/stage2/04-wrolpi/01-packages @@ -62,10 +62,10 @@ postgresql-contrib protobuf-c-compiler python3-mapnik python3-pip -python3.9-dev -python3.9-doc -python3.9-minimal -python3.9-venv +python3-dev +python3-doc +python3-minimal +python3-venv renderd software-properties-common tar diff --git a/pi-gen/stage2/04-wrolpi/02-run.sh b/pi-gen/stage2/04-wrolpi/02-run.sh index 8946c269..bdb56714 100755 --- a/pi-gen/stage2/04-wrolpi/02-run.sh +++ b/pi-gen/stage2/04-wrolpi/02-run.sh @@ -1,4 +1,12 @@ #! /bin/bash -e mkdir -p "${ROOTFS_DIR}/opt/wrolpi-blobs" -cp files/district-of-columbia-230116.dump.gz "${ROOTFS_DIR}/opt/wrolpi-blobs/gis-map.dump.gz" +cp files/district-of-columbia-*.dump.gz "${ROOTFS_DIR}/opt/wrolpi-blobs/gis-map.dump.gz" + +set +x + +echo +echo "======================================================================" +echo "02-run.sh completed" +echo "======================================================================" +echo diff --git a/pi-gen/stage2/04-wrolpi/03-run-chroot.sh b/pi-gen/stage2/04-wrolpi/03-run-chroot.sh index 9aa17123..a160c049 100755 --- a/pi-gen/stage2/04-wrolpi/03-run-chroot.sh +++ b/pi-gen/stage2/04-wrolpi/03-run-chroot.sh @@ -6,7 +6,7 @@ set -x apt purge -y hostapd apt autoremove -y -cat >> /etc/profile << 'EOF' +cat >>/etc/profile <<'EOF' alias ll='ls -lh' alias la='ll -a' EOF @@ -15,22 +15,24 @@ EOF sed -i 's/port = 5433/port = 5432/' /etc/postgresql/13/main/postgresql.conf # Install Node console commands. -single-file --version || npm i -g serve@12.0.1 single-file-cli@1.0.15 'git+https://github.com/lrnselfreliance/readability-extractor' carto@1.2.0 +single-file --version || npm i -g serve@12.0.1 single-file-cli@1.0.15 readability-extractor@0.0.6 carto@1.2.0 # Put the latest WROLPi release in /opt/wrolpi. rm -rf /opt/wrolpi -git clone -b master https://github.com/lrnselfreliance/wrolpi.git /opt/wrolpi -chown -R ${FIRST_USER_NAME}:${FIRST_USER_NAME} /opt/wrolpi +git clone -b release https://github.com/lrnselfreliance/wrolpi.git /opt/wrolpi # Install Python requirements. pip3 install --upgrade pip -pip3 install -r /opt/wrolpi/requirements.txt +python3 -m venv /opt/wrolpi/venv +/opt/wrolpi/venv/bin/pip3 install -r /opt/wrolpi/requirements.txt # Install webapp. cd /opt/wrolpi/app npm install npm run build +chown -R ${FIRST_USER_NAME}:${FIRST_USER_NAME} /opt/wrolpi + # Configure nginx. cp /opt/wrolpi/nginx.conf /etc/nginx/nginx.conf cp /opt/wrolpi/50x.html /var/www/50x.html @@ -53,9 +55,9 @@ chmod 660 /etc/sudoers.d/90-wrolpi [ -d /media/wrolpi ] || mkdir /media/wrolpi chown ${FIRST_USER_NAME}:${FIRST_USER_NAME} /media/wrolpi -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi-api.service /etc/systemd/system/ -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi-app.service /etc/systemd/system/ -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi.target /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-api.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-app.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi.target /etc/systemd/system/ systemctl enable wrolpi-api.service systemctl enable wrolpi-app.service @@ -63,3 +65,10 @@ systemctl enable wrolpi-app.service # NetworkManager for the hotspot. systemctl enable NetworkManager +set +x + +echo +echo "======================================================================" +echo "03-run-chroot.sh completed" +echo "======================================================================" +echo diff --git a/pi-gen/stage2/04-wrolpi/04-run-chroot.sh b/pi-gen/stage2/04-wrolpi/04-run-chroot.sh index 98cf0258..f070eda2 100755 --- a/pi-gen/stage2/04-wrolpi/04-run-chroot.sh +++ b/pi-gen/stage2/04-wrolpi/04-run-chroot.sh @@ -5,7 +5,7 @@ set -x # Create mapnik config. if [ ! -d /opt/openstreetmap-carto ]; then - git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto + git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto fi (cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff) chown -R ${FIRST_USER_NAME}:${FIRST_USER_NAME} /opt/openstreetmap-carto @@ -15,7 +15,7 @@ if [[ ! -f /opt/openstreetmap-carto/mapnik.xml || ! -s /opt/openstreetmap-carto/ fi # WROLPi user can access WROLPi and Map database. -cat > /home/wrolpi/.pgpass << 'EOF' +cat >/home/wrolpi/.pgpass <<'EOF' 127.0.0.1:5432:gis:_renderd:wrolpi 127.0.0.1:5432:wrolpi:wrolpi:wrolpi EOF @@ -23,19 +23,27 @@ chown -R ${FIRST_USER_NAME}:${FIRST_USER_NAME} /home/wrolpi chmod 0600 /home/wrolpi/.pgpass # Configure renderd. -cp /opt/wrolpi/etc/debian11/renderd.conf /etc/renderd.conf +cp /opt/wrolpi/etc/raspberrypios/renderd.conf /etc/renderd.conf # Configure Apache2 to listen on 8084. Import renderd into apache. -cp /opt/wrolpi/etc/debian11/ports.conf /etc/apache2/ports.conf -cp /opt/wrolpi/etc/debian11/000-default.conf /etc/apache2/sites-available/000-default.conf -cp /opt/wrolpi/etc/ubuntu20.04/mod_tile.conf /etc/apache2/conf-enabled/mod_tile.conf +cp /opt/wrolpi/etc/raspberrypios/ports.conf /etc/apache2/ports.conf +cp /opt/wrolpi/etc/raspberrypios/000-default.conf /etc/apache2/sites-available/000-default.conf +cp /opt/wrolpi/etc/raspberrypios/mod_tile.conf /etc/apache2/conf-enabled/mod_tile.conf # Copy Leaflet files to Apache's directory so they can be used offline. -cp /opt/wrolpi/etc/debian11/index.html \ +cp /opt/wrolpi/etc/raspberrypios/index.html \ /opt/wrolpi/modules/map/leaflet.js \ /opt/wrolpi/modules/map/leaflet.css /var/www/html/ chmod 644 /var/www/html/* -mkdir /var/lib/mod_tile -chown -R _renderd:_renderd /var/lib/mod_tile - # Allow wrolpi user to delete map tiles. usermod -aG _renderd wrolpi + +[ -d /var/cache/renderd/tiles ] || mkdir /var/cache/renderd/tiles +chown -R _renderd:_renderd /var/cache/renderd/tiles + +set +x + +echo +echo "======================================================================" +echo "04-run-chroot.sh completed" +echo "======================================================================" +echo diff --git a/requirements.txt b/requirements.txt index d310cfc0..f5b35d7c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,7 @@ Pillow==9.3.0 Pint==0.18 PyPDF2[crypto]==2.11.2 -PyYAML==5.3.1 # this is the only version available on RPi +PyYAML==6.0 SQLAlchemy==1.3.22 aiohttp==3.8.3 alembic==1.7.5 diff --git a/scripts/build_api_and_app.sh b/scripts/build_api_and_app.sh new file mode 100755 index 00000000..e2693270 --- /dev/null +++ b/scripts/build_api_and_app.sh @@ -0,0 +1,21 @@ +#! /usr/bin/env bash +# This script will rebuild and install dependencies of the API and App. + +set -x +set -e + +# Install any App dependencies. +cd /opt/wrolpi/app || exit 1 +npm install || npm install || npm install || npm install # try install multiple times :( +# Build app in the background. +npm run build & + +# Install python requirements in background job. +/opt/wrolpi/venv/bin/pip3 install -r /opt/wrolpi/requirements.txt & + +wait $(jobs -p) + +# Upgrade the WROLPi database. +(cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py db upgrade) + +systemctl start wrolpi.target diff --git a/scripts/create_db.sh b/scripts/create_db.sh index 82cd4624..75f84956 100755 --- a/scripts/create_db.sh +++ b/scripts/create_db.sh @@ -2,13 +2,13 @@ # Create WROLPi API database. sudo -u postgres psql -c '\l' | grep wrolpi || ( - sudo -u postgres createuser wrolpi + sudo -u postgres createuser -s wrolpi # superuser so maps can be imported sudo -u postgres psql -c "alter user postgres password 'wrolpi'" sudo -u postgres psql -c "alter user wrolpi password 'wrolpi'" sudo -u postgres createdb -E UTF8 -O wrolpi wrolpi echo "Created wrolpi database" ) -sudo -u wrolpi /bin/bash -c 'cd /opt/wrolpi && ./main.py db upgrade' +sudo -u wrolpi /bin/bash -c 'cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 main.py db upgrade' # Create gis (map) database. sudo -u postgres psql -c '\l' | grep gis || ( @@ -25,4 +25,4 @@ sudo -u postgres psql -d gis -c 'ALTER TABLE geography_columns OWNER TO _renderd sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO _renderd' sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO _renderd" # Clear map tile cache files. -[ -d /var/lib/mod_tile/ajt ] && sudo rm -r /var/lib/mod_tile/ajt +[ -d /var/cache/renderd/tiles/ajt ] && sudo rm -r /var/cache/renderd/tiles/ajt diff --git a/scripts/import_map.sh b/scripts/import_map.sh index 9cc4a3f3..5e352d9a 100755 --- a/scripts/import_map.sh +++ b/scripts/import_map.sh @@ -18,7 +18,7 @@ if [ "${1}" == "" ]; then fi HAS_ICE_SHEET=false -psql gis wrolpi -c '\d' | grep icesheet_polygons && HAS_ICE_SHEET=true +psql gis -c '\d' | grep icesheet_polygons && HAS_ICE_SHEET=true if [ ${HAS_ICE_SHEET} = false ]; then echo "WROLPi: Missing icesheet_polygons. Run reset_map.sh" exit 4 @@ -58,12 +58,30 @@ if [[ ${1} == *.osm.pbf ]]; then pbf_path=${MERGED_TMP_FILE} fi - nice -n 18 osm2pgsql -d gis --create --slim -G --hstore -U _renderd -H 127.0.0.1 \ + # Import as wrolpi user (the user running this script), then change ownership to _renderd. + nice -n 18 osm2pgsql -d gis --create --slim -G --hstore \ --tag-transform-script /opt/openstreetmap-carto/openstreetmap-carto.lua \ -C ${MAX_CACHE} \ --number-processes 3 \ -S /opt/openstreetmap-carto/openstreetmap-carto.style \ "${pbf_path}" + # Assign all data to _renderd so the map can be rendered. + psql -d gis -c 'ALTER TABLE external_data OWNER TO _renderd' + psql -d gis -c 'ALTER TABLE geography_columns OWNER TO _renderd' + psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO _renderd' + psql -d gis -c "ALTER TABLE icesheet_outlines OWNER TO _renderd" + psql -d gis -c "ALTER TABLE icesheet_polygons OWNER TO _renderd" + psql -d gis -c "ALTER TABLE ne_100m_admin_0_boundary_lines_land OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_line OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_nodes OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_point OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_polygon OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_rels OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_roads OWNER TO _renderd" + psql -d gis -c "ALTER TABLE planet_osm_ways OWNER TO _renderd" + psql -d gis -c "ALTER TABLE simplified_water_polygons OWNER TO _renderd" + psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO _renderd" + psql -d gis -c "ALTER TABLE water_polygons OWNER TO _renderd" elif [[ ${1} == *.dump ]]; then if [ ! -f "${1}" ]; then echo "WROLPi: File does not exist! ${1}" diff --git a/scripts/initialize_map.sh b/scripts/initialize_map.sh index 54dc8dbd..c2e0db6b 100755 --- a/scripts/initialize_map.sh +++ b/scripts/initialize_map.sh @@ -8,9 +8,9 @@ fi set -x # Use D.C. to initialized DB because it is so small. -DC_MAP=district-of-columbia-latest.osm.pbf +DC_MAP=district-of-columbia-230222.osm.pbf DC_MAP_PATH=/tmp/${DC_MAP} -wget --continue https://download.geofabrik.de/north-america/us/${DC_MAP} -O ${DC_MAP_PATH} || : +[ -f "${DC_MAP_PATH}" ] || wget --continue https://wrolpi.org/downloads/${DC_MAP} -O ${DC_MAP_PATH} if [ ! -f "${DC_MAP_PATH}" ]; then echo "Could not download D.C. map" @@ -28,15 +28,15 @@ dc_import=false indexes=false while [ "${COUNT}" -lt "${MAX_TRIES}" ]; do [[ "${external_data}" = false ]] && - sudo -u wrolpi nice -n 18 /opt/openstreetmap-carto/scripts/get-external-data.py \ + sudo -u _renderd nice -n 18 /opt/openstreetmap-carto/scripts/get-external-data.py \ -C \ - -d gis -U wrolpi && + -d gis -U _renderd && external_data=true [[ "${dc_import}" = false ]] && - sudo -u wrolpi nice -n 18 /opt/wrolpi/scripts/import_map.sh ${DC_MAP_PATH} && + sudo -u _renderd nice -n 18 /opt/wrolpi/scripts/import_map.sh ${DC_MAP_PATH} && dc_import=true [[ "${indexes}" = false ]] && - sudo -u wrolpi psql -d gis -f /opt/openstreetmap-carto/indexes.sql && + sudo -u _renderd psql -d gis -f /opt/openstreetmap-carto/indexes.sql && indexes=true if [[ "${external_data}" == true && "${dc_import}" == true && "${indexes}" == true ]]; then SUCCESS=true diff --git a/scripts/install_debian_11.sh b/scripts/install_debian_11.sh index d02b7e1a..b262020e 100755 --- a/scripts/install_debian_11.sh +++ b/scripts/install_debian_11.sh @@ -4,28 +4,42 @@ echo "Debian 11 install start: $(date '+%Y-%m-%d %H:%M:%S')" set -x set -e -# Install npm repos. -npm --version || curl -fsSL https://deb.nodesource.com/setup_18.x | bash - - # Update if we haven't updated in the last day. [ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update # Install dependencies -apt install -y apt-transport-https ca-certificates curl gnupg-agent gcc libpq-dev software-properties-common \ - postgresql-13 nginx-full nginx-doc python3-full python3-dev python3-doc python3-venv htop vim ffmpeg hostapd \ - chromium chromium-driver cpufrequtils network-manager nodejs +apt install -y \ + apt-transport-https \ + ca-certificates \ + chromium \ + chromium-driver \ + cpufrequtils \ + curl \ + ffmpeg \ + gcc \ + gnupg-agent \ + hostapd \ + htop \ + libpq-dev \ + network-manager \ + nginx-doc \ + nginx-full \ + postgresql-13 \ + python3-dev \ + python3-doc \ + python3-full \ + python3-pip \ + python3-venv \ + software-properties-common \ + vim + +# Install npm. +npm --version || curl -fsSL https://deb.nodesource.com/setup_18.x | bash - +apt install -y nodejs +node -v +npm -v # Install serve, and archiving tools. -sudo npm i -g serve single-file-cli@1.0.33 'git+https://github.com/lrnselfreliance/readability-extractor' & - -# Setup the virtual environment that main.py expects -pip3 --version || ( - # If no pip, install pip - curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py && - python3 /tmp/get-pip.py & -) - -# Wait for npm and pip3. -wait $(jobs -p) +sudo npm i -g serve@12.0.1 single-file-cli@1.0.33 readability-extractor@0.0.6 # Build React app in background job. cd /opt/wrolpi/app || exit 5 @@ -33,7 +47,8 @@ npm install || npm install || npm install || npm install # try install multiple npm run build & # Install python requirements in background job. -pip3 install -r /opt/wrolpi/requirements.txt & +python3 -m venv /opt/wrolpi/venv +/opt/wrolpi/venv/bin/pip3 install -r /opt/wrolpi/requirements.txt & wait $(jobs -p) @@ -44,7 +59,12 @@ cp /opt/wrolpi/50x.html /var/www/50x.html # Create the WROLPi user grep wrolpi /etc/passwd || useradd -md /home/wrolpi wrolpi -s "$(command -v bash)" -chown -R wrolpi:wrolpi /opt/wrolpi +cat >/home/wrolpi/.pgpass <<'EOF' +127.0.0.1:5432:gis:_renderd:wrolpi +127.0.0.1:5432:wrolpi:wrolpi:wrolpi +EOF +chown -R wrolpi:wrolpi /home/wrolpi /opt/wrolpi +chmod 0600 /home/wrolpi/.pgpass # Give WROLPi group a few privileged commands via sudo without password. cat >/etc/sudoers.d/90-wrolpi <<'EOF' @@ -62,9 +82,9 @@ visudo -c -f /etc/sudoers.d/90-wrolpi chown wrolpi:wrolpi /media/wrolpi # Install the systemd services -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi-api.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-api.service /etc/systemd/system/ cp /opt/wrolpi/etc/debian11/wrolpi-app.service /etc/systemd/system/ -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi.target /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi.target /etc/systemd/system/ /usr/bin/systemctl daemon-reload systemctl enable wrolpi-api.service systemctl enable wrolpi-app.service @@ -74,13 +94,16 @@ systemctl stop wrolpi-app.service # Configure Postgresql. Do this after the API is stopped. sudo -u postgres psql -c '\l' | grep wrolpi || ( - sudo -u postgres createuser wrolpi && + sudo -u postgres createuser -s wrolpi && sudo -u postgres psql -c "alter user postgres password 'wrolpi'" && sudo -u postgres psql -c "alter user wrolpi password 'wrolpi'" && sudo -u postgres createdb -E UTF8 -O wrolpi wrolpi ) # Initialize/upgrade the WROLPi database. -(cd /opt/wrolpi && /usr/bin/python3 /opt/wrolpi/main.py db upgrade) +(cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py db upgrade) + +# Install map only if that script hasn't finished. +[ -f /var/www/html/leaflet.css ] || /opt/wrolpi/scripts/install_map_debian_11.sh set +x diff --git a/scripts/install_map_debian_11.sh b/scripts/install_map_debian_11.sh index c7baa5f0..0bd1ff54 100755 --- a/scripts/install_map_debian_11.sh +++ b/scripts/install_map_debian_11.sh @@ -9,48 +9,52 @@ set -e [ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update # Install map dependencies. apt install -y libapache2-mod-tile renderd tar unzip wget bzip2 apache2 lua5.1 mapnik-utils python3-mapnik gdal-bin \ - fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont osmium-tool + fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont osmium-tool python3-psycopg2 python3-yaml -/usr/bin/npm install -g carto -/usr/bin/carto -v +/usr/bin/npm install -g carto@1.2.0 +carto -v # Create `gis` database for map. -sudo -u postgres psql -c '\l' | grep gis || sudo -u postgres createdb -E UTF8 -O wrolpi gis -sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis;" || : -sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;" || : -sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO wrolpi;" -sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO wrolpi;" +sudo -u postgres psql -c '\l' | grep gis || ( + sudo -u postgres createuser _renderd || : + sudo -u postgres psql -c "alter user _renderd password 'wrolpi'" || : + sudo -u postgres createdb -E UTF8 -O _renderd gis + sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis" || : + sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore" || : + echo "Created gis database" +) +sudo -u postgres psql -d gis -c 'ALTER TABLE geography_columns OWNER TO _renderd' +sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO _renderd' +sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO _renderd" # Install Debian 11 compatible mod_tile. git clone -b switch2osm https://github.com/lrnselfreliance/mod_tile.git /tmp/mod_tile || : # Install init/systemd config to start map service on startup. -cp /tmp/mod_tile/debian/renderd.init /etc/init.d/renderd +cp /opt/wrolpi/etc/raspberrypios/renderd /etc/init.d/renderd sudo chmod +x /etc/init.d/renderd -sed -i -e 's/^RUNASUSER.*/RUNASUSER=wrolpi/' /etc/init.d/renderd -sed -i -e 's/^DAEMON=.*/DAEMON=\/usr\/bin\/$NAME/' /etc/init.d/renderd -sed -i -e 's/^DAEMON_ARGS=.*/DAEMON_ARGS="-c \/etc\/renderd.conf"/' /etc/init.d/renderd cp /tmp/mod_tile/debian/renderd.service /lib/systemd/system/ chmod +x /etc/init.d/renderd # Initialize gis database. [ -d /opt/openstreetmap-carto ] && chown -R wrolpi:wrolpi /opt/openstreetmap-carto git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto || : -chown -R wrolpi:wrolpi /opt/openstreetmap-carto -sudo -u wrolpi /bin/bash -c '(cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff)' +chown -R _renderd:_renderd /opt/openstreetmap-carto +sudo -u _renderd /bin/bash -c '(cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff)' cd /opt/openstreetmap-carto if [[ ! -f /opt/openstreetmap-carto/mapnik.xml || ! -s /opt/openstreetmap-carto/mapnik.xml ]]; then - /usr/bin/carto project.mml >/opt/openstreetmap-carto/mapnik.xml + carto project.mml >/opt/openstreetmap-carto/mapnik.xml fi +chown -R _renderd:_renderd /opt/openstreetmap-carto # Reset map. Force it. /opt/wrolpi/scripts/reset_map.sh -f -cp /opt/wrolpi/etc/debian11/renderd.conf /etc/renderd.conf +cp /opt/wrolpi/etc/raspberrypios/renderd.conf /etc/renderd.conf # Configure Apache2 to listen on 8084. -cp /opt/wrolpi/etc/debian11/ports.conf /etc/apache2/ports.conf +cp /opt/wrolpi/etc/raspberrypios/ports.conf /etc/apache2/ports.conf cp /opt/wrolpi/etc/debian11/000-default.conf /etc/apache2/sites-available/000-default.conf # Copy Leaflet files to Apache's directory so they can be used offline. -cp /opt/wrolpi/etc/debian11/index.html \ +cp /opt/wrolpi/etc/raspberrypios/index.html \ /opt/wrolpi/modules/map/leaflet.js \ /opt/wrolpi/modules/map/leaflet.css /var/www/html/ chmod 644 /var/www/html/* diff --git a/scripts/install_map_raspberrypios.sh b/scripts/install_map_raspberrypios.sh new file mode 100755 index 00000000..ae429a76 --- /dev/null +++ b/scripts/install_map_raspberrypios.sh @@ -0,0 +1,76 @@ +#! /usr/bin/env bash +# This script installs OpenStreetMap tile server. +# This server uses Apache2, renderd, and mod_tile to render and display map tiles. + +set -x +set -e + +# Update if we haven't updated in the last day. +[ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update +# Install map dependencies. +apt install -y libapache2-mod-tile libboost-all-dev git tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev \ + libgeos++-dev libpq-dev libbz2-dev libproj-dev protobuf-c-compiler libfreetype6-dev libtiff5-dev \ + libicu-dev libgdal-dev libcairo2-dev libcairomm-1.0-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-0-dev \ + postgis postgresql-13-postgis-3 postgresql-13-postgis-3-scripts osm2pgsql gdal-bin libmapnik-dev mapnik-utils \ + python3-mapnik apache2 apache2-dev libcurl4-gnutls-dev libiniparser-dev libmemcached-dev librados-dev \ + fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted postgresql-contrib osmium-tool python3-psycopg2 python3-yaml + +/usr/bin/npm install -g carto@1.2.0 +carto -v + +# Create `gis` database for map. +sudo -u postgres psql -c '\l' | grep gis || ( + sudo -u postgres createuser _renderd || : + sudo -u postgres psql -c "alter user _renderd password 'wrolpi'" || : + sudo -u postgres createdb -E UTF8 -O _renderd gis + sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis" || : + sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore" || : + echo "Created gis database" +) +sudo -u postgres psql -d gis -c 'ALTER TABLE geography_columns OWNER TO _renderd' +sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO _renderd' +sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO _renderd" + +# Install Raspberry Pi OS compatible mod_tile. +git clone -b switch2osm https://github.com/lrnselfreliance/mod_tile.git /tmp/mod_tile || : +# Install init/systemd config to start map service on startup. +cp /opt/wrolpi/etc/raspberrypios/renderd /etc/init.d/renderd +sudo chmod +x /etc/init.d/renderd +cp /tmp/mod_tile/debian/renderd.service /lib/systemd/system/ +chmod +x /etc/init.d/renderd + +# Initialize gis database. +[ -d /opt/openstreetmap-carto ] && chown -R wrolpi:wrolpi /opt/openstreetmap-carto +git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto || : +chown -R _renderd:_renderd /opt/openstreetmap-carto +sudo -u _renderd /bin/bash -c '(cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff)' +cd /opt/openstreetmap-carto +if [[ ! -f /opt/openstreetmap-carto/mapnik.xml || ! -s /opt/openstreetmap-carto/mapnik.xml ]]; then + carto project.mml >/opt/openstreetmap-carto/mapnik.xml +fi +chown -R _renderd:_renderd /opt/openstreetmap-carto + +# Reset map. Force it. +/opt/wrolpi/scripts/reset_map.sh -f + +cp /opt/wrolpi/etc/raspberrypios/renderd.conf /etc/renderd.conf +# Enable mod_tile. +cp /opt/wrolpi/etc/raspberrypios/mod_tile.conf /etc/apache2/conf-available/mod_tile.conf + +/usr/sbin/a2enconf mod_tile + +# Configure Apache2 to listen on 8084. +cp /opt/wrolpi/etc/raspberrypios/ports.conf /etc/apache2/ports.conf +cp /opt/wrolpi/etc/raspberrypios/000-default.conf /etc/apache2/sites-available/000-default.conf +# Copy Leaflet files to Apache's directory so they can be used offline. +cp /opt/wrolpi/etc/raspberrypios/index.html \ + /opt/wrolpi/modules/map/leaflet.js \ + /opt/wrolpi/modules/map/leaflet.css /var/www/html/ +chmod 644 /var/www/html/* + +systemctl enable renderd +systemctl stop renderd +systemctl start renderd + +systemctl stop apache2 +systemctl start apache2 diff --git a/scripts/install_map_ubuntu_20.04.sh b/scripts/install_map_ubuntu_20.04.sh deleted file mode 100755 index 315adffd..00000000 --- a/scripts/install_map_ubuntu_20.04.sh +++ /dev/null @@ -1,84 +0,0 @@ -#! /usr/bin/env bash -# This script installs OpenStreetMap tile server. -# This server uses Apache2, renderd, and mod_tile to render and display map tiles. - -function yes_or_no { - while true; do - read -p "$* [y/n]:" yn - case $yn in - [Yy]*) return 0 ;; - [Nn]*) return 1 ;; - esac - done -} - -set -x -set -e - -# Update if we haven't updated in the last day. -[ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update -# Install map dependencies. -apt install -y libboost-all-dev git tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev \ - libgeos++-dev libpq-dev libbz2-dev libproj-dev protobuf-c-compiler libfreetype6-dev libtiff5-dev \ - libicu-dev libgdal-dev libcairo2-dev libcairomm-1.0-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-0-dev \ - postgis postgresql-12-postgis-3 postgresql-12-postgis-3-scripts osm2pgsql gdal-bin libmapnik-dev mapnik-utils \ - python3-mapnik apache2 apache2-dev libcurl4-gnutls-dev libiniparser-dev libmemcached-dev librados-dev \ - fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted postgresql-contrib osmium-tool - -/usr/bin/npm install -g carto -/usr/bin/carto -v - -# Create `gis` database for map. -sudo -u postgres psql -c '\l' | grep gis || sudo -u postgres createdb -E UTF8 -O wrolpi gis -sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis;" || : -sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;" || : -sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO wrolpi;" -sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO wrolpi;" - -# Install Ubuntu 20.04 compatible mod_tile. -git clone -b switch2osm https://github.com/lrnselfreliance/mod_tile.git /tmp/mod_tile || : -if [ ! -f /usr/lib/apache2/modules/mod_tile.so ]; then - (cd /tmp/mod_tile && ./autogen.sh && ./configure && make -j4 && make install && make install-mod_tile && ldconfig) -fi -# Install init/systemd config to start map service on startup. -cp /tmp/mod_tile/debian/renderd.init /etc/init.d/renderd -sudo chmod +x /etc/init.d/renderd -cp /tmp/mod_tile/debian/renderd.service /lib/systemd/system/ -sed -ie 's/^RUNASUSER.*/RUNASUSER=wrolpi/' /etc/init.d/renderd -chmod +x /etc/init.d/renderd -[ ! -d /var/lib/mod_tile ] && mkdir /var/lib/mod_tile -chown wrolpi /var/lib/mod_tile - -# Initialize gis database. -[ -d /opt/openstreetmap-carto ] && chown -R wrolpi:wrolpi /opt/openstreetmap-carto -git clone https://github.com/lrnselfreliance/openstreetmap-carto.git /opt/openstreetmap-carto || : -chown -R wrolpi:wrolpi /opt/openstreetmap-carto -sudo -u wrolpi /bin/bash -c '(cd /opt/openstreetmap-carto && git fetch && git checkout master && git reset --hard origin/master && git pull --ff)' -cd /opt/openstreetmap-carto -if [[ ! -f /opt/openstreetmap-carto/mapnik.xml || ! -s /opt/openstreetmap-carto/mapnik.xml ]]; then - /usr/bin/carto project.mml >/opt/openstreetmap-carto/mapnik.xml -fi - -# Reset map. Force it. -/opt/wrolpi/scripts/reset_map.sh -f - -cp /opt/wrolpi/etc/ubuntu20.04/renderd.conf /usr/local/etc/renderd.conf -# Enable mod_tile. -cp /opt/wrolpi/etc/ubuntu20.04/mod_tile.conf /etc/apache2/conf-available/mod_tile.conf - -/usr/sbin/a2enconf mod_tile - -# Configure Apache2 to listen on 8084. -cp /opt/wrolpi/etc/ubuntu20.04/ports.conf /etc/apache2/ports.conf -cp /opt/wrolpi/etc/ubuntu20.04/000-default.conf /etc/apache2/sites-available/000-default.conf -cp /opt/wrolpi/etc/ubuntu20.04/index.html /var/www/html/index.html -# Copy Leaflet files to Apache's directory so they can be used offline. -cp /opt/wrolpi/modules/map/leaflet.js /var/www/html/leaflet.js -cp /opt/wrolpi/modules/map/leaflet.css /var/www/html/leaflet.css - -systemctl enable renderd -systemctl stop renderd -systemctl start renderd - -systemctl stop apache2 -systemctl start apache2 diff --git a/scripts/install_ubuntu_20.04.sh b/scripts/install_raspberrypios.sh similarity index 70% rename from scripts/install_ubuntu_20.04.sh rename to scripts/install_raspberrypios.sh index a62fcc4d..af723f94 100755 --- a/scripts/install_ubuntu_20.04.sh +++ b/scripts/install_raspberrypios.sh @@ -1,31 +1,45 @@ #!/usr/bin/env bash -echo "Ubuntu 20.04 install start: $(date '+%Y-%m-%d %H:%M:%S')" +echo "Raspberry Pi OS install start: $(date '+%Y-%m-%d %H:%M:%S')" set -x set -e -# Install npm repos. -npm --version || curl -fsSL https://deb.nodesource.com/setup_18.x | bash - - # Update if we haven't updated in the last day. [ -z "$(find -H /var/lib/apt/lists -maxdepth 0 -mtime -1)" ] && apt update # Install dependencies -apt install -y apt-transport-https ca-certificates curl gnupg-agent gcc libpq-dev software-properties-common \ - postgresql-12 nginx-full nginx-doc python3.8-minimal python3.8-dev python3.8-doc python3.8-venv \ - ffmpeg hostapd nodejs chromium-browser chromium-chromedriver cpufrequtils network-manager +apt install -y \ + apt-transport-https \ + ca-certificates \ + chromium \ + chromium-driver \ + cpufrequtils \ + curl \ + ffmpeg \ + gcc \ + gnupg-agent \ + hostapd \ + htop \ + libpq-dev \ + network-manager \ + nginx-doc \ + nginx-full \ + postgresql-13 \ + python3-dev \ + python3-doc \ + python3-full \ + python3-pip \ + python3-venv \ + software-properties-common \ + vim + +# Install npm. +npm --version || curl -fsSL https://deb.nodesource.com/setup_18.x | bash - +apt install -y nodejs +node -v +npm -v # Install serve, and archiving tools. -sudo npm i -g serve single-file-cli@1.0.33 'git+https://github.com/lrnselfreliance/readability-extractor' & - -# Setup the virtual environment that main.py expects -pip3 --version || ( - # If no pip, install pip - curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py && - python3 /tmp/get-pip.py & -) - -# Wait for npm and pip3. -wait $(jobs -p) +sudo npm i -g serve@12.0.1 single-file-cli@1.0.33 readability-extractor@0.0.6 # Build React app in background job. cd /opt/wrolpi/app || exit 5 @@ -33,7 +47,8 @@ npm install || npm install || npm install || npm install # try install multiple npm run build & # Install python requirements in background job. -pip3 install -r /opt/wrolpi/requirements.txt & +python3 -m venv /opt/wrolpi/venv +/opt/wrolpi/venv/bin/pip3 install -r /opt/wrolpi/requirements.txt & wait $(jobs -p) @@ -62,9 +77,9 @@ visudo -c -f /etc/sudoers.d/90-wrolpi chown wrolpi:wrolpi /media/wrolpi # Install the systemd services -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi-api.service /etc/systemd/system/ -cp /opt/wrolpi/etc/debian11/wrolpi-app.service /etc/systemd/system/ -cp /opt/wrolpi/etc/ubuntu20.04/wrolpi.target /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-api.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi-app.service /etc/systemd/system/ +cp /opt/wrolpi/etc/raspberrypios/wrolpi.target /etc/systemd/system/ /usr/bin/systemctl daemon-reload systemctl enable wrolpi-api.service systemctl enable wrolpi-app.service @@ -74,13 +89,16 @@ systemctl stop wrolpi-app.service # Configure Postgresql. Do this after the API is stopped. sudo -u postgres psql -c '\l' | grep wrolpi || ( - sudo -u postgres createuser wrolpi && + sudo -u postgres createuser -s wrolpi && sudo -u postgres psql -c "alter user postgres password 'wrolpi'" && sudo -u postgres psql -c "alter user wrolpi password 'wrolpi'" && sudo -u postgres createdb -E UTF8 -O wrolpi wrolpi ) # Initialize/upgrade the WROLPi database. -(cd /opt/wrolpi && /usr/bin/python3 /opt/wrolpi/main.py db upgrade) +(cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py db upgrade) + +# Install map only if that script hasn't finished. +[ -f /var/www/html/leaflet.css ] || /opt/wrolpi/scripts/install_map_raspberrypios.sh set +x diff --git a/scripts/reset_map.sh b/scripts/reset_map.sh index f5a5a62f..cdfb4bb3 100755 --- a/scripts/reset_map.sh +++ b/scripts/reset_map.sh @@ -66,11 +66,11 @@ if [ -d /var/lib/mod_tile ]; then else mkdir /var/lib/mod_tile fi -chown -R wrolpi:wrolpi /var/lib/mod_tile +chown -R _renderd:_renderd /var/lib/mod_tile sudo -u postgres dropdb gis || : -sudo -u postgres createdb -E UTF8 -O wrolpi gis +sudo -u postgres createdb -E UTF8 -O _renderd gis sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis;" || : sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;" || : sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO wrolpi;" diff --git a/upgrade.sh b/upgrade.sh new file mode 100755 index 00000000..66ec98a8 --- /dev/null +++ b/upgrade.sh @@ -0,0 +1,52 @@ +#! /usr/bin/env bash +# This script will upgrade the WROLPi API and App. It is required that you install WROLPi first. + +Help() { + # Display Help + echo "Upgrade WROLPi API and App on this machine." + echo + echo "Syntax: upgrade.sh [-h] [-b BRANCH]" + echo "options:" + echo "h Print this help." + echo "b Upgrade from this git BRANCH." + echo +} + +BRANCH="release" +while getopts ":hb:" option; do + case $option in + h) # display Help + Help + exit + ;; + b) + BRANCH="${OPTARG}" + ;; + *) # invalid argument(s) + echo "Error: Invalid option" + exit 1 + ;; + esac +done + +if [ ! -d /opt/wrolpi ] || [ ! -d /opt/wrolpi/wrolpi ]; then + echo "You must install WROLPi first. Try: /opt/wrolpi/install.sh" + exit 2 +fi + +set -x +set -e + +systemctl stop wrolpi-api +systemctl stop wrolpi-app + +# Pull the latest commit of the requested branch. +(cd /opt/wrolpi && git fetch && git checkout "${BRANCH}" && git reset --hard origin/"${BRANCH}") || exit 4 + +/opt/wrolpi/scripts/build_api_and_app.sh 2>&1 | tee /opt/wrolpi/upgrade.log + +set +x + +echo "Upgrade end $(date '+%Y-%m-%d %H:%M:%S')" >>/opt/wrolpi/upgrade.log + +echo "WROLPi upgrade has completed" diff --git a/wrolpi/cmd.py b/wrolpi/cmd.py index 310d81f8..561ab4f5 100644 --- a/wrolpi/cmd.py +++ b/wrolpi/cmd.py @@ -60,19 +60,19 @@ BASH_BIN = which('bash', '/bin/bash', warn=True) # Archive SINGLE_FILE_BIN = which('single-file', - '/usr/bin/single-file', # rpi ubuntu + '/usr/bin/single-file', # rpi os '/usr/local/bin/single-file', # debian warn=True, flag='singlefile_installed', ) CHROMIUM = which('chromium-browser', 'chromium', - '/usr/bin/chromium-browser', # rpi ubuntu + '/usr/bin/chromium-browser', # rpi os '/usr/bin/chromium', # debian warn=True, flag='chromium_installed', ) READABILITY_BIN = which('readability-extractor', - '/usr/bin/readability-extractor', # rpi ubuntu + '/usr/bin/readability-extractor', # rpi os '/usr/local/bin/readability-extractor', # debian warn=True, flag='readability_installed',