From 756c063d15c8c06c263761d47b5ddab394a7cd13 Mon Sep 17 00:00:00 2001 From: Roland Date: Sun, 5 Jul 2026 15:40:47 -0600 Subject: [PATCH] Upgrade/repair: a down database must never block the pipeline. scripts/upgrade.sh ran `main.py db upgrade` under set -e before repair.sh, so a broken Postgres bricked the very upgrade carrying the fix for it (seen live: the bootstrap bind-mount fix could not be delivered because the binds had already killed the cluster). repair.sh had the same problem with initialize_api_db.sh and the superuser psql. Warn and continue instead: the migration is retried by initialize_api_db.sh during repair, and if the database is still down afterwards the API reports it loudly while the system-level work (configs, services, bootstrap refresh) still completes. Co-Authored-By: Claude Fable 5 --- repair.sh | 9 ++++++--- scripts/upgrade.sh | 7 +++++-- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/repair.sh b/repair.sh index 66ed22fb..367b30e6 100755 --- a/repair.sh +++ b/repair.sh @@ -143,10 +143,13 @@ chmod 0440 /etc/sudoers.d/90-wrolpi # Verify this new file is valid. visudo -c -f /etc/sudoers.d/90-wrolpi -# Configure Postgresql. Do this after the API is stopped. -/opt/wrolpi/scripts/initialize_api_db.sh +# Configure Postgresql. Do this after the API is stopped. Never fatal: a +# repair/upgrade must complete its system-level work even when the database +# is down (e.g. it is the reboot after this repair that revives Postgres). +/opt/wrolpi/scripts/initialize_api_db.sh || \ + echo "WARNING: could not initialize the database; the API will report if it stays down." # wrolpi user is superuser so they can import maps. -sudo -iu postgres psql -c "alter user wrolpi with superuser" +sudo -iu postgres psql -c "alter user wrolpi with superuser" || : # Create the media directory. This should be mounted by the maintainer. [ -d /media/wrolpi ] || mkdir /media/wrolpi diff --git a/scripts/upgrade.sh b/scripts/upgrade.sh index 9a959585..df602aca 100755 --- a/scripts/upgrade.sh +++ b/scripts/upgrade.sh @@ -96,8 +96,11 @@ find /opt/wrolpi/venv -name "*.pyc" -delete 2>/dev/null || : # Install any new Python requirements. /opt/wrolpi/venv/bin/pip3 install --upgrade -r /opt/wrolpi/requirements.txt -# Upgrade the WROLPi database. -(cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py db upgrade) +# Upgrade the WROLPi database. Never fatal: the database may be down and +# repair.sh (which runs at the end of this upgrade) may be exactly what fixes +# it; initialize_api_db.sh re-runs this migration there. +(cd /opt/wrolpi && /opt/wrolpi/venv/bin/python3 /opt/wrolpi/main.py db upgrade) || \ + echo "WARNING: db upgrade failed (database down?); it will be retried during repair." # Upgrade WROLPi Help. /opt/wrolpi/scripts/install_help_service.sh || echo "Help install failed."