From cd9fb4c6de4e297fc95ef030e29166b90310cbd8 Mon Sep 17 00:00:00 2001 From: Roland Date: Sun, 31 May 2026 12:21:41 -0600 Subject: [PATCH] release-watch: don't auto-retry failed builds (default MAX_RETRIES=1) Builds take hours and a failure almost always needs a code fix, not a blind retry -- retrying 3x burned ~5h overnight. Default to a single attempt: on failure, record the marker, fire the notify hook, and wait for a human to investigate and clear it. The MAX_RETRIES override remains for N attempts. Co-Authored-By: Claude Opus 4.8 (1M context) --- release/release-watch.sh | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/release/release-watch.sh b/release/release-watch.sh index 1e7701d8..c2f9ed9b 100755 --- a/release/release-watch.sh +++ b/release/release-watch.sh @@ -8,15 +8,16 @@ # release.sh to build and publish it. latest.json is the source of truth for # "already built" -- there is no local bookkeeping for success. # -# A local state file is used only to throttle retries of a release that fails -# to build, so a permanently-broken release does not rebuild every hour -# forever. +# A local state file records a tag that failed to build. Builds are expensive +# (hours), so by default a failed tag is attempted once and then NOT auto-retried +# -- the notify hook alerts you, and you investigate and clear the marker (or run +# release.sh manually) instead of burning hours on blind retries. # # ./release-watch.sh # check once and act # # Honors $S3CFG (passed through to release.sh) and these optional env vars: # STATE_DIR where to keep the lock + failure marker (default /var/lib/wrolpi-release) -# MAX_RETRIES how many times to auto-retry a failing tag (default 3) +# MAX_RETRIES build attempts before giving up on a tag (default 1 = no auto-retry) set -euo pipefail @@ -25,7 +26,7 @@ SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) REPO="lrnselfreliance/wrolpi" CDN_BASE="https://wrolpi.nyc3.cdn.digitaloceanspaces.com" STATE_DIR="${STATE_DIR:-/var/lib/wrolpi-release}" -MAX_RETRIES="${MAX_RETRIES:-3}" +MAX_RETRIES="${MAX_RETRIES:-1}" LOCK="${STATE_DIR}/build.lock" FAIL_STATE="${STATE_DIR}/failed-tag"