release-watch: don't auto-retry failed builds (default MAX_RETRIES=1)
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Controller Tests / controller-tests (push) Waiting to run
Jest Tests / jest-tests (push) Waiting to run

Builds take hours and a failure almost always needs a code fix, not a blind
retry -- retrying 3x burned ~5h overnight. Default to a single attempt: on
failure, record the marker, fire the notify hook, and wait for a human to
investigate and clear it. The MAX_RETRIES override remains for N attempts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Roland 2026-05-31 12:21:41 -06:00
parent 5625f17b5b
commit cd9fb4c6de
No known key found for this signature in database
GPG key ID: 1BA620CE6C9A4680

View file

@ -8,15 +8,16 @@
# release.sh to build and publish it. latest.json is the source of truth for
# "already built" -- there is no local bookkeeping for success.
#
# A local state file is used only to throttle retries of a release that fails
# to build, so a permanently-broken release does not rebuild every hour
# forever.
# A local state file records a tag that failed to build. Builds are expensive
# (hours), so by default a failed tag is attempted once and then NOT auto-retried
# -- the notify hook alerts you, and you investigate and clear the marker (or run
# release.sh manually) instead of burning hours on blind retries.
#
# ./release-watch.sh # check once and act
#
# Honors $S3CFG (passed through to release.sh) and these optional env vars:
# STATE_DIR where to keep the lock + failure marker (default /var/lib/wrolpi-release)
# MAX_RETRIES how many times to auto-retry a failing tag (default 3)
# MAX_RETRIES build attempts before giving up on a tag (default 1 = no auto-retry)
set -euo pipefail
@ -25,7 +26,7 @@ SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
REPO="lrnselfreliance/wrolpi"
CDN_BASE="https://wrolpi.nyc3.cdn.digitaloceanspaces.com"
STATE_DIR="${STATE_DIR:-/var/lib/wrolpi-release}"
MAX_RETRIES="${MAX_RETRIES:-3}"
MAX_RETRIES="${MAX_RETRIES:-1}"
LOCK="${STATE_DIR}/build.lock"
FAIL_STATE="${STATE_DIR}/failed-tag"