feat✨: check the shutdown budget against every stop deadline

The budget is one number in config/settings.yml. The deadlines that have to
cover it are in four other files, none of which anybody edits while thinking
about shutdown - so raising the budget passes every test, deploys, and has the
cleanup callbacks killed on the next release.

Two checks share one arithmetic and one five-second margin.

shutdown-budget-overruns-grace compares preStop + drain + server + cleanup
against terminationGracePeriodSeconds in the shipped manifest. Those two files
are not merely adjacent examples: scripts/k8s/prerun.sh builds the
settings-admin ConfigMap out of config/settings.yml and the Deployment mounts
it, so the manifest deploys that file.

docker-stop-cuts-shutdown-short covers the three ways this container is
stopped: `docker stop` in the release workflow, the same in the Makefile, and
stop_grace_period on a compose service that runs this repository's own image. A
service running a database is not this process and is left alone. The duration
is parsed rather than scanned for digits - compose accepts 1m30s, and reading
the first number out of it would call ninety seconds one.

All three spellings of the deadline are read: --timeout, the deprecated --time,
and the short -t. A deadline the check cannot read is reported as no deadline at
all, so recognising only one of them would call a correct command broken and
send whoever fixed it towards the spelling docker is retiring. The message
quotes the flag back in the spelling it was written in, for the same reason:
suggesting a flag the line does not use is how a tool teaches people to
disbelieve it.

What neither covers is `docker rm -f`, which has no deadline to compare
against: it is SIGKILL by definition. That gap is deliberate, and it is why the
previous commit changed the one place that used it on a container that might
still be running.

Both report at two levels. A budget that already overruns is an ERROR; one that
fits with nothing to spare is a WARN, because it works today and failing the
build on a working configuration is how a project teaches people to ignore its
warnings. The two are exclusive: an overrun satisfies the headroom condition as
well, and an ERROR that always drags a duplicate WARN behind it teaches the same
lesson.

preStop is in the sum although the shipped manifest has no hook. That is the
point - a hook added later is spent before the process is told anything, and a
self-check that could not see it would understate the real budget by however
long somebody set it to, which is worse than not checking. A hook whose duration
cannot be read is reported rather than counted as zero.

The fallbacks for fields the settings file leaves out are read from the
constants in the scanned tree, not copied here; if they are renamed the run
stops instead of going quiet with the wrong numbers.

The wording differs by audience on purpose. At run time this is somebody else's
deployment under constraints the process cannot see, so the log states a
minimum. These checks read files this repository owns, where there is standing
to ask for headroom, so they name a target.

The table in AGENTS.md is relisted while it is being touched: the two new
checks, plus datascope-route-unguarded, which has been missing since it was
added. The hard-coded count is gone - it said seven and there were ten, which is
what a written-down count does. AGENTS.md and docs/contract.md both sent readers
to `go run ./tools/checksilent -h` for the list of checks; that prints
command-line flags and has never printed a check, so both now point at
runChecks.

The yaml parser moves from an indirect requirement to a direct one - it was
already in the module graph - and tidy drops four go.sum lines left over from
two older releases of core.

.github/workflows/build.yml

@@ -115,7 +115,14 @@ jobs:
             if sudo docker ps -a --format '{{.Names}}' | grep -qx "$NAME"; then
               sudo docker rm -f "$PREV" >/dev/null 2>&1 || true
               sudo docker rename "$NAME" "$PREV"
-              sudo docker stop "$PREV" >/dev/null
+              # --timeout, because the default is 10 seconds and the process
+              # spends drain + server + cleanup from extend.shutdown before it
+              # exits - 8 seconds out of the box, and more for anyone who
+              # configures a drain window. Past the deadline docker sends
+              # SIGKILL and the cleanup callbacks are cut off part-way through.
+              # checksilent's docker-stop-cuts-shutdown-short check compares
+              # this number against config/settings.yml.
+              sudo docker stop --timeout 30 "$PREV" >/dev/null
             fi
 
             sudo docker run -d -p 8000:8000 \

.github/workflows/go.yml

@@ -15,6 +15,27 @@ jobs:
   build:
     name: Build
     runs-on: ubuntu-latest
+
+    # The queue's ordering rule - consumers registered before the queue is
+    # started - is invisible on the memory backend, which is the default and
+    # therefore what every other test runs on: queue.Memory's Register starts a
+    # consumer goroutine whatever the state. Only redis refuses a late
+    # registration, so without a server here the tests that cover it would skip
+    # and the suite would report success for a queue that accepts no consumers.
+    services:
+      redis:
+        image: redis:7-alpine
+        ports:
+          - 6379:6379
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 5s
+          --health-timeout 3s
+          --health-retries 10
+
+    env:
+      GO_ADMIN_TEST_REDIS_ADDR: 127.0.0.1:6379
+
     steps:
 
     - name: Set up Go 1.26

AGENTS.md

@@ -227,8 +227,9 @@ go run -tags sqlite3 . server  -c config/settings.sqlite.yml
 
 ## 静默失败校验
 
-`make checksilent` 检查七类**不报错、不记日志、行为悄悄变得不对**的问题，
-CI 会跑，命中 ERROR 即失败：
+`make checksilent` 逐条检查那些**不报错、不记日志、行为悄悄变得不对**的问题，
+CI 会跑，命中 ERROR 即失败。这里不写条数——写死的数字会悄悄过时，
+真正的清单是 `tools/checksilent/checks.go` 里 `runChecks` 跑的那几个：
 
 | 检查 | 级别 | 静默后果 |
 |---|---|---|
@@ -238,8 +239,15 @@ CI 会跑，命中 ERROR 即失败：
 | `menu-id-collision` | ERROR | 两个模块硬编码同一菜单 ID，互相覆盖 |
 | `contract-import-boundary` | ERROR | 契约包 import `app/`，应用无法独立编译 |
 | `contract-shim-alias` | ERROR | 契约薄壳写成 defined type 而非别名，方法集丢失，本仓可能照常编译、第三方应用编译不过 |
+| `datascope-route-unguarded` | ERROR | handler 读调用方的数据权限，而注册它的路由组没装提供权限的中间件。取不到时拿到零值、走 fail-closed 分支，查询被塞进 `1 = 0`：接口对确实存在的行返回「查不到」，且只在 `enabledp: true` 的部署上出现 |
+| `shutdown-budget-overruns-grace` | ERROR / WARN | `settings.yml` 的 `extend.shutdown` 预算（含清单里的 `preStop`）放不进自带 k8s 清单的 `terminationGracePeriodSeconds`，SIGKILL 在清理回调跑到一半时到达 |
+| `docker-stop-cuts-shutdown-short` | ERROR / WARN | 停止容器的两条路径——脚本/工作流里的 `docker stop`，和 `docker-compose.yml` 的 `stop_grace_period`——没写或写得不够关闭预算用。两边默认都是 10 秒，而这个数字离命令很远，调大预算的人不会想起它 |
 | `menu-name-mismatch` | WARN | 菜单名与前端组件 `name` 不一致，keep-alive 缓存静默失效 |
 
+两条关闭预算检查分两级，用的是同一条算术和同一个 5 秒边际：真的超限报 ERROR，
+放得进但余量不足 5 秒报 WARN。余量不足做 WARN 不做 ERROR，是因为那是个技术上
+跑得通的配置——**一条在正确配置下也会响的 ERROR，训练的是忽略它**。
+
 最后一条要跨仓库比对，只能做正则启发式，因此是 WARN，**不影响退出码**，
 且默认跳过；要跑它得指定前端目录：
 

Makefile

@@ -15,7 +15,16 @@ build-sqlite:
 # make run
 run:
     # delete go-admin-api container
-	@if [ $(shell docker ps -aq --filter name=go-admin --filter publish=8000) ]; then docker rm -f go-admin; fi
+    #
+    # stop then rm, rather than `rm -f`. The force flag kills a running
+    # container with SIGKILL and no grace at all, so restarting locally cut
+    # short every shutdown this application does - the drain window was never
+    # once reached on a developer's machine. --timeout has to cover
+    # extend.shutdown's drain + server + cleanup; checksilent's
+    # docker-stop-cuts-shutdown-short check compares it against
+    # config/settings.yml. On a container that has already stopped, stop is a
+    # no-op and the removal is unchanged.
+	@if [ $(shell docker ps -aq --filter name=go-admin --filter publish=8000) ]; then docker stop --timeout 30 go-admin && docker rm go-admin; fi
 
     # 启动方法一 run go-admin-api container  docker-compose 启动方式
     # 进入到项目根目录 执行 make run 命令

app/jobs/jobbase.go

@@ -1,6 +1,7 @@
 package jobs
 
 import (
+	"context"
 	"fmt"
 	log "github.com/go-admin-team/go-admin-core/v2/logger"
 	"github.com/go-admin-team/go-admin-core/v2/sdk"
@@ -145,11 +146,36 @@ func setup(key string, db *gorm.DB) {
 	}
 
 	// 其中任务
-	crontab.Start()
+	startCrontab(crontab)
+}
+
+// startCrontab starts c and arranges for it to be stopped on the way out.
+//
+// The stop used to be `defer crontab.Stop()` followed by `select {}`. The
+// select never returned, so the defer never ran and the scheduler was never
+// stopped; and because setup never returned, the loop in Setup never reached
+// the second tenant - only whichever database came first out of the map ever
+// got a scheduler at all. cron.Start is itself `go c.run()`, so the select was
+// blocking for nothing.
+//
+// cron.Stop returns a context that closes once the jobs already running have
+// finished. That is the wait the shutdown budget exists to bound: giving up on
+// it leaves those jobs running until the process exits, which is better than
+// holding the whole shutdown open for one job that will not end.
+func startCrontab(c *cron.Cron) {
+	c.Start()
 	fmt.Println(time.Now().Format(timeFormat), " [INFO] JobCore start success.")
+
 	// 关闭任务
-	defer crontab.Stop()
-	select {}
+	sdk.Runtime.SetShutdown(func(ctx context.Context) {
+		stopped := c.Stop()
+		select {
+		case <-stopped.Done():
+			fmt.Println(time.Now().Format(timeFormat), " [INFO] JobCore stopped.")
+		case <-ctx.Done():
+			fmt.Println(time.Now().Format(timeFormat), " [WARN] JobCore stop gave up waiting for running jobs")
+		}
+	})
 }
 
 // AddJob 添加任务 AddJob(invokeTarget string, jobId int, jobName string, cronExpression string)

app/jobs/jobbase_test.go

@@ -0,0 +1,52 @@
+package jobs
+
+import (
+	"context"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/go-admin-team/go-admin-core/v2/sdk"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/pkg/cronjob"
+)
+
+// The scheduler had never been stopped. `defer crontab.Stop()` sat directly
+// above a `select {}` that never returned, so the deferred call was
+// unreachable for the life of the process.
+//
+// There is one test rather than several because BeforeExit closes to further
+// registration once it has run: a second RunShutdown in this binary would find
+// an empty registry and pass while proving nothing.
+func TestTheSchedulerIsStoppedOnTheWayOut(t *testing.T) {
+	var ticks atomic.Int64
+
+	c := cronjob.NewWithSeconds()
+	if _, err := c.AddFunc("* * * * * *", func() { ticks.Add(1) }); err != nil {
+		t.Fatalf("AddFunc: %v", err)
+	}
+
+	startCrontab(c)
+
+	// It has to be running before stopping it can mean anything.
+	deadline := time.Now().Add(5 * time.Second)
+	for ticks.Load() == 0 && time.Now().Before(deadline) {
+		time.Sleep(20 * time.Millisecond)
+	}
+	if ticks.Load() == 0 {
+		t.Fatal("the scheduler never ran the job, so this test cannot show it was stopped")
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
+	defer cancel()
+	if err := sdk.Runtime.RunShutdown(ctx); err != nil {
+		t.Fatalf("RunShutdown: %v", err)
+	}
+
+	// Two and a half seconds is two more firings of a job that runs every
+	// second, so silence here is the assertion.
+	at := ticks.Load()
+	time.Sleep(2500 * time.Millisecond)
+	if n := ticks.Load() - at; n > 0 {
+		t.Errorf("the job fired %d more times after shutdown: the scheduler is still running", n)
+	}
+}

app/other/router/monitor.go

@@ -1,23 +1,82 @@
 package router
 
 import (
+	"context"
 	"net/http"
+	"time"
 
 	"github.com/gin-gonic/gin"
 	"github.com/go-admin-team/go-admin-core/v2/tools/transfer"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
+
+	"go-admin/common/health"
 )
 
 func init() {
-	routerNoCheckRole = append(routerNoCheckRole, registerMonitorRouter)
+	routerNoCheckRole = append(routerNoCheckRole, RegisterMonitorRouter)
 }
 
-// 需认证的路由代码
-func registerMonitorRouter(v1 *gin.RouterGroup) {
+// readyTimeout bounds the whole probe. What constrains it is the orchestrator's
+// per-check timeout rather than its polling period: Kubernetes allows a probe
+// one second by default, so a dependency that answers in 1.2s is recorded as a
+// failed check however promptly this handler returns. A manifest that mounts
+// this probe has to raise timeoutSeconds above this value, and
+// scripts/k8s/deploy.yml does.
+const readyTimeout = 2 * time.Second
+
+// HealthPath and ReadyPath are the two probe routes, relative to APIPrefix.
+//
+// Exported for the same reason as the prefix: the rate limiter has to be told
+// to skip them, and it is installed in a package that cannot import this one.
+const (
+	HealthPath = "/health"
+	ReadyPath  = "/ready"
+)
+
+// RegisterMonitorRouter mounts the metrics endpoint and the two probes on v1.
+//
+// Exported so that a test can put the real probes on a server of its own. The
+// alternative - a test that re-implements the handler it means to check - is
+// how a probe comes to be asserted against a copy of itself.
+//
+// 无需认证的路由代码
+func RegisterMonitorRouter(v1 *gin.RouterGroup) {
 	v1.GET("/metrics", transfer.Handler(promhttp.Handler()))
-	//健康检查
-	v1.GET("/health", func(c *gin.Context) {
+
+	// 健康检查（存活）
+	//
+	// Stays a bare 200 on purpose. This is the answer to "should I restart
+	// you", and a process whose database is unreachable does not want
+	// restarting - that turns one outage into a crash loop and throws away the
+	// connection pool, the cache and every in-flight request along the way.
+	v1.GET(HealthPath, func(c *gin.Context) {
 		c.Status(http.StatusOK)
 	})
 
-}
+	// 就绪检查
+	//
+	// The answer to "should I send you requests". It fails while a dependency
+	// is unreachable, and from the moment shutdown begins - for as long as
+	// extend.shutdown.drain says, which is zero unless it is configured. The
+	// package comment in common/health says what that window is worth, and to
+	// whom.
+	v1.GET(ReadyPath, func(c *gin.Context) {
+		if health.Draining() {
+			c.JSON(http.StatusServiceUnavailable, gin.H{
+				"status": "draining",
+				"checks": []health.Check{},
+			})
+			return
+		}
+		ctx, cancel := context.WithTimeout(c.Request.Context(), readyTimeout)
+		defer cancel()
+
+		checks := health.Ready(ctx)
+		status := http.StatusOK
+		if !health.Healthy(checks) {
+			status = http.StatusServiceUnavailable
+		}
+		c.JSON(status, gin.H{"status": http.StatusText(status), "checks": checks})
+	})
+
+}

app/other/router/router.go

@@ -10,6 +10,13 @@ var (
 	routerCheckRole   = make([]func(v1 *gin.RouterGroup, authMiddleware *jwt.GinJWTMiddleware), 0)
 )
 
+// APIPrefix is the group every route below is registered under.
+//
+// Exported because the middleware chain in cmd/api has to name two of those
+// routes in full - the rate limiter is installed on the engine and must skip
+// the probes - and a prefix spelled in two places is a prefix that drifts.
+const APIPrefix = "/api/v1"
+
 // initRouter 路由示例
 func initRouter(r *gin.Engine, authMiddleware *jwt.GinJWTMiddleware) *gin.Engine {
 
@@ -24,7 +31,7 @@ func initRouter(r *gin.Engine, authMiddleware *jwt.GinJWTMiddleware) *gin.Engine
 // noCheckRoleRouter 无需认证的路由示例
 func noCheckRoleRouter(r *gin.Engine) {
 	// 可根据业务需求来设置接口版本
-	v1 := r.Group("/api/v1")
+	v1 := r.Group(APIPrefix)
 
 	for _, f := range routerNoCheckRole {
 		f(v1)
@@ -34,7 +41,7 @@ func noCheckRoleRouter(r *gin.Engine) {
 // checkRoleRouter 需要认证的路由示例
 func checkRoleRouter(r *gin.Engine, authMiddleware *jwt.GinJWTMiddleware) {
 	// 可根据业务需求来设置接口版本
-	v1 := r.Group("/api/v1")
+	v1 := r.Group(APIPrefix)
 
 	for _, f := range routerCheckRole {
 		f(v1, authMiddleware)

cmd/api/phase_test.go

@@ -0,0 +1,160 @@
+package api
+
+import (
+	"fmt"
+	"net"
+	"net/http"
+	"testing"
+	"time"
+
+	"github.com/go-admin-team/go-admin-core/v2/sdk"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/config"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/runtime"
+)
+
+// freePort returns a port nothing is listening on. It is inherently a guess -
+// the port is free when it is handed back and could be taken a moment later -
+// but every alternative needs the caller to hold the listener, which is the one
+// thing these tests cannot do.
+func freePort(t *testing.T) int {
+	t.Helper()
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatalf("probe listen: %v", err)
+	}
+	port := ln.Addr().(*net.TCPAddr).Port
+	_ = ln.Close()
+	return port
+}
+
+// AfterListen promises a hook that the port is reachable. Both halves of that
+// are asserted here, and in one test rather than two, because the phase seals
+// itself once it has run: a second test calling RunPhase again would find a
+// closed registry and pass while proving nothing.
+//
+// The failing bind comes first for the same reason. It must leave the phase
+// unsealed, which is only visible if nothing has sealed it yet.
+func TestAfterListenIsAnnouncedOnlyOnceThePortIsBound(t *testing.T) {
+	// The pause makes the "announced synchronously" claim testable: if the
+	// announcement were moved onto a goroutine, startServing would return
+	// while the hook was still sleeping and the count below would be zero.
+	var ran int
+	sdk.Runtime.SetPhase(runtime.AfterListen, func() {
+		time.Sleep(50 * time.Millisecond)
+		ran++
+	})
+
+	// Somebody else already has the port. Under ListenAndServe this surfaced
+	// on the serving goroutine, far too late to stop the announcement.
+	taken, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatalf("occupy: %v", err)
+	}
+	defer func() { _ = taken.Close() }()
+
+	blocked := &http.Server{Addr: taken.Addr().String(), Handler: http.NewServeMux()}
+	if err := startServing(blocked, false, "", ""); err == nil {
+		t.Fatal("startServing returned no error for a port that was already taken")
+	}
+	if ran != 0 {
+		t.Errorf("AfterListen ran %d times after a failed bind; a hook there is told the port is reachable", ran)
+	}
+	if sdk.Runtime.PhaseSealed(runtime.AfterListen) {
+		t.Error("a failed bind sealed AfterListen, so the phase could never run for a server that did start")
+	}
+
+	// A certificate that cannot be read is the other way to fail before there
+	// is anything to announce. ServeTLS reads it on the serving goroutine, so
+	// without the check in startServing this would be a hook told the port was
+	// reachable while the server was already on its way down.
+	if err := startServing(&http.Server{Addr: "127.0.0.1:0"}, true, "no-such.pem", "no-such.key"); err == nil {
+		t.Fatal("startServing returned no error for a certificate that does not exist")
+	}
+	if ran != 0 {
+		t.Errorf("AfterListen ran %d times after a certificate failure", ran)
+	}
+	if sdk.Runtime.PhaseSealed(runtime.AfterListen) {
+		t.Error("a certificate failure sealed AfterListen")
+	}
+
+	// And now a bind that works.
+	port := freePort(t)
+	srv := &http.Server{Addr: fmt.Sprintf("127.0.0.1:%d", port), Handler: http.NewServeMux()}
+	if err := startServing(srv, false, "", ""); err != nil {
+		t.Fatalf("startServing on a free port: %v", err)
+	}
+	defer func() { _ = srv.Close() }()
+
+	// Checked the instant startServing returns, so this is also the assertion
+	// that it did not return early: an asynchronous announcement would still
+	// be inside the sleep. Synchrony matters because an announcement that
+	// overlaps the wait below could, on a fast SIGTERM, have the shutdown
+	// callbacks finish before the startup ones.
+	if ran != 1 {
+		t.Fatalf("AfterListen ran %d times, want 1", ran)
+	}
+
+	// The claim is not "Serve was called" but "the port answers". Dial it.
+	c, err := net.DialTimeout("tcp", srv.Addr, 5*time.Second)
+	if err != nil {
+		t.Fatalf("AfterListen ran but the port does not answer: %v", err)
+	}
+	_ = c.Close()
+}
+
+// BeforeRouter is the last point at which a module can still affect how routes
+// are built, so it has to run while there is no engine yet. The before registry
+// is a different moment despite the name: those callbacks run after initRouter
+// has built the engine.
+//
+// The two are two lines apart in buildRouter, and calling them equivalent is a
+// mistake this repository has already made in writing. Until this test the
+// ordering was checked by reading - which is how the stop signals came to be
+// armed after the readiness banner in the same file.
+func TestBeforeRouterRunsWhileThereIsNoEngine(t *testing.T) {
+	freshRuntime(t)
+
+	// AuthInit reads these two package-level values and nothing else. No
+	// database is involved in building a router: the handlers are registered,
+	// not called.
+	config.ApplicationConfig.Mode = "dev"
+	config.JwtConfig.Secret = "test-secret-for-the-router-build"
+
+	type observation struct {
+		ran        int
+		engineWas  interface{}
+		engineSeen bool
+	}
+	var phase, before observation
+
+	sdk.Runtime.SetPhase(runtime.BeforeRouter, func() {
+		phase.ran++
+		phase.engineWas = sdk.Runtime.GetEngine()
+		phase.engineSeen = true
+	})
+	sdk.Runtime.SetBefore(func() {
+		before.ran++
+		before.engineWas = sdk.Runtime.GetEngine()
+		before.engineSeen = true
+	})
+
+	buildRouter()
+
+	if phase.ran != 1 {
+		t.Fatalf("BeforeRouter ran %d times, want 1", phase.ran)
+	}
+	if !phase.engineSeen || phase.engineWas != nil {
+		t.Errorf("BeforeRouter saw engine %v, want nil: it is meant to run before initRouter builds one", phase.engineWas)
+	}
+
+	if before.ran != 1 {
+		t.Fatalf("the before registry ran %d times, want 1", before.ran)
+	}
+	if before.engineWas == nil {
+		t.Error("a before callback saw no engine; that registry is meant to run after initRouter, and describing it as equivalent to BeforeRouter is the error this asserts against")
+	}
+
+	if sdk.Runtime.GetEngine() == nil {
+		t.Error("buildRouter returned with no engine built")
+	}
+}

cmd/api/queue_test.go

@@ -0,0 +1,167 @@
+package api
+
+import (
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	corestorage "github.com/go-admin-team/go-admin-core/v2/storage"
+)
+
+// recordingQueue records what was done to it, in order. Register and Run are
+// the two calls whose order is the point of this file; Append and Shutdown are
+// here to satisfy the interface.
+type recordingQueue struct {
+	mu     sync.Mutex
+	events []string
+	ran    chan struct{}
+}
+
+func newRecordingQueue() *recordingQueue {
+	return &recordingQueue{ran: make(chan struct{}, 4)}
+}
+
+func (q *recordingQueue) record(e string) {
+	q.mu.Lock()
+	q.events = append(q.events, e)
+	q.mu.Unlock()
+}
+
+func (q *recordingQueue) seen() []string {
+	q.mu.Lock()
+	defer q.mu.Unlock()
+	return append([]string(nil), q.events...)
+}
+
+func (q *recordingQueue) String() string                    { return "recording" }
+func (q *recordingQueue) Append(corestorage.Messager) error { return nil }
+func (q *recordingQueue) Register(name string, _ corestorage.ConsumerFunc) {
+	q.record("register:" + name)
+}
+func (q *recordingQueue) Shutdown() {}
+
+func (q *recordingQueue) Run() {
+	q.record("run")
+	select {
+	case q.ran <- struct{}{}:
+	default:
+	}
+}
+
+// waitForRun waits for Run, which is started on a goroutine.
+func (q *recordingQueue) waitForRun(t *testing.T) {
+	t.Helper()
+	select {
+	case <-q.ran:
+	case <-time.After(5 * time.Second):
+		t.Fatalf("Run was never called; saw: %v", q.seen())
+	}
+}
+
+// The consumers must be registered before the queue is started. A queue that
+// is already running refuses further registration - the contract
+// implementations answer storage.ErrQueueAlreadyStarted - and the legacy
+// adapter this path goes through drops that error, so the wrong order loses
+// consumers with nothing said about it. The memory backend does not care,
+// which is exactly why this cannot be left to be noticed in use.
+func TestConsumersAreRegisteredBeforeTheQueueIsStarted(t *testing.T) {
+	attachedQueue.Store(0)
+	t.Cleanup(func() { attachedQueue.Store(0) })
+
+	q := newRecordingQueue()
+	attachConsumersOnce(1, q)
+	q.waitForRun(t)
+
+	seen := q.seen()
+	runAt := -1
+	registers := 0
+	for i, e := range seen {
+		switch {
+		case e == "run":
+			if runAt < 0 {
+				runAt = i
+			}
+		case strings.HasPrefix(e, "register:"):
+			registers++
+			if runAt >= 0 {
+				t.Errorf("%q came after Run; a running queue refuses registration", e)
+			}
+		}
+	}
+	if registers != 3 {
+		t.Errorf("registered %d consumers, want 3; saw %v", registers, seen)
+	}
+	if runAt < 0 {
+		t.Errorf("the queue was never started; saw %v", seen)
+	}
+}
+
+// AfterResource runs again on every configuration reload, so the hook has to
+// be idempotent with respect to a given queue - not "does nothing the second
+// time". Registering twice on the same queue would give every message two
+// consumers and write every log row twice.
+func TestTheSameQueueIsNotGivenConsumersTwice(t *testing.T) {
+	attachedQueue.Store(0)
+	t.Cleanup(func() { attachedQueue.Store(0) })
+
+	q := newRecordingQueue()
+	attachConsumersOnce(1, q)
+	q.waitForRun(t)
+	attachConsumersOnce(1, q)
+
+	// Nothing to wait for on the second call, so give a wrong implementation
+	// the time it would need to show up.
+	time.Sleep(200 * time.Millisecond)
+	if n := len(q.seen()); n != 4 {
+		t.Errorf("%d calls after attaching twice to the same queue, want 4 (3 registers + 1 run); saw %v", n, q.seen())
+	}
+}
+
+// The other half of the same rule: a reload builds a new adapter, and the
+// consumers on the old one are attached to a queue nobody publishes to any
+// more. A new generation must get its own set.
+func TestANewQueueGetsItsOwnConsumers(t *testing.T) {
+	attachedQueue.Store(0)
+	t.Cleanup(func() { attachedQueue.Store(0) })
+
+	first := newRecordingQueue()
+	attachConsumersOnce(1, first)
+	first.waitForRun(t)
+
+	second := newRecordingQueue()
+	attachConsumersOnce(2, second)
+	second.waitForRun(t)
+
+	if n := len(second.seen()); n != 4 {
+		t.Errorf("the queue from the second generation saw %d calls, want 4; saw %v", n, second.seen())
+	}
+	if n := len(first.seen()); n != 4 {
+		t.Errorf("the queue from the first generation saw %d calls, want 4 - it should not have been touched again; saw %v", n, first.seen())
+	}
+}
+
+// Generation 0 means the configuration has no queue section at all, so nothing
+// was installed and the runtime hands back its own memory queue. That case
+// still has to get consumers - the registration it replaces was unconditional,
+// and dropping it would stop the login and operation logs for anyone who
+// commented the section out.
+func TestAnUnconfiguredQueueStillGetsConsumers(t *testing.T) {
+	attachedQueue.Store(0)
+	t.Cleanup(func() { attachedQueue.Store(0) })
+
+	q := newRecordingQueue()
+	attachConsumersOnce(0, q)
+	q.waitForRun(t)
+
+	if n := len(q.seen()); n != 4 {
+		t.Errorf("an unconfigured queue saw %d calls, want 4; saw %v", n, q.seen())
+	}
+
+	// And still only once.
+	attachConsumersOnce(0, q)
+	time.Sleep(200 * time.Millisecond)
+	if n := len(q.seen()); n != 4 {
+		t.Errorf("generation 0 was attached to twice: %d calls, want 4; saw %v", n, q.seen())
+	}
+}

cmd/api/server.go

@@ -2,10 +2,13 @@ package api
 
 import (
 	"context"
+	"crypto/tls"
 	"fmt"
+	"net"
 	"net/http"
 	"os"
 	"os/signal"
+	"sync/atomic"
 	"syscall"
 	"time"
 
@@ -14,16 +17,21 @@ import (
 	log "github.com/go-admin-team/go-admin-core/v2/logger"
 	"github.com/go-admin-team/go-admin-core/v2/sdk"
 	"github.com/go-admin-team/go-admin-core/v2/sdk/api"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/bootstrap"
 	"github.com/go-admin-team/go-admin-core/v2/sdk/config"
 	"github.com/go-admin-team/go-admin-core/v2/sdk/pkg"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/runtime"
+	corestorage "github.com/go-admin-team/go-admin-core/v2/storage"
 	"github.com/pkg/errors"
 	"github.com/spf13/cobra"
 
 	"go-admin/app/admin/models"
 	"go-admin/app/admin/router"
 	"go-admin/app/jobs"
+	otherrouter "go-admin/app/other/router"
 	"go-admin/common/database"
 	"go-admin/common/global"
+	"go-admin/common/health"
 	common "go-admin/common/middleware"
 	"go-admin/common/middleware/handler"
 	"go-admin/common/storage"
@@ -60,30 +68,115 @@ func init() {
 func setup() {
 	// 注入配置扩展项
 	config.ExtendConfig = &ext.ExtConfig
+
+	// Registered before the configuration is read. SetupConfig announces
+	// AfterResource as soon as the callbacks that build the resources have
+	// run, so a hook added after that call would miss the first round and the
+	// queue would have no consumers until somebody edited the config file.
+	sdk.Runtime.SetPhase(runtime.AfterResource, attachQueueConsumers)
+
+	// On AfterListen rather than on a bare goroutine from run(). Two reasons:
+	// the phase runs behind core's panic guard, which does not reach across a
+	// goroutine boundary - a panic while loading jobs used to take the whole
+	// process down with a stack that named this file - and the jobs it starts
+	// can call the API, which is only true once the socket is accepting.
+	sdk.Runtime.SetPhase(runtime.AfterListen, startCronJobs)
+
 	//1. 读取配置
-	config.Setup(
+	bootstrap.SetupConfig(
 		file.NewSource(file.WithPath(configYml)),
 		database.Setup,
 		storage.Setup,
 	)
-	//注册监听函数
-	queue := sdk.Runtime.GetQueuePrefix("")
-	queue.Register(global.LoginLog, models.SaveLoginLog)
-	queue.Register(global.OperateLog, models.SaveOperaLog)
-	queue.Register(global.ApiCheck, models.SaveSysApi)
-	go queue.Run()
 
 	usageStr := `starting api server...`
 	log.Info(usageStr)
 }
 
+// startCronJobs registers the job implementations and starts a scheduler for
+// every tenant database.
+//
+// It is synchronous, like the phase that runs it. jobs.Setup returns now that
+// the `select {}` at the end of its per-tenant setup is gone, which is what
+// makes that possible; while it was there this could only be a goroutine, and
+// a goroutine is outside the panic guard.
+func startCronJobs() {
+	jobs.InitJob()
+	jobs.Setup(sdk.Runtime.GetAllDb())
+}
+
+// attachedQueue is the queue generation the consumers are attached to, plus
+// one, so that the zero value means "attached to nothing yet". Written from
+// the goroutine running the phase, read from the next one - rounds never
+// overlap, but they are not the same goroutine.
+var attachedQueue atomic.Uint64
+
+// attachQueueConsumers registers the log consumers against the queue that is
+// current, and starts it.
+//
+// It runs on AfterResource, so it runs again after every configuration reload
+// - and it has to. A reload rebuilds the queue adapter, and consumers
+// registered against the one that existed at start-up are attached to an
+// adapter nobody publishes to any more, so the login and operation logs stop
+// being written with nothing said about it.
+//
+// It is therefore idempotent with respect to a given queue rather than "does
+// nothing the second time": a new adapter gets a fresh set of consumers, the
+// same one gets none. Registering twice on the same queue would give every
+// message two consumers and write every log row twice.
+//
+// Generation 0 means the configuration has no queue section, so nothing was
+// installed and GetQueuePrefix hands back the runtime's own memory queue.
+// That case still gets consumers - it is what the previous unconditional
+// registration did, and dropping it would silently stop logging for anyone who
+// commented the section out - it just never gets them twice.
+func attachQueueConsumers() {
+	attachConsumersOnce(storage.QueueGeneration(), sdk.Runtime.GetQueuePrefix(""))
+}
+
+// attachConsumersOnce puts the log consumers on q and starts it, unless gen
+// says this queue already has them.
+//
+// Split out from attachQueueConsumers so that the order and the once-ness can
+// be checked against a queue the test controls: the sequence that matters here
+// cannot be read back out of a real adapter.
+func attachConsumersOnce(gen uint64, q corestorage.AdapterQueue) {
+	if attachedQueue.Load() == gen+1 {
+		return
+	}
+	attachedQueue.Store(gen + 1)
+
+	//注册监听函数
+	q.Register(global.LoginLog, models.SaveLoginLog)
+	q.Register(global.OperateLog, models.SaveOperaLog)
+	q.Register(global.ApiCheck, models.SaveSysApi)
+
+	// Started only now, and by whoever registered. setupQueue deliberately
+	// leaves it stopped: a queue that is already running refuses further
+	// registration, and the adapter in this path drops that error on the
+	// floor, so starting first loses consumers without a word.
+	go q.Run()
+}
+
 func run() error {
+	// Resolved first, and used both for the line it prints and for the
+	// shutdown that spends it. Reading the configuration again at signal time
+	// would let the two disagree, and the sum that gets printed is the whole
+	// point of printing it.
+	//
+	// Refused rather than corrected, and refused before anything is built: a
+	// budget that cannot be spent as written is a configuration error, and the
+	// moment to say so is while nothing depends on this process yet.
+	seconds, err := ext.ExtConfig.Shutdown.Budget()
+	if err != nil {
+		return err
+	}
+	reportShutdownBudget(seconds)
+
 	if config.ApplicationConfig.Mode == pkg.ModeProd.String() {
 		gin.SetMode(gin.ReleaseMode)
 	}
-	initRouter()
-
-	runStartupHooks()
+	buildRouter()
 
 	srv := &http.Server{
 		Addr:         fmt.Sprintf("%s:%d", config.ApplicationConfig.Host, config.ApplicationConfig.Port),
@@ -92,12 +185,6 @@ func run() error {
 		WriteTimeout: time.Duration(config.ApplicationConfig.WriterTimeout) * time.Second,
 	}
 
-	go func() {
-		jobs.InitJob()
-		jobs.Setup(sdk.Runtime.GetAllDb())
-
-	}()
-
 	if apiCheck {
 		var routers = sdk.Runtime.GetRouter()
 		q := sdk.Runtime.GetQueuePrefix("")
@@ -122,18 +209,10 @@ func run() error {
 	// arming is separate from waiting.
 	quit, disarmStopSignals := armStopSignals()
 
-	go func() {
-		// 服务连接
-		if config.SslConfig.Enable {
-			if err := srv.ListenAndServeTLS(config.SslConfig.Pem, config.SslConfig.KeyStr); err != nil && !errors.Is(err, http.ErrServerClosed) {
-				log.Fatal("listen: ", err)
-			}
-		} else {
-			if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
-				log.Fatal("listen: ", err)
-			}
-		}
-	}()
+	if err := startServing(srv, config.SslConfig.Enable, config.SslConfig.Pem, config.SslConfig.KeyStr); err != nil {
+		return err
+	}
+
 	fmt.Println(pkg.Red(string(global.LogoContent)))
 	tip()
 	fmt.Println(pkg.Green("Server run at:"))
@@ -145,26 +224,187 @@ func run() error {
 	fmt.Printf("%s Enter Control + C Shutdown Server \r\n", pkg.GetCurrentTimeStr())
 
 	<-quit
-	// Restored here, not deferred: from this point a second signal must reach
-	// the default handler, so a shutdown that hangs can still be interrupted.
-	disarmStopSignals()
-
-	log.Info("Shutdown Server ... ")
-	if err := shutdownServer(srv, shutdownTimeout); err != nil {
+	serverErr, cleanupErr := gracefulShutdown(srv, quit, disarmStopSignals, budgetFrom(seconds))
+	if serverErr != nil {
 		// Not log.Fatal: that is an unconditional os.Exit(1), and Shutdown
 		// reports an error exactly when connections were still in flight -
-		// which is when the cleanup that follows matters most.
-		log.Error("Server Shutdown: ", err)
+		// which is when the cleanup that ran after it mattered most.
+		log.Error("Server Shutdown: ", serverErr)
+	}
+	if cleanupErr != nil {
+		log.Error("Cleanup: ", cleanupErr)
 	}
-	log.Info("Server exiting")
 
 	return nil
 }
 
-// shutdownTimeout is how long Shutdown waits for in-flight requests. It plus
-// whatever cleanup follows has to stay inside the orchestrator's grace period
-// - `docker stop` allows 10s by default before it sends SIGKILL.
-const shutdownTimeout = 5 * time.Second
+// budget is the three waits a shutdown spends, in the order it spends them.
+type budget struct {
+	drain   time.Duration
+	server  time.Duration
+	cleanup time.Duration
+}
+
+// budgetFrom turns the resolved seconds into the durations the sequence waits
+// on.
+func budgetFrom(s ext.ShutdownBudget) budget {
+	return budget{
+		drain:   time.Duration(s.Drain) * time.Second,
+		server:  time.Duration(s.Server) * time.Second,
+		cleanup: time.Duration(s.Cleanup) * time.Second,
+	}
+}
+
+// defaultBudget is what a process with no extend.shutdown section spends.
+func defaultBudget() budget {
+	return budget{drain: drainTimeout, server: shutdownTimeout, cleanup: cleanupTimeout}
+}
+
+// gracefulShutdown takes the process down in the order that gives something
+// else a chance to notice first.
+//
+// The whole order lives here, and run() is not the only caller: the signal
+// tests run this function rather than reproducing it. A test that reproduces a
+// sequence asserts against its own copy and stays green while the sequence it
+// was written for regresses.
+//
+// The caller has already taken the first signal off quit. quit is handed on
+// because a second signal during the drain window ends the window early -
+// somebody sending another kill wants this over with sooner - and because
+// until the window is over that signal must not reach the default handler and
+// kill the process outright.
+//
+// disarm is therefore called at the end of the window rather than on the first
+// signal. After it, a second signal is handled by the default disposition
+// again, which is the only way out of a Shutdown or a cleanup callback that
+// never returns. Restoring it any earlier would put every ordinary shutdown
+// inside that escape hatch for the whole length of the drain, where before
+// this window existed only a hung callback could reach it.
+//
+// The two waits' errors are returned separately rather than logged: they fail
+// for different reasons, and the caller decides what each is worth.
+func gracefulShutdown(srv *http.Server, quit <-chan os.Signal, disarm func(), b budget) (serverErr, cleanupErr error) {
+	// Said before anything is taken apart. A configuration reload arriving in
+	// this window would otherwise re-run AfterResource - rebuilding the pool
+	// and the queue adapter, and re-registering consumers - on top of cleanup
+	// that has already run.
+	sdk.Runtime.BeginShutdown()
+
+	// Readiness fails from here, which is before the server stops accepting.
+	// That order is necessary and not sufficient: with nothing between this
+	// line and the listener closing, the two are microseconds apart and a
+	// poller on a multi-second interval sees the refused connection instead of
+	// the 503. The window below is what turns the order into something
+	// observable - extend.shutdown.drain, which is zero unless it is
+	// configured.
+	health.BeginDraining()
+
+	// Keep-alive off for the same window, and for the same reason. The server
+	// keeps connections alive while !disableKeepAlives && !shuttingDown(), and
+	// shuttingDown() is only set by Shutdown itself - so without this line
+	// every pooled connection stays open for the whole drain and is cut at the
+	// end of it anyway, which is the cost of the window without its benefit.
+	// This is the switch Shutdown flips, moved earlier by the window's length:
+	// answers now carry Connection: close, and the idle connections a balancer
+	// is holding are closed at once rather than when it next tries to use one.
+	srv.SetKeepAlivesEnabled(false)
+
+	drain(quit, b.drain)
+
+	// Restored here, not on the first signal: from this point a second signal
+	// must reach the default handler, so a shutdown that hangs can still be
+	// interrupted.
+	disarm()
+
+	log.Info("Shutdown Server ... ")
+	serverErr = shutdownServer(srv, b.server)
+	// Runs whether or not the wait above failed, and deliberately so: Shutdown
+	// reports an error exactly when connections were still in flight, which is
+	// when there is most left to clean up after.
+	cleanupErr = runShutdownHooks(b.cleanup)
+	log.Info("Server exiting")
+
+	return serverErr, cleanupErr
+}
+
+// drain keeps serving for d, or until another stop signal arrives.
+//
+// Requests are answered normally throughout. Refusing them would move the
+// outage earlier rather than avoid it - the point of the window is that this
+// instance is still able to work while whoever routes to it stops routing.
+func drain(quit <-chan os.Signal, d time.Duration) {
+	if d <= 0 {
+		return
+	}
+	log.Infof("Draining for %s: still serving, /ready answers 503 from here", d)
+
+	timer := time.NewTimer(d)
+	defer timer.Stop()
+	select {
+	case <-quit:
+		log.Info("Second signal during the drain window, closing the listener now")
+	case <-timer.C:
+	}
+}
+
+// Reference stop grace periods, printed when nothing was configured to compare
+// against. They are three times apart, which is why the check below needs a
+// configured value rather than a constant of its own: a budget that overruns
+// under one of them fits comfortably under the other.
+const (
+	dockerStopGraceSeconds = 10
+	kubernetesGraceSeconds = 30
+)
+
+// reportShutdownBudget states what a shutdown will spend and whether it fits.
+//
+// The sum is taken from the resolved values, not from the configuration file:
+// a field left out of extend.shutdown still costs its default, so adding up
+// what was written down understates the budget by exactly the fields nobody
+// wrote.
+func reportShutdownBudget(s ext.ShutdownBudget) {
+	log.Infof("shutdown budget: drain %ds + server %ds + cleanup %ds = %ds",
+		s.Drain, s.Server, s.Cleanup, s.Total())
+
+	if s.Grace <= 0 {
+		log.Infof("shutdown budget: extend.shutdown.grace is not set, so nothing is compared against it - "+
+			"for reference `docker stop` allows %ds and Kubernetes terminationGracePeriodSeconds defaults to %ds",
+			dockerStopGraceSeconds, kubernetesGraceSeconds)
+		return
+	}
+	if over := s.Overrun(); over > 0 {
+		// A minimum, not a target. This is somebody else's deployment under
+		// constraints this process cannot see, so the honest thing to state is
+		// how much is missing - the repository's own files are where there is
+		// standing to ask for headroom on top, and checksilent does that.
+		log.Warnf("shutdown budget of %ds does not fit inside the %ds of extend.shutdown.grace: "+
+			"SIGKILL arrives while the cleanup callbacks are still running, and the work they "+
+			"were about to finish is lost. It needs at least %ds more, or %ds less budget.",
+			s.Total(), s.Grace, over, over)
+		return
+	}
+	log.Infof("shutdown budget of %ds fits inside the %ds of extend.shutdown.grace", s.Total(), s.Grace)
+}
+
+// The budgets a shutdown spends when extend.shutdown configures nothing:
+// drainTimeout keeps the process serving after the stop signal, then
+// shutdownTimeout waits for in-flight requests, then cleanupTimeout is what
+// the BeforeExit callbacks get.
+//
+// The seconds come from config, which is where an absent field falls back, so
+// the default is one number rather than two that can drift apart.
+//
+// They are consumed one after the other, so their sum is what has to stay
+// inside the orchestrator's grace period: `docker stop` allows 10s by default
+// before it sends SIGKILL, and 0+5+3 leaves room for the process to finish
+// returning. Raising one without lowering another buys nothing - the budget
+// that runs out is the orchestrator's, and reportShutdownBudget is what says
+// so at start-up.
+var (
+	drainTimeout    = time.Duration(ext.DefaultDrainSeconds) * time.Second
+	shutdownTimeout = time.Duration(ext.DefaultServerSeconds) * time.Second
+	cleanupTimeout  = time.Duration(ext.DefaultCleanupSeconds) * time.Second
+)
 
 // armStopSignals registers for the stop signals and returns the channel they
 // arrive on together with the function that restores the default disposition.
@@ -185,6 +425,66 @@ func armStopSignals() (<-chan os.Signal, func()) {
 	return quit, func() { signal.Stop(quit) }
 }
 
+// startServing binds srv.Addr, hands the listener to srv on its own goroutine,
+// and announces AfterListen.
+//
+// The bind is done here rather than left to ListenAndServe, which binds on the
+// goroutine that serves. That put the failure every deployment actually hits -
+// "address already in use" - on a goroutine nobody was reading, so the banner
+// went on to claim the server was up, and there would be no way to keep
+// AfterListen from announcing a socket that does not exist. A hook there is
+// promised a reachable port; the only way to keep that promise is for the bind
+// to have already happened on this goroutine.
+//
+// AfterListen is announced synchronously. Running it in a goroutine to save the
+// few milliseconds would let it overlap the shutdown: on a fast SIGTERM the
+// cleanup callbacks could finish before the startup ones had.
+//
+// Both ways of failing to start are therefore checked before the announcement:
+// the bind, and - with ssl enabled - the certificate.
+func startServing(srv *http.Server, useTLS bool, pem, key string) error {
+	if useTLS {
+		// Read the certificate before anything is announced. ServeTLS reads
+		// these files itself, but on the serving goroutine - so a bad
+		// certificate used to surface after AfterListen had already promised a
+		// reachable port. Loading it here costs one extra read and moves the
+		// failure onto this goroutine, where run() can return it.
+		//
+		// ServeTLS still does the real work below rather than this handing it a
+		// tls.Listener: that is what sets up HTTP/2 negotiation, and taking it
+		// over here would quietly drop h2 for every TLS deployment.
+		if _, err := tls.LoadX509KeyPair(pem, key); err != nil {
+			return errors.Wrap(err, "tls certificate")
+		}
+	}
+
+	ln, err := net.Listen("tcp", srv.Addr)
+	if err != nil {
+		return errors.Wrap(err, "listen")
+	}
+
+	go func() {
+		// 服务连接
+		var err error
+		if useTLS {
+			err = srv.ServeTLS(ln, pem, key)
+		} else {
+			err = srv.Serve(ln)
+		}
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			// Still fatal, as it was. Neither the bind nor the certificate is
+			// among the errors that reach here any more - both are checked
+			// above, on the caller's goroutine. What is left is a serve that
+			// failed after the port was taken, and carrying on would park the
+			// process on <-quit with nothing serving.
+			log.Fatal("serve: ", err)
+		}
+	}()
+
+	sdk.Runtime.RunPhase(runtime.AfterListen)
+	return nil
+}
+
 // shutdownServer stops srv, giving in-flight requests up to timeout to finish.
 //
 // It returns the error instead of exiting on it. A caller that exits here skips
@@ -196,6 +496,34 @@ func shutdownServer(srv *http.Server, timeout time.Duration) error {
 	return srv.Shutdown(ctx)
 }
 
+// runShutdownHooks runs the BeforeExit callbacks with timeout to share.
+//
+// What the budget bounds is the wait, not the work. When it is gone RunShutdown
+// stops waiting and returns; a callback that never looks at its context carries
+// on until the process exits, and may leave a partial write behind. Go cannot
+// cancel a function that does not check for cancellation, which is why the
+// callbacks are handed a context at all.
+func runShutdownHooks(timeout time.Duration) error {
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+	return sdk.Runtime.RunShutdown(ctx)
+}
+
+// buildRouter announces BeforeRouter, builds the engine, and then drains the
+// startup registries.
+//
+// The order is the contract. BeforeRouter is the last point at which a module
+// can still affect how routes are built, so it has to run while there is no
+// engine yet. The before registry runStartupHooks drains is a different moment
+// despite the name: those callbacks run after initRouter has built the engine.
+// Two lines apart, and describing them as equivalent is a mistake this
+// repository has already made once in writing.
+func buildRouter() {
+	sdk.Runtime.RunPhase(runtime.BeforeRouter)
+	initRouter()
+	runStartupHooks()
+}
+
 // runStartupHooks runs the router registries and then the before callbacks.
 //
 // The package-level slice runs first and in its existing order, so a fork that
@@ -243,10 +571,38 @@ func initRouter() {
 		r.Use(handler.TlsHandler())
 	}
 	//r.Use(middleware.Metrics())
-	r.Use(common.Sentinel()).
+	r.Use(exemptProbes(common.Sentinel())).
 		Use(common.RequestId(pkg.TrafficKey)).
 		Use(api.SetRequestLogger)
 
 	common.InitMiddleware(r)
 
 }
+
+// probePaths are the two routes the rate limiter must not answer for.
+var probePaths = map[string]bool{
+	otherrouter.APIPrefix + otherrouter.HealthPath: true,
+	otherrouter.APIPrefix + otherrouter.ReadyPath:  true,
+}
+
+// exemptProbes wraps a middleware so the health and readiness routes skip it.
+//
+// The limiter is installed on the engine and the probes are routes like any
+// other, so above the threshold they are answered with 429 as well. A liveness
+// probe that collects 429s fails its threshold and the container is restarted,
+// which takes capacity out of a deployment that is already short of it and
+// pushes the rest closer to the threshold - the limiter working exactly as
+// intended is what causes it. It is the argument common/health makes about
+// restarting a process whose database is unreachable, applied to load.
+//
+// Wrapping rather than teaching the limiter about these paths: the limiter
+// lives under common/, which may not import the package that registers them.
+func exemptProbes(h gin.HandlerFunc) gin.HandlerFunc {
+	return func(c *gin.Context) {
+		if probePaths[c.FullPath()] {
+			c.Next()
+			return
+		}
+		h(c)
+	}
+}

cmd/api/shutdown_test.go

@@ -0,0 +1,139 @@
+package api
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"github.com/gin-gonic/gin"
+
+	otherrouter "go-admin/app/other/router"
+	"go-admin/common/health"
+	ext "go-admin/config"
+)
+
+// The seconds in the configuration and the durations the sequence waits on are
+// two spellings of one budget, and only one of them is printed at start-up.
+func TestBudgetFromSeconds(t *testing.T) {
+	got := budgetFrom(ext.ShutdownBudget{Drain: 10, Server: 5, Cleanup: 3})
+	want := budget{
+		drain:   10 * time.Second,
+		server:  5 * time.Second,
+		cleanup: 3 * time.Second,
+	}
+	if got != want {
+		t.Errorf("budgetFrom = %+v, want %+v", got, want)
+	}
+}
+
+// The package variables and config.Default*Seconds have to say the same thing.
+// They are the same default written twice - once as durations for the shutdown
+// and once as seconds for the fallback - and a deployment that configures
+// nothing is entitled to one answer, not two.
+func TestDefaultBudgetIsTheConfiguredFallback(t *testing.T) {
+	unconfigured, err := ext.Shutdown{}.Budget()
+	if err != nil {
+		t.Fatalf("the empty section did not resolve: %v", err)
+	}
+	if got, want := defaultBudget(), budgetFrom(unconfigured); got != want {
+		t.Errorf("defaultBudget = %+v, want the unconfigured budget %+v", got, want)
+	}
+}
+
+// The rate limiter must not answer for the probes.
+//
+// It is installed on the engine, so without this the probes are limited like
+// any other route and answer 429 above the threshold. A liveness probe that
+// collects 429s fails its threshold and the container is restarted - taking
+// capacity out of a deployment that is already short of it and pushing the
+// rest closer to the threshold. The limiter working exactly as designed is
+// what would cause it.
+//
+// The stand-in rejects everything rather than being a real limiter: what is
+// under test is which requests reach it, and a real one would need the traffic
+// to cross a threshold before it said anything.
+func TestTheProbesSkipTheRateLimiter(t *testing.T) {
+	gin.SetMode(gin.TestMode)
+
+	var reached []string
+	r := gin.New()
+	r.Use(exemptProbes(func(c *gin.Context) {
+		reached = append(reached, c.FullPath())
+		c.AbortWithStatus(http.StatusTooManyRequests)
+	}))
+	v1 := r.Group(otherrouter.APIPrefix)
+	otherrouter.RegisterMonitorRouter(v1)
+	v1.GET("/business", func(c *gin.Context) { c.Status(http.StatusOK) })
+
+	for _, tc := range []struct {
+		path    string
+		limited bool
+	}{
+		{otherrouter.APIPrefix + otherrouter.HealthPath, false},
+		{otherrouter.APIPrefix + otherrouter.ReadyPath, false},
+		// Not a probe, and deliberately not exempt: the exemption is for the
+		// two routes an orchestrator acts on, not for everything under
+		// /api/v1 that happens to be unauthenticated.
+		{otherrouter.APIPrefix + "/metrics", true},
+		{otherrouter.APIPrefix + "/business", true},
+	} {
+		t.Run(tc.path, func(t *testing.T) {
+			w := httptest.NewRecorder()
+			r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, tc.path, nil))
+
+			if tc.limited {
+				if w.Code != http.StatusTooManyRequests {
+					t.Errorf("answered %d, want the middleware's 429 - it was skipped for a route that is not a probe", w.Code)
+				}
+				return
+			}
+			if w.Code == http.StatusTooManyRequests {
+				t.Errorf("answered 429; a probe that can be rate-limited gets the container restarted under load")
+			}
+		})
+	}
+
+	// Said separately, because a probe could also answer 429 by itself: what
+	// has to be true is that the middleware never saw the request.
+	for _, p := range reached {
+		if probePaths[p] {
+			t.Errorf("the middleware ran for %s", p)
+		}
+	}
+}
+
+// /health has to stay 200 while draining, and it is the assertion most easily
+// lost by accident: making the liveness probe follow the readiness flag reads
+// like tidying up, and it turns every rolling restart into a kubelet-issued
+// kill part-way through the drain.
+func TestHealthStaysUpWhileDraining(t *testing.T) {
+	gin.SetMode(gin.TestMode)
+
+	r := gin.New()
+	v1 := r.Group(otherrouter.APIPrefix)
+	otherrouter.RegisterMonitorRouter(v1)
+
+	ask := func(path string) int {
+		w := httptest.NewRecorder()
+		r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
+		return w.Code
+	}
+
+	if got := ask(otherrouter.APIPrefix + otherrouter.HealthPath); got != http.StatusOK {
+		t.Fatalf("/health answered %d before draining, want 200", got)
+	}
+
+	// Process-wide and one-way - nothing clears it - so this is the last thing
+	// in this package that may run in-process and care. Everything else that
+	// exercises draining does so in a child process of its own.
+	health.BeginDraining()
+
+	if got := ask(otherrouter.APIPrefix + otherrouter.HealthPath); got != http.StatusOK {
+		t.Errorf("/health answered %d while draining, want 200 - liveness is "+
+			"\"should I restart you\", and the answer during a drain is no", got)
+	}
+	if got := ask(otherrouter.APIPrefix + otherrouter.ReadyPath); got != http.StatusServiceUnavailable {
+		t.Errorf("/ready answered %d while draining, want 503", got)
+	}
+}

cmd/api/signal_test.go

@@ -1,33 +1,65 @@
 package api
 
 import (
+	"context"
 	"fmt"
+	"io"
 	"net"
 	"net/http"
 	"os"
 	"os/exec"
+	"strconv"
 	"strings"
+	"sync"
 	"syscall"
 	"testing"
 	"time"
+
+	"github.com/gin-gonic/gin"
+	"github.com/go-admin-team/go-admin-core/v2/sdk"
+
+	otherrouter "go-admin/app/other/router"
 )
 
 // The signal path cannot be exercised in-process: delivering a signal to the
 // test binary would race with the test framework, and the disposition changes
-// are global. So the test re-executes itself as a child, and the child runs the
-// same armStopSignals / shutdownServer the server does.
+// are global. So the test re-executes itself as a child, and the child runs
+// gracefulShutdown - the same function run() runs, not a second copy of the
+// sequence. A test that reproduces the sequence asserts against its own copy:
+// move BeginDraining after the drain window and the process regresses while
+// the test stays green, which is the failure mode this file exists to avoid.
 //
-// The child deliberately serves an empty http.Server rather than the real one:
-// this repository's CI has no database (.github/workflows/go.yml runs neither
-// MySQL nor a sqlite-tagged build), and none of what is under test needs one.
+// The child serves the real probe routes on an http.Server of its own rather
+// than the configured one: this repository's CI has no database
+// (.github/workflows/go.yml runs neither MySQL nor a sqlite-tagged build), and
+// none of what is under test needs one. /ready answers 503 either way - with
+// no database its checks fail - so the assertions below are on the draining
+// answer specifically, not on the status code alone.
 const (
-	childEnv       = "GO_ADMIN_SIGNAL_CHILD"
-	childStuckEnv  = "GO_ADMIN_SIGNAL_CHILD_STUCK"
-	childHangConn  = "GO_ADMIN_SIGNAL_CHILD_HANGCONN"
-	markerReady    = "CHILD-READY"
-	markerSignal   = "CHILD-SIGNAL"
-	markerShutdown = "CHILD-SHUTDOWN-OK"
-	markerExiting  = "CHILD-EXITING"
+	childEnv         = "GO_ADMIN_SIGNAL_CHILD"
+	childStuckEnv    = "GO_ADMIN_SIGNAL_CHILD_STUCK"
+	childHangConn    = "GO_ADMIN_SIGNAL_CHILD_HANGCONN"
+	childSlowCleanup = "GO_ADMIN_SIGNAL_CHILD_SLOWCLEANUP"
+	childDrainMS     = "GO_ADMIN_SIGNAL_CHILD_DRAIN_MS"
+	markerAddr       = "CHILD-ADDR"
+	markerReady      = "CHILD-READY"
+	markerSignal     = "CHILD-SIGNAL"
+	markerShutdown   = "CHILD-SHUTDOWN-OK"
+	markerCleanup    = "CHILD-CLEANUP-RAN"
+	markerTook       = "CHILD-TOOK-NS"
+	markerExiting    = "CHILD-EXITING"
+)
+
+// childPingRoute is an ordinary route, registered beside the probes so the
+// window can be checked for what it promises: requests arriving inside it are
+// served, not refused. Refusing them would move the outage earlier instead of
+// avoiding it.
+const childPingRoute = "/signal-test-ping"
+
+var (
+	readyPath  = otherrouter.APIPrefix + otherrouter.ReadyPath
+	healthPath = otherrouter.APIPrefix + otherrouter.HealthPath
+	pingPath   = otherrouter.APIPrefix + childPingRoute
 )
 
 // TestSignalChild is the child process. It is skipped in a normal run.
@@ -36,6 +68,12 @@ func TestSignalChild(t *testing.T) {
 		t.Skip("child process entry point")
 	}
 
+	gin.SetMode(gin.TestMode)
+	engine := gin.New()
+	v1 := engine.Group(otherrouter.APIPrefix)
+	otherrouter.RegisterMonitorRouter(v1)
+	v1.GET(childPingRoute, func(c *gin.Context) { c.String(http.StatusOK, "pong") })
+
 	ln, err := net.Listen("tcp", "127.0.0.1:0")
 	if err != nil {
 		fmt.Println("listen:", err)
@@ -47,7 +85,7 @@ func TestSignalChild(t *testing.T) {
 	// finds nothing to wait for and returns immediately.
 	accepted := make(chan struct{}, 1)
 	srv := &http.Server{
-		Handler: http.NewServeMux(),
+		Handler: engine,
 		ConnState: func(_ net.Conn, state http.ConnState) {
 			if state == http.StateNew {
 				select {
@@ -59,6 +97,45 @@ func TestSignalChild(t *testing.T) {
 	}
 	go func() { _ = srv.Serve(ln) }()
 
+	// The budget the child spends. Nothing here calls bootstrap.SetupConfig, so
+	// with no environment set this is the budget of a deployment that
+	// configures no extend.shutdown section at all.
+	b := defaultBudget()
+	if ms := os.Getenv(childDrainMS); ms != "" {
+		n, err := strconv.Atoi(ms)
+		if err != nil {
+			fmt.Println("drain:", err)
+			os.Exit(4)
+		}
+		b.drain = time.Duration(n) * time.Millisecond
+	}
+
+	// A BeforeExit callback, registered the way a module would. What the tests
+	// below care about is whether it runs at all - after a Shutdown that
+	// failed, and after its own budget has been spent.
+	sdk.Runtime.SetShutdown(func(ctx context.Context) {
+		switch {
+		case os.Getenv(childStuckEnv) == "1":
+			// Stands in for a cleanup hook that never finishes. The point of
+			// restoring the signal disposition after the drain window is that
+			// a second signal still reaches the default handler and kills this.
+			time.Sleep(2 * time.Minute)
+		case os.Getenv(childSlowCleanup) == "1":
+			// Outlasts the budget on purpose, and does not consult ctx - which
+			// is the case the contract is explicit about: what the context
+			// bounds is the wait, not the work.
+			time.Sleep(2 * time.Second)
+		}
+		fmt.Println(markerCleanup)
+		_ = os.Stdout.Sync()
+	})
+	switch {
+	case os.Getenv(childStuckEnv) == "1":
+		b.cleanup = 2 * time.Minute
+	case os.Getenv(childSlowCleanup) == "1":
+		b.cleanup = 300 * time.Millisecond
+	}
+
 	// Arm before announcing readiness. Doing it the other way round leaves a
 	// window in which the parent's signal reaches the default handler and
 	// kills the child before any of this runs - which is exactly the failure
@@ -66,22 +143,14 @@ func TestSignalChild(t *testing.T) {
 	// accident.
 	quit, disarm := armStopSignals()
 
+	fmt.Println(markerAddr, ln.Addr().String())
 	fmt.Println(markerReady)
-	os.Stdout.Sync()
+	_ = os.Stdout.Sync()
 
 	sig := <-quit
-	disarm()
 	fmt.Println(markerSignal, sig)
-	os.Stdout.Sync()
+	_ = os.Stdout.Sync()
 
-	if os.Getenv(childStuckEnv) == "1" {
-		// Stand in for a cleanup hook that never finishes. The point of
-		// restoring the signal disposition is that a second signal still
-		// reaches the default handler and kills this.
-		time.Sleep(2 * time.Minute)
-	}
-
-	timeout := shutdownTimeout
 	if os.Getenv(childHangConn) == "1" {
 		// Dialled here, not at start-up. net/http stops counting a StateNew
 		// connection against Shutdown once it is more than five seconds old,
@@ -107,21 +176,29 @@ func TestSignalChild(t *testing.T) {
 		// only treats a StateNew connection as idle once it is more than five
 		// seconds old. A short budget makes the timeout deterministic without
 		// waiting out the real one.
-		timeout = 300 * time.Millisecond
+		b.server = 300 * time.Millisecond
 	}
 
-	if err := shutdownServer(srv, timeout); err != nil {
+	started := time.Now()
+	serverErr, cleanupErr := gracefulShutdown(srv, quit, disarm, b)
+	spent := time.Since(started)
+
+	if serverErr != nil {
 		// Deliberately not fatal, and deliberately not a bare return: the
 		// point is that whatever follows still runs.
-		fmt.Println("shutdown error:", err)
+		fmt.Println("shutdown error:", serverErr)
 	} else {
 		fmt.Println(markerShutdown)
 	}
+	if cleanupErr != nil {
+		fmt.Println("cleanup error:", cleanupErr)
+	}
+	fmt.Println(markerTook, spent.Nanoseconds())
 	fmt.Println(markerExiting)
-	os.Stdout.Sync()
+	_ = os.Stdout.Sync()
 }
 
-func startChild(t *testing.T, stuck bool, extraEnv ...string) (*exec.Cmd, *os.File, chan string) {
+func startChild(t *testing.T, stuck bool, extraEnv ...string) (*exec.Cmd, chan string) {
 	t.Helper()
 
 	r, w, err := os.Pipe()
@@ -141,7 +218,7 @@ func startChild(t *testing.T, stuck bool, extraEnv ...string) (*exec.Cmd, *os.Fi
 	}
 	_ = w.Close()
 
-	lines := make(chan string, 64)
+	lines := make(chan string, 256)
 	go func() {
 		defer close(lines)
 		buf := make([]byte, 4096)
@@ -175,12 +252,13 @@ func startChild(t *testing.T, stuck bool, extraEnv ...string) (*exec.Cmd, *os.Fi
 		_, _ = cmd.Process.Wait()
 		_ = r.Close()
 	})
-	return cmd, r, lines
+	return cmd, lines
 }
 
 // await drains lines until one contains want, or the deadline passes. It
-// returns everything it saw, so a failure says what the child actually did.
-func await(t *testing.T, lines chan string, want string, d time.Duration) []string {
+// returns everything it saw, so a failure says what the child actually did,
+// and the matching line, so a marker can carry a value.
+func await(t *testing.T, lines chan string, want string, d time.Duration) ([]string, string) {
 	t.Helper()
 	var seen []string
 	deadline := time.After(d)
@@ -192,7 +270,7 @@ func await(t *testing.T, lines chan string, want string, d time.Duration) []stri
 			}
 			seen = append(seen, l)
 			if strings.Contains(l, want) {
-				return seen
+				return seen, l
 			}
 		case <-deadline:
 			t.Fatalf("timed out waiting for %q; saw:\n%s", want, strings.Join(seen, "\n"))
@@ -200,6 +278,134 @@ func await(t *testing.T, lines chan string, want string, d time.Duration) []stri
 	}
 }
 
+// childAddr waits for the address the child is listening on.
+func childAddr(t *testing.T, lines chan string) string {
+	t.Helper()
+	_, line := await(t, lines, markerAddr, 30*time.Second)
+	fields := strings.Fields(line)
+	return fields[len(fields)-1]
+}
+
+// took reads the nanoseconds gracefulShutdown spent, as the child measured
+// them. Measured inside the child on purpose: the parent's own clock includes
+// process scheduling, which is the noise the tightest assertion here cannot
+// afford.
+func took(t *testing.T, lines chan string, d time.Duration) time.Duration {
+	t.Helper()
+	_, line := await(t, lines, markerTook, d)
+	fields := strings.Fields(line)
+	ns, err := strconv.ParseInt(fields[len(fields)-1], 10, 64)
+	if err != nil {
+		t.Fatalf("unreadable %s line %q: %v", markerTook, line, err)
+	}
+	return time.Duration(ns)
+}
+
+// sample is one answer, or the refusal that replaced it.
+type sample struct {
+	at   time.Time
+	path string
+	// status is zero when the connection could not be made at all, which is
+	// what a closed listener looks like from outside.
+	status   int
+	draining bool
+	// willClose is what the server answered about the connection: the header
+	// it sends is Connection: close, which the transport consumes and reports
+	// here rather than leaving in Response.Header.
+	willClose bool
+}
+
+// probe asks once, on a connection of its own.
+//
+// A new transport per request, because a connection opened before the signal
+// can still be served after the listener is closed: reusing one would let this
+// test pass against a shutdown that had already broken the listener. Keep-alive
+// is left enabled so the server's own Connection: close is observable - a
+// client that asked for close would get that header back either way, and the
+// assertion would prove nothing.
+func probe(addr, path string) sample {
+	tr := &http.Transport{}
+	defer tr.CloseIdleConnections()
+	c := &http.Client{Transport: tr, Timeout: 3 * time.Second}
+
+	s := sample{at: time.Now(), path: path}
+	resp, err := c.Get("http://" + addr + path)
+	if err != nil {
+		return s
+	}
+	defer func() { _ = resp.Body.Close() }()
+	body, _ := io.ReadAll(resp.Body)
+	s.status = resp.StatusCode
+	s.willClose = resp.Close
+	s.draining = strings.Contains(string(body), `"status":"draining"`)
+	return s
+}
+
+// watcher polls the child until it stops accepting connections, keeping every
+// answer.
+type watcher struct {
+	mu      sync.Mutex
+	samples []sample
+	done    chan struct{}
+}
+
+func watch(addr string, paths ...string) *watcher {
+	w := &watcher{done: make(chan struct{})}
+	go func() {
+		defer close(w.done)
+		for {
+			refused := false
+			for _, p := range paths {
+				s := probe(addr, p)
+				w.mu.Lock()
+				w.samples = append(w.samples, s)
+				w.mu.Unlock()
+				if s.status == 0 {
+					refused = true
+				}
+			}
+			if refused {
+				return
+			}
+			time.Sleep(20 * time.Millisecond)
+		}
+	}()
+	return w
+}
+
+// sawDraining reports whether /ready has answered "draining" yet.
+func (w *watcher) sawDraining() bool {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	for _, s := range w.samples {
+		if s.path == readyPath && s.draining {
+			return true
+		}
+	}
+	return false
+}
+
+func (w *watcher) wait(t *testing.T, d time.Duration) []sample {
+	t.Helper()
+	select {
+	case <-w.done:
+	case <-time.After(d):
+		t.Fatal("the child never stopped accepting connections")
+	}
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	return w.samples
+}
+
+func describe(samples []sample) string {
+	var b strings.Builder
+	for _, s := range samples {
+		fmt.Fprintf(&b, "  %s %s -> %d draining=%v willClose=%v\n",
+			s.at.Format("15:04:05.000"), s.path, s.status, s.draining, s.willClose)
+	}
+	return b.String()
+}
+
 // Acceptance 19. Registering only os.Interrupt meant SIGTERM - the signal
 // `docker stop`, Kubernetes and systemd all send - terminated the process
 // before any of the shutdown path ran. Both must now reach it.
@@ -212,7 +418,7 @@ func TestBothSignalsRunTheShutdownPath(t *testing.T) {
 		{"SIGTERM", syscall.SIGTERM},
 	} {
 		t.Run(tc.name, func(t *testing.T) {
-			cmd, _, lines := startChild(t, false)
+			cmd, lines := startChild(t, false)
 			await(t, lines, markerReady, 30*time.Second)
 
 			if err := cmd.Process.Signal(tc.sig); err != nil {
@@ -234,8 +440,14 @@ func TestBothSignalsRunTheShutdownPath(t *testing.T) {
 // without restoring the disposition a second signal only refills the buffer:
 // once SIGTERM is registered, a shutdown that hangs could not be interrupted by
 // anything short of SIGKILL.
+//
+// The hang is now a cleanup callback that never returns, which is where a
+// shutdown actually hangs, and it is reached through gracefulShutdown - so this
+// also pins where the disposition is restored. Restore it before the drain
+// window and the window itself becomes the interruptible part; restore it never
+// and this test hangs.
 func TestASecondSignalStillKillsAStuckShutdown(t *testing.T) {
-	cmd, _, lines := startChild(t, true)
+	cmd, lines := startChild(t, true)
 	await(t, lines, markerReady, 30*time.Second)
 
 	if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
@@ -243,21 +455,33 @@ func TestASecondSignalStillKillsAStuckShutdown(t *testing.T) {
 	}
 	await(t, lines, markerSignal, 10*time.Second)
 
-	// The child is now inside a cleanup that will not finish on its own.
-	if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
-		t.Fatalf("second signal: %v", err)
-	}
-
+	// The child is now on its way into a cleanup that will not finish on its
+	// own. Signalled repeatedly rather than once: the marker is printed just
+	// before gracefulShutdown is entered, and the disposition is not restored
+	// until the drain window is over - zero seconds here, but not zero
+	// instructions - so a single signal sent immediately after the marker can
+	// still land in the buffered channel and be dropped. Which of them does
+	// the killing is not the assertion; that one of them can is.
 	done := make(chan error, 1)
 	go func() { done <- cmd.Wait() }()
 
-	select {
-	case err := <-done:
-		if err == nil {
-			t.Fatal("child exited cleanly; it was supposed to be killed by the second signal")
+	retry := time.NewTicker(200 * time.Millisecond)
+	defer retry.Stop()
+	deadline := time.After(15 * time.Second)
+	for {
+		select {
+		case err := <-done:
+			if err == nil {
+				t.Fatal("child exited cleanly; it was supposed to be killed by the second signal")
+			}
+			return
+		case <-retry.C:
+			if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
+				t.Fatalf("second signal: %v", err)
+			}
+		case <-deadline:
+			t.Fatal("the second signal did not kill a stuck shutdown - the escape hatch is gone")
 		}
-	case <-time.After(15 * time.Second):
-		t.Fatal("the second signal did not kill a stuck shutdown - the escape hatch is gone")
 	}
 }
 
@@ -266,7 +490,7 @@ func TestASecondSignalStillKillsAStuckShutdown(t *testing.T) {
 // unconditional os.Exit(1). Everything after it, which is where the cleanup
 // hooks will hang, never ran. A failed Shutdown must not end the process.
 func TestShutdownTimeoutDoesNotStopWhatFollows(t *testing.T) {
-	cmd, _, lines := startChild(t, false, childHangConn+"=1")
+	cmd, lines := startChild(t, false, childHangConn+"=1")
 	await(t, lines, markerReady, 30*time.Second)
 
 	if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
@@ -274,7 +498,7 @@ func TestShutdownTimeoutDoesNotStopWhatFollows(t *testing.T) {
 	}
 	await(t, lines, markerSignal, 10*time.Second)
 
-	seen := await(t, lines, markerExiting, 20*time.Second)
+	seen, _ := await(t, lines, markerExiting, 20*time.Second)
 
 	var timedOut bool
 	for _, l := range seen {
@@ -286,7 +510,230 @@ func TestShutdownTimeoutDoesNotStopWhatFollows(t *testing.T) {
 		t.Fatalf("Shutdown did not time out, so this test proves nothing; saw:\n%s",
 			strings.Join(seen, "\n"))
 	}
+	var cleaned bool
+	for _, l := range seen {
+		if strings.Contains(l, markerCleanup) {
+			cleaned = true
+		}
+	}
+	if !cleaned {
+		t.Fatalf("the BeforeExit callback did not run after a failed Shutdown; saw:\n%s",
+			strings.Join(seen, "\n"))
+	}
 	if err := cmd.Wait(); err != nil {
 		t.Fatalf("child exited with %v after a failed Shutdown, want a clean exit", err)
 	}
 }
+
+// A callback that outlasts its budget must not take the process with it, and
+// must not be waited for: RunShutdown reports the deadline and returns, the
+// callback carries on, and the process still exits cleanly. This is the half of
+// the contract that is easy to get backwards - the context bounds the wait, not
+// the work, because Go cannot cancel a function that does not check for it.
+func TestACleanupThatOutlastsItsBudgetIsAbandonedNotAwaited(t *testing.T) {
+	cmd, lines := startChild(t, false, childSlowCleanup+"=1")
+	await(t, lines, markerReady, 30*time.Second)
+
+	if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
+		t.Fatalf("signal: %v", err)
+	}
+	await(t, lines, markerSignal, 10*time.Second)
+
+	// The budget is 300ms and the callback sleeps two seconds. If RunShutdown
+	// waited for it, this marker would not arrive for two seconds; the one
+	// second here is what makes "abandoned, not awaited" the thing asserted.
+	seen, _ := await(t, lines, markerExiting, 1*time.Second)
+
+	var reported bool
+	for _, l := range seen {
+		if strings.Contains(l, "cleanup error:") {
+			reported = true
+		}
+		if strings.Contains(l, markerCleanup) {
+			t.Fatalf("the slow callback finished before the process moved on, so nothing was abandoned; saw:\n%s",
+				strings.Join(seen, "\n"))
+		}
+	}
+	if !reported {
+		t.Fatalf("RunShutdown returned no error for a callback that outlasted the budget; saw:\n%s",
+			strings.Join(seen, "\n"))
+	}
+	if err := cmd.Wait(); err != nil {
+		t.Fatalf("child exited with %v, want a clean exit despite the abandoned callback", err)
+	}
+}
+
+// The core acceptance: with a drain window configured, something outside the
+// process can observe that this instance is draining, on a connection it opens
+// after the signal, and can still be served while it does.
+//
+// Two windows rather than one. A single value proves only that something takes
+// that long, which a hard-coded sleep anywhere in the sequence would satisfy;
+// two say the wait is the configured one.
+//
+// What each answer is for:
+//
+//   - /ready reporting "draining" is the window being observable at all. The
+//     status code alone would not say it: with no database configured the
+//     probe's own checks fail and 503 is also the answer before the signal.
+//   - The server refusing to keep those connections alive is the window being
+//     useful. It keeps them alive until Shutdown sets shuttingDown(), so
+//     without switching keep-alive off here a balancer's pool would sit
+//     untouched for the whole window and be cut at the end of it anyway. The
+//     header saying so is Connection: close; the transport consumes it and
+//     reports it as Response.Close, which is what a sample records.
+//   - /health staying 200 is the window not asking to be restarted, and the
+//     ordinary route staying 200 is the window not refusing work. Draining is
+//     "stop sending me new work", not "reject what arrives".
+func TestTheDrainWindowIsObservableWhileStillServing(t *testing.T) {
+	for _, drain := range []time.Duration{300 * time.Millisecond, 1200 * time.Millisecond} {
+		t.Run(drain.String(), func(t *testing.T) {
+			cmd, lines := startChild(t, false,
+				fmt.Sprintf("%s=%d", childDrainMS, drain.Milliseconds()))
+			addr := childAddr(t, lines)
+			await(t, lines, markerReady, 30*time.Second)
+
+			w := watch(addr, readyPath, healthPath, pingPath)
+			// Long enough for a round of answers from a server that is not yet
+			// draining, which is what the keep-alive assertion below compares
+			// against.
+			time.Sleep(150 * time.Millisecond)
+
+			signalAt := time.Now()
+			if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
+				t.Fatalf("signal: %v", err)
+			}
+
+			samples := w.wait(t, drain+30*time.Second)
+			spent := took(t, lines, 10*time.Second)
+			await(t, lines, markerExiting, 10*time.Second)
+
+			if spent < drain {
+				t.Errorf("the shutdown took %s, want at least the %s window", spent, drain)
+			}
+
+			var refusedAt = -1
+			for i, s := range samples {
+				if s.status == 0 {
+					refusedAt = i
+					break
+				}
+			}
+			if refusedAt < 0 {
+				t.Fatalf("the child never stopped accepting; saw:\n%s", describe(samples))
+			}
+
+			var keptAliveBefore, drainingInside, closedInside bool
+			for _, s := range samples[:refusedAt] {
+				switch s.path {
+				case readyPath:
+					if s.at.Before(signalAt) && !s.draining && !s.willClose {
+						keptAliveBefore = true
+					}
+					if s.at.After(signalAt) && s.draining {
+						drainingInside = true
+						if s.willClose {
+							closedInside = true
+						}
+					}
+				case healthPath, pingPath:
+					if s.status != http.StatusOK {
+						t.Errorf("%s answered %d before the listener closed, want 200;\n%s",
+							s.path, s.status, describe(samples))
+					}
+				}
+			}
+
+			if !keptAliveBefore {
+				t.Fatalf("no answer before the signal kept the connection alive, so the header assertion below proves nothing;\n%s",
+					describe(samples))
+			}
+			if !drainingInside {
+				t.Errorf("no answer inside the window reported draining; the flip and the closed listener were not far enough apart to observe;\n%s",
+					describe(samples))
+			}
+			if !closedInside {
+				t.Errorf("answers inside the window still kept the connection alive, so a pooled connection survives the whole window and is cut at the end of it anyway;\n%s",
+					describe(samples))
+			}
+
+			if err := cmd.Wait(); err != nil {
+				t.Fatalf("child exited with %v, want a clean exit", err)
+			}
+		})
+	}
+}
+
+// The default has to be no window at all: a process that configures no
+// extend.shutdown section must shut down the way it did before the section
+// existed.
+//
+// Asserted as a sequence rather than as a duration. How long a shutdown takes
+// is decided by how much the cleanup callbacks have to do, so "as fast as
+// before" is not falsifiable; "nothing was inserted between the signal and the
+// listener closing" is.
+func TestAnUnconfiguredShutdownAddsNoWindow(t *testing.T) {
+	cmd, lines := startChild(t, false)
+	await(t, lines, markerReady, 30*time.Second)
+
+	if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
+		t.Fatalf("signal: %v", err)
+	}
+	await(t, lines, markerSignal, 10*time.Second)
+
+	spent := took(t, lines, 10*time.Second)
+	if spent > 100*time.Millisecond {
+		t.Errorf("an unconfigured shutdown spent %s between the signal and exiting; "+
+			"with no drain window and no cleanup callbacks it must be immediate", spent)
+	}
+
+	if err := cmd.Wait(); err != nil {
+		t.Fatalf("child exited with %v, want a clean exit", err)
+	}
+}
+
+// A second signal during the window ends it early rather than killing the
+// process. Somebody sending another kill wants this over with sooner, and the
+// answer to that is to stop draining - not to skip the cleanup, which is what
+// the default disposition would do.
+//
+// This is the pair to TestASecondSignalStillKillsAStuckShutdown: the escape
+// hatch has to be closed for the length of the window and open after it.
+func TestASecondSignalEndsTheDrainWindowEarly(t *testing.T) {
+	// Long enough that the shutdown cannot plausibly have taken this long on
+	// its own, short enough that the test does not sit out the whole window
+	// when the early exit is missing - it fails on the reported duration
+	// instead of on a timeout, which says which of the two broke.
+	const window = 10 * time.Second
+	cmd, lines := startChild(t, false,
+		fmt.Sprintf("%s=%d", childDrainMS, window.Milliseconds()))
+	addr := childAddr(t, lines)
+	await(t, lines, markerReady, 30*time.Second)
+
+	w := watch(addr, readyPath)
+	if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
+		t.Fatalf("first signal: %v", err)
+	}
+
+	deadline := time.Now().Add(15 * time.Second)
+	for !w.sawDraining() {
+		if time.Now().After(deadline) {
+			t.Fatal("the child never reported draining, so the second signal below would not land inside the window")
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+
+	if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
+		t.Fatalf("second signal: %v", err)
+	}
+
+	spent := took(t, lines, window+20*time.Second)
+	if spent >= window {
+		t.Errorf("the window ran its full %s despite a second signal (%s); the signal was ignored", window, spent)
+	}
+	await(t, lines, markerExiting, 10*time.Second)
+
+	if err := cmd.Wait(); err != nil {
+		t.Fatalf("child exited with %v; a second signal inside the window must end the window, not the process", err)
+	}
+}

common/health/health.go

@@ -0,0 +1,177 @@
+// Package health answers whether this process should be sent traffic.
+//
+// The two questions an orchestrator asks are not the same one, and go-admin
+// answers them at two endpoints:
+//
+//   - /health is liveness: is the process there at all. It stays a bare 200,
+//     because the honest answer to "should I restart you" is almost always no.
+//     Restarting a process because its database is unreachable turns one
+//     outage into a crash loop that also loses the connection pool, the cache
+//     and every in-flight request.
+//   - /ready is readiness: should this instance receive requests now. It fails
+//     while the dependencies are unreachable, and - the part that only exists
+//     because of the life-cycle phases - it fails as soon as shutdown begins,
+//     before the server stops accepting.
+//
+// # What the draining answer is worth
+//
+// Order alone does not produce a window. Answering before the server stops
+// accepting is the right order - the reverse reports the state after the
+// connections are already cut - but with nothing between the two they are
+// microseconds apart, and a poller on a multi-second interval never sees the
+// 503.
+//
+// The delay between them is extend.shutdown.drain, which is zero unless it is
+// configured. On the shipped defaults this is therefore still an answer that
+// can be read rather than one anything acts on; a deployment that sets a drain
+// window is the one that gets a window to act in.
+//
+// What acts on it depends on who does the removing. A load balancer that polls
+// /ready takes this instance out when it reads the 503, and the window has to
+// cover its check interval times its failure threshold, plus however long the
+// removal takes to apply. On Kubernetes the endpoint is withdrawn when the Pod
+// receives a deletionTimestamp, concurrently with SIGTERM and regardless of
+// what the probe returns - there the window covers the delay in that removal
+// reaching every node, and the 503 is what makes the state observable.
+package health
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"sync/atomic"
+	"time"
+
+	"github.com/go-admin-team/go-admin-core/v2/sdk"
+)
+
+// draining is set when the process starts shutting down.
+//
+// It is kept here rather than read back from core: BeginShutdown sets a flag on
+// the Application, but nothing exports it, and one host wanting to know is not
+// yet a reason to widen that interface.
+var draining atomic.Bool
+
+// BeginDraining records that shutdown has started, so readiness fails from now
+// on. It is called with BeginShutdown, before anything is taken apart.
+func BeginDraining() { draining.Store(true) }
+
+// Draining reports whether shutdown has begun.
+func Draining() bool { return draining.Load() }
+
+// Check is one dependency and what asking it produced.
+type Check struct {
+	Name string `json:"name"`
+	OK   bool   `json:"ok"`
+	Err  string `json:"error,omitempty"`
+}
+
+// Ready asks every dependency this process cannot serve a request without.
+//
+// The queue is deliberately absent. Nothing on AdapterQueue answers "are you
+// reachable" without publishing something, the memory backend cannot fail, and
+// a queue that is down degrades logging rather than stopping requests - which
+// is a reason to alert, not a reason to leave the load balancer pool.
+func Ready(ctx context.Context) []Check {
+	return []Check{
+		safely("database", func() error { return pingDB(ctx) }),
+		safely("cache", probeCache),
+	}
+}
+
+// safely turns a panic into a failed check.
+//
+// Not defensive habit: the accessors hand back wrappers, not the resources.
+// sdk.Runtime.GetCacheAdapter builds a runtime.Cache around whatever is
+// configured and returns it even when nothing is - so the value is not nil, the
+// cache inside it is, and the first call dereferences it. A nil check cannot
+// see that, and the same is true of GetQueueAdapter.
+//
+// Whatever the reason, a probe is the last thing that should be able to take
+// the process down: the caller is asking whether this instance is well, and
+// killing it to answer is the wrong reply.
+func safely(name string, fn func() error) (c Check) {
+	c = Check{Name: name}
+	defer func() {
+		if r := recover(); r != nil {
+			c.OK, c.Err = false, fmt.Sprintf("the check panicked: %v", r)
+		}
+	}()
+	if err := fn(); err != nil {
+		c.Err = err.Error()
+		return c
+	}
+	c.OK = true
+	return c
+}
+
+// Healthy reports whether every check passed.
+func Healthy(checks []Check) bool {
+	for _, c := range checks {
+		if !c.OK {
+			return false
+		}
+	}
+	return true
+}
+
+func pingDB(ctx context.Context) error {
+	db := sdk.Runtime.GetDb()
+	if db == nil {
+		return errors.New("no database configured")
+	}
+	sqlDB, err := db.DB()
+	if err != nil {
+		return err
+	}
+	return sqlDB.PingContext(ctx)
+}
+
+// cacheProbePrefix names the probe's keys. The key itself is per probe, not
+// fixed: two /ready requests arriving together - or two instances sharing one
+// redis, which is the normal deployment - would otherwise overwrite each
+// other's value between the write and the read and each conclude the cache was
+// broken. A readiness probe that reports false negatives under load takes
+// healthy instances out of the pool, which is worse than not probing.
+const cacheProbePrefix = "go-admin:health:"
+
+// cacheProbeTTL is short because these keys are write-once and never read
+// again by anyone else; it only has to outlive the read that follows.
+const cacheProbeTTL = 30
+
+func probeCache() error {
+	adapter := sdk.Runtime.GetCacheAdapter()
+	if adapter == nil {
+		return errors.New("no cache configured")
+	}
+
+	suffix := make([]byte, 8)
+	if _, err := rand.Read(suffix); err != nil {
+		return fmt.Errorf("could not build a probe key: %w", err)
+	}
+	key := cacheProbePrefix + hex.EncodeToString(suffix)
+
+	// Written and read back rather than only read: a cache that answers "miss"
+	// for every key - a client pointed at the wrong server - is
+	// indistinguishable from a healthy one on a read alone.
+	want := time.Now().Format(time.RFC3339Nano)
+	if err := adapter.Set(key, want, cacheProbeTTL); err != nil {
+		return err
+	}
+	// Best effort, and its error is deliberately dropped: the verdict is
+	// already decided by the read below, and a cache that cannot delete a key
+	// it just wrote is not a reason to refuse traffic. The TTL is the real
+	// cleanup.
+	defer func() { _ = adapter.Del(key) }()
+
+	got, err := adapter.Get(key)
+	if err != nil {
+		return err
+	}
+	if got != want {
+		return errors.New("the cache returned a different value than was written")
+	}
+	return nil
+}

common/health/health_test.go

@@ -0,0 +1,244 @@
+package health
+
+import (
+	"context"
+	"errors"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/go-admin-team/go-admin-core/v2/sdk"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/runtime"
+	corestorage "github.com/go-admin-team/go-admin-core/v2/storage"
+)
+
+func freshRuntime(t *testing.T) {
+	t.Helper()
+	previous := sdk.Runtime
+	t.Cleanup(func() { sdk.Runtime = previous })
+	sdk.Runtime = runtime.NewConfig()
+}
+
+// fakeCache answers whatever the test needs it to.
+type fakeCache struct {
+	mu      sync.Mutex
+	setErr  error
+	getErr  error
+	getBack string // returned instead of what was written, when non-empty
+	stored  map[string]string
+
+	// oneSlot makes the cache keep a single value however many keys are
+	// written, which is what a shared probe key turns any cache into.
+	oneSlot bool
+	slot    string
+
+	// setBarrier, when set, holds every writer until all of them have written.
+	// Without it the probes are short enough that the scheduler usually runs
+	// them one after another, and a shared key survives by luck rather than by
+	// design - which would leave the test below asserting nothing.
+	setBarrier *barrier
+}
+
+// barrier releases every waiter once n of them have arrived.
+type barrier struct {
+	n   int
+	mu  sync.Mutex
+	got int
+	ch  chan struct{}
+}
+
+func newBarrier(n int) *barrier { return &barrier{n: n, ch: make(chan struct{})} }
+
+func (b *barrier) wait() {
+	b.mu.Lock()
+	b.got++
+	if b.got == b.n {
+		close(b.ch)
+	}
+	b.mu.Unlock()
+	<-b.ch
+}
+
+func (c *fakeCache) String() string { return "fake" }
+
+func (c *fakeCache) Set(key string, val interface{}, _ int) error {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if c.setErr != nil {
+		return c.setErr
+	}
+	v, _ := val.(string)
+	if c.oneSlot {
+		c.slot = v
+		return nil
+	}
+	if c.stored == nil {
+		c.stored = map[string]string{}
+	}
+	c.stored[key] = v
+	c.mu.Unlock()
+	if c.setBarrier != nil {
+		// Outside the lock on purpose: waiting while holding it would deadlock
+		// every other writer before the barrier could fill.
+		c.setBarrier.wait()
+	}
+	c.mu.Lock()
+	return nil
+}
+
+func (c *fakeCache) Get(key string) (string, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if c.getErr != nil {
+		return "", c.getErr
+	}
+	if c.getBack != "" {
+		return c.getBack, nil
+	}
+	if c.oneSlot {
+		return c.slot, nil
+	}
+	return c.stored[key], nil
+}
+
+func (c *fakeCache) Del(key string) error {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	delete(c.stored, key)
+	return nil
+}
+
+func (c *fakeCache) HashGet(_, _ string) (string, error) { return "", nil }
+func (c *fakeCache) HashDel(_, _ string) error           { return nil }
+func (c *fakeCache) Increase(string) error               { return nil }
+func (c *fakeCache) Decrease(string) error               { return nil }
+func (c *fakeCache) Expire(string, time.Duration) error  { return nil }
+
+var _ corestorage.AdapterCache = (*fakeCache)(nil)
+
+func named(checks []Check, name string) Check {
+	for _, c := range checks {
+		if c.Name == name {
+			return c
+		}
+	}
+	return Check{Name: name, Err: "check not reported at all"}
+}
+
+// A cache that accepts writes and answers every read with a different value is
+// the failure this probe exists for - a client pointed at the wrong server, or
+// one that silently drops everything. A read alone cannot tell that apart from
+// a healthy cache with a cold key, which is why the probe writes first.
+func TestCacheProbeFailsWhenTheValueDoesNotComeBack(t *testing.T) {
+	freshRuntime(t)
+	sdk.Runtime.SetCacheAdapter(&fakeCache{getBack: "something else"})
+
+	got := named(Ready(context.Background()), "cache")
+	if got.OK {
+		t.Error("the cache check passed although the value written was not the value read back")
+	}
+	if got.Err == "" {
+		t.Error("the failing check reported no reason")
+	}
+}
+
+func TestCacheProbePassesWhenTheValueComesBack(t *testing.T) {
+	freshRuntime(t)
+	sdk.Runtime.SetCacheAdapter(&fakeCache{})
+
+	if got := named(Ready(context.Background()), "cache"); !got.OK {
+		t.Errorf("the cache check failed for a cache that works: %s", got.Err)
+	}
+}
+
+func TestCacheProbeReportsAWriteFailure(t *testing.T) {
+	freshRuntime(t)
+	sdk.Runtime.SetCacheAdapter(&fakeCache{setErr: errors.New("connection refused")})
+
+	got := named(Ready(context.Background()), "cache")
+	if got.OK {
+		t.Error("the cache check passed although the write failed")
+	}
+}
+
+// Nothing configured is the case that used to take the process down rather
+// than answer. GetCacheAdapter builds a wrapper around whatever is configured
+// and returns it even when nothing is, so the value is not nil, the cache
+// inside it is, and Set dereferences it - a probe that panics is the worst
+// possible answer to "are you well".
+//
+// Every check has to be reported, passing or not. A probe that omits what it
+// could not reach reads as a shorter list of healthy things.
+func TestEveryDependencyIsReportedEvenWithNothingConfigured(t *testing.T) {
+	freshRuntime(t)
+
+	checks := Ready(context.Background())
+	for _, name := range []string{"database", "cache"} {
+		c := named(checks, name)
+		if c.Err == "check not reported at all" {
+			t.Errorf("%s was not reported", name)
+		}
+		if c.OK {
+			t.Errorf("%s passed with nothing configured", name)
+		}
+	}
+	if Healthy(checks) {
+		t.Error("Healthy said yes for a process with no database and no cache")
+	}
+}
+
+// BeginDraining sets the flag and Draining reports it, before anything else is
+// taken apart. That is the whole of what can be checked from inside the
+// process: whether anyone outside gets to read it depends on
+// extend.shutdown.drain, which is zero unless it is configured, and on who is
+// routing traffic here - the package comment has both. The subprocess tests in
+// cmd/api are where a reader on the other end of a socket sees the 503.
+func TestDrainingIsObservableOnceItBegins(t *testing.T) {
+	previous := draining.Load()
+	t.Cleanup(func() { draining.Store(previous) })
+
+	draining.Store(false)
+	if Draining() {
+		t.Fatal("Draining reported true before shutdown began")
+	}
+	BeginDraining()
+	if !Draining() {
+		t.Error("Draining still reported false after BeginDraining")
+	}
+}
+
+// Two probes at once must both pass. With one fixed key they overwrite each
+// other's value between the write and the read, and a readiness probe that
+// reports false negatives under load takes healthy instances out of the pool -
+// which is worse than not probing at all.
+func TestConcurrentProbesDoNotOverwriteEachOther(t *testing.T) {
+	freshRuntime(t)
+	const probes = 16
+	sdk.Runtime.SetCacheAdapter(&fakeCache{setBarrier: newBarrier(probes)})
+
+	var wg sync.WaitGroup
+	failures := make(chan string, probes)
+	for i := 0; i < probes; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			if c := named(Ready(context.Background()), "cache"); !c.OK {
+				failures <- c.Err
+			}
+		}()
+	}
+	wg.Wait()
+	close(failures)
+
+	var n int
+	var first string
+	for err := range failures {
+		if n == 0 {
+			first = err
+		}
+		n++
+	}
+	if n > 0 {
+		t.Errorf("%d of %d concurrent probes called a healthy cache broken; first: %s", n, probes, first)
+	}
+}

common/middleware/settings.go

@@ -34,6 +34,7 @@ var CasbinExclude = []UrlInfo{
 	{Url: "/api/v1/user/pwd", Method: "PUT"},
 	{Url: "/api/v1/metrics", Method: "GET"},
 	{Url: "/api/v1/health", Method: "GET"},
+	{Url: "/api/v1/ready", Method: "GET"},
 	{Url: "/", Method: "GET"},
 	{Url: "/api/v1/server-monitor", Method: "GET"},
 	{Url: "/api/v1/public/uploadFile", Method: "POST"},

common/storage/initialize.go

@@ -9,10 +9,11 @@ package storage
 
 import (
 	"log"
+	"sync"
 
+	"github.com/go-admin-team/go-admin-core/v2/captcha"
 	"github.com/go-admin-team/go-admin-core/v2/sdk"
 	"github.com/go-admin-team/go-admin-core/v2/sdk/config"
-	"github.com/go-admin-team/go-admin-core/v2/captcha"
 )
 
 // Setup 配置storage组件
@@ -34,17 +35,75 @@ func setupCaptcha() {
 	captcha.SetStore(captcha.NewCacheStore(sdk.Runtime.GetCacheAdapter(), 600))
 }
 
+var (
+	queueMu sync.Mutex
+	// installed is the adapter setupQueue built, kept so the next reload can
+	// shut it down, and counted so a consumer can tell one from the next.
+	installed    interface{ Shutdown() }
+	installedGen uint64
+)
+
+// QueueGeneration reports how many times this package has installed a queue
+// adapter. It changes every time setupQueue builds a new one, which is on
+// every configuration reload, and stays 0 for as long as the configuration has
+// no queue section at all - in which case nothing is installed and callers are
+// working with the runtime's own fallback queue.
+//
+// It exists because there is no way to ask for the adapter's identity from the
+// outside. sdk.Runtime.GetQueueAdapter and GetQueuePrefix build a fresh
+// runtime.Queue wrapper on every call, so comparing what two calls return
+// compares two wrappers and never matches, however many times the underlying
+// adapter has been replaced. This package creates the adapter, so this is the
+// only place that knows. A counter rather than the adapter itself keeps the
+// comparison on a uint64: an adapter type that is not comparable would panic
+// an `==` between two interface values.
+func QueueGeneration() uint64 {
+	queueMu.Lock()
+	defer queueMu.Unlock()
+	return installedGen
+}
+
 func setupQueue() {
 	if config.QueueConfig.Empty() {
 		return
 	}
-	if q := sdk.Runtime.GetQueueAdapter(); q != nil {
-		q.Shutdown()
-	}
+
+	queueMu.Lock()
+	defer queueMu.Unlock()
+
 	queueAdapter, err := config.QueueConfig.Setup()
 	if err != nil {
 		log.Fatalf("queue setup error, %s\n", err.Error())
 	}
+
+	previous := installed
 	sdk.Runtime.SetQueueAdapter(queueAdapter)
-	go queueAdapter.Run()
+	installed = queueAdapter
+	installedGen++
+
+	// The previous adapter goes down after the new one is installed, not
+	// before. Shutdown waits for its consumers to deliver what it still holds,
+	// and for that whole wait the runtime would otherwise be handing producers
+	// a queue that has stopped accepting: every Append in the window comes back
+	// ErrQueueClosed, and both call sites in common/middleware log it. Swapping
+	// first leaves no such window - a producer gets the new queue or the old
+	// one, and both work.
+	//
+	// Only an adapter this package installed. GetQueueAdapter never returns
+	// nil - with nothing configured the runtime falls back to its own memory
+	// queue and wraps that - so the `if q := GetQueueAdapter(); q != nil` this
+	// replaces was always true, and shut down the fallback queue on the very
+	// first start, before anything had used it.
+	if previous != nil {
+		previous.Shutdown()
+	}
+
+	// Deliberately not started here. Run has to come after the consumers have
+	// registered: the contract implementations refuse a registration once the
+	// queue is running (storage.ErrQueueAlreadyStarted), and the legacy
+	// adapter this repository still goes through swallows that error rather
+	// than reporting it - its own comment says the interface gives it no way
+	// to tell the caller. Starting here and registering afterwards is
+	// therefore a race that loses consumers in silence. Whoever registers is
+	// the one that starts it.
 }

common/storage/queue_redis_test.go

@@ -0,0 +1,130 @@
+package storage
+
+import (
+	"errors"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/go-admin-team/go-admin-core/v2/sdk/config"
+	corestorage "github.com/go-admin-team/go-admin-core/v2/storage"
+	"github.com/go-admin-team/go-admin-core/v2/storage/queue"
+)
+
+// redisAddrEnv points these tests at a server. They are skipped without it, so
+// a developer with no redis running still gets a green run - and CI sets it,
+// which is the point: the ordering rule they cover is invisible on the memory
+// backend, and memory is the default. A suite that only ever exercised the
+// default would report success for a queue that silently drops every consumer.
+const redisAddrEnv = "GO_ADMIN_TEST_REDIS_ADDR"
+
+func redisAddr(t *testing.T) string {
+	t.Helper()
+	addr := os.Getenv(redisAddrEnv)
+	if addr != "" {
+		return addr
+	}
+	// Skipping locally is the point; skipping in CI is the failure this whole
+	// file exists to prevent. A workflow that renamed the variable, or dropped
+	// the service, would otherwise go green while these two tests quietly did
+	// nothing - which is the same shape as the defect they cover.
+	if os.Getenv("CI") != "" {
+		t.Fatalf("%s is not set while CI is: the redis-backed queue tests must not skip here", redisAddrEnv)
+	}
+	t.Skipf("%s is not set; skipping the redis-backed queue tests", redisAddrEnv)
+	return ""
+}
+
+// newRedisQueue builds the queue the same way setupQueue does - through
+// config.QueueConfig.Setup - so that what is under test is the adapter this
+// repository actually gets, LegacyQueueAdapter and all, rather than a redis
+// client wired up by the test.
+func newRedisQueue(t *testing.T, prefix string) corestorage.AdapterQueue {
+	t.Helper()
+	previous := config.QueueConfig
+	t.Cleanup(func() { config.QueueConfig = previous })
+
+	config.QueueConfig = &config.Queue{
+		Redis: &config.RedisQueue{
+			RedisOptions: config.RedisOptions{Addr: redisAddr(t)},
+			Group:        prefix,
+			KeyPrefix:    prefix,
+		},
+	}
+	q, err := config.QueueConfig.Setup()
+	if err != nil {
+		t.Fatalf("queue setup: %v", err)
+	}
+	t.Cleanup(q.Shutdown)
+	return q
+}
+
+func message(t *testing.T, stream string) corestorage.Messager {
+	t.Helper()
+	m := &queue.Message{}
+	m.SetStream(stream)
+	m.SetValues(map[string]interface{}{"hello": "world"})
+	return m
+}
+
+// Registered first, then started: the consumer gets the message. This is the
+// order setupQueue and attachQueueConsumers now produce between them.
+func TestRedisQueueDeliversToAConsumerRegisteredBeforeTheStart(t *testing.T) {
+	stream := "t-ordered"
+	q := newRedisQueue(t, "gotest-ordered")
+
+	got := make(chan struct{}, 1)
+	q.Register(stream, func(corestorage.Messager) error {
+		select {
+		case got <- struct{}{}:
+		default:
+		}
+		return nil
+	})
+	go q.Run()
+
+	// Give Start a moment to reach its read loop before publishing.
+	time.Sleep(500 * time.Millisecond)
+	if err := q.Append(message(t, stream)); err != nil {
+		t.Fatalf("append: %v", err)
+	}
+
+	select {
+	case <-got:
+	case <-time.After(15 * time.Second):
+		t.Fatal("the consumer never received the message")
+	}
+}
+
+// Started first, then registered: the registration is refused and every
+// publish afterwards fails.
+//
+// Subscribe answers ErrQueueAlreadyStarted, and LegacyQueueAdapter.Register
+// returns nothing, so the caller cannot know - that part is silent. What is not
+// silent is the consequence: no consumer group was created, so Publish refuses
+// the topic with ErrNoHandler on every single request, and go-admin's call
+// sites log that at error level while the login and operation log rows are
+// never written.
+//
+// This is the test the memory backend cannot provide. queue.Memory's Register
+// starts another consumer goroutine whatever the state, so the same code passes
+// there - which is how the defect survived, memory being the default.
+func TestRedisQueueRefusesAConsumerRegisteredAfterTheStart(t *testing.T) {
+	stream := "t-late"
+	q := newRedisQueue(t, "gotest-late")
+
+	go q.Run()
+	time.Sleep(500 * time.Millisecond)
+
+	q.Register(stream, func(corestorage.Messager) error { return nil })
+
+	err := q.Append(message(t, stream))
+	if err == nil {
+		t.Fatal("a message was accepted for a topic whose registration came after Start; " +
+			"if the backend now accepts late registration, the ordering rule in setupQueue can be revisited")
+	}
+	if !errors.Is(err, corestorage.ErrNoHandler) {
+		t.Fatalf("append failed with %v, want %v - the test is meant to pin the "+
+			"missing-consumer path, not any error at all", err, corestorage.ErrNoHandler)
+	}
+}

common/storage/queue_swap_test.go

@@ -0,0 +1,161 @@
+package storage
+
+import (
+	"errors"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/go-admin-team/go-admin-core/v2/sdk"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/config"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/runtime"
+	corestorage "github.com/go-admin-team/go-admin-core/v2/storage"
+	"github.com/go-admin-team/go-admin-core/v2/storage/queue"
+)
+
+// sampleSize is how many publishes have to land inside the reload before the
+// measurement is taken. Waiting on the count rather than on wall clock keeps
+// the window the test covers the same on a loaded runner as on an idle one.
+const sampleSize = 200
+
+func swapMsg() corestorage.Messager {
+	m := new(queue.Message)
+	m.SetStream("t")
+	m.SetValues(map[string]interface{}{"a": "b"})
+	return m
+}
+
+// A reload must never leave producers holding a queue that has stopped
+// accepting.
+//
+// Shutdown waits for its consumers to deliver what the queue still holds. Taking
+// the old adapter down before installing the new one meant the runtime pointed
+// at a closed queue for that entire wait: every Append in the window came back
+// ErrQueueClosed, and both call sites in common/middleware log it at error
+// level. Installing first leaves no window - a producer gets the new queue or
+// the old one, and both accept.
+//
+// The difference is only visible during that wait, which is why the test holds
+// a consumer rather than checking the state after Setup has returned: by then
+// the two orders look identical.
+//
+// One refusal survives the fix and is not something this ordering can reach.
+// GetQueuePrefix hands back a wrapper that captured the adapter, so a producer
+// that fetched before the swap and appends after Shutdown has begun is still
+// holding the old one. That window is one call wide and closing it means
+// resolving the adapter inside Append, which is core's to change. What the
+// ordering removes is the sustained window: every producer that fetches during
+// the wait. The test publishes from a single goroutine, so at most one of its
+// calls can straddle the swap - which is what makes "more than one" the line
+// between the two orders rather than a tolerance.
+func TestAReloadNeverPointsProducersAtAClosedQueue(t *testing.T) {
+	prevQ, prevC := config.QueueConfig, config.CacheConfig
+	prevRuntime := sdk.Runtime
+	prevInstalled, prevGen := installed, installedGen
+	t.Cleanup(func() {
+		config.QueueConfig, config.CacheConfig = prevQ, prevC
+		sdk.Runtime = prevRuntime
+		queueMu.Lock()
+		installed, installedGen = prevInstalled, prevGen
+		queueMu.Unlock()
+	})
+	sdk.Runtime = runtime.NewConfig()
+	config.CacheConfig = &config.Cache{Memory: struct{}{}}
+	// Sized so the buffer cannot fill while the consumer is held: a full queue
+	// returns an error of its own, and this test needs every error other than
+	// ErrQueueClosed to mean something it does not model has happened.
+	config.QueueConfig = &config.Queue{Memory: &config.QueueMemory{PoolSize: 4096}}
+
+	Setup()
+
+	// A consumer that will not finish until this test lets it, so the reload's
+	// Shutdown has something to wait for.
+	release := make(chan struct{})
+	consuming := make(chan struct{})
+	var picked sync.Once
+	first := sdk.Runtime.GetQueuePrefix("")
+	first.Register("t", func(corestorage.Messager) error {
+		picked.Do(func() { close(consuming) })
+		<-release
+		return nil
+	})
+	go first.Run()
+	for i := 0; i < 4; i++ {
+		if err := first.Append(swapMsg()); err != nil {
+			t.Fatalf("seed append %d: %v", i, err)
+		}
+	}
+	select {
+	case <-consuming:
+	case <-time.After(10 * time.Second):
+		t.Fatal("the consumer never picked a message up, so the reload has nothing to wait for")
+	}
+
+	reloaded := make(chan struct{})
+	go func() { Setup(); close(reloaded) }()
+
+	// Publish continuously while the reload is in progress.
+	var refused atomic.Int64
+	var attempts atomic.Int64
+	unexpected := make(chan error, 1)
+	stop := make(chan struct{})
+	// publishing is closed by the producer on its way out. The test joins on it
+	// before returning: t.Cleanup restores sdk.Runtime, and a producer still in
+	// flight would be reading the variable that restore writes.
+	publishing := make(chan struct{})
+	go func() {
+		defer close(publishing)
+		for {
+			select {
+			case <-stop:
+				return
+			default:
+			}
+			attempts.Add(1)
+			err := sdk.Runtime.GetQueuePrefix("").Append(swapMsg())
+			switch {
+			case err == nil:
+			case errors.Is(err, corestorage.ErrQueueClosed):
+				refused.Add(1)
+			default:
+				// Kept rather than counted: an Append refused for some other
+				// reason would otherwise leave refused at zero and the test
+				// green while nothing was reaching a queue at all.
+				select {
+				case unexpected <- err:
+				default:
+				}
+			}
+			time.Sleep(time.Millisecond)
+		}
+	}()
+
+	deadline := time.After(30 * time.Second)
+	for attempts.Load() < sampleSize {
+		select {
+		case <-deadline:
+			t.Fatalf("only %d publishes landed inside the reload; the window was never sampled", attempts.Load())
+		case <-time.After(time.Millisecond):
+		}
+	}
+	close(release)
+
+	select {
+	case <-reloaded:
+	case <-time.After(30 * time.Second):
+		t.Fatal("the reload never finished")
+	}
+	close(stop)
+	<-publishing
+
+	select {
+	case err := <-unexpected:
+		t.Fatalf("a publish failed for a reason this test does not model: %v", err)
+	default:
+	}
+	if n := refused.Load(); n > 1 {
+		t.Errorf("%d of %d publishes during the reload were refused: producers were pointed at the closed queue",
+			n, attempts.Load())
+	}
+}

common/storage/setup_gen_test.go

@@ -0,0 +1,53 @@
+package storage
+
+import (
+	"testing"
+
+	"github.com/go-admin-team/go-admin-core/v2/sdk"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/config"
+	"github.com/go-admin-team/go-admin-core/v2/sdk/runtime"
+)
+
+// Issue #892: a configuration reload replaces the queue adapter and the
+// consumers registered against the previous one are attached to a queue nobody
+// publishes to any more.
+//
+// The fix has two halves. attachQueueConsumers gives a new queue its own
+// consumers and the same queue none, which cmd/api covers against a queue the
+// test controls. This is the other half: that a reload actually produces a new
+// queue for it to notice. Setup is what config re-runs on every change, so
+// calling it twice is what a reload does to this package.
+func TestSetupBumpsTheQueueGenerationOnEveryReload(t *testing.T) {
+	// Setup writes the process-wide sdk.Runtime - the cache and queue adapters -
+	// and this package's own record of what it installed. Restoring all of it
+	// keeps the test from deciding what a later test in this binary sees,
+	// which is the same isolation cmd/api's freshRuntime provides.
+	prevQ, prevC := config.QueueConfig, config.CacheConfig
+	prevRuntime := sdk.Runtime
+	prevInstalled, prevGen := installed, installedGen
+	t.Cleanup(func() {
+		config.QueueConfig, config.CacheConfig = prevQ, prevC
+		sdk.Runtime = prevRuntime
+		queueMu.Lock()
+		installed, installedGen = prevInstalled, prevGen
+		queueMu.Unlock()
+	})
+	sdk.Runtime = runtime.NewConfig()
+
+	config.CacheConfig = &config.Cache{Memory: struct{}{}}
+	config.QueueConfig = &config.Queue{Memory: &config.QueueMemory{PoolSize: 10}}
+
+	before := QueueGeneration()
+	Setup()
+	first := QueueGeneration()
+	Setup()
+	second := QueueGeneration()
+
+	t.Logf("before=%d first=%d second=%d", before, first, second)
+	if first == before {
+		t.Fatal("the first Setup did not install a queue")
+	}
+	if second == first {
+		t.Fatal("a second Setup - which is what a configuration reload does - did not install a new one")
+	}
+}

config/extend.go

@@ -1,5 +1,10 @@
 package config
 
+import (
+	"fmt"
+	"strings"
+)
+
 var ExtConfig Extend
 
 // Extend 扩展配置
@@ -13,6 +18,7 @@ type Extend struct {
 	AMap      AMap // 这里配置对应配置文件的结构即可
 	FileStore FileStore
 	RateLimit RateLimit
+	Shutdown  Shutdown
 }
 
 // DefaultInboundQPS is the limit applied when nothing is configured. It is the
@@ -72,3 +78,133 @@ type ObjectStore struct {
 func (o ObjectStore) Configured() bool {
 	return o.Endpoint != "" && o.AccessKeyID != "" && o.AccessKeySecret != "" && o.BucketName != ""
 }
+
+// Default budgets for a graceful shutdown, in seconds. Each applies to the
+// matching field of extend.shutdown when that field is absent, and together
+// they are what the process spent before the section existed - so a deployment
+// that configures nothing keeps the shutdown it already had.
+//
+// The drain default is zero deliberately. The three budgets are spent one
+// after the other, and once their sum reaches the orchestrator's stop grace
+// period the process is killed part-way through its cleanup callbacks, which
+// is worse than not draining at all. `docker stop` allows ten seconds by
+// default and 5+3 already leaves little room, so a non-zero default here would
+// slow down every existing shutdown to buy something only a load balancer that
+// polls /ready can collect.
+const (
+	DefaultDrainSeconds   = 0
+	DefaultServerSeconds  = 5
+	DefaultCleanupSeconds = 3
+)
+
+// Shutdown is how long a graceful shutdown may spend, stage by stage.
+//
+//	extend:
+//	  shutdown:
+//	    drain: 0
+//	    server: 5
+//	    cleanup: 3
+//	    grace: 30
+//
+// Every field is a pointer for the reason RateLimit.InboundQPS is: nil means
+// "not configured" and takes the default, while a value that was written down
+// is taken literally, zero included. Without that separation `server: 0` - do
+// not wait for in-flight requests, which is a reasonable thing to ask under a
+// very short grace period - could not be said at all, and `drain: 0` would
+// have to mean something different from `server: 0` in the same section.
+type Shutdown struct {
+	// Drain is how long to keep serving normally after a stop signal arrives.
+	// Throughout it /ready answers 503 and keep-alive is switched off, which
+	// is what gives whatever routes traffic here time to stop routing it
+	// before the listener closes. Zero is no window: the readiness flip and
+	// the listener closing are then microseconds apart and nothing observes
+	// the first.
+	//
+	// What the window is worth depends on who does the removing and on what
+	// basis; the package comment in common/health has the two cases, and they
+	// do not want the same value.
+	Drain *int
+	// Server is how long the server waits for in-flight requests once the
+	// listener is closed.
+	Server *int
+	// Cleanup is how long the BeforeExit callbacks get after that.
+	Cleanup *int
+	// Grace is the stop grace period the orchestrator gives this process -
+	// `docker stop --timeout`, or terminationGracePeriodSeconds. Nothing reads
+	// it during a shutdown; it exists so start-up can say whether the budget
+	// fits inside it. Absent means no comparison is made, because the
+	// reference values differ threefold between runtimes and a fixed threshold
+	// would warn about configurations that are correct.
+	Grace *int
+}
+
+// ShutdownBudget is what a shutdown will actually spend, in seconds, after the
+// fallbacks have been applied.
+type ShutdownBudget struct {
+	Drain   int
+	Server  int
+	Cleanup int
+	// Grace is zero when extend.shutdown.grace was not configured.
+	Grace int
+}
+
+// Budget resolves the configured section into the values that will be spent.
+//
+// A negative is refused rather than corrected. A wait cannot be negative, so
+// there is no reading of one to honour, and quietly turning it into zero would
+// be the failure this whole section exists to remove: written down, accepted,
+// and not what happens. It is returned as an error rather than reported here
+// so that the rule can be checked without ending the process.
+func (s Shutdown) Budget() (ShutdownBudget, error) {
+	var negative []string
+	for _, f := range []struct {
+		name  string
+		value *int
+	}{
+		{"drain", s.Drain},
+		{"server", s.Server},
+		{"cleanup", s.Cleanup},
+		{"grace", s.Grace},
+	} {
+		if f.value != nil && *f.value < 0 {
+			negative = append(negative, fmt.Sprintf("%s: %d", f.name, *f.value))
+		}
+	}
+	if len(negative) > 0 {
+		return ShutdownBudget{}, fmt.Errorf(
+			"extend.shutdown was given a negative number of seconds (%s); "+
+				"a wait cannot be negative, and 0 is how to say \"do not wait\"",
+			strings.Join(negative, ", "))
+	}
+
+	return ShutdownBudget{
+		Drain:   budgetSeconds(s.Drain, DefaultDrainSeconds),
+		Server:  budgetSeconds(s.Server, DefaultServerSeconds),
+		Cleanup: budgetSeconds(s.Cleanup, DefaultCleanupSeconds),
+		Grace:   budgetSeconds(s.Grace, 0),
+	}, nil
+}
+
+func budgetSeconds(configured *int, fallback int) int {
+	if configured != nil {
+		return *configured
+	}
+	return fallback
+}
+
+// Total is the whole of the shutdown, since the three stages run one after the
+// other.
+func (b ShutdownBudget) Total() int { return b.Drain + b.Server + b.Cleanup }
+
+// Overrun reports how many seconds have to be found for the budget to fit
+// inside the configured grace period. It is zero when no grace period was
+// configured and when the budget already fits.
+//
+// Fitting means strictly less: the grace period is when SIGKILL is sent, so a
+// budget that ends exactly then leaves the last callback no time to return.
+func (b ShutdownBudget) Overrun() int {
+	if b.Grace <= 0 || b.Total() < b.Grace {
+		return 0
+	}
+	return b.Total() - b.Grace + 1
+}

config/extend_test.go

@@ -1,6 +1,9 @@
 package config
 
-import "testing"
+import (
+	"strings"
+	"testing"
+)
 
 func TestObjectStoreConfigured(t *testing.T) {
 	if (ObjectStore{}).Configured() {
@@ -32,3 +35,166 @@ func TestRateLimitThreshold(t *testing.T) {
 		t.Errorf("configured limit = %v, want %v", got, custom)
 	}
 }
+
+func ptr(v int) *int { return &v }
+
+// The zero-value rule is the same for all four fields, and it is the one the
+// section would otherwise need a paragraph of documentation to survive: nil
+// takes the default, a number that was written down is spent literally. A
+// `server: 0` that quietly became five seconds would be the same class of
+// failure this whole batch is about - configuration accepted and not applied.
+func TestShutdownBudgetFallbacks(t *testing.T) {
+	for _, tc := range []struct {
+		name string
+		in   Shutdown
+		want ShutdownBudget
+	}{
+		{
+			// What an existing settings.yml hits after an upgrade: no
+			// extend.shutdown section at all, and therefore the shutdown it
+			// already had.
+			name: "nothing configured",
+			in:   Shutdown{},
+			want: ShutdownBudget{Drain: 0, Server: 5, Cleanup: 3},
+		},
+		{
+			name: "all four configured",
+			in:   Shutdown{Drain: ptr(10), Server: ptr(8), Cleanup: ptr(4), Grace: ptr(30)},
+			want: ShutdownBudget{Drain: 10, Server: 8, Cleanup: 4, Grace: 30},
+		},
+		{
+			// The case a plain int could not express: do not wait for
+			// in-flight requests, which is a reasonable thing to ask for when
+			// the grace period is very short.
+			name: "explicit zeros are spent, not replaced",
+			in:   Shutdown{Drain: ptr(0), Server: ptr(0), Cleanup: ptr(0)},
+			want: ShutdownBudget{Drain: 0, Server: 0, Cleanup: 0},
+		},
+		{
+			name: "one field configured, the rest default",
+			in:   Shutdown{Drain: ptr(15)},
+			want: ShutdownBudget{Drain: 15, Server: 5, Cleanup: 3},
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			got, err := tc.in.Budget()
+			if err != nil {
+				t.Fatalf("Budget() = %v", err)
+			}
+			if got != tc.want {
+				t.Errorf("Budget() = %+v, want %+v", got, tc.want)
+			}
+		})
+	}
+}
+
+// A negative is refused, not corrected. Turning it into zero would be the
+// failure this section exists to remove - written down, accepted, and not what
+// happens - and there is no reading of a negative wait to honour.
+//
+// The last row is what makes the other four mean anything: an implementation
+// that refused every value would pass them all.
+func TestShutdownBudgetRefusesNegativeSeconds(t *testing.T) {
+	for _, tc := range []struct {
+		name    string
+		in      Shutdown
+		wantErr bool
+	}{
+		{name: "negative drain", in: Shutdown{Drain: ptr(-1)}, wantErr: true},
+		{name: "negative server", in: Shutdown{Server: ptr(-1)}, wantErr: true},
+		{name: "negative cleanup", in: Shutdown{Cleanup: ptr(-1)}, wantErr: true},
+		{name: "negative grace", in: Shutdown{Grace: ptr(-1)}, wantErr: true},
+		{name: "explicit zeros are not negative", in: Shutdown{Drain: ptr(0), Server: ptr(0), Cleanup: ptr(0)}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			_, err := tc.in.Budget()
+			if tc.wantErr && err == nil {
+				t.Fatal("Budget() accepted a negative number of seconds")
+			}
+			if !tc.wantErr && err != nil {
+				t.Fatalf("Budget() = %v, want the zeros taken literally", err)
+			}
+		})
+	}
+}
+
+// The message has to name every field that is wrong, not the first one: a
+// caller who fixes one and gets the same error back learns to distrust it.
+func TestShutdownBudgetNamesEveryNegativeField(t *testing.T) {
+	_, err := Shutdown{Drain: ptr(-1), Server: ptr(-30), Cleanup: ptr(-3), Grace: ptr(-9)}.Budget()
+	if err == nil {
+		t.Fatal("Budget() accepted four negative values")
+	}
+	for _, name := range []string{"drain", "server", "cleanup", "grace"} {
+		if !strings.Contains(err.Error(), name) {
+			t.Errorf("%q does not name %s", err, name)
+		}
+	}
+}
+
+// The sum is what has to fit inside the orchestrator's grace period, and the
+// verdict is only reached when a grace period was configured. A fixed
+// threshold instead would warn about the manifest this repository ships.
+func TestShutdownBudgetOverrun(t *testing.T) {
+	resolved := func(s Shutdown) ShutdownBudget {
+		b, err := s.Budget()
+		if err != nil {
+			t.Fatalf("Budget() = %v", err)
+		}
+		return b
+	}
+	for _, tc := range []struct {
+		name        string
+		budget      ShutdownBudget
+		wantTotal   int
+		wantOverrun int
+	}{
+		{
+			name:      "defaults, no grace period to judge against",
+			budget:    resolved(Shutdown{}),
+			wantTotal: 8,
+		},
+		{
+			name:      "fits with room to spare",
+			budget:    resolved(Shutdown{Drain: ptr(10), Grace: ptr(30)}),
+			wantTotal: 18,
+		},
+		{
+			// Equal is not a fit. The grace period is when SIGKILL is sent, so
+			// a budget that ends exactly then leaves the last callback no time
+			// to return.
+			name:        "exactly equal still overruns",
+			budget:      resolved(Shutdown{Drain: ptr(22), Grace: ptr(30)}),
+			wantTotal:   30,
+			wantOverrun: 1,
+		},
+		{
+			name:        "over by five",
+			budget:      resolved(Shutdown{Drain: ptr(26), Grace: ptr(30)}),
+			wantTotal:   34,
+			wantOverrun: 5,
+		},
+		{
+			// The reason the threshold is a configured value rather than a
+			// constant: the same budget is wrong under `docker stop` and right
+			// under a Kubernetes default.
+			name:        "the docker default is the tighter one",
+			budget:      resolved(Shutdown{Drain: ptr(10), Grace: ptr(10)}),
+			wantTotal:   18,
+			wantOverrun: 9,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := tc.budget.Total(); got != tc.wantTotal {
+				t.Errorf("Total() = %d, want %d", got, tc.wantTotal)
+			}
+			if got := tc.budget.Overrun(); got != tc.wantOverrun {
+				t.Errorf("Overrun() = %d, want %d", got, tc.wantOverrun)
+			}
+			if over := tc.budget.Overrun(); over > 0 && tc.budget.Total()-over >= tc.budget.Grace {
+				t.Errorf("Overrun() = %d does not bring %d under the %d grace period",
+					over, tc.budget.Total(), tc.budget.Grace)
+			}
+		})
+	}
+}

config/settings.full.yml

@@ -82,6 +82,40 @@ settings:
     # 会被负载均衡、监控和压测统计成成功）。
     rateLimit:
       inboundQPS: 200
+    # shutdown budgets, in seconds. The three are spent one after the other,
+    # and their sum has to stay inside the stop grace period the orchestrator
+    # allows - once it is up, SIGKILL arrives part-way through the cleanup
+    # callbacks, which is worse than not draining at all.
+    shutdown:
+      # How long to keep serving normally after a stop signal arrives. For that
+      # long /ready answers 503 and keep-alive is switched off, which is what
+      # gives a load balancer time to take this instance out of rotation before
+      # the listener closes.
+      #
+      # What to set depends on who removes this instance and on what basis. A
+      # load balancer that polls /ready itself needs at least "check interval x
+      # failure threshold + however long removal takes to apply". A Kubernetes
+      # Service removes the endpoint when the Pod is deleted, concurrently with
+      # SIGTERM and regardless of what the probe returns, so here this covers
+      # the delay in that removal reaching every node.
+      #
+      # 0 by default: a deployment that leaves this alone shuts down exactly as
+      # it did before this section existed. It also means /ready never reports
+      # draining - the flip and the closed listener are microseconds apart, and
+      # no poller reads anything in between.
+      drain: 0
+      # How long to wait for in-flight requests once the listener is closed.
+      server: 5
+      # How long the BeforeExit cleanup callbacks get after that.
+      cleanup: 3
+      # The stop grace period the orchestrator gives this process - `docker stop
+      # --timeout`, or terminationGracePeriodSeconds. Nothing reads it during a
+      # shutdown; start-up uses it to say whether the three budgets above fit
+      # inside it, and warns when they do not. Left out, nothing is compared:
+      # the reference values are 10s for docker and 30s for Kubernetes, three
+      # times apart, and a fixed threshold would warn about correct
+      # configurations.
+      #grace: 30
     # fileStore 对象存储。上传接口的 source 参数决定走哪一家：
     #   source=1 只存本地，source=2 阿里云 OSS，source=3 七牛 Kodo
     # 没有填的那一家在被请求时会返回明确错误，不会静默存到别处。

config/settings.yml

@@ -66,6 +66,40 @@ settings:
     # 会被负载均衡、监控和压测统计成成功）。
     rateLimit:
       inboundQPS: 200
+    # shutdown budgets, in seconds. The three are spent one after the other,
+    # and their sum has to stay inside the stop grace period the orchestrator
+    # allows - once it is up, SIGKILL arrives part-way through the cleanup
+    # callbacks, which is worse than not draining at all.
+    shutdown:
+      # How long to keep serving normally after a stop signal arrives. For that
+      # long /ready answers 503 and keep-alive is switched off, which is what
+      # gives a load balancer time to take this instance out of rotation before
+      # the listener closes.
+      #
+      # What to set depends on who removes this instance and on what basis. A
+      # load balancer that polls /ready itself needs at least "check interval x
+      # failure threshold + however long removal takes to apply". A Kubernetes
+      # Service removes the endpoint when the Pod is deleted, concurrently with
+      # SIGTERM and regardless of what the probe returns, so here this covers
+      # the delay in that removal reaching every node.
+      #
+      # 0 by default: a deployment that leaves this alone shuts down exactly as
+      # it did before this section existed. It also means /ready never reports
+      # draining - the flip and the closed listener are microseconds apart, and
+      # no poller reads anything in between.
+      drain: 0
+      # How long to wait for in-flight requests once the listener is closed.
+      server: 5
+      # How long the BeforeExit cleanup callbacks get after that.
+      cleanup: 3
+      # The stop grace period the orchestrator gives this process - `docker stop
+      # --timeout`, or terminationGracePeriodSeconds. Nothing reads it during a
+      # shutdown; start-up uses it to say whether the three budgets above fit
+      # inside it, and warns when they do not. Left out, nothing is compared:
+      # the reference values are 10s for docker and 30s for Kubernetes, three
+      # times apart, and a fixed threshold would warn about correct
+      # configurations.
+      #grace: 30
   cache:
 #    redis:
 #      addr: 127.0.0.1:6379

config/settings_shutdown_test.go

@@ -0,0 +1,68 @@
+package config
+
+import (
+	"testing"
+
+	coreconfig "github.com/go-admin-team/go-admin-core/v2/config"
+	"github.com/go-admin-team/go-admin-core/v2/config/source/file"
+)
+
+// shippedSettings is the shape the loader fills in, cut down to the part under
+// test. The reader is JSON-based, so the keys are matched against field names
+// case-insensitively - which is exactly the matching that silently drops a
+// section the struct has no field for.
+type shippedSettings struct {
+	Settings struct {
+		Extend Extend
+	}
+}
+
+func (*shippedSettings) OnChange() {}
+
+// The shutdown section has to arrive where it is read from, and with the
+// values the documentation claims.
+//
+// This is the failure this batch exists to remove, one level up: the loader
+// discards keys no field matches, without an error and without a log line, so
+// a section put in the wrong place is written, accepted, and never applied.
+// Nothing but loading the shipped file through the real loader can tell the
+// two apart - the struct compiles either way.
+//
+// The values are asserted as well as the arrival. A settings.yml that shipped
+// a different default from config.Default*Seconds would give two answers to
+// "what does an unconfigured deployment spend", and the file is the one people
+// read.
+func TestTheShippedSettingsReachTheShutdownStruct(t *testing.T) {
+	for _, name := range []string{"settings.yml", "settings.full.yml"} {
+		t.Run(name, func(t *testing.T) {
+			var loaded shippedSettings
+			c, err := coreconfig.NewConfig(
+				coreconfig.WithSource(file.NewSource(file.WithPath(name))),
+				coreconfig.WithEntity(&loaded),
+			)
+			if err != nil {
+				t.Fatalf("load %s: %v", name, err)
+			}
+			t.Cleanup(func() { _ = c.Close() })
+
+			s := loaded.Settings.Extend.Shutdown
+			if s.Drain == nil || s.Server == nil || s.Cleanup == nil {
+				t.Fatalf("%s left extend.shutdown unfilled (%+v); the section is written but nothing reads it",
+					name, s)
+			}
+
+			want := ShutdownBudget{
+				Drain:   DefaultDrainSeconds,
+				Server:  DefaultServerSeconds,
+				Cleanup: DefaultCleanupSeconds,
+			}
+			got, err := s.Budget()
+			if err != nil {
+				t.Fatalf("%s does not resolve: %v", name, err)
+			}
+			if got != want {
+				t.Errorf("%s ships %+v, want the documented defaults %+v", name, got, want)
+			}
+		})
+	}
+}

docker-compose.yml

@@ -7,6 +7,13 @@ services:
     restart: always
     ports:
       - 8000:8000
+    # Compose allows 10 seconds by default, and this process spends
+    # drain + server + cleanup from extend.shutdown before it exits - 8 out of
+    # the box, more for anyone who configures a drain window. Past the deadline
+    # it is sent SIGKILL and the cleanup callbacks are cut off part-way
+    # through. checksilent's docker-stop-cuts-shutdown-short check compares
+    # this against config/settings.yml.
+    stop_grace_period: 30s
     volumes:
       - ./config/:/go-admin-api/config/
       - ./static/:/go-admin-api/static/

docs/contract.md

@@ -377,7 +377,7 @@ if res.RowsAffected == 0 { return ErrAlreadyPaid }   // 别人先改了
 | 应用间调用 | 零定义。A 应用要调 B 应用只能直接 import 对方的包，循环依赖就回来了 |
 | 领域事件 / EventBus | 无 |
 | 缓存的租户隔离 | `service.Service` 有 `Cache` 字段，**是否按租户隔离未验证**。当作没隔离来写 |
-| 异步任务 | 有队列，但热更新后消费者会丢（issue #892） |
+| 生命周期钩子之外的时点 | 只有下面那四个。没有「路由装好之后、开始监听之前」这一档 |
 
 这几条留给后续批次，按真实需求补——现在凭空设计只会设计错。
 如果你的应用卡在这里，在 issue 里说一声，那正是我们要的输入。
@@ -672,6 +672,91 @@ if sdk.Runtime.AppRoutersSealed() { /* RunAppRouters 已经跑过了 */ }
 
 ---
 
+## 生命周期挂载点
+
+除了注册路由和迁移，应用还可以把工作挂在进程生命的四个时点上，不必等宿主按名字来调自己。
+契约本身在 core，见
+[go-admin-core `docs/contract.md`](https://github.com/go-admin-team/go-admin-core/blob/main/docs/contract.md)
+的「Life-cycle phases」一节。这里只写**在本仓里它们分别落在哪一行**。
+
+| 阶段 | 在 `cmd/api/server.go` 的位置 | 此时可用 |
+|---|---|---|
+| `AfterResource` | `bootstrap.SetupConfig` 跑完 `database.Setup` / `storage.Setup` 之后 | 配置、库、缓存、队列、casbin |
+| `BeforeRouter` | `initRouter()` **之前** | 以上，加引擎尚未构建这一事实 |
+| `AfterListen` | `startServing()` 里，`net.Listen` 返回之后 | 全部，端口**已绑定**、连接进得来 |
+| `BeforeExit` | `srv.Shutdown` 返回之后（无论它是否报错） | 全部，正在被拆掉 |
+
+`AfterListen` 承诺的是**端口已绑定**，不是「`Serve` 已经在 accept 循环里」——
+`srv.Serve` 在另一个 goroutine 上。这个区别是真实的：绑定成功之后内核就会把连接排进
+backlog，所以钩子里去连自己的端口不会被拒;但此刻 `Serve` 可能还没跑到第一次 `Accept`。
+绑定失败则**根本不会有这个阶段**：`net.Listen` 的错误直接从 `run()` 返回，
+横幅不打印，进程非零退出。
+
+```go
+sdk.Runtime.SetPhase(runtime.AfterResource, func() { /* ... */ })
+sdk.Runtime.SetShutdown(func(ctx context.Context) { /* ... */ })
+```
+
+### `BeforeRouter` 不等于 `before` 注册表
+
+**这两个不是同一个时点，文档里别混着写。** `SetBefore` 的回调由
+`runStartupHooks()` 执行，而那是在 `initRouter()` **之后**——引擎已经建好了。
+`BeforeRouter` 在它之前。
+
+顺带：`BeforeRouter` 是「硬约束：注册要赶在启动钩子之前」那一节所说的合法注册窗口之一。
+它早于 `runStartupHooks()`，所以在这里调 `sdk.Runtime.SetAppRouters` 仍然来得及。
+
+### `AfterResource` 会跑很多次，回调必须扛得住
+
+它在**每次配置热更新之后**都会再跑一遍，因为热更新会重建它所命名的那些资源。
+所以这里的回调要求是**「对同一个资源幂等」，不是「第二次什么都不做」**。
+
+本仓自己的队列消费者就是这条规则的样板，也是它存在的理由
+（`cmd/api/server.go` 的 `attachQueueConsumers`）：
+
+- 热更新重建了队列适配器，挂在旧适配器上的消费者连着一个**再没人往里发消息**的队列，
+  登录日志和操作日志就此停写且不出声。所以新适配器**必须**重新注册。
+- 但同一个适配器不能注册两次，否则每条消息有两个消费者，每行日志写两遍。
+
+**身份不能从访问器取。** `sdk.Runtime.GetQueueAdapter()` 与 `GetQueuePrefix()`
+每次调用都新造一个 `runtime.Queue` 包装，比较两次返回等于比较两个包装，
+**底层适配器换过多少次都不相等**。要在**创建资源的地方**记身份——
+本仓是 `common/storage.QueueGeneration()`。
+
+### 注册消费者要赶在队列启动之前
+
+走哪条实现，取决于配置里有没有 `redis:` 段（`config.QueueConfig.Setup()`）：
+
+| 配置 | 实际类型 | 启动后还能注册吗 |
+|---|---|---|
+| `queue: memory:` | `queue.NewMemory` | **能**。它的 `Register` 每次起一个消费 goroutine，不看是否已 `Run` |
+| `queue: redis:` | `storage.LegacyQueueAdapter` 包着新契约实现 | **不能**。`Register` 内部调 `Subscribe`，启动后返回 `storage.ErrQueueAlreadyStarted` |
+
+而 `LegacyQueueAdapter.Register` **没有返回值**——它只能把这个错误写进 slog，
+core 里那行注释自己写着「The interface has no way to report this to the caller」。
+**静默的是注册这一步，不是之后。** 没有建立消费组，redis 会用
+`storage.ErrNoHandler` 拒绝**之后的每一次投递**，而本仓两个调用点
+（`common/middleware/logger.go`、`common/middleware/handler/auth.go`）
+都把它记为 error——于是日志行一条都不落库，同时每个请求刷一条错误日志。
+
+所以顺序是硬的：**先 `Register` 完，再由注册方 `Run()`。**
+`common/storage` 的 `setupQueue` 有意不启动队列。
+
+默认配置选的是 memory 后端,它不在乎顺序——**这个缺陷在默认部署里看不见,
+只在配了 redis 的部署上发作**,而丢掉的正是登录日志、操作日志和 api 检查。
+
+### `BeforeExit` 反序执行，预算约束的是等待
+
+清理按**注册的逆序**执行。`SetShutdown` 拿到宿主剩余的预算，
+但**它约束的是等待，不是工作**：预算用尽时 `RunShutdown` 停止等待并返回，
+而不检查 context 的回调会一直跑到进程退出。Go 没法取消一个不检查取消的函数。
+
+本仓的样板是 cron（`app/jobs/jobbase.go` 的 `startCrontab`）：
+`cron.Stop()` 返回一个在**已经在跑的任务结束时**关闭的 context，
+钩子在它和预算之间二选一。
+
+---
+
 ## 安全边界：装一个应用等于信任它
 
 **这一层划不出安全边界，本文不假装划得出。**
@@ -718,5 +803,6 @@ if sdk.Runtime.AppRoutersSealed() { /* RunAppRouters 已经跑过了 */ }
 `checksilent` 一个文件都看不到。所以它保的是**这个仓库和它的 fork**，
 不是你的应用——你的应用要自己跑自己的检查。
 
-`checksilent` 还检查另外五类"不出声的失败"，写模块时值得先看一眼
-`go run ./tools/checksilent -h`。
+`checksilent` 还检查其他几类"不出声的失败"，写模块时值得先看一眼
+`AGENTS.md` 的「静默失败校验」一节，或者 `tools/checksilent/checks.go` 里的
+`runChecks`（`-h` 只打印命令行参数，不列检查）。

go.mod

@@ -11,7 +11,7 @@ require (
 	github.com/casbin/casbin/v3 v3.8.1
 	github.com/gin-gonic/gin v1.12.0
 	github.com/glebarez/sqlite v1.11.0
-	github.com/go-admin-team/go-admin-core/v2 v2.5.0
+	github.com/go-admin-team/go-admin-core/v2 v2.7.0
 	github.com/google/uuid v1.6.0
 	github.com/huaweicloud/huaweicloud-sdk-go-obs v3.26.6+incompatible
 	github.com/mssola/user_agent v0.6.0
@@ -26,6 +26,7 @@ require (
 	github.com/swaggo/gin-swagger v1.6.1
 	github.com/swaggo/swag v1.16.6
 	github.com/unrolled/secure v1.17.0
+	go.yaml.in/yaml/v3 v3.0.5
 	golang.org/x/crypto v0.54.0
 	gorm.io/driver/mysql v1.6.0
 	gorm.io/driver/postgres v1.6.2
@@ -126,7 +127,6 @@ require (
 	go.uber.org/atomic v1.11.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.27.1 // indirect
-	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/arch v0.30.0 // indirect
 	golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 // indirect
 	golang.org/x/image v0.41.0 // indirect

go.sum

@@ -145,8 +145,8 @@ github.com/glebarez/go-sqlite v1.22.0 h1:uAcMJhaA6r3LHMTFgP0SifzgXg46yJkgxqyuyec
 github.com/glebarez/go-sqlite v1.22.0/go.mod h1:PlBIdHe0+aUEFn+r2/uthrWq4FxbzugL0L8Li6yQJbc=
 github.com/glebarez/sqlite v1.11.0 h1:wSG0irqzP6VurnMEpFGer5Li19RpIRi2qvQz++w0GMw=
 github.com/glebarez/sqlite v1.11.0/go.mod h1:h8/o8j5wiAsqSPoWELDUdJXhjAhsVliSn7bWZjOhrgQ=
-github.com/go-admin-team/go-admin-core/v2 v2.5.0 h1:aD1SALklBxizGB9u8cOgm4OT8z656FM83F4fD6dMz9g=
-github.com/go-admin-team/go-admin-core/v2 v2.5.0/go.mod h1:LG/XvEfOplbuadKrPTPm0Nu5pN06aQUNZZC3ao4B4gs=
+github.com/go-admin-team/go-admin-core/v2 v2.7.0 h1:1qV0/5iFBvkE3BRtm4ip0v0QYG9Fgx4UtOTd8zkQT9c=
+github.com/go-admin-team/go-admin-core/v2 v2.7.0/go.mod h1:LG/XvEfOplbuadKrPTPm0Nu5pN06aQUNZZC3ao4B4gs=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.10.0/go.mod h1:xUsJbQ/Fp4kEt7AFgCuvyX4a71u8h9jB8tj/ORgOZ7o=

scripts/k8s/deploy.yml

@@ -22,6 +22,11 @@ metadata:
     app: go-admin
     version: v1
 spec:
+  # One replica, and the drain window below buys nothing at one replica: there
+  # is nowhere to send the traffic this pod stops taking. Raising it needs one
+  # more change than the number - the volume below is shared by every replica,
+  # and the log path in settings.yml lives on it, so a second pod would append
+  # to the same rotating file.
   replicas: 1
   selector:
     matchLabels:
@@ -39,6 +44,40 @@ spec:
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 8000
+        # Readiness answers "send me requests". It fails while the database or
+        # the cache is unreachable, so this pod stays out of the Service until
+        # the datastore settings.yml names is really there - which is a change
+        # from having no probe at all, where a pod with an unreachable database
+        # was still sent traffic.
+        #
+        # timeoutSeconds is 3 rather than the default 1 because the handler
+        # allows its checks 2 seconds (readyTimeout in
+        # app/other/router/monitor.go). At the default, a database that answers
+        # in 1.2s is recorded as a failed check while the handler is returning
+        # 200.
+        readinessProbe:
+          httpGet:
+            path: /api/v1/ready
+            port: 8000
+          periodSeconds: 5
+          timeoutSeconds: 3
+          failureThreshold: 3
+        # Liveness answers "restart me", which is a different question: a
+        # process whose database is unreachable does not want restarting, so
+        # this points at /health, which is a bare 200. Both probes skip the
+        # rate limiter - see exemptProbes in cmd/api/server.go - because a
+        # liveness probe that collects 429s under load gets the container
+        # restarted at the moment the deployment can least afford to lose it.
+        #
+        # initialDelaySeconds covers the migrations, which run before the
+        # listener opens.
+        livenessProbe:
+          httpGet:
+            path: /api/v1/health
+            port: 8000
+          initialDelaySeconds: 15
+          periodSeconds: 10
+          failureThreshold: 3
         volumeMounts:
         - name: go-admin
           mountPath: /temp
@@ -47,6 +86,18 @@ spec:
         - name: go-admin-config
           mountPath: /config/
           readOnly: true
+      # SIGKILL arrives when this is up, so it has to be longer than what the
+      # process spends shutting down: extend.shutdown's drain + server +
+      # cleanup, which settings.yml ships as 0 + 5 + 3. Raise drain here and
+      # this number has to follow, or the cleanup callbacks are cut off
+      # part-way through - checksilent's shutdown-budget-overruns-grace check
+      # is what notices.
+      #
+      # No preStop hook on purpose. A sleep there would be spent before the
+      # process is told anything, so BeginDraining never runs and /ready
+      # answers 200 for the whole of it - and it would be added to the budget
+      # above rather than replacing any of it.
+      terminationGracePeriodSeconds: 30
       volumes:
       - name: go-admin
         persistentVolumeClaim:

tools/checksilent/checks.go

@@ -21,6 +21,8 @@ const (
 	checkImportBoundary = "contract-import-boundary"
 	checkShimAlias      = "contract-shim-alias"
 	checkDataScopeRoute = "datascope-route-unguarded"
+	checkShutdownGrace  = "shutdown-budget-overruns-grace"
+	checkDockerStop     = "docker-stop-cuts-shutdown-short"
 )
 
 // Package paths, relative to the module. Spelled once so a module rename
@@ -50,6 +52,17 @@ func runChecks(s *snapshot, opt options) ([]Finding, error) {
 	out = append(out, checkContractShimAlias(s)...)
 	out = append(out, checkDataScopeRoutes(s)...)
 
+	for _, run := range []func(*snapshot) ([]Finding, error){
+		checkShutdownBudgetFitsGrace,
+		checkDockerStopGrace,
+	} {
+		fs, err := run(s)
+		if err != nil {
+			return nil, err
+		}
+		out = append(out, fs...)
+	}
+
 	if opt.UIDir != "" {
 		fs, err := checkMenuNames(s, opt.UIDir)
 		if err != nil {

tools/checksilent/main.go

@@ -1,13 +1,23 @@
 // Command checksilent reports the failures in this repository that do not
 // announce themselves: no error, no log line, behaviour quietly wrong.
 //
-// Seven checks, six of them ERROR and one WARN. An ERROR fails the run; a WARN
-// prints and does not. The split is not about how bad the consequence is - all
-// seven are bad - but about how certain the detection is. Everything reported as
-// an ERROR is decided from this repository's own syntax. The one WARN compares
-// against a second repository through a regular expression, and a check that
-// can be wrong must not be able to stop a build, or the first response to it
-// will be an ignore comment.
+// An ERROR fails the run; a WARN prints and does not. The split is not about
+// how bad the consequence is - every one of these is bad - but about how much
+// room is left to act.
+//
+// Most of them report only ERROR: each is decided from this repository's own
+// files and is either true or not. The menu-name check reports only WARN,
+// because it compares against a second repository through a regular
+// expression, and a check that can be wrong must not be able to stop a build
+// or the first response to it will be an ignore comment. The two
+// shutdown-budget checks report at both levels from one arithmetic: a budget
+// that already overruns is an ERROR, and one that fits with no headroom left
+// is a WARN - it works today, so failing the build on it would be failing a
+// correct configuration.
+//
+// The list of checks is runChecks in checks.go. It is deliberately not
+// repeated here as a count: the two places that carried one were both wrong by
+// the time anybody looked.
 //
 // Usage:
 //

tools/checksilent/shutdownbudget.go

@@ -0,0 +1,651 @@
+package main
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strconv"
+	"strings"
+
+	yaml "go.yaml.in/yaml/v3"
+)
+
+// The files this check compares, and the package the fallbacks come from.
+const (
+	settingsFile     = "config/settings.yml"
+	k8sDeployFile    = "scripts/k8s/deploy.yml"
+	pkgHostConfig    = "config"
+	drainConstName   = "DefaultDrainSeconds"
+	serverConstName  = "DefaultServerSeconds"
+	cleanupConstName = "DefaultCleanupSeconds"
+)
+
+// graceMarginSeconds is the headroom a shutdown budget needs beyond itself.
+//
+// Spelled once and used by both checks below, because they fail the same way:
+// somebody raises a budget in config/settings.yml and does not go looking for
+// the two other places that have to allow room for it. Two margins would
+// eventually be two different numbers.
+const graceMarginSeconds = 5
+
+// checkShutdownBudgetFitsGrace compares the shutdown budget this repository
+// ships against the stop grace period its own Kubernetes manifest allows.
+//
+// The two are not merely adjacent examples. scripts/k8s/prerun.sh builds the
+// settings-admin ConfigMap out of config/settings.yml, and the Deployment
+// mounts that ConfigMap - so the manifest deploys that file.
+//
+// The budgets are spent one after the other, and when their sum reaches
+// terminationGracePeriodSeconds the kubelet sends SIGKILL while the cleanup
+// callbacks are still running. Nothing reports it: the pod disappears
+// mid-shutdown and it reads as a crash rather than as a number that was raised
+// in one file and not the other. Which is how it would be raised - drain is
+// the interesting knob and the grace period is in a different directory.
+//
+// Two levels, and an overrun is not also reported as a shortage of headroom:
+// every Error satisfies the Warn condition too, and an Error that always drags
+// a duplicate Warn behind it teaches people to skip Warns.
+//
+// A preStop hook counts, even though the shipped manifest has none. It is
+// spent before the process is told anything, so it is added to the budget
+// rather than overlapping it - and a self-check that cannot see it would
+// understate the real cost by however long somebody set it to, which is worse
+// than not checking.
+//
+// It reports nothing when either file is absent and when the manifest sets no
+// grace period, because there is then no second number to disagree with.
+func checkShutdownBudgetFitsGrace(s *snapshot) ([]Finding, error) {
+	budget, ok, err := shippedShutdownBudget(s)
+	if err != nil || !ok {
+		return nil, err
+	}
+	m, ok, err := readManifest(s)
+	if err != nil || !ok {
+		return nil, err
+	}
+	if m.grace == nil {
+		return nil, nil
+	}
+
+	var out []Finding
+	if m.preStopUnreadable {
+		out = append(out, Finding{
+			Check:    checkShutdownGrace,
+			Severity: Warn.String(),
+			File:     k8sDeployFile,
+			Line:     m.preStopLine,
+			Col:      1,
+			Message: "this preStop hook is not a sleep, so how long it takes cannot be read here " +
+				"and is not in the sum below; it is spent before the process is told anything, " +
+				"so whatever it costs has to fit inside terminationGracePeriodSeconds as well.",
+			severity: Warn,
+		})
+	}
+
+	total := m.preStop + budget.drain + budget.server + budget.cleanup
+	grace := *m.grace
+	spelled := fmt.Sprintf("preStop %d + drain %d + server %d + cleanup %d",
+		m.preStop, budget.drain, budget.server, budget.cleanup)
+
+	switch {
+	case total >= grace:
+		out = append(out, Finding{
+			Check:    checkShutdownGrace,
+			Severity: Error.String(),
+			File:     k8sDeployFile,
+			Line:     m.graceLine,
+			Col:      1,
+			Message: fmt.Sprintf(
+				"terminationGracePeriodSeconds is %d and the shutdown takes %d (%s, from %s); "+
+					"SIGKILL would arrive while the cleanup callbacks are still running. "+
+					"Raise it to %d, or take %d off the budget.",
+				grace, total, spelled, settingsFile,
+				total+graceMarginSeconds, total+graceMarginSeconds-grace),
+			severity: Error,
+		})
+	case total+graceMarginSeconds > grace:
+		out = append(out, Finding{
+			Check:    checkShutdownGrace,
+			Severity: Warn.String(),
+			File:     k8sDeployFile,
+			Line:     m.graceLine,
+			Col:      1,
+			Message: fmt.Sprintf(
+				"terminationGracePeriodSeconds is %d and the shutdown takes %d (%s, from %s), "+
+					"which leaves under %ds of headroom; a callback that runs slightly long is "+
+					"cut off. Raise it to %d.",
+				grace, total, spelled, settingsFile, graceMarginSeconds, total+graceMarginSeconds),
+			severity: Warn,
+		})
+	}
+	return out, nil
+}
+
+// dockerStopArgs matches a stop command in a script or a workflow.
+var (
+	dockerStopArgs = regexp.MustCompile(`\bdocker\s+stop\b`)
+	// --timeout is the current name, --time its deprecated spelling and -t the
+	// short form; docker still accepts all three, so all three are read. The
+	// long name comes first because --time is a prefix of it, and a flag that
+	// the check cannot read is reported as no deadline at all - which would
+	// have this tool pressing people towards the deprecated spelling.
+	dockerStopTime = regexp.MustCompile(`(--timeout|--time|-t)[=\s]*(\d+)`)
+)
+
+// checkDockerStopGrace reports a stop path that does not allow this process
+// the time it spends shutting down.
+//
+// docker allows ten seconds unless told otherwise, and that number is nowhere
+// near the command - so a budget raised in config/settings.yml passes every
+// test, deploys, and then has its cleanup callbacks killed on the next
+// release. Same failure as the manifest's grace period, same arithmetic, same
+// margin; only the file it lives in is different.
+//
+// Both ways of stopping this repository's container are covered, because
+// covering one of two identical paths is what produces a clean run that means
+// nothing: `docker stop` in a workflow or a script, and stop_grace_period in
+// the compose file the Makefile's own `make run` uses.
+//
+// An absent deadline is reported rather than assumed to be ten: the value that
+// applies is then invisible at the call site and cannot follow the budget.
+func checkDockerStopGrace(s *snapshot) ([]Finding, error) {
+	budget, ok, err := shippedShutdownBudget(s)
+	if err != nil || !ok {
+		return nil, err
+	}
+	total := budget.drain + budget.server + budget.cleanup
+	spelled := fmt.Sprintf("drain %d + server %d + cleanup %d",
+		budget.drain, budget.server, budget.cleanup)
+
+	sites, err := findStopDeadlines(s)
+	if err != nil {
+		return nil, err
+	}
+
+	var out []Finding
+	for _, site := range sites {
+		finding := Finding{
+			Check: checkDockerStop,
+			File:  site.file,
+			Line:  site.line,
+			Col:   1,
+		}
+		switch {
+		case !site.set:
+			finding.Severity, finding.severity = Error.String(), Error
+			finding.Message = fmt.Sprintf(
+				"%s allows the default %d seconds, and this process spends %d shutting down "+
+					"(%s, from %s). %s.",
+				site.what, dockerDefaultGraceSeconds, total, spelled, settingsFile,
+				site.fix(total+graceMarginSeconds))
+		case site.seconds <= total:
+			finding.Severity, finding.severity = Error.String(), Error
+			finding.Message = fmt.Sprintf(
+				"%s allows %d seconds and this shutdown takes %d (%s, from %s); the cleanup "+
+					"callbacks are killed part-way through. %s.",
+				site.what, site.seconds, total, spelled, settingsFile,
+				site.fix(total+graceMarginSeconds))
+		case site.seconds < total+graceMarginSeconds:
+			finding.Severity, finding.severity = Warn.String(), Warn
+			finding.Message = fmt.Sprintf(
+				"%s allows %d seconds over a shutdown that takes %d (%s, from %s), which leaves "+
+					"under %ds of headroom. %s.",
+				site.what, site.seconds, total, spelled, settingsFile, graceMarginSeconds,
+				site.fix(total+graceMarginSeconds))
+		default:
+			continue
+		}
+		out = append(out, finding)
+	}
+	return out, nil
+}
+
+// dockerDefaultGraceSeconds is what docker allows a container to stop in when
+// nothing says otherwise. It applies to `docker stop` and to compose alike.
+const dockerDefaultGraceSeconds = 10
+
+// stopSite is one place this repository decides how long a container gets.
+type stopSite struct {
+	file string
+	line int
+	// what names the setting in the finding, in the spelling of the file it
+	// was found in.
+	what string
+	// compose says which of the two fixes to suggest.
+	compose bool
+	seconds int
+	set     bool
+}
+
+func (s stopSite) fix(seconds int) string {
+	if s.compose {
+		return fmt.Sprintf("Set stop_grace_period: %ds", seconds)
+	}
+	return fmt.Sprintf("Pass --timeout %d", seconds)
+}
+
+func findStopDeadlines(s *snapshot) ([]stopSite, error) {
+	sites, err := findDockerStops(s.Root)
+	if err != nil {
+		return nil, err
+	}
+	compose, err := findComposeServices(s)
+	if err != nil {
+		return nil, err
+	}
+	return append(sites, compose...), nil
+}
+
+// dockerStopExtensions and dockerStopNames are where a stop command can be
+// written in this repository: workflows, shell scripts and the Makefile.
+var (
+	dockerStopExtensions = map[string]bool{".yml": true, ".yaml": true, ".sh": true, ".bash": true}
+	dockerStopNames      = map[string]bool{"Makefile": true, "makefile": true}
+)
+
+func findDockerStops(root string) ([]stopSite, error) {
+	var out []stopSite
+	err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		if info.IsDir() {
+			if path != root && skippedDirs[info.Name()] {
+				return filepath.SkipDir
+			}
+			return nil
+		}
+		if !dockerStopExtensions[filepath.Ext(path)] && !dockerStopNames[info.Name()] {
+			return nil
+		}
+		b, err := os.ReadFile(path)
+		if err != nil {
+			return err
+		}
+		rel, err := filepath.Rel(root, path)
+		if err != nil {
+			return err
+		}
+		for i, line := range strings.Split(string(b), "\n") {
+			// A commented-out command is not one that runs, and the settings
+			// file describes `docker stop` in prose right beside the budget
+			// this check reads.
+			if trimmed := strings.TrimSpace(line); strings.HasPrefix(trimmed, "#") {
+				continue
+			}
+			if !dockerStopArgs.MatchString(line) {
+				continue
+			}
+			site := stopSite{
+				file: filepath.ToSlash(rel),
+				line: i + 1,
+				what: "`docker stop` with no --timeout",
+			}
+			if m := dockerStopTime.FindStringSubmatch(line); m != nil {
+				seconds, err := strconv.Atoi(m[2])
+				if err != nil {
+					continue
+				}
+				// Quoted back in the spelling it was written in, so the
+				// message cannot misreport what the line says.
+				site.what = fmt.Sprintf("`docker stop %s %d`", m[1], seconds)
+				site.seconds, site.set = seconds, true
+			}
+			out = append(out, site)
+		}
+		return nil
+	})
+	return out, err
+}
+
+type shutdownSeconds struct{ drain, server, cleanup int }
+
+// shippedShutdownBudget reads extend.shutdown out of the settings file this
+// repository ships, filling in whatever it leaves out from the Go constants
+// that do the same at run time.
+//
+// Taking the fallbacks from the snapshot rather than repeating 0/5/3 here is
+// what keeps this honest when the defaults move: a tool that carries its own
+// copy of the number it is checking eventually checks the wrong one.
+//
+// A negative value is left alone. config.Shutdown.Budget refuses it and the
+// server does not start, so it is not a failure that passes unnoticed - and
+// adding a negative into the sums above would understate them.
+func shippedShutdownBudget(s *snapshot) (shutdownSeconds, bool, error) {
+	raw, ok, err := readRepoFile(s, settingsFile)
+	if err != nil || !ok {
+		return shutdownSeconds{}, false, err
+	}
+
+	var doc struct {
+		Settings struct {
+			Extend struct {
+				Shutdown *struct {
+					Drain   *int `yaml:"drain"`
+					Server  *int `yaml:"server"`
+					Cleanup *int `yaml:"cleanup"`
+				} `yaml:"shutdown"`
+			} `yaml:"extend"`
+		} `yaml:"settings"`
+	}
+	if err := yaml.Unmarshal(raw, &doc); err != nil {
+		return shutdownSeconds{}, false, fmt.Errorf("%s: %w", settingsFile, err)
+	}
+	section := doc.Settings.Extend.Shutdown
+	if section == nil {
+		return shutdownSeconds{}, false, nil
+	}
+
+	defaults, ok := s.hostConfigDefaults()
+	if !ok {
+		// The constants moved or were renamed. Reporting nothing would let the
+		// check go quiet, which is the failure it exists to catch, so this
+		// stops the run instead.
+		return shutdownSeconds{}, false, fmt.Errorf(
+			"%s has extend.shutdown but package %s declares no %s/%s/%s to fall back on",
+			settingsFile, pkgHostConfig, drainConstName, serverConstName, cleanupConstName)
+	}
+
+	budget := shutdownSeconds{
+		drain:   orDefault(section.Drain, defaults.drain),
+		server:  orDefault(section.Server, defaults.server),
+		cleanup: orDefault(section.Cleanup, defaults.cleanup),
+	}
+	if budget.drain < 0 || budget.server < 0 || budget.cleanup < 0 {
+		return shutdownSeconds{}, false, nil
+	}
+	return budget, true, nil
+}
+
+func orDefault(configured *int, fallback int) int {
+	if configured != nil {
+		return *configured
+	}
+	return fallback
+}
+
+// hostConfigDefaults reads the three fallback constants out of the parsed tree.
+func (s *snapshot) hostConfigDefaults() (shutdownSeconds, bool) {
+	for _, sf := range s.Files {
+		if sf.Pkg != s.pkg(pkgHostConfig) {
+			continue
+		}
+		drain, okDrain := sf.consts[drainConstName]
+		server, okServer := sf.consts[serverConstName]
+		cleanup, okCleanup := sf.consts[cleanupConstName]
+		if okDrain && okServer && okCleanup {
+			return shutdownSeconds{int(drain), int(server), int(cleanup)}, true
+		}
+	}
+	return shutdownSeconds{}, false
+}
+
+// manifest is what the shipped Deployment says about how long it will wait.
+type manifest struct {
+	grace     *int
+	graceLine int
+	// preStop is the longest sleep any container's hook performs, since the
+	// hooks of several containers run at the same time.
+	preStop     int
+	preStopLine int
+
+	preStopUnreadable bool
+}
+
+var preStopSleep = regexp.MustCompile(`\bsleep\s+(\d+)s?\b`)
+
+// readManifest finds the grace period and the preStop hooks in the shipped
+// manifest, with the lines they are on so a finding can be opened at them.
+//
+// The file holds several documents and only the Deployment carries a pod
+// template, so every document is decoded and the first one with a grace period
+// wins.
+func readManifest(s *snapshot) (manifest, bool, error) {
+	raw, ok, err := readRepoFile(s, k8sDeployFile)
+	if err != nil || !ok {
+		return manifest{}, false, err
+	}
+
+	dec := yaml.NewDecoder(bytes.NewReader(raw))
+	for {
+		var doc struct {
+			Spec struct {
+				Template struct {
+					Spec struct {
+						Grace      *int `yaml:"terminationGracePeriodSeconds"`
+						Containers []struct {
+							Lifecycle struct {
+								// A value, not a pointer: yaml.v3 only hands
+								// the raw node to a field of type yaml.Node,
+								// and a *yaml.Node field is allocated and left
+								// empty - which reads as "the hook is there but
+								// unreadable" for every manifest that has one.
+								PreStop yaml.Node `yaml:"preStop"`
+							} `yaml:"lifecycle"`
+						} `yaml:"containers"`
+					} `yaml:"spec"`
+				} `yaml:"template"`
+			} `yaml:"spec"`
+		}
+		switch err := dec.Decode(&doc); {
+		case errors.Is(err, io.EOF):
+			return manifest{}, false, nil
+		case err != nil:
+			return manifest{}, false, fmt.Errorf("%s: %w", k8sDeployFile, err)
+		}
+
+		pod := doc.Spec.Template.Spec
+		if pod.Grace == nil && len(pod.Containers) == 0 {
+			continue
+		}
+
+		m := manifest{
+			grace:     pod.Grace,
+			graceLine: lineOf(raw, "terminationGracePeriodSeconds:"),
+		}
+		for _, c := range pod.Containers {
+			hook := c.Lifecycle.PreStop
+			if hook.Kind == 0 {
+				continue
+			}
+			m.preStopLine = hook.Line
+			if seconds, ok := preStopSeconds(&hook); ok {
+				// The longest one, not the sum: the hooks of several
+				// containers run at the same time.
+				if seconds > m.preStop {
+					m.preStop = seconds
+				}
+				continue
+			}
+			m.preStopUnreadable = true
+		}
+		if m.grace == nil {
+			continue
+		}
+		return m, true, nil
+	}
+}
+
+// preStopSeconds reads how long a hook sleeps for.
+//
+// Every scalar under the hook is joined and searched, because the sleep can be
+// written as one argument or as several: ["sh","-c","sleep 10"] and
+// ["sleep","10"] both wait ten seconds.
+func preStopSeconds(node *yaml.Node) (int, bool) {
+	var words []string
+	var walk func(*yaml.Node)
+	walk = func(n *yaml.Node) {
+		if n == nil {
+			return
+		}
+		if n.Kind == yaml.ScalarNode {
+			words = append(words, n.Value)
+		}
+		for _, child := range n.Content {
+			walk(child)
+		}
+	}
+	walk(node)
+
+	m := preStopSleep.FindStringSubmatch(strings.Join(words, " "))
+	if m == nil {
+		return 0, false
+	}
+	seconds, err := strconv.Atoi(m[1])
+	if err != nil {
+		return 0, false
+	}
+	return seconds, true
+}
+
+// lineOf locates a key for a finding's position. A miss reports line 1 rather
+// than failing: the position is where to look, and the message is the finding.
+func lineOf(content []byte, key string) int {
+	for i, l := range strings.Split(string(content), "\n") {
+		if strings.Contains(l, key) && !strings.HasPrefix(strings.TrimSpace(l), "#") {
+			return i + 1
+		}
+	}
+	return 1
+}
+
+// readRepoFile reads a file relative to the scanned root, reporting absence
+// rather than failing on it: the checks run over fixtures that carry only what
+// the check under test needs.
+func readRepoFile(s *snapshot, rel string) ([]byte, bool, error) {
+	b, err := os.ReadFile(filepath.Join(s.Root, filepath.FromSlash(rel)))
+	switch {
+	case errors.Is(err, os.ErrNotExist):
+		return nil, false, nil
+	case err != nil:
+		return nil, false, err
+	}
+	return b, true, nil
+}
+
+// composeFiles are the names Docker Compose looks for, in its own order of
+// preference.
+var composeFiles = []string{"compose.yaml", "compose.yml", "docker-compose.yaml", "docker-compose.yml"}
+
+// composeDuration matches the durations compose accepts for
+// stop_grace_period: a bare number of seconds, or hours, minutes and seconds
+// in that order.
+var composeDuration = regexp.MustCompile(`^(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s?)?$`)
+
+// findComposeServices reports the stop_grace_period of every compose service
+// that runs this repository's own image.
+//
+// Only those services. The grace period of a database or a cache alongside it
+// is not this process's shutdown budget, and reporting one against the other
+// would be arithmetic about two unrelated things.
+func findComposeServices(s *snapshot) ([]stopSite, error) {
+	var out []stopSite
+	for _, name := range composeFiles {
+		raw, ok, err := readRepoFile(s, name)
+		if err != nil {
+			return nil, err
+		}
+		if !ok {
+			continue
+		}
+
+		var root yaml.Node
+		if err := yaml.Unmarshal(raw, &root); err != nil {
+			return nil, fmt.Errorf("%s: %w", name, err)
+		}
+		if len(root.Content) == 0 {
+			continue
+		}
+		services := mapValue(root.Content[0], "services")
+		if services == nil {
+			continue
+		}
+
+		for i := 0; i+1 < len(services.Content); i += 2 {
+			key, service := services.Content[i], services.Content[i+1]
+			if !runsThisRepo(service, s.ModulePath) {
+				continue
+			}
+			site := stopSite{
+				file:    name,
+				line:    key.Line,
+				what:    fmt.Sprintf("service %s, which sets no stop_grace_period,", key.Value),
+				compose: true,
+			}
+			if grace := mapValue(service, "stop_grace_period"); grace != nil {
+				seconds, ok := composeSeconds(grace.Value)
+				if !ok {
+					// A duration this cannot read is left alone rather than
+					// guessed at: compose knows what it means, and inventing a
+					// number here would report against a value nobody wrote.
+					continue
+				}
+				site.line = grace.Line
+				site.what = fmt.Sprintf("stop_grace_period on service %s", key.Value)
+				site.seconds, site.set = seconds, true
+			}
+			out = append(out, site)
+		}
+	}
+	return out, nil
+}
+
+// runsThisRepo reports whether a compose service starts the image this
+// repository builds - by building it, or by naming it.
+func runsThisRepo(service *yaml.Node, modulePath string) bool {
+	if mapValue(service, "build") != nil {
+		return true
+	}
+	image := mapValue(service, "image")
+	if image == nil {
+		return false
+	}
+	repository := image.Value
+	if i := strings.LastIndex(repository, ":"); i > strings.LastIndex(repository, "/") {
+		repository = repository[:i]
+	}
+	return baseName(repository) == baseName(modulePath)
+}
+
+func baseName(path string) string {
+	if i := strings.LastIndex(path, "/"); i >= 0 {
+		return path[i+1:]
+	}
+	return path
+}
+
+func composeSeconds(value string) (int, bool) {
+	m := composeDuration.FindStringSubmatch(strings.TrimSpace(value))
+	if m == nil || m[1]+m[2]+m[3] == "" {
+		return 0, false
+	}
+	var total int
+	for i, unit := range []int{3600, 60, 1} {
+		if m[i+1] == "" {
+			continue
+		}
+		n, err := strconv.Atoi(m[i+1])
+		if err != nil {
+			return 0, false
+		}
+		total += n * unit
+	}
+	return total, true
+}
+
+// mapValue returns the value a mapping node holds for key.
+func mapValue(node *yaml.Node, key string) *yaml.Node {
+	if node == nil || node.Kind != yaml.MappingNode {
+		return nil
+	}
+	for i := 0; i+1 < len(node.Content); i += 2 {
+		if node.Content[i].Value == key {
+			return node.Content[i+1]
+		}
+	}
+	return nil
+}

tools/checksilent/shutdownbudget_test.go

@@ -0,0 +1,484 @@
+package main
+
+import (
+	"strings"
+	"testing"
+)
+
+// hostConfigSource is the part of config/extend.go this check reads: the
+// fallbacks it applies to whatever the settings file leaves out.
+const hostConfigSource = `package config
+
+const (
+	DefaultDrainSeconds   = 0
+	DefaultServerSeconds  = 5
+	DefaultCleanupSeconds = 3
+)
+`
+
+// factorySettings is what this repository ships: 0 + 5 + 3.
+const factorySettings = "settings:\n  extend:\n    shutdown:\n      drain: 0\n      server: 5\n      cleanup: 3\n"
+
+func settingsWith(shutdown string) string {
+	return "settings:\n  extend:\n" + shutdown
+}
+
+// deployWith builds a manifest with the given container extras and pod-level
+// lines, in the shape the shipped one has.
+func deployWith(containerExtra, podExtra string) string {
+	return `---
+apiVersion: v1
+kind: Service
+metadata:
+  name: go-admin
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: go-admin-v1
+spec:
+  replicas: 1
+  template:
+    spec:
+      containers:
+      - name: go-admin
+        image: go-admin
+` + containerExtra + podExtra
+}
+
+func graceOf(seconds string) string {
+	return "      terminationGracePeriodSeconds: " + seconds + "\n"
+}
+
+const preStopSleep25 = `        lifecycle:
+          preStop:
+            exec:
+              command: ["sh", "-c", "sleep 25"]
+`
+
+// The six scenarios worked through in the technical plan, plus the one that
+// only fails when preStop is left out of the sum.
+//
+// The values matter. "raise server to 25" gives 28, which is under a grace
+// period of 30 and reaches only the WARN level - it would not show that the
+// ERROR level works at all.
+func TestShutdownBudgetAgainstTheGracePeriod(t *testing.T) {
+	for _, tc := range []struct {
+		name     string
+		settings string
+		deploy   string
+		want     Severity
+		contains string
+	}{
+		{
+			name:     "the shipped defaults, with headroom",
+			settings: factorySettings,
+			deploy:   deployWith("", graceOf("30")),
+			want:     -1,
+		},
+		{
+			// The one this check exists for: drain is the interesting knob and
+			// the grace period is in another directory, so raising one and not
+			// the other is the natural mistake.
+			name:     "the budget was raised and the manifest was not",
+			settings: settingsWith("    shutdown:\n      drain: 0\n      server: 30\n      cleanup: 3\n"),
+			deploy:   deployWith("", graceOf("30")),
+			want:     Error,
+			contains: "preStop 0 + drain 0 + server 30 + cleanup 3",
+		},
+		{
+			// Equal is not a fit: the grace period is when SIGKILL is sent, so
+			// a budget that ends exactly then leaves nothing time to return.
+			name:     "the grace period was lowered to the budget",
+			settings: factorySettings,
+			deploy:   deployWith("", graceOf("8")),
+			want:     Error,
+		},
+		{
+			name:     "fits, but with nothing to spare",
+			settings: factorySettings,
+			deploy:   deployWith("", graceOf("12")),
+			want:     Warn,
+			contains: "leaves under 5s of headroom",
+		},
+		{
+			// The hook is spent before the process is told anything, so it is
+			// added to the budget rather than overlapping it.
+			name:     "a preStop hook is part of the budget",
+			settings: factorySettings,
+			deploy:   deployWith(preStopSleep25, graceOf("30")),
+			want:     Error,
+			contains: "preStop 25 + drain 0 + server 5 + cleanup 3",
+		},
+		{
+			// The example from the review: 10 + 10 + 5 + 3 against 30.
+			name:     "preStop and a drain window together, just fitting",
+			settings: settingsWith("    shutdown:\n      drain: 10\n      server: 5\n      cleanup: 3\n"),
+			deploy: deployWith(`        lifecycle:
+          preStop:
+            exec:
+              command: ["sleep", "10"]
+`, graceOf("30")),
+			want: Warn,
+		},
+		{
+			name:     "no shutdown section",
+			settings: settingsWith("    rateLimit:\n      inboundQPS: 200\n"),
+			deploy:   deployWith("", graceOf("30")),
+			want:     -1,
+		},
+		{
+			// Nothing to disagree with. A manifest without a grace period gets
+			// the Kubernetes default, which this file cannot see, and guessing
+			// at it would make the check wrong rather than quiet.
+			name:     "the manifest sets no grace period",
+			settings: settingsWith("    shutdown:\n      drain: 300\n"),
+			deploy:   deployWith("", ""),
+			want:     -1,
+		},
+		{
+			// config.Shutdown.Budget refuses this and the server does not
+			// start, so it is not a failure that passes unnoticed - and adding
+			// a negative into the sum would understate it.
+			name:     "a negative budget is left to the run-time refusal",
+			settings: settingsWith("    shutdown:\n      drain: -100\n      server: 5\n      cleanup: 3\n"),
+			deploy:   deployWith("", graceOf("5")),
+			want:     -1,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			root := fixture(t, map[string]string{
+				"config/extend.go":       hostConfigSource,
+				"config/settings.yml":    tc.settings,
+				"scripts/k8s/deploy.yml": tc.deploy,
+			})
+			got := only(t, check(t, root, options{}), checkShutdownGrace)
+			if tc.want < 0 {
+				if len(got) != 0 {
+					t.Fatalf("reported %d findings, want none:\n%v", len(got), got)
+				}
+				return
+			}
+			if len(got) != 1 {
+				// An ERROR also satisfies the WARN condition, so a second
+				// finding here means the two levels were not made exclusive -
+				// and an ERROR that always drags a duplicate WARN behind it
+				// teaches people to skip WARNs.
+				t.Fatalf("reported %d findings, want exactly 1:\n%v", len(got), got)
+			}
+			if got[0].severity != tc.want {
+				t.Errorf("reported %s, want %s: %s", got[0].Severity, tc.want, got[0].Message)
+			}
+			if tc.contains != "" && !strings.Contains(got[0].Message, tc.contains) {
+				t.Errorf("message %q does not contain %q", got[0].Message, tc.contains)
+			}
+			if got[0].File != k8sDeployFile {
+				t.Errorf("reported against %s, want %s", got[0].File, k8sDeployFile)
+			}
+			if want := lineOf([]byte(tc.deploy), "terminationGracePeriodSeconds:"); got[0].Line != want {
+				t.Errorf("reported line %d, want %d", got[0].Line, want)
+			}
+		})
+	}
+}
+
+// A hook whose duration cannot be read is said out loud rather than counted as
+// nothing. It is still spent inside the grace period, and a self-check that
+// silently valued it at zero would be the understatement this check exists to
+// prevent.
+func TestAnUnreadablePreStopIsReported(t *testing.T) {
+	root := fixture(t, map[string]string{
+		"config/extend.go":    hostConfigSource,
+		"config/settings.yml": factorySettings,
+		"scripts/k8s/deploy.yml": deployWith(`        lifecycle:
+          preStop:
+            httpGet:
+              path: /drain
+              port: 8000
+`, graceOf("30")),
+	})
+	got := only(t, check(t, root, options{}), checkShutdownGrace)
+	if len(got) != 1 {
+		t.Fatalf("reported %d findings, want 1:\n%v", len(got), got)
+	}
+	if got[0].severity != Warn {
+		t.Errorf("reported %s, want WARN", got[0].Severity)
+	}
+	if !strings.Contains(got[0].Message, "not a sleep") {
+		t.Errorf("message %q does not say why the hook could not be read", got[0].Message)
+	}
+}
+
+// Either file missing means there is nothing to compare, which is the state
+// every other check's fixture is in.
+func TestShutdownBudgetIsSkippedWithoutBothFiles(t *testing.T) {
+	for _, files := range []map[string]string{
+		{"config/extend.go": hostConfigSource},
+		{"config/extend.go": hostConfigSource, "config/settings.yml": settingsWith("    shutdown:\n      drain: 300\n")},
+		{"config/extend.go": hostConfigSource, "scripts/k8s/deploy.yml": deployWith("", graceOf("30"))},
+	} {
+		root := fixture(t, files)
+		if got := only(t, check(t, root, options{}), checkShutdownGrace); len(got) != 0 {
+			t.Errorf("reported %d findings with only %d file(s):\n%v", len(got), len(files), got)
+		}
+	}
+}
+
+// A tool that cannot find the defaults it is meant to apply has to say so.
+// Reporting nothing would be the failure this whole tool is about: a check
+// that stops checking and goes on printing a clean run.
+func TestShutdownBudgetStopsWhenTheFallbacksAreGone(t *testing.T) {
+	root := fixture(t, map[string]string{
+		"config/extend.go":       "package config\n\nconst DefaultDrainSeconds = 0\n",
+		"config/settings.yml":    settingsWith("    shutdown:\n      drain: 1\n"),
+		"scripts/k8s/deploy.yml": deployWith("", graceOf("30")),
+	})
+	s, err := load(root)
+	if err != nil {
+		t.Fatalf("load: %v", err)
+	}
+	if _, err := runChecks(s, options{}); err == nil {
+		t.Fatal("runChecks succeeded with the fallback constants renamed away")
+	} else if !strings.Contains(err.Error(), serverConstName) {
+		t.Errorf("error %q does not name the missing constant", err)
+	}
+}
+
+// The same arithmetic and the same margin as the manifest check, against the
+// other place a shutdown gets cut short.
+func TestDockerStopAgainstTheShutdownBudget(t *testing.T) {
+	for _, tc := range []struct {
+		name     string
+		script   string
+		want     Severity
+		contains string
+	}{
+		{
+			name:   "explicit and generous",
+			script: "sudo docker stop --timeout 30 \"$PREV\"\n",
+			want:   -1,
+		},
+		{
+			// --time is the deprecated spelling of the same flag and docker
+			// still honours it. A check that could not read it would report a
+			// deadline that exists as missing, and push whoever fixed that
+			// towards a flag that is on its way out.
+			name:   "the deprecated spelling still counts",
+			script: "docker stop --time 30 go-admin\n",
+			want:   -1,
+		},
+		{
+			name:   "the short form counts too",
+			script: "docker stop -t 30 go-admin\n",
+			want:   -1,
+		},
+		{
+			// docker's default is 10 and this process spends 8, so it happens
+			// to work today - and would stop working the first time anybody
+			// configures a drain window, without the command changing.
+			name:     "no deadline at all",
+			script:   "sudo docker stop \"$PREV\" >/dev/null\n",
+			want:     Error,
+			contains: "Pass --timeout 13",
+		},
+		{
+			name:     "shorter than the shutdown",
+			script:   "docker stop --timeout 5 go-admin\n",
+			want:     Error,
+			contains: "allows 5 seconds and this shutdown takes 8",
+		},
+		{
+			// Quoted back in the spelling that was written, so the message
+			// cannot misreport the line it is pointing at.
+			name:     "the message quotes the flag that was used",
+			script:   "docker stop -t 5 go-admin\n",
+			want:     Error,
+			contains: "`docker stop -t 5`",
+		},
+		{
+			name:   "longer than the shutdown but inside the margin",
+			script: "docker stop --timeout=10 go-admin\n",
+			want:   Warn,
+		},
+		{
+			name:   "exactly the margin",
+			script: "docker stop --timeout 13 go-admin\n",
+			want:   -1,
+		},
+		{
+			// The settings file describes `docker stop` in prose right beside
+			// the budget this check reads.
+			name:   "a commented-out command is not one that runs",
+			script: "# docker stop go-admin\n",
+			want:   -1,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			root := fixture(t, map[string]string{
+				"config/extend.go":    hostConfigSource,
+				"config/settings.yml": factorySettings,
+				"scripts/deploy.sh":   "#!/bin/sh\n" + tc.script,
+			})
+			got := only(t, check(t, root, options{}), checkDockerStop)
+			if tc.want < 0 {
+				if len(got) != 0 {
+					t.Fatalf("reported %d findings, want none:\n%v", len(got), got)
+				}
+				return
+			}
+			if len(got) != 1 {
+				t.Fatalf("reported %d findings, want 1:\n%v", len(got), got)
+			}
+			if got[0].severity != tc.want {
+				t.Errorf("reported %s, want %s: %s", got[0].Severity, tc.want, got[0].Message)
+			}
+			if tc.contains != "" && !strings.Contains(got[0].Message, tc.contains) {
+				t.Errorf("message %q does not contain %q", got[0].Message, tc.contains)
+			}
+			if got[0].File != "scripts/deploy.sh" || got[0].Line != 2 {
+				t.Errorf("reported %s:%d, want scripts/deploy.sh:2", got[0].File, got[0].Line)
+			}
+		})
+	}
+}
+
+func composeWith(service string) string {
+	return "version: '3.8'\nservices:\n" + service
+}
+
+// The compose file is the other way this repository's container is stopped -
+// `make run` starts it that way - and it fails identically: the default is ten
+// seconds and it is nowhere near the budget it has to cover.
+func TestComposeStopGraceAgainstTheShutdownBudget(t *testing.T) {
+	for _, tc := range []struct {
+		name     string
+		service  string
+		want     Severity
+		contains string
+	}{
+		{
+			name:    "generous",
+			service: "  api:\n    image: go-admin:latest\n    stop_grace_period: 30s\n",
+			want:    -1,
+		},
+		{
+			name:     "not set at all",
+			service:  "  api:\n    image: go-admin:latest\n",
+			want:     Error,
+			contains: "Set stop_grace_period: 13s",
+		},
+		{
+			name:     "shorter than the shutdown",
+			service:  "  api:\n    image: go-admin:latest\n    stop_grace_period: 5s\n",
+			want:     Error,
+			contains: "stop_grace_period on service api allows 5 seconds",
+		},
+		{
+			name:    "longer than the shutdown but inside the margin",
+			service: "  api:\n    image: go-admin:latest\n    stop_grace_period: 10s\n",
+			want:    Warn,
+		},
+		{
+			// Compose takes hours and minutes as well as seconds, and a check
+			// that only read the digits would call 1m30s ninety times too
+			// short.
+			name:    "minutes and seconds",
+			service: "  api:\n    image: go-admin:latest\n    stop_grace_period: 1m30s\n",
+			want:    -1,
+		},
+		{
+			// A service running something else is not this process, and its
+			// grace period has nothing to do with this budget.
+			name:    "another image is not this application",
+			service: "  db:\n    image: mysql:8\n",
+			want:    -1,
+		},
+		{
+			// Built from this repository, so it is this application whatever
+			// the image ends up being called.
+			name:     "built here rather than named",
+			service:  "  api:\n    build: .\n",
+			want:     Error,
+			contains: "service api, which sets no stop_grace_period",
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			root := fixture(t, map[string]string{
+				"config/extend.go":    hostConfigSource,
+				"config/settings.yml": factorySettings,
+				"docker-compose.yml":  composeWith(tc.service),
+			})
+			got := only(t, check(t, root, options{}), checkDockerStop)
+			if tc.want < 0 {
+				if len(got) != 0 {
+					t.Fatalf("reported %d findings, want none:\n%v", len(got), got)
+				}
+				return
+			}
+			if len(got) != 1 {
+				t.Fatalf("reported %d findings, want 1:\n%v", len(got), got)
+			}
+			if got[0].severity != tc.want {
+				t.Errorf("reported %s, want %s: %s", got[0].Severity, tc.want, got[0].Message)
+			}
+			if tc.contains != "" && !strings.Contains(got[0].Message, tc.contains) {
+				t.Errorf("message %q does not contain %q", got[0].Message, tc.contains)
+			}
+			if got[0].File != "docker-compose.yml" {
+				t.Errorf("reported against %s, want docker-compose.yml", got[0].File)
+			}
+		})
+	}
+}
+
+func TestComposeDurations(t *testing.T) {
+	for _, tc := range []struct {
+		in     string
+		want   int
+		wantOK bool
+	}{
+		{in: "30s", want: 30, wantOK: true},
+		{in: "30", want: 30, wantOK: true},
+		{in: "1m30s", want: 90, wantOK: true},
+		{in: "2m", want: 120, wantOK: true},
+		{in: "1h", want: 3600, wantOK: true},
+		{in: "1h0m30s", want: 3630, wantOK: true},
+		{in: "", wantOK: false},
+		{in: "forever", wantOK: false},
+		{in: "500ms", wantOK: false},
+	} {
+		t.Run(tc.in, func(t *testing.T) {
+			got, ok := composeSeconds(tc.in)
+			if ok != tc.wantOK {
+				t.Fatalf("composeSeconds(%q) ok = %v, want %v", tc.in, ok, tc.wantOK)
+			}
+			if ok && got != tc.want {
+				t.Errorf("composeSeconds(%q) = %d, want %d", tc.in, got, tc.want)
+			}
+		})
+	}
+}
+
+// The command can be written in a workflow or in the Makefile as easily as in
+// a shell script, and a check that only looked at one of them would be quiet
+// about the others.
+func TestDockerStopIsFoundInEveryKindOfFile(t *testing.T) {
+	root := fixture(t, map[string]string{
+		"config/extend.go":           hostConfigSource,
+		"config/settings.yml":        factorySettings,
+		".github/workflows/ship.yml": "jobs:\n  deploy:\n    steps:\n      - run: docker stop app\n",
+		"Makefile":                   "stop:\n\tdocker stop app\n",
+		"scripts/deploy.sh":          "docker stop app\n",
+	})
+	got := only(t, check(t, root, options{}), checkDockerStop)
+	if len(got) != 3 {
+		t.Fatalf("found %d commands, want 3:\n%v", len(got), got)
+	}
+}
+
+func TestLineOfIgnoresComments(t *testing.T) {
+	content := []byte("a: 1\n  # terminationGracePeriodSeconds: 99\n  terminationGracePeriodSeconds: 30\n")
+	if got := lineOf(content, "terminationGracePeriodSeconds:"); got != 3 {
+		t.Errorf("lineOf = %d, want 3 - a commented-out key is not the setting", got)
+	}
+}