Files
go-admin/common/storage/initialize.go
T
zhangwenjian 84bd87dcc9 fix🐛: drain the queue on the way out instead of abandoning it
Nothing stopped the queue when the process exited. core v2.7.0 made the
drain work - Memory.Shutdown closes the queue and waits for every
consumer to finish what it holds, and the legacy adapter cancels its
context and closes the underlying queue - but no call site ever reached
it. The only Shutdown() in this repository applies to the previous
adapter during a reload, so the installed one was simply left. The login
log, the operation log and the API sync all publish through it, so a
rolling restart dropped whatever had not been consumed, on the path
where the process exits 0 and reports "Server exiting".

Setup now registers a BeforeExit callback that shuts down the adapter
this package installed.

Three things it has to get right, each with a test.

It reads `installed` when it runs, not when it registers. A reload
replaces the adapter, and the one from start-up is a queue nobody has
published to since.

It never goes through sdk.Runtime.GetQueueAdapter. That accessor never
returns nil - with no queue section configured it wraps the runtime's
own fallback - so it would look like it worked while closing a queue
this package neither built nor started. That is the same trap setupQueue
already had to drop an `if q != nil` for.

It registers once. Setup is re-run on every configuration change, and a
callback per reload would leave the shutdown phase holding a row of
identical entries, each eligible to be named as the one that overran the
budget.

That last one needed a seam. shutdownQueue takes the adapter on its
first run, so the second and third callbacks find nothing and return -
three registrations produce exactly the same observable result as one,
and a test going through the effect passes either way. It did: the
counter-proof for "register on every reload" came back green until the
registration was counted at the seam instead.

The wait is bounded here rather than left to the phase. Shutdown takes
no context, so a consumer that never finishes would hold the process
until SIGKILL; the callback gives up and says what is being lost, which
the phase's generic overrun message cannot.

Ordering falls out of the phase rather than being arranged: callbacks
run in reverse registration order, this one registers during setup and
the job scheduler's registers on AfterListen, so the schedulers stop
before the queue drains. Verified against core v2.7.0 rather than read
off the source.

Closes #911.
2026-09-07 17:10:13 +08:00

197 lines
7.0 KiB
Go

/*
* @Author: zhangwenjian
* @Date: 2025/04/13 22:03
* @Last Modified by: zhangwenjian
* @Last Modified time: 2025/04/13 22:03
*/
package storage
import (
"context"
"log"
"sync"
"github.com/go-admin-team/go-admin-core/v2/captcha"
corelog "github.com/go-admin-team/go-admin-core/v2/logger"
"github.com/go-admin-team/go-admin-core/v2/sdk"
"github.com/go-admin-team/go-admin-core/v2/sdk/config"
)
// Setup 配置storage组件
func Setup() {
setupCache()
setupCaptcha()
setupQueue()
registerQueueDrain()
}
func setupCache() {
cacheAdapter, err := config.CacheConfig.Setup()
if err != nil {
log.Fatalf("cache setup error, %s\n", err.Error())
}
sdk.Runtime.SetCacheAdapter(cacheAdapter)
}
func setupCaptcha() {
captcha.SetStore(captcha.NewCacheStore(sdk.Runtime.GetCacheAdapter(), 600))
}
var (
queueMu sync.Mutex
// installed is the adapter setupQueue built, kept so the next reload can
// shut it down, and counted so a consumer can tell one from the next.
installed interface{ Shutdown() }
installedGen uint64
// drainRegistered records that the BeforeExit callback is on the runtime,
// so that a reload does not add another one.
drainRegistered bool
)
// setShutdown is sdk.Runtime.SetShutdown, indirected so that registering can
// be observed.
//
// It has to be: the runtime does not report how many callbacks a phase holds,
// and shutdownQueue takes the adapter on its first run, so every registration
// after the first returns immediately and changes nothing anybody can see. A
// reload adding one callback per round would therefore be invisible from the
// outside - which is exactly how it would survive.
var setShutdown = func(f func(context.Context)) { sdk.Runtime.SetShutdown(f) }
// registerQueueDrain puts shutdownQueue on the BeforeExit phase, once.
//
// Setup is one of the callbacks bootstrap.SetupConfig re-runs on every
// configuration change, so registering from it without a guard would leave one
// callback per reload - each shutting down the same adapter, each reported
// separately when the budget runs out.
//
// A flag under the existing mutex rather than a sync.Once: the tests in this
// package already save and restore installed and installedGen to keep one test
// from deciding what the next one sees, and a sync.Once cannot be put back.
func registerQueueDrain() {
queueMu.Lock()
first := !drainRegistered
drainRegistered = true
queueMu.Unlock()
if first {
setShutdown(shutdownQueue)
}
}
// shutdownQueue drains the queue this package installed, on the way out.
//
// Nothing used to. core's Memory.Shutdown closes the queue and waits for every
// consumer to finish what it is holding, and the legacy adapter cancels its
// context and closes the underlying queue - but neither ran at exit, so the
// process left with the login log, the operation log and the API sync still
// buffered, and left reporting success.
//
// The adapter is read here rather than captured at registration because a
// reload replaces it. Registration happens once per process; this runs against
// whatever is current when the signal arrives.
//
// Only an adapter this package installed. sdk.Runtime.GetQueueAdapter never
// returns nil - with no queue section configured the runtime wraps its own
// fallback queue - so going through that accessor would shut down a queue this
// package neither built nor started.
//
// The adapter is taken, not read: after this the package owns nothing, so a
// reload arriving mid-shutdown builds a new one instead of being handed a
// closed one to shut down again. Both implementations tolerate a second
// Shutdown, so this is about who owns it rather than about a crash.
func shutdownQueue(ctx context.Context) {
queueMu.Lock()
q := installed
installed = nil
queueMu.Unlock()
if q == nil {
return
}
done := make(chan struct{})
go func() {
defer close(done)
q.Shutdown()
}()
select {
case <-done:
corelog.Info("queue: drained")
case <-ctx.Done():
// The wait is what the budget bounds, not the work: Shutdown takes no
// context and is still running in that goroutine. Saying so here names
// what is being lost, which the generic overrun message cannot.
corelog.Warnf("queue: the shutdown budget ran out while the queue was still draining - " +
"whatever it had not delivered goes with the process. Raise extend.shutdown.cleanup " +
"if this recurs.")
}
}
// QueueGeneration reports how many times this package has installed a queue
// adapter. It changes every time setupQueue builds a new one, which is on
// every configuration reload, and stays 0 for as long as the configuration has
// no queue section at all - in which case nothing is installed and callers are
// working with the runtime's own fallback queue.
//
// It exists because there is no way to ask for the adapter's identity from the
// outside. sdk.Runtime.GetQueueAdapter and GetQueuePrefix build a fresh
// runtime.Queue wrapper on every call, so comparing what two calls return
// compares two wrappers and never matches, however many times the underlying
// adapter has been replaced. This package creates the adapter, so this is the
// only place that knows. A counter rather than the adapter itself keeps the
// comparison on a uint64: an adapter type that is not comparable would panic
// an `==` between two interface values.
func QueueGeneration() uint64 {
queueMu.Lock()
defer queueMu.Unlock()
return installedGen
}
func setupQueue() {
if config.QueueConfig.Empty() {
return
}
queueMu.Lock()
defer queueMu.Unlock()
queueAdapter, err := config.QueueConfig.Setup()
if err != nil {
log.Fatalf("queue setup error, %s\n", err.Error())
}
previous := installed
sdk.Runtime.SetQueueAdapter(queueAdapter)
installed = queueAdapter
installedGen++
// The previous adapter goes down after the new one is installed, not
// before. Shutdown waits for its consumers to deliver what it still holds,
// and for that whole wait the runtime would otherwise be handing producers
// a queue that has stopped accepting: every Append in the window comes back
// ErrQueueClosed, and both call sites in common/middleware log it. Swapping
// first leaves no such window - a producer gets the new queue or the old
// one, and both work.
//
// Only an adapter this package installed. GetQueueAdapter never returns
// nil - with nothing configured the runtime falls back to its own memory
// queue and wraps that - so the `if q := GetQueueAdapter(); q != nil` this
// replaces was always true, and shut down the fallback queue on the very
// first start, before anything had used it.
if previous != nil {
previous.Shutdown()
}
// Deliberately not started here. Run has to come after the consumers have
// registered: the contract implementations refuse a registration once the
// queue is running (storage.ErrQueueAlreadyStarted), and the legacy
// adapter this repository still goes through swallows that error rather
// than reporting it - its own comment says the interface gives it no way
// to tell the caller. Starting here and registering afterwards is
// therefore a race that loses consumers in silence. Whoever registers is
// the one that starts it.
}