The wait was one select over done and ctx.Done(). Both can be ready when
it runs, select picks at random among ready cases, and so a queue that
drained in the same instant the budget expired was reported as an
overrun about half the times it landed there - often enough to be read
as noise, and pointing at the wrong thing when it was not. core's own
RunShutdown re-checks for this reason; this did not.
The tie-break is now a function taking channels rather than a queue,
which is what lets a test hand it a closed done and an expired ctx
together. That state is the whole of the bug and cannot be arrived at
reliably from the outside; over 1000 iterations the single-select
version fails, and the second look does not.
The test for giving up on the deadline read the call counter straight
after shutdownQueue returned, while Shutdown runs on a goroutine nobody
joins. It passed because the goroutine is scheduled promptly, not
because anything ordered the two. The fake now signals that Shutdown has
been entered and the test waits for it.
Both raised by Copilot on #918.
The first attempt at the tie-break test was wrong and is not what
landed: it asserted that an immediately-returning Shutdown always counts
as drained under an already-expired context, which is not true and
should not be - if the goroutine has not run, nothing has drained. That
test failed, correctly. What is being claimed is narrower: when both are
ready, done wins.
Nothing stopped the queue when the process exited. core v2.7.0 made the
drain work - Memory.Shutdown closes the queue and waits for every
consumer to finish what it holds, and the legacy adapter cancels its
context and closes the underlying queue - but no call site ever reached
it. The only Shutdown() in this repository applies to the previous
adapter during a reload, so the installed one was simply left. The login
log, the operation log and the API sync all publish through it, so a
rolling restart dropped whatever had not been consumed, on the path
where the process exits 0 and reports "Server exiting".
Setup now registers a BeforeExit callback that shuts down the adapter
this package installed.
Three things it has to get right, each with a test.
It reads `installed` when it runs, not when it registers. A reload
replaces the adapter, and the one from start-up is a queue nobody has
published to since.
It never goes through sdk.Runtime.GetQueueAdapter. That accessor never
returns nil - with no queue section configured it wraps the runtime's
own fallback - so it would look like it worked while closing a queue
this package neither built nor started. That is the same trap setupQueue
already had to drop an `if q != nil` for.
It registers once. Setup is re-run on every configuration change, and a
callback per reload would leave the shutdown phase holding a row of
identical entries, each eligible to be named as the one that overran the
budget.
That last one needed a seam. shutdownQueue takes the adapter on its
first run, so the second and third callbacks find nothing and return -
three registrations produce exactly the same observable result as one,
and a test going through the effect passes either way. It did: the
counter-proof for "register on every reload" came back green until the
registration was counted at the seam instead.
The wait is bounded here rather than left to the phase. Shutdown takes
no context, so a consumer that never finishes would hold the process
until SIGKILL; the callback gives up and says what is being lost, which
the phase's generic overrun message cannot.
Ordering falls out of the phase rather than being arranged: callbacks
run in reverse registration order, this one registers during setup and
the job scheduler's registers on AfterListen, so the schedulers stop
before the queue drains. Verified against core v2.7.0 rather than read
off the source.
Closes#911.
Shutdown now waits for its consumers to deliver what the queue still holds, and
setupQueue called it first. For that whole wait sdk.Runtime still pointed at the
adapter that had stopped accepting, so every Append landing in the window came
back ErrQueueClosed - and both call sites in common/middleware log that at error
level while the row never reaches the database.
Measured with a held consumer: 177 of 178 publishes during one reload.
Installing first leaves no window. A producer fetches the adapter per call and
gets either the new queue or the old one, and both accept; the old one still
drains, because Shutdown is what waits for that.
The difference only exists during the wait - after Setup returns the two orders
look identical, which is why the test holds a consumer and publishes throughout
the reload rather than checking the state afterwards.
The counter-proof compiles and reports the 177.
setupQueue ended with `go queueAdapter.Run()`, and the three log consumers were
registered afterwards, from setup(). The contract implementations refuse a
registration once the queue is running - memqueue and the redis queue both
answer storage.ErrQueueAlreadyStarted - and Register cannot report it: it
returns nothing, which its own comment in core records as the reason the
interface is deprecated. Start first and register second, across two
goroutines, and the registration is dropped without the caller being able to
tell.
What follows is not quiet. No consumer group was created, so redis refuses
every later publish with storage.ErrNoHandler, and go-admin logs that at error
level from both call sites while the login and operation log rows are simply
never written. The silence is in the registration; the cost shows up on every
request after it.
Which implementation is behind the interface depends on the configuration.
config.QueueConfig.Setup returns queue.NewMemory directly when there is no
redis section - and that one does not care about the order, because its
Register just starts another consumer goroutine. Only a redis section reaches
storage.LegacyQueueAdapter, which wraps the contract implementation and
therefore refuses. So the defect is invisible in the default deployment and
shows up only where redis is configured, dropping the login log, the operation
log and the api check - the three things #892 was about.
The start therefore moves to the code that registers, and nothing starts the
queue but that.
The registration also moves onto AfterResource. It has to: a reload rebuilds
the adapter, and consumers attached to the one that existed at start-up are
attached to a queue nobody publishes to any more. Being on that phase means
running again on every reload, so the callback is idempotent with respect to a
given queue rather than "does nothing the second time" - registering twice on
the same queue would give every message two consumers and write every log row
twice.
Identity for that comes from common/storage, where the adapter is built, as a
generation counter. It cannot come from the accessors: GetQueueAdapter and
GetQueuePrefix build a fresh runtime.Queue wrapper on every call, so comparing
two of them compares two wrappers and never matches however many times the
adapter underneath has been replaced. A counter also keeps the comparison on a
uint64 rather than an `==` between two interface values, which would panic on
an adapter type that is not comparable.
Generation 0 means the configuration has no queue section, so nothing was
installed and the runtime hands back its own memory queue. That case still gets
consumers, because the registration this replaces was unconditional and
dropping it would stop the logs for anyone who commented the section out.
Two things fixed on the way past:
- `if q := sdk.Runtime.GetQueueAdapter(); q != nil { q.Shutdown() }` was
always true. GetQueueAdapter never returns nil - with nothing configured
the runtime falls back to its own memory queue and wraps that - so the
first start shut down the fallback queue before anything had used it. Only
an adapter this package installed is shut down now.
- config.Setup becomes bootstrap.SetupConfig, which is what announces
AfterResource, and announces it after the callbacks that build the
resources rather than before.
attachConsumersOnce is split out so the order and the once-ness can be checked
against a queue the test controls; neither can be read back out of a real
adapter. Four tests cover the ordering, both directions of the idempotency
rule, and the unconfigured case. Both counter-proofs compile and fail: calling
Run before the registrations reports each of the three as "came after Run", and
dropping the generation guard reports eight calls where four are wanted.
One honest limit: the counter-proof for the ordering makes Run synchronous.
The original arrangement started the queue on another goroutine, and a race
cannot be made to fail every time - which is the reason the order is enforced
by structure here instead of being left to be noticed in use.
Every import of the module changes, not only the seven packages that
moved out of sdk/pkg: Go requires the major version in the path from v2
on. Both happen in one pass —
go run github.com/go-admin-team/go-admin-core/tools/coreupgrade@v2.0.0 -w -v2 .
go mod tidy
— which is the command the release notes give, run here as a consumer
would run it. 210 imports across 95 files.
The compatibility shims this used are gone in v2, so the paths that
moved had to move: sdk/pkg/captcha, sdk/pkg/jwtauth and its user
package, sdk/pkg/response and sdk/pkg/casbin.
The count of unformatted files is unchanged at 34, none of them touched
by this: the tool reformats a file only if it was already gofmt clean,
so a migration cannot disappear into whitespace.
The pinned core dated from April, before sdk stopped being a separate module,
so the build resolved sdk packages from the old module and core packages from
the new one. Dropping the separate requirement is what makes the two agree
again.
Most of the diff is renames that came with that: the tenant accessors gained a
ByTenant suffix, GetDb now returns one database and GetAllDb the map, and
casbin moved to v3.
The change that matters is four call sites moving from GetMemoryQueue to
GetQueuePrefix. GetMemoryQueue returns a queue fixed at construction, so the
login log, the operate log and the api check ran in process no matter what the
settings file selected — a second instance saw none of it. GetQueuePrefix
returns whatever the configuration built, which is the point of being able to
configure a queue at all.
Verified against core at main: build and vet clean. The two file_store failures
are unchanged from before this branch; they need cloud credentials.
zhangwenjian