The three budgets are spent one after the other, so what has to fit inside the
orchestrator's grace period is their sum - and nothing said what that sum was.
Working it out meant reading a configuration file, remembering which fields
were absent, and knowing what each one falls back to.
Start-up now prints it: the three values and the total, taken from the resolved
budget rather than from the file. A field left out still costs its default, so
adding up what was written down understates the total by exactly the fields
nobody wrote - which is the arithmetic somebody doing this by hand gets wrong.
Whether the total fits is a separate question, and the framework cannot answer
it alone: `docker stop` allows ten seconds and Kubernetes thirty, three times
apart. A fixed threshold would have warned about the manifest this repository
is about to ship. So extend.shutdown.grace is optional, nothing reads it during
a shutdown, and when it is absent the line says so and prints both reference
values instead of judging.
When it is set and the budget does not fit, the warning names the shortfall:
how many more seconds are needed. A minimum, not a target - this is somebody
else's deployment under constraints this process cannot see, and asking them to
leave headroom on top is not this line's business. Equal does not fit either;
the grace period is when SIGKILL is sent, so a budget that ends exactly then
leaves the last callback no time to return.
/ready has failed from the moment shutdown begins since the readiness probe was
added, and the order it does that in is right: reversed, the state would be
reported after the connections were already cut. But order alone does not
produce a window. Nothing waited between the flip and Shutdown, so the two were
microseconds apart, and a poller on a multi-second interval never saw the 503 -
it saw a refused connection, which is the thing the probe was supposed to
avoid. Polling a container through a SIGTERM on the demo host recorded exactly
that: 200, then connection refused, and no 503 in between.
extend.shutdown.drain is that wait. The process keeps serving normally for it -
answering requests, not refusing them, because refusing them would move the
outage earlier rather than avoid it - and only then closes the listener.
It is zero by default, so nothing changes for a deployment that does not ask
for it. That is not timidity: the budgets are spent one after another, and a
non-zero default would push every existing shutdown closer to the orchestrator's
grace period, where being cut off part-way through the cleanup callbacks is
worse than never draining at all.
Keep-alive is switched off with the flip. The server keeps connections alive
until Shutdown sets shuttingDown() itself, so without this the pooled
connections a balancer holds would sit untouched for the whole window and be
cut at the end of it anyway - the cost of the window without its benefit. This
is the switch Shutdown flips, moved earlier by the window's length.
The signal disposition is restored after the window rather than on the first
signal. Before there was a window, the interval where a second signal killed
the process outright was only reachable while a cleanup callback hung; putting
a multi-second wait inside it would have made every ordinary shutdown
interruptible for the length of the drain. A second signal during the window is
taken by the channel and ends the window early instead - somebody sending
another kill wants this over with sooner - and the escape hatch comes back the
moment the window does.
What the window is worth depends on who removes this instance. A balancer that
polls /ready acts on the 503 and needs the window to cover its check interval
times its failure threshold; a Kubernetes Service withdraws the endpoint when
the Pod is deleted, concurrently with SIGTERM and regardless of what the probe
returns, and there the window covers the delay in that removal reaching every
node. The three comments that used to say a balancer "has a chance to" take the
instance out said it without either qualification, which is how a claim comes to
be repeated after a live test has refuted it.
The subprocess test polls the real probes on a connection it opens after the
signal - a reused one can be served after the listener is closed, which would
let this pass against a shutdown that had already broken it - and asserts on the
draining answer in the body, not on the status code. With no database the status
is 503 from start-up, so a status-code assertion would hold even with
BeginDraining deleted. Two window lengths, because one proves only that
something takes that long.
How long a shutdown may spend waiting for in-flight requests, and how long the
cleanup callbacks get after that, were compile-time constants. The two together
have to fit inside whatever grace period the orchestrator allows before it
sends SIGKILL, and that number is not the same everywhere - `docker stop`
allows ten seconds, Kubernetes thirty by default - so the one deployment shape
these constants suited was the one they were written for.
They now come from extend.shutdown, beside rateLimit. Not from application:
that section is a fixed struct in core, and the decoder discards keys it has no
field for without an error, so a budget written there would be accepted and
never applied. That is the failure this whole change is about, and putting the
configuration where it cannot be read would have reproduced it.
Both fields are pointers, following RateLimit.InboundQPS: nil means "not
configured" and takes the default, and a number that was written down is spent
literally, zero included. Without that separation `server: 0` - do not wait for
in-flight requests at all, which is a reasonable thing to ask when the grace
period is very short - could not be expressed, and the section would need a
paragraph explaining which zeros mean what.
A negative is refused rather than clamped. Correcting a value quietly is the
same failure in a different costume, and Budget returns the error instead of
ending the process so that the rule can be tested without a subprocess.
The defaults live in config as seconds and in cmd/api as durations, both from
the same constants, and a test asserts the two agree - a deployment that
configures nothing is entitled to one answer about what it spends, not two.
The last test loads the two settings files this repository ships through the
real loader and asserts the section arrives with the documented values. Nothing
weaker can tell "the key is read" from "the key is discarded": the struct
compiles either way.
Two settings that decide whether a deployment survives load, neither of which
appeared in any template.
The connection pool. Left unset, Go's defaults apply, and MaxIdleConns is 2:
under load almost every request opens a TCP connection and closes it again,
local ports run out, and the process answers "can't assign requested address"
to everything. Not slower - unavailable. A sweep against MySQL collapsed to
zero successful responses at 64 concurrent requests without these, and served
13,846 req/s with no errors once they were set.
The queue buffer. poolSize is the point at which messages start being dropped,
not a tuning knob: a full queue discards the message and returns an error
rather than blocking, and each stream has one consumer goroutine writing to the
database. At the previous default of 100 a load test lost over 60% of them; at
1000, none. Login and operation logs travel this queue, so what gets lost is
audit data - though only when logger.enableddb is on.
Both carry the reasoning in the file, because the failure mode of each is
invisible until it happens in production.
A rejected request answered 200 with the failure only in the body, so every
layer that reads the status line counted it as served: load balancers, metrics,
client-side retry. A load test against this reported the limiter's own
rejections as successful traffic and overstated throughput more than tenfold.
The threshold was a constant in the middleware, which made 200 QPS the ceiling
of every deployment with nothing in the configuration to reveal it. It now
reads extend.rateLimit.inboundQPS; an absent value keeps 200, so an upgrade
changes nothing, and zero disables the limiter for a deployment behind its own
gateway.
Also drops Strategy: system.BBR. Reading sentinel's source, the adaptive
strategy is consulted only for Load and CpuUsage - for InboundQPS the trigger
count is compared directly - so it read as if the limit adapted to the machine
when it never did.
The sample had producer/consumer nested keys (streamMaxLength,
approximateMaxLength, visibilityTimeout, bufferSize, concurrency,
blockingTimeout, reclaimInterval) that don't exist on config.RedisQueue —
checked against sdk/config/queue.go, which only reads addr, password, and the
embedded RedisOptions fields, plus group, key_prefix and max_attempts. Filling
in the old sample as written would compile and start fine, since it's YAML
under a key the struct doesn't declare, and every one of those settings would
be silently ignored.
Replaced with the fields the struct actually has. Still commented out —
redis stays opt-in, this only fixes what filling it in would produce.
zhangwenjian