app-order registered its migrations and its menus and nothing else, so
`migrate install order` answered that no application in the binary registers a
manifest. Which was true, and made the installer untestable against the one
application this repository ships.
The manifest goes in the migration package rather than one of its own because
that is the package a host has to import for the application to exist at all -
its migrations register from there too. A second package would be a second
thing to remember to import, and forgetting it would leave an application
whose migrations run and which no installer can name.
Its Version is not the migration version and the two move independently:
adding a migration file without renaming the application is normal, and so is
a release that changes no schema. The migration versions decide what runs;
this decides what the installed row says.
go-admin-core moves to v2.8.0, which is where contract/app lives.
The fourth registered driver, and the one that disagrees with the other three
about NULL. A suite that never pointed at it reported success for a migration
no SQL Server database could apply - the same shape as the PostgreSQL gap that
put the postgres service here, one driver further along.
SQL Server has no equivalent of POSTGRES_DB, so the database the DSN names is
created in a step before the tests. The test helper fails rather than skips
when CI is set and the variable is not, so dropping the service or renaming
the variable cannot quietly go green.
1786700008000 could not be applied to any SQL Server database. Not an old one
with awkward data - any of them, including an empty one:
Msg 1505 ... duplicate key ... The duplicate key value is (, <NULL>, 0).
MySQL, PostgreSQL and SQLite treat two NULLs in a unique index as different
values, so any number of rows missing a seed_code coexist under
uk_sys_menu_app_seed_code_del. SQL Server treats them as equal and permits
exactly one. 1786700001000 seeds five menus and none of them has a seed_code,
so the second one already collides with the first. sys_api's index has the
same shape over two nullable columns, path and action.
On SQL Server the index is now filtered to the rows that carry a value, which
is what the other three engines do by not comparing their NULLs. The filter is
not added elsewhere: MySQL has no filtered index at all, and on PostgreSQL and
SQLite it would only restate what those engines already do.
Nothing that has applied this migration is affected, and no SQL Server
database can have.
Verified against SQL Server 2022. The migration completes; the filtered index
still rejects a second (order, dir) and still lets another app reuse "dir",
so filtering removed the NULL rows from the index rather than the index's
teeth. Two degradations turn that red: dropping the filter, and naming only
path in sys_api's - the second one needed a fixture row with a path and no
action, because rows missing both are excluded either way and the first
attempt at that degradation came out green.
There is also a control test asserting the unfiltered statement still fails on
this engine, so the first test is passing because of the fix rather than
because SQL Server turned out not to mind.
Two comments added in this branch cite docs-prd/008-.../数据库变更.md by path.
That directory is not tracked here, so the citation reads as a file the reader
can open and cannot. The reasoning it pointed at is short enough to state in
place.
Three comments from the previous batch cite the same path and are left alone;
they belong to a different change.
Copilot could not review this branch - the account is over its review quota -
so these are what a second pass over the uninstaller turned up. No defect: the
three cases were uncovered rather than wrong.
findOrphanPolicies batches its OR chain because a driver runs out of
placeholders long before an application runs out of endpoints, and nothing
exercised the boundary. 205 paths across three batches, the last one short,
plus one policy no key names as a control. Taking one fewer per batch,
advancing one too far, and stopping after the first batch each turn it red.
An application with apis and no menus, and one with menus and no apis, are
both normal - endpoints another service calls, or a section with no endpoints
of its own - and each leaves one of the two id lists the uninstall reads
empty.
That last pair also corrected a comment. The guard in front of the join-table
delete was described as being there because an empty IN list is a syntax
error. It is in raw SQL, but GORM renders IN with an empty slice as a
condition that matches nothing, and removing the guard leaves the new test
green. It stays as a statement of intent, and now says so.
`migrate uninstall <code>` removes what an application's install wrote and
leaves the application's own tables alone. Removing an order module is not the
same decision as destroying the orders, and nothing here can tell an operator
who is done with it from one who will reinstall tomorrow.
One transaction, and this one really is one: every statement is DML or a
SELECT, so unlike an install there is no DDL to commit it out from under
itself. Child rows go first, while the ids that identify them can still be
read from their parents, and the api paths are read before the rows carrying
them are deleted.
The two join tables need no ledger and get none. menu_id is a surrogate key,
so a sys_role_menu or sys_menu_api_rule row can only have come from a menu
this application wrote - there is no "looks like it but is not". A column on
sys_role_menu would have been worse than unnecessary: SysRole.Update deletes a
role's rows and writes them back through GORM's many2many, which does not
carry extra columns, so the column would be blanked the first time anybody
edited a role, silently. There is a test that performs that edit and then
uninstalls.
casbin_rule is the opposite case, because its key is business text somebody
else may have written for their own reasons. Policies are removed one at a
time, by exact tuple, and only the ones the ledger says this install created.
A tuple the ledger names that is no longer there is reported, not treated as a
failure - the uninstall wanted it gone and it is gone. Then, with everything
the ledger could speak for already dealt with, a read-only pass lists the
policies still naming this application's paths: those are grants somebody made
by hand, they are about to point at APIs that no longer exist, and they are
not this command's to delete. The two lists stay separate because they mean
different things - one is something of ours that had already gone, the other
is somebody else's grant now pointing at nothing - and merged into one "could
not remove" list neither would be actionable.
sys_migration's rows for the application go too. Without that a reinstall
finds every version applied, runs no migration, seeds nothing, and reports
success. It is the easiest step to leave out, because a migration record does
not look like the application's data.
A sys_app row is not required. `migrate` with no subcommand applies every
registered migration, an application's included, so an application can have
all of its rows without ever having gone through the installer - and that is
the case where nothing else can clean up after it.
Eleven degradations were applied one at a time, each red on the assertion it
was aimed at: skipping either join table, deleting sys_role_menu without its
filter, skipping sys_migration, deleting sys_migration without its filter,
matching policies by path instead of by ledger tuple, dropping the orphan
pass, treating a missing policy as a failure, leaving the ledger behind, soft
deleting sys_menu instead of removing it, and running the whole thing outside
a transaction.
uk_sys_menu_app_seed_code_del covers (app_code, seed_code, deleted_at) and is
created by 1786700008000 with explicit SQL. The struct tag named the same
index on SeedCode alone, and a named uniqueIndex tag collects only the fields
carrying that name - so AutoMigrate on this model would build a unique index
on seed_code by itself: stricter than the real one, and forbidding two
applications from both having a "dir" node, which the composite key exists to
allow.
Worse than being stricter, it would win. The migration only creates its index
when HasIndex says the name is free, so a schema built by AutoMigrate first
keeps the wrong index and the migration steps over it without a word.
The tag cannot express the real index: deleted_at comes from the ModelTime
embed shared by every table, which no single model can add a tag to. So the
tag goes and the migration is the only thing that creates it.
No database is affected. The initial table migration AutoMigrates a frozen
snapshot of this model that has neither app_code nor seed_code, and nothing
else in the repository AutoMigrates the live one - which is why this stayed
invisible until a test built the schema from the live model and seeded two
applications, and got a unique-constraint failure on a seed code they are
supposed to be able to share.
sys_app_casbin_grant has existed since the registry tables were added and
nothing ever wrote to it. An uninstaller reading it would have found it empty,
deleted no policy at all, and reported every one of them as an unattributable
leftover - which is what "report and skip" looks like when the ledger was
simply never written, and is indistinguishable from it working.
grantToAdminRole now writes an entry for each policy it creates. The entry
carries the tuple casbin_rule is unique on rather than a foreign key into it,
because casbin_rule is not this project's table: the gorm adapter's SavePolicy
truncates it and writes it back from memory, and SysRole.Update replaces a
role's policy rows wholesale. Both rebuild the same tuple from the same
sys_menu/sys_api data, so a match on the tuple survives what a row id does
not.
Only policies this install actually created are recorded - the insert is
conditional and its RowsAffected says which. A policy that was already there
was granted by somebody else and is not this app's to take away.
The two ways that can be wrong are not equally bad, which is what settles it.
Under-recording leaves a policy behind and the uninstall says so, because a
policy naming this app's own path with no ledger entry is exactly what it
reports as an orphan. Over-recording deletes somebody's authorization,
silently. Between a visible leftover and an invisible deletion, take the
leftover.
The ledger insert is itself conditional, for a case the obvious retry test
does not reach: on a plain re-run the policy still exists, so the insert is
skipped before the ledger is touched. It is reached when the policy row was
removed while its entry stayed, and a plain insert would then abort the whole
seed on the ledger's unique index. There is a test for that specific shape,
and replacing the insert with a plain one turns it red - which the plain retry
test does not.
Ordering: the ledger table is created by a framework migration, and version
strings sort bare digits ahead of any app-prefixed one, so it exists before
any application's seed runs. Nothing in the framework's own migrations calls
SeedMenus.
`migrate install <code>` brings one application up to the version its manifest
declares: it runs that application's outstanding migrations and records what
it did in sys_app.
It goes under migrate rather than under the existing `app` command, which
already means "generate the skeleton of a new application" - a directory that
does not exist yet, not an application already compiled into this binary.
Installing one is running its migrations, which is what this command is, so
--domain, resolveDB and the guard that refuses a mistyped code instead of
reporting a successful no-op are all already here.
Three phases, each committing on its own, and they are not one transaction.
An application's versions are separate migration files, and on MySQL a DDL
statement commits the transaction around it - destroying an outer transaction
and every savepoint taken from it. So this does not promise that a
half-installed application cannot happen. It promises one is visible when it
does: phase A writes "installing" before anything that can fail, phase B runs
the migrations, phase C turns that into "installed" or into "failed" with the
version it stopped on.
What is left to apply comes from sys_migration, never from sys_app. sys_app
is a derived view - a summary, and the answer to "which version does this app
think it is at". If it were the authority, an operator who deleted
sys_migration rows by hand would be told an application is installed while its
schema is not, which is worse than not knowing. So "already installed, nothing
to do" needs all three: nothing outstanding, recorded as installed, and the
same version. A row stuck at "installing" - what it reads as after the process
was killed partway - is not installed, and retrying is just running the
command again.
An upgrade is in place and keeps the first install's time; a downgrade is
refused, and refused before phase A writes anything, so a refusal cannot cost
the operator the row that told them what they had. An unparseable recorded
version is refused the same way, while it is still readable.
The report ends by saying the code is not running yet. That is not a
pleasantry: Go links at build time and Vite resolves its import globs at build
time, so installing an application writes its menus, its APIs and its
permissions and cannot make one line of its code run - and the menus appearing
is exactly what makes an operator believe otherwise.
Ten degradations were applied one at a time to check the tests name the
behaviour rather than the shape: deciding the no-op from sys_app alone,
always writing installed_at, allowing the downgrade, keeping the previous
attempt's diagnostics on a row that now says installed, treating "installing"
as installed, truncating last_error by bytes so a Chinese message is cut
mid-rune, not recording the failure at all, skipping code normalization, and
writing phase A before either the downgrade or the version-parse refusal.
Each went red on the assertion it was aimed at. An eleventh was discarded
rather than counted: it failed in the first install's setup, not on the claim.
run() called log.Fatalf on the first migration that failed, which ended the
process from inside the migration engine. Nothing above it could record what
happened - an installer needs to write down which version an attempt stopped
on - and no test could exercise a failing migration at all without taking the
test binary with it, which is why the one test that covers a failed migration
drove the registered function directly and left the scheduler uncovered.
run(), Migrate() and MigrateApp() now return an error, and the exit moved to
the command layer where the exit code is the command's business.
Two of those errors say more than "it failed". A migration that fails comes
back as a *VersionFailure naming the version, because an installer records
that as a diagnostic snapshot - the authoritative answer to where a retry
resumes is always recomputed from sys_migration, never read back, and asking
the database what is still pending answers a different question that merely
has the same answer most of the time. An app code nothing registered under is
now an error rather than a log line, so an installer asking for one app by
name cannot be told that installing an app that does not exist succeeded; the
command layer still rejects a typo before any database work.
exitOnError is what makes the command exit non-zero, and it covers more than
it replaces. Every path out of migrateModel used to return without an exit
code: an unreachable tenant database or a failed AutoMigrate printed a line
and exited 0, so a caller that migrates before starting a server - the deploy
workflow does exactly that - carried on onto a schema that had not been
brought forward. A failing migration function was the only failure reported,
and only as a side effect of the log.Fatalf this commit removes.
Each of these was checked by degrading it and watching the named assertion
go red: returning nil instead of the failure, naming the first version rather
than the one that failed, accepting an unregistered app code, and not exiting.
One gap is left open deliberately. Go allows a call whose only result is an
error to stand as a statement, so `migration.Migrate.Migrate()` still compiles
while dropping what it returns - `go build` passed while migrateModel was
doing exactly that during this change. Both call sites now return the value,
which the compiler does check, but nothing guards against the statement form
coming back. A checksilent rule was considered and dropped: that tool parses
without type information, so it could only match the method name, and a guard
that fires on any type with a Migrate method is noise.
Cleaning up after a successful deployment never runs on the host that needs it.
The pull is the first thing in this script that needs space and it is where a
full disk stops it, so the run ends before reaching any cleanup - and so does
the next run, and the one after that. That is not hypothetical: a deployment
failed on the pull with no space left on the device, and rerunning the workflow
unchanged failed at the same place. The disk had to be cleared by hand before a
deployment could go through.
The pipeline is now a function called twice, before the pull and after the
health check, so the window is bounded on both sides.
Verified against a real docker daemon, in the function form rather than the
inlined one: with five images newer than the running one, so position alone no
longer protects it, it leaves three and does not select the live one; removing
the id exclusion from the same function does select it. With NAME pointing at a
container that does not exist it selects nothing - as it also does without the
explicit guard, which is there because grep -v on an empty id reads like the
opposite of what it does, not because it changes the outcome.
Every deployment pulls an image tagged with its commit and nothing removed the
previous one, so they only accumulated. 68 had built up when a deployment failed
on a pull with no space left on the device. That is the harmless place to fail -
the site kept serving the image it already had - but no later run would have
recovered on its own.
Three are kept so a release can be re-run by tag by hand. Only this repository's
images are listed, because the host runs other services. The image the new
container is on is excluded by id rather than by position, and rmi is called
without -f so an image a container still holds is refused rather than taken from
it.
Verified against a real docker daemon: with five images newer than the running
one, so position alone no longer protects it, the pipeline leaves three and does
not select the live one. Removing the id exclusion from the same pipeline does
select it, so that guard is load-bearing rather than decorative.
gcc and g++ were 273MB of a 381MB image, and nothing in the container ever
invoked them: the binary is compiled and statically linked before the image is
built and arrives as a COPY, and Go is not installed here either.
The layer cost more than its size. apk resolves against an index that moves, so
its digest differed on every build and no two images shared it - a host that
keeps one image per deployed commit paid the full 273MB each time rather than
storing it once.
libc6-compat is kept although nothing measured needs it: a container built
without it resolves a hostname and opens a database connection exactly as one
built with it, but it costs half a megabyte and covers a ./main that was linked
dynamically, which this Dockerfile cannot check.
Verified by building this Dockerfile and running the result under the check the
deploy script uses - captcha answering 200 and the log reporting the datastore
connected. A control built from the current recipe passes the same check and
carries a 273MB apk layer this one does not; a third build with a deliberately
truncated binary fails the check, so it distinguishes a serving process from a
dead one.
The check grouped by (app_code, path, action) and refused whatever
appeared more than once. GROUP BY treats two NULLs as the same value; a
unique index treats them as different ones and allows both. So a
database holding rows with a null path or action was refused for
duplicates the index it was blocking would have accepted - and the
migration stopped, on a database with nothing wrong with it.
Both columns are nullable: neither carries a not-null tag, so gorm built
them that way. Measured on MySQL 8.0, PostgreSQL 15 and SQLite: two rows
with both columns null are one group to GROUP BY, and the unique index
builds over them without complaint. Standard SQL, not a dialect quirk.
They are now excluded from the check rather than grouped. Each column
needs its own exclusion and has its own test: one null column is enough
to make the index accept the pair, so removing either condition alone
lets that half through - which is what the two subtests are for, and
each fails only for its own half.
The message could not name the rows either. MySQL's CONCAT returns NULL
when any argument is, and scanning that into a string fails with
"converting NULL to string is unsupported" - so the check reported a
driver error instead of the duplicates it exists to report. SQLite and
PostgreSQL treat a null argument as empty and say nothing, which is why
this never surfaced in the tests: they run on SQLite, and this
repository has no MySQL in CI. The comment says so, so that a
postgres-only test file is not mistaken for cover.
No COALESCE was added to paper over that. It would have had nothing left
to guard once the nulls are excluded, and it would make a future
regression quieter: someone dropping the exclusions would get a report
naming rows that are not duplicates, which reads as a real answer,
rather than a scan error that reads as a broken query.
Leaving path and action nullable is deliberate. Tightening them is a
migration of its own - existing null rows have to be given values, and
what those values should be belongs to whoever owns the data, not to a
migration whose job is adding an index.
The idempotency check added in the previous commit stopped a retry
inserting a second copy, and introduced a quieter failure in its place:
a retry that found an existing sys_menu row skipped everything after the
insert. Those are the sys_menu_api_rule bindings and the materialized
path, and neither is written by the statement that writes the menu -
paths is a separate UPDATE, and on MySQL an earlier DDL has already
committed the transaction that was supposed to hold them together.
So an install interrupted between those steps left a menu that exists,
sits outside the tree with an empty path, and is bound to no API. What
core's contract.md says about such a menu is that it is invisible to
every role and its apis are authorized for no one - while the installer
reports success.
Reusing now repairs. paths is compared before it is written, so a row
that is already right is not touched. Bindings are inserted with WHERE
NOT EXISTS rather than deleted and rebuilt: an administrator can bind an
api to a menu from the menu screen, and delete-then-rebuild would take
that with it on the next retry - the same accident as sys_role.go's
Association.Delete, pointing the other way.
Confirmed as a defect before it was fixed, by building the half-written
state and watching the assertions fail:
dir.Paths = "", want "/0/1"
binding count for list = 0, want 1
Four paths through the repair, each with a degradation that reds its own
test and leaves the others green: missing bindings only, missing paths
only, both, and neither. The fourth asserts no UPDATE is issued for a
row already correct.
A fifth covers what the repair must not do. Rebuilding bindings instead
of inserting them leaves every other test green while silently deleting
a binding this code did not create; that one now fails with "a retry
silently deleted a binding it does not own".
Bindings an older version of a manifest created and a newer one no
longer lists are left alone. Removing them is a delete, and a delete
needs the same certainty about ownership that uninstall does - this
function cannot tell a stale binding from one somebody added by hand.
seedApis and seedMenuTree were bare tx.Create calls. A migration that
failed partway and was run again re-inserted everything it had already
written - which is not hypothetical: the demo site collected eighteen
duplicate sys_menu rows this way, and three duplicate menus were visible
in its sidebar.
Both now look for a live row already holding the natural key and reuse
it. Only live rows count: a row an earlier soft-delete retired does not
stand in the way of a fresh insert under the same key, which is also
what the unique indexes allow.
The app_code half of each key has a test of its own. Without it the
lookups still passed every existing test while quietly letting one
application adopt another's rows - and an uninstall would then delete
rows the other application believed were its own, on both sides without
an error. Removing app_code from either lookup now fails with
"has 1 row(s) ... want 2 - one per app".
Seeding needs something to look for before it inserts, or a retry writes
a second copy of everything it already wrote. sys_api already had one in
(app_code, path, action). sys_menu had nothing usable: menu_name is
pascalCase(appCode) + pascalCase(code), which is not injective -
"list-all", "listAll" and "list_all" all become "ListAll" - so the
original code cannot be recovered from it. Hence a new column.
seed_code is nullable, against this repository's habit of NOT NULL
DEFAULT '' for a new column, and deliberately. Every row that predates
it has no meaningful value, and under a unique index an empty string
collides with every other empty string while NULL collides with nothing.
The convention exists because deleted_at's nullability broke a unique
index; here nullability is what makes one possible.
The unique index on sys_api cannot simply be created: a live database is
known to hold historical duplicates - the demo site had eighteen. The
migration looks first and refuses while naming the offending rows,
rather than letting CREATE UNIQUE INDEX fail with a constraint error
that names none. Same shape as 1786700003000's refuseOnDuplicates.
Run against SQLite, MySQL 8.0 and PostgreSQL 15, including the CONCAT
duplicate check, which had only ever been executed by SQLite's driver.
sys_app is one row per installed application. It is physically deleted
on uninstall rather than following the millisecond soft-delete marker
the other sys_ tables use: an installed-app registry has no "deleted by
accident, needs recovering" case, and a physical delete is what lets the
same code be installed again afterwards.
status is installing/installed/failed rather than a boolean, because an
install spanning several migration files is not atomic on MySQL - DDL
commits implicitly, so a run can stop in the middle. failed_version and
last_error are diagnostic snapshots for a person to read; nothing may
decide anything from them, and the field comments say so. Where to
resume is answered by sys_migration, which cannot drift from what was
actually applied.
sys_app_casbin_grant records which casbin_rule rows an install created,
keyed by casbin_rule's own natural key. That table is not extended
instead: gorm-adapter's SavePolicy truncates and reloads it from an
in-memory model, which would drop any column added here without a word.
Built against SQLite, MySQL 8.0 and PostgreSQL 15.
v2.8.0 adds sdk/contract/app - an application's manifest, and the one
comparator for its version - which the installer in this batch is built
on. Nothing here uses it yet; this is the dependency arriving.
Checked that the release is consumable rather than only tagged: a
program built against the published module registers a manifest, reads
it back, and gets -1 from Compare("1.9.0", "1.10.0"), which is the
multi-digit case a string comparison would order backwards.
25 packages pass and checksilent is clean on the new version.
Every Create and Raw dropped its error. Most of them would have failed
an assertion further down anyway, with a message describing the wrong
problem - but one of them would not.
The last count in the index test reads how many indexes survived and
expects zero. An unchecked query that fails leaves the variable at zero,
and zero is what success looks like: a test that cannot reach the
database reports that the indexes were dropped.
Demonstrated rather than assumed, by breaking that one query both ways:
with the check: FAIL counting the indexes after: relation
"pg_indexes_nope" does not exist
without it: PASS
Raised by Copilot on #922, as a consistency point with the SQLite tests
in this package. It is that as well, but the reason it is worth doing is
the row above.
The rest of this package's tests run on SQLite, where dropping an index
through the migrator works. That is why a migration which failed on
every PostgreSQL database it was pointed at had a green suite: the
defect cannot occur on the backend being tested.
A test alone would not have helped either - without a service it skips,
and a test that never runs is the same as no test. So the workflow gains
a postgres service and the DSN, and the helper refuses to skip when CI
is set: a workflow that drops the service or renames the variable fails
rather than going quietly green, which is the shape of the original
defect.
Counter-proved by putting Migrator().DropIndex back, which reproduces
the statement verbatim:
DROP INDEX CURRENT_SCHEMA()."idx_sd_pg_user_deleted_at"
ERROR: syntax error at or near "CURRENT_SCHEMA" (SQLSTATE 42601)
The conversion test also checks the timestamp survives as a marker,
since a conversion that dropped it would bring deleted rows back live
while still passing a column-type assertion.
The soft-delete conversion dropped the indexes on deleted_at through
gorm's Migrator().DropIndex. Its PostgreSQL driver resolves a schema for
the statement and falls back to an expression when it cannot:
currentSchema, _ := m.CurrentSchema(stmt, stmt.Table)
m.DB.Exec("DROP INDEX ?.?", currentSchema, clause.Column{Name: name})
DROP INDEX takes an identifier in that position, so what reached the
server was
DROP INDEX CURRENT_SCHEMA()."idx_sys_api_deleted_at"
ERROR: syntax error at or near "CURRENT_SCHEMA" (SQLSTATE 42601)
The schema is unresolvable for every call this migration makes, because
it passes a table name as a string rather than a model. So it failed on
every PostgreSQL database rather than intermittently, and stopped the
whole conversion at the first table.
What that looked like from outside is go-admin#919: an upgrade that
could not complete, and a login rejecting a correct password, because
deleted_at was still a timestamptz while the current query compares it
to 0. Neither symptom names a migration.
Written per dialect, for the same reason addBigIntColumn and
renameColumn already are. MySQL and SQL Server name the table and have
no IF EXISTS for it; PostgreSQL and SQLite name the index alone.
Verified by running the shipped migration against PostgreSQL 15 and
MySQL 8.0 in containers, and SQLite through this package's tests. The
SQL Server form is from its documentation and has not been run - there
is no SQL Server here to run it against, and saying so is better than
implying four dialects were checked.
The process started, both probes passed, and the first sign that the
schema did not match was a login failing with a driver-level encoding
error - go-admin#919, where an operator upgraded the binary and
restarted the API without running migrate. Nothing between those two
events had an opinion about the schema.
Readiness is where this belongs. Liveness asks "restart me", and a
process whose database is on the wrong schema comes back to the same
schema. Readiness asks "send me requests", and the answer is no. A
rolling update then stalls at the deploy - new instances never become
ready, the old ones keep serving - rather than at somebody's login, and
running migrate clears it without a restart because the check is
evaluated per request.
Any tenant database being behind fails the check, not only the one being
served: migrations are applied to every database in one run, so one
behind means that run did not finish, and serving the rest would let a
half-applied deploy look like a partial success.
A missing sys_migration table is nothing applied rather than an error.
That is a first deploy, where every migration is pending and the
operator can act on being told so.
The one test that matters is the one that cannot be written normally.
The registry is filled by init() in packages cmd/api does not import, so
a test that imported them to look at it would pass whatever the real
binary links - and a binary that links none of them gives a check that
reports every database current, forever, with every other test here
still green. TestTheServingBinaryLinksTheMigrationRegistry asks the
build instead, with a negative control so that a query matching
everything fails rather than passes.
Closes#920.
Whether the schema matches what the binary expects is answered by the
migration registry, which lives under cmd/. common/ has never imported
cmd/, and starting with this would put the shared layer behind the
command layer for one check.
Register instead, from where both are already in scope. A duplicate name
panics rather than appending: two checks under one name make the failing
one impossible to identify from the response body, and registering the
same one twice is a wiring mistake better heard at start-up than never.
The registered check is run through the same guard as the built-in ones,
so one that panics fails its check rather than taking down the probe
that asked.
Status answers what is registered, what is applied, and what is applied
while nothing registers it - and needs a database to do it. A readiness
check needs only the first half, and it already holds the databases it
is asking about.
Without this it would have to call SetDb to reuse Status, writing this
package's shared state from a request path, for a question that does not
depend on any database at all.
The wait was one select over done and ctx.Done(). Both can be ready when
it runs, select picks at random among ready cases, and so a queue that
drained in the same instant the budget expired was reported as an
overrun about half the times it landed there - often enough to be read
as noise, and pointing at the wrong thing when it was not. core's own
RunShutdown re-checks for this reason; this did not.
The tie-break is now a function taking channels rather than a queue,
which is what lets a test hand it a closed done and an expired ctx
together. That state is the whole of the bug and cannot be arrived at
reliably from the outside; over 1000 iterations the single-select
version fails, and the second look does not.
The test for giving up on the deadline read the call counter straight
after shutdownQueue returned, while Shutdown runs on a goroutine nobody
joins. It passed because the goroutine is scheduled promptly, not
because anything ordered the two. The fake now signals that Shutdown has
been entered and the test waits for it.
Both raised by Copilot on #918.
The first attempt at the tie-break test was wrong and is not what
landed: it asserted that an immediately-returning Shutdown always counts
as drained under an already-expired context, which is not true and
should not be - if the goroutine has not run, nothing has drained. That
test failed, correctly. What is being claimed is narrower: when both are
ready, done wins.
Nothing stopped the queue when the process exited. core v2.7.0 made the
drain work - Memory.Shutdown closes the queue and waits for every
consumer to finish what it holds, and the legacy adapter cancels its
context and closes the underlying queue - but no call site ever reached
it. The only Shutdown() in this repository applies to the previous
adapter during a reload, so the installed one was simply left. The login
log, the operation log and the API sync all publish through it, so a
rolling restart dropped whatever had not been consumed, on the path
where the process exits 0 and reports "Server exiting".
Setup now registers a BeforeExit callback that shuts down the adapter
this package installed.
Three things it has to get right, each with a test.
It reads `installed` when it runs, not when it registers. A reload
replaces the adapter, and the one from start-up is a queue nobody has
published to since.
It never goes through sdk.Runtime.GetQueueAdapter. That accessor never
returns nil - with no queue section configured it wraps the runtime's
own fallback - so it would look like it worked while closing a queue
this package neither built nor started. That is the same trap setupQueue
already had to drop an `if q != nil` for.
It registers once. Setup is re-run on every configuration change, and a
callback per reload would leave the shutdown phase holding a row of
identical entries, each eligible to be named as the one that overran the
budget.
That last one needed a seam. shutdownQueue takes the adapter on its
first run, so the second and third callbacks find nothing and return -
three registrations produce exactly the same observable result as one,
and a test going through the effect passes either way. It did: the
counter-proof for "register on every reload" came back green until the
registration was counted at the seam instead.
The wait is bounded here rather than left to the phase. Shutdown takes
no context, so a consumer that never finishes would hold the process
until SIGKILL; the callback gives up and says what is being lost, which
the phase's generic overrun message cannot.
Ordering falls out of the phase rather than being arranged: callbacks
run in reverse registration order, this one registers during setup and
the job scheduler's registers on AfterListen, so the schedulers stop
before the queue drains. Verified against core v2.7.0 rather than read
off the source.
Closes#911.
The warning said to set application.mode and stopped there. Following
that on a running process does not close anything: buildRouter has one
call site, in run(), and route registration is on no phase and no reload
callback, so a configuration reload moves the mode and leaves the routes
exactly where they were.
The reader is then worse off than before they acted. The mode now says
prod, GenWriteRoutesEnabled agrees, and the endpoints are still served -
so the one thing they could check to confirm the fix reports success
while the exposure is untouched, until something restarts the process.
A test pins the gap rather than the prose: build under dev, move the
mode to prod, and the routes are still in the engine. It fails if
registration ever becomes dynamic, which is the change that would make
the new sentence wrong.
That test degrades differently from the others - making it fail means
rewriting registration, not weakening it - so what was checked instead
is that it cannot go vacuous. Both of its premises are guarded: with the
gate always refusing it reports building under dev without the writing
routes, and with the gate always allowing it reports the predicate still
allowing prod. Neither failure can be mistaken for the assertion passing.
Raised by Copilot on #917.
registeredRoutes set config.ApplicationConfig.Mode and gave it back with
t.Cleanup, which runs at the end of the test rather than at the end of
the helper. Everything the caller did after the call therefore ran under
the mode the helper had been asked about, not one the caller chose.
Nothing was wrong yet: the one caller that reads the mode afterwards
sets it itself, and the two cleanups happen to unwind in an order that
leaves the right value. Both of those are accidents, and neither is
visible at the call site.
A defer inside the helper makes the borrowing end where it starts. The
doc comment said the mode was put back before returning while the code
did not, so that is now true rather than aspirational.
The counter-proof is the reason this has a test of its own: with
t.Cleanup back in place TestRegisteredRoutesRestoresTheModeBeforeReturning
fails and nothing else does, which is what a leak this quiet looks like
when something is actually watching for it.
Raised by Copilot on #917.
The gate in the previous commit is decided by application.mode, and the
shipped configuration says dev. So the deployment most likely to be
serving the writing endpoints is the one that changed nothing, and that
is also the one least likely to go looking for them. A gate whose
default is open needs to say so.
Nothing is said in demo mode. The routes are registered there, but
DemoEvn refuses all three by name, so a warning would describe an
exposure that is not present.
The decision is split from the logging so it can be tested. Three
counter-proofs: warning in demo as well fails mode=demo; a warning that
never fires fails mode=dev, which is what shows the line can be reached
at all; and one that always fires fails every mode but dev.
Three of the code generator's endpoints do not read. /gen/toproject
writes seven Go and Vue source files onto the host, one of them under
the path gen.frontpath names; /gen/apitofile writes a migration;
/gen/todb inserts menus and APIs. All three are GET, and all three are
listed in CasbinExclude - which AuthCheckRole skips - so Enforce never
runs for them. Any account that can log in could call them, on every
deployment.
They are now registered only where application.mode is dev or demo. dev
is the shipped default and is where the generator is meant to be used.
demo keeps them because demo mode already has a better answer than a
404: DemoEvn refuses these three by name and explains itself, which is
what the demo is for. test and prod get nothing, and so does a process
whose mode was never set.
This does not make the endpoints safe where they exist; it stops them
existing where nobody should be calling them. A host left on the shipped
dev is still open, which is why the next commit says so at start-up.
CasbinExclude is left alone on purpose. Taking the three off that list
would make them require a permission no existing deployment has granted,
so every non-admin user would start getting 403 from a tool that worked
yesterday. That is a migration, not a guard, and it belongs with a
release that can carry one.
Four counter-proofs, each red on the test that names the behaviour and
green everywhere else: a gate that always allows fails test/prod/unset
only; a gate that always refuses fails dev/demo and takes
TestEveryRouteDemoModeRefusesStillExists with it; moving a read-only
route inside the gate fails the reading test; and spelling the condition
at the registration site instead of calling the predicate fails the
agreement test, which is what keeps that test from being a tautology.
Two pre-existing deviations: a space before the comma in
sysNoCheckRoleRouter's parameter list, and no newline at end of file.
Separated from the change that follows so its diff is only the change.
The comment at the top of this workflow says documentation-only changes
skip it, because a push to master pushes an image, runs the migrations
and restarts the demo container. The ignore list did not cover
scripts/k8s, so editing a manifest that the deploy never reads bought
the site an outage.
The pattern is scripts/k8s/** rather than scripts/** because
scripts/Dockerfile is a build input - go.yml builds the release image
from it on a tag.
This workflow file stays outside the list on purpose. paths-ignore skips
only when every changed path matches, so a change that edits the deploy
still runs it, which is the point.
The comment on replicas gave one obstacle to raising it, the shared log
volume, which reads as the only one. Someone who moves the log path off
that volume would conclude the way is clear.
The scheduler in app/jobs is the second, and it is the one that does not
announce itself. Its handle on a job lives in sys_job.entry_id, one
column shared by every process, and startup zeroes the whole column
before writing its own ids. A second pod therefore erases the first
pod's, and both pods run the full enabled list. Stopping a job from the
UI then removes an entry from whichever process is asked, by an id that
belongs to another one, and answers 200.
See #915.
DemoEvn decided by HTTP method: GET and OPTIONS through, everything else
refused. Three of the code generator's routes are registered as GET and write
anyway - two emit Go source files onto the server's filesystem, and the third
inserts menus, APIs and casbin rules into the database. They sit in a group
whose own name says it does no role check, and a demo deployment lets anybody
log in. So on the demo host any visitor could write to the machine and to the
database, and the one that writes menus had in fact been used: three generated
SysCasbinRule entries is how this was noticed.
The guard now also looks at the matched route. The method cannot answer the
question - whether a request changes anything is not something the verb reports
truthfully here - so the three are named, as gin route patterns, which is what
Context.FullPath returns and how CasbinExclude already spells them.
Changing them to POST would be the better shape and is not this change. sys_api
records an endpoint by method and path and the casbin policy follows it, so
flipping the verb needs a migration and a policy resync; until both land, every
existing deployment would start answering 403 to a role that could use the
generator the day before.
The read-only half stays reachable: preview, the table tree, and the two
database listings. A demo host that cannot demonstrate the generator is as
broken as one that lets visitors write to it - refusing too much is the same
defect facing the other way, and there is a test for that direction too.
Half of the general hole is closed and the other half is written down. The
closed half is a test beside the route registrations: it builds the generator's
routes, enumerates them, and fails if any entry in the guard has stopped being
a real route, so renaming one turns the list red instead of quietly making it
match nothing. It lives there because common/ may not import app/ - which is
also why the guard cannot check its own list from where it is. The open half is
that no static check can tell a handler that writes from one that reads, so the
next GET that writes has to be added by hand. The comment says that rather than
leaving the impression the class is covered.
application.demomsg was configuration nothing read. The message was hard-coded
in the middleware, and the demo host's configured string happened to be
identical, so the setting looked like it worked and never had. It is read now,
with the old string kept verbatim as the fallback, so a deployment that never
set it is answered exactly as before.
This covers demo mode only. On a deployment that is not a demo those three
routes remain in CasbinExclude and stay reachable by any authenticated user
whatever their role; that is a separate decision and is not touched here.
The budget is one number in config/settings.yml. The deadlines that have to
cover it are in four other files, none of which anybody edits while thinking
about shutdown - so raising the budget passes every test, deploys, and has the
cleanup callbacks killed on the next release.
Two checks share one arithmetic and one five-second margin.
shutdown-budget-overruns-grace compares preStop + drain + server + cleanup
against terminationGracePeriodSeconds in the shipped manifest. Those two files
are not merely adjacent examples: scripts/k8s/prerun.sh builds the
settings-admin ConfigMap out of config/settings.yml and the Deployment mounts
it, so the manifest deploys that file.
docker-stop-cuts-shutdown-short covers the three ways this container is
stopped: `docker stop` in the release workflow, the same in the Makefile, and
stop_grace_period on a compose service that runs this repository's own image. A
service running a database is not this process and is left alone. The duration
is parsed rather than scanned for digits - compose accepts 1m30s, and reading
the first number out of it would call ninety seconds one.
All three spellings of the deadline are read: --timeout, the deprecated --time,
and the short -t. A deadline the check cannot read is reported as no deadline at
all, so recognising only one of them would call a correct command broken and
send whoever fixed it towards the spelling docker is retiring. The message
quotes the flag back in the spelling it was written in, for the same reason:
suggesting a flag the line does not use is how a tool teaches people to
disbelieve it.
What neither covers is `docker rm -f`, which has no deadline to compare
against: it is SIGKILL by definition. That gap is deliberate, and it is why the
previous commit changed the one place that used it on a container that might
still be running.
Both report at two levels. A budget that already overruns is an ERROR; one that
fits with nothing to spare is a WARN, because it works today and failing the
build on a working configuration is how a project teaches people to ignore its
warnings. The two are exclusive: an overrun satisfies the headroom condition as
well, and an ERROR that always drags a duplicate WARN behind it teaches the same
lesson.
preStop is in the sum although the shipped manifest has no hook. That is the
point - a hook added later is spent before the process is told anything, and a
self-check that could not see it would understate the real budget by however
long somebody set it to, which is worse than not checking. A hook whose duration
cannot be read is reported rather than counted as zero.
The fallbacks for fields the settings file leaves out are read from the
constants in the scanned tree, not copied here; if they are renamed the run
stops instead of going quiet with the wrong numbers.
The wording differs by audience on purpose. At run time this is somebody else's
deployment under constraints the process cannot see, so the log states a
minimum. These checks read files this repository owns, where there is standing
to ask for headroom, so they name a target.
The table in AGENTS.md is relisted while it is being touched: the two new
checks, plus datascope-route-unguarded, which has been missing since it was
added. The hard-coded count is gone - it said seven and there were ten, which is
what a written-down count does. AGENTS.md and docs/contract.md both sent readers
to `go run ./tools/checksilent -h` for the list of checks; that prints
command-line flags and has never printed a check, so both now point at
runChecks.
The yaml parser moves from an indirect requirement to a direct one - it was
already in the module graph - and tidy drops four go.sum lines left over from
two older releases of core.
Stopping this process takes drain + server + cleanup seconds: eight out of the
box, and more for anyone who configures a drain window. Three places decide
whether it gets that long, and none of them was written with it in mind.
The release workflow stopped the previous container with the default deadline,
which docker sets at ten seconds. The compose file - which the Makefile calls
the first way to run this - set no stop_grace_period, so it took the same ten.
Under either, a drain window over two seconds would have been cut off by
SIGKILL part-way through the cleanup callbacks: this project's own deployments
could not have run the capability it ships.
The third was worse. `make run` removed the previous container with `docker rm
-f`, and the force flag kills a running container outright - "uses SIGKILL", in
docker's own words - with no grace at all. Restarting locally cut every
shutdown short, so the drain window would never once have been reached on a
developer's machine. It now stops with a deadline and then removes, which
leaves what gets removed unchanged: on a container that has already stopped,
stop is a no-op.
So: --timeout 30 in the workflow, stop_grace_period: 30s on the compose
service, and stop --timeout 30 before the removal in the Makefile. --timeout
rather than --time, which docker still honours but has deprecated - it prints a
warning on every use, and a deploy log that always carries a warning is one
nobody reads.
The three remaining `docker rm -f` calls in the workflow are left alone. Two
remove containers that have already been stopped and one is the rollback path,
and nothing static can tell those apart from a container that is still running -
which is also why the check added next does not look at `rm -f` at all: a forced
removal has no deadline to compare against. What keeps that path honest is the
line above it, not a check.
Thirty will drift the first time somebody raises a budget. The next commit is
what notices, which is also why these comments name a check that does not exist
yet.
The manifest in this repository had no probes at all. A pod was sent traffic as
soon as its container was running, whether or not the database it needs was
reachable, and it was stopped with whatever grace period Kubernetes defaults to
rather than one chosen against what this process actually spends shutting down.
It now mounts both probes, at the endpoint that answers each question:
readiness at /ready, which fails while a dependency is unreachable, and
liveness at /health, which is a bare 200 because restarting a process whose
database is down turns one outage into a crash loop. Both skip the rate
limiter, which is why that had to land first.
timeoutSeconds is 3, not the default 1. The handler allows its checks two
seconds, so at the default a database answering in 1.2s would be recorded as a
failed check while the handler was returning 200 - the probe would be failing on
the orchestrator's stopwatch, not on its own. The comment beside that constant
said the constraint was the polling period; the constraint is the per-check
timeout, and it is now written down correctly.
terminationGracePeriodSeconds is 30, against a shipped budget of 0 + 5 + 3.
Raising drain means raising this too, in the same commit; the check that
notices when somebody does not arrives two commits from here.
replicas stays at 1, and the comment says why that makes the drain window worth
nothing: there is nowhere to send the traffic this pod stops taking. Raising it
needs one more change than the number - the volume is shared by every replica
and the log path lives on it, so a second pod would append to the same rotating
file. The reason not to raise it is not the one the review assumed: the claim
was that the PVC is ReadWriteOnce, and it is not, it is ReadWriteMany on nfs-csi.
There is no preStop hook. How long one should sleep depends on how fast the
thing in front removes this instance, which the repository cannot know, and a
manifest carrying both a preStop sleep and a drain window is the double-counting
trap - the budget would be spent twice and the start-up line would report half
of it.
The three budgets are spent one after the other, so what has to fit inside the
orchestrator's grace period is their sum - and nothing said what that sum was.
Working it out meant reading a configuration file, remembering which fields
were absent, and knowing what each one falls back to.
Start-up now prints it: the three values and the total, taken from the resolved
budget rather than from the file. A field left out still costs its default, so
adding up what was written down understates the total by exactly the fields
nobody wrote - which is the arithmetic somebody doing this by hand gets wrong.
Whether the total fits is a separate question, and the framework cannot answer
it alone: `docker stop` allows ten seconds and Kubernetes thirty, three times
apart. A fixed threshold would have warned about the manifest this repository
is about to ship. So extend.shutdown.grace is optional, nothing reads it during
a shutdown, and when it is absent the line says so and prints both reference
values instead of judging.
When it is set and the budget does not fit, the warning names the shortfall:
how many more seconds are needed. A minimum, not a target - this is somebody
else's deployment under constraints this process cannot see, and asking them to
leave headroom on top is not this line's business. Equal does not fit either;
the grace period is when SIGKILL is sent, so a budget that ends exactly then
leaves the last callback no time to return.
zhangwenjian