serial processing, it turns out, is significantly faster and more efficient for tested use-cases, which came as a surprise. As the parallelization option buys nothing, serial processing is set as default. Parallel processing remains as an option (where available, as before). (assisted by Claude-Code)
Fix issue with consistency (flags run serial & parallel inconsistenly). Both write one file per document per language and share no handle, so they belong on the list with html, epub, text and sqlite_discrete. The guard above the list still takes out --pod, --pod2, --source and the shared sqlite db actions before it is reached, so those stay serial as they were; checked. However measurement tests show parallel turn out to result in a processing slowdown, on the (35) sample markup documents, run on multiple passes: run on 16 cores a slow down of about 20% for eleven times the cpu! --text 4.83-5.10 s wall 48-52 s user --text --serial 3.94-4.05 s wall 4.2-4.4 s user will make serial run the default. (assisted by Claude-Code)
fix of a couple of issues with tests. test-search-cgi.sh reported PASS while printing "Schema version gate: skipped, sqlite3 not on PATH", and the values it searches for were the hardcoded fallbacks rather than values read out of the database it had just built. Both halves need sqlite3, both are checks, and a run that leaves them out still exits 0. The other three scripts re-run themselves under nix shell "nixpkgs#sqlite"; this one now does the same. sqlite3 is still not required: with neither sqlite3 nor nix the test runs as before and says what it skipped. test-abstraction-ssp.sh depends on --abstraction for its meaning, and nothing in it said so. --show-abstraction does not appear in spine's parallelise() list, so dropping that one flag turns the current run serial, and the parallel-versus-serial comparison below it then compares a serial run with itself - the guard gone, with nothing failing to report it. That is how the multi-lingual write race survived. A comment at the call now says not to drop it. (assisted by Claude-Code)
abstraction directory cleared once, by whichever language is first The .ssp writer clears stale files out of pod/<doc>/media/abstraction/, and that one directory is shared by every language of a document. The clearing is now done once per directory per run, by whichever language reaches it first, with the lock held across it. A language that finds the directory already prepared has passed through that same lock before writing, so the clearing it skipped had completed before its own write began: no .ssp produced on a given run can be removed during it. (removes possibility of a race condition on parallelisation) (assisted by Claude-Code)
Spine now declares sqlite_db_schema_version and stamps it into the
database as PRAGMA user_version when the tables are created, in
both the shared and the discrete DDL blocks. On opening an
existing database it compares, and says once per run which version
it found and which it writes.
Failures are now tallied (shared, the output can run in parallel),
reported one line each on stderr naming the operation, and main
exits 1 without printing "run complete, ok".
Two tests under test/, both taking the spine binary as their first
argument and building their own database from data/pod unless
$SpinePOD says otherwise:
test-search-db-schema.sh names: every column the search form
uses exists, and spine's declaration, the database's stamp
and the search form's expectation all agree
test-search-cgi.sh behaviour: the real search binary answers
real requests against a fresh database, no web server
involved, the probe values read out of whichever database it
is given
(assisted by Claude-Code)
The metadata page gains a line between the markup source and the source digests. (assisted by Claude-Code)
ocda (object centric document abstraction) flag --ocda-db (or --abstraction-db) replaces --show-abstraction-db rename results in consequently large diff
pod/ holds all document source representations: pod/<doc>/ source tree pod/<doc>/media/abstraction/<uid>.ssp abstraction, as text pod/<doc>.zip tree, zipped, .ssp included pod/<doc>.digests.txt sha256s of what is in them pod/<uid>.ocda.db abstraction, as sqlite db <lang>/abstraction/ is gone. pod/<doc>/media/abstraction/<uid>.ssp preferred as having the images (found within the pod tree) which .ssp needs to reproduce a document but does not carry on its own. The .ocda.db sits carries the images as well and (like the pod.zip) can be used to reproduce a document directly. digests.txt now covers the database as well as the zip, the source and the .ssp; (as does the metadata html page). Two ordering issues addressed: - the pod builder clean-slates pod/<doc>/ before regenerating it, and the .ssp is now written before that runs. The clean slate now leaves media/abstraction/ alone, and the .ssp writer clears that directory itself on the first language of a run, so a .ssp for a language the document no longer is removed and cannot be bundled. - for a multi-language document the .ssp files accumulate one language at a time and are bundled on the last, which is why the directory cannot simply be emptied by whichever (language) gets there first. (assisted by Claude-Code)
sisudoc.ocda.abstraction.load names the five things a document can be read from, tells them apart, and loads the two that are self-describing artefacts: .sst / .ssm + images the markup source pod (dir) + images the same, bundled pod .zip the same, zipped .ssp + images the abstraction, as text .ocda.db the abstraction, sqlite, images inside abstractionSourceOf(path) is the detection, by name and for a directory by whether it holds pod.manifest. abstractionLoad(path) returns a LoadedAbstraction: the source kind, whether it was loaded, why not when it was not, and the document itself. The three source forms are deliberately not loaded here. Reading them is the parser's job (sisudoc.ocda.meta.metadoc spineAbstraction) and it needs the manifest, environment and configuration that spine.d assembles, none of which belongs in a loader. What this gives that case is the dispatch and a plain statement of where it is handled, rather than a silent empty result. spine --abstraction-source=<path> says what a path is and, for an artefact, loads it and reports what came back: the document, title and author, header block sizes, object counts and the objects in each section. Exit 0 when an abstraction was loaded, 1 when not. (assisted by Claude-Code)