A URL ending in .ocda.db is now downloaded and processed in place, through the path that already did it for .zip: the same curl call, the same size and timeout limits, the same refusal of local and private addresses, the same --allow-downloads guard, the same temp file cleanup. Only the pattern had to widen. rgx_url_zip ^https?://...[.]zip$ rgx_url_source ^https?://...([.]zip|[.]ocda[.]db)$ downloadZipUrl is no downloadSourceUrl and the download temp directory is spine-download rather than spine-zip-pod; (the extraction directory keeps its name, being still only for zips). (assisted by Claude-Code)
The document markup sample collection builds byte identically from either artefact: 1457 files, no differences, from markup, from the 35 .ssp files, and from the 35 .ocda.db files. A .ocda.db carries its images as blobs. dbReadFiles() reads them and they are written where the output writers look for images: a directory of this run's making, removed when the run ends, not the pod beside the artefact, (which could be a working published tree). The document's source path moves with it, image_dir_path being reached from the document's own file rather than from the pod. Each blob is checked against the sha256 the database recorded beside it. The two were written together, so a mismatch means the file is damaged. (the point of having recorded the digest). A .ssp only describes its images, so those are the ones in the pod it sits in, and they are checked against the digests it recorded. That asymmetry arises from of what the two artefacts are: the ocda.db is self-sufficient and can prove it is looking at the right image, the .ssp describes a pod it sits alongside. Found while wiring it: setting the pod directory alone was not enough, since image_dir_path is derived from the source file's path (../../image from media/text/<lang>/), so the source path has to be moved with it. (assisted by Claude-Code)
Process a document from its .ocda.db or .ssp abstraction. A .ssp or a .ocda.db given as an argument is now a document like any other: read rather than parsed, and handed to the output writers as the same doc they already take. The 35 document sample collection built from its .ssp files is byte identical to the same collection built from markup: 1457 files, no differences. Three things had to be rebuilt rather than read, all of them a value parsed rather than a structure derived: - classify_topic_register_arr and its expanded twin. The split is not a plain one, so the rule moved to sisudoc.ocda.meta.topic_register and both the yaml reader and this one call it. - creator_author_arr, which is creator.author split on the ", " it was joined with. - title_sub, a copy of title_subtitle made where the header is read, and what epub3 puts in dc:title id="subtitle". Fixed ordering bug found by the acceptance test. A heading's own anchor can be a bare number taken from its text and this can collide with the ocn of an unrelated object. Whole output comparison found that before the fix there was one wrong link in one epub's table of contents. From a .ocda.db, 1446 of 1457 files are identical. The 11 that are not are the images: a database carries its own image blobs and nothing yet extracts them, so the five sisu_markup images are not copied and the epub that embeds them differs. That is the next step and is not a defect in this one. --source and --pod2 are refused with a warning rather than half done, no artefact carrying the markup. (assisted by Claude-Code)
ST_DocumentMatters was declared inside spineAbstraction(), closing over the parser's locals, so only the parser could build it. A document loaded from a .ssp or a .ocda.db has to arrive at the same value from what the artefact carries, and had nothing to build. It is moved to sisudoc.ocda.meta.doc_matters as docMattersMake(), taking the seven things a document is described by: the run (program_info, opt_action), where it is and what it is called (manifest), its header and the site config (conf_make_meta), its counts and indexes (ST_DocHas), its source digests, and the files its markup inserted. The parser fills them from what it has just parsed; the loader will fill them from the artefact. Pure refactor: the struct's members are unchanged, only where it is declared and where its inputs come from. (assisted by Claude-Code)
The segment a cross reference to a heading lands on, gixed upstream in
ocda, where the field is created, rather than inferred by the loader.
A heading above level 4 opens no html segment of its own, so a link to
it has to land on the level 4 heading that follows. The build loop
cannot know that when it reads the heading, and says so in a comment of
its own: "for html segname need following lv4 not yet known". It
back-fills tag_assoc when the level 4 heading arrives (lv0to3_tags), and
the answer then lives only in that map.
tags.segment_lv4_is now holds it, resolved in a pass over the finished
head and body sections: walking backwards, the last level 4 heading seen
is the next one for everything above it, which is the same answer the
back-fill gives. Emitted in the .ssp only where it differs from
.segment_html_is, which is every heading at level 4 or below, so it is
sparse: 317 lines over the 35 document reference set, none removed.
Carried in the database as a column of its own, and read back by both
readers.
docHasFromAbstraction reads it instead of working it out. With that, and
on top of the two defect fixes, a document loaded from an artefact and
one parsed from markup agree:
key sets identical, all 35 documents
values 3 entries differ of some 30,000, all of them
_the_title, where the abstraction supplies an epub
segment the parser leaves unset
link targets all 1,651 agree, against 89 differing before any
of this work and 1 after the defect fixes alone
Format stays v1.1. That version is new in this same run of work and
nothing outside spine has read it, so this belongs in it rather than in
a bump of its own.
(assisted by Claude-Code)
docHasFromAbstraction() builds ST_DocHas from the objects and the
@doc_has block, so a document read from a .ssp or a .ocda.db can reach
the value the parser reaches. Everything it builds is an index over
properties the artefact already carries, not a recomputation of
something it does not: the counts come from the header, imagelist from
the .image records, the segment name lists and the tag associations from
.anchor, .segment, .segment_epub, .heading_lev_anchor and
.segment_*_is, and section_keys_sequenced from which sections are
non-empty plus the run's own flags.
*The two defect fixes this now sits on did most of the closing.*
Measured before them and after, over some 30,000 tag_associations
entries and the 1,651 keys a document actually links to:
imagelist 8 documents differed, now none. The parser was
the one that was wrong, and rgx.image being
anchored to image markup brought it into line
with what the .image records always said.
key sets four keys differed (_part_eof, "0", the empty
key, and "toc" the other way about), now none.
_part_eof came back the moment @tail was
carried, and the rest with it.
values 89 link targets resolved differently, now 1.
The one left is free_culture's ocn 5, and it is the case this cannot
reach: a heading above level 4 takes its segment by back-filling when
the next level 4 heading arrives, so the object never holds the answer
and reading the objects in order only approximates it. Fixed properly in
the commit that follows, in ocda where the field is made, rather than
guessed at here.
(assisted by Claude-Code)
ST_DocHas in metadoc_object_setter, filled by the parser, in place of DocHas_, a struct of accessors nested inside docAbstraction() that closed over the parser's locals. The member names are the ones the output writers already use, so nothing downstream changes. The point is that a struct closing over parser locals can only ever be built by the parser. A document loaded from a .ssp or a .ocda.db has to arrive at the same value with only what the artefact carries, and now there is something for it to fill. Two small things fall out. imagelist is a string[] rather than the lazy uniq range it was, and images() is therefore imagelist.length rather than counting the commas in the range's string representation, which carried a "TODO not ideal rethink". Same answer unless a filename contains a comma; the reference test agrees on all 35. (assisted by Claude-Code)