Subj : Persist archive content listings in a per-area data/dirs/<code>.conten
To   : GitLab note in main/sbbs
From : Rob Swindell
Date : Mon Sep 21 2026 22:08:17

https://gitlab.synchro.net/main/sbbs/-/work_items/1247#note_10413

## Storage changed: the listing lives in the file record's auxdata

The per-area `data/dirs/<code>.contents` store described above has been
replaced by the file record's own `auxdata`, under an `archive_contents` key.
The read cost is what settled it.

Serving one listing from a per-area file means parsing a single JSON object
holding every listing in that area. Measured against the store as the crawler
had actually populated it here:

| | per-area `.contents` | auxdata |
|---|---|---|
| a view in the largest area (4.78 MB, 76 records) | 31 ms, **+34.8 MB RSS** to return 1 record | n/a |
| the largest real listing (295 KB, 6,612 entries) | as above, whatever the record | **2 ms, +1.6 MB** |
| a typical record | as above | **0 ms, no measurable heap** |

That 34.8 MB is mozjs185 heap on the web server, per request, concurrent with
every other request, which is the same pressure the stored listings exist to
avoid. A cold view was worse still: it re-serialized the whole file to add one
record. In auxdata the cost is the record's own data blocks and does not scale
with how many archives the area holds.

Everything else follows from that:

* The listing is removed with the file record, so the orphan-sweep question in
  this issue is moot. No lock file, no second format, no separate backup.
* The size projection is unchanged, about 105 MB base-wide at a measured mean
  of 729 bytes per record (p50 257 B, p99 4.2 KB, max 295 KB across the 18,650
  records the crawler had stored).
* `auxdata` is a general-purpose per-file slot, so the record goes under its
  own key and anything else found there is preserved. Auxdata that the library
  cannot parse is left alone rather than replaced.
* No migration. A missing record is extracted and stored on first view, so the
  base warms itself. The `data/dirs/*.contents` files are simply unused.

## What made auxdata viable

Writing auxdata used to mean `FileBase.update()` removing the record and adding
it back, because `smb_putfile()` writes only the header and the index entry.
That is a full `.sid` read, rewrite and truncate under the base-wide lock, with
a window in which the file does not exist at all.

`smb_updatefile()` (smblib) now replaces a file record's data fields in place:
new blocks are written, the record is pointed at them, and the old blocks are
freed last. The record keeps its number, its header and its index position, and
a failure part way through leaks blocks (which chksmb reports and fixsmb
reclaims) instead of losing the record. `FileBase.update()` uses it whenever the
stored text differs; `readd_always` still removes and re-adds, since dating the
file as newly imported is the point of that option.

Two related defects turned up while doing it, both fixed:

* `FileBase.update()` wrote NULL over any extended description or auxdata the
  file object did not carry, so a script that read a record and handed it back
  silently dropped both. `rehashfiles.js`, `updatefiles.js`, `readdfiles.js` and
  `testbuild.js` all do exactly that, and `fileman.js` dropped auxdata whenever
  a sysop edited an extended description. An absent property now means "leave it
  alone"; an empty string still clears.
* `parse_file_properties()` rejected a file object made up of nothing but
  `extdesc`, `auxdata`, `cost`, `added`, `times_downloaded` or `anon`, returning
  SMB_ERR_NOT_FOUND and writing nothing.

## Open questions from the original report

* Should `?view=` stay crawlable: yes, decided.
* Is tier 2 (extract and store on the web path) allowed: yes, and verified live.
  The crawler warming the base is what populated 1,158 areas in a day.
* Orphan sweeping: no longer applicable.

Related: #1252, the same free-before-allocate ordering in `editmsg()`.

-- *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
 * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

.