Subj : exec/chat_llm.js exec/llm_index/dokuwiki.js To : Git commit to main/sbbs/master From : Rob Swindell (on Debian Linux) Date : Mon Aug 10 2026 03:17:45 https://gitlab.synchro.net/main/sbbs/-/commit/36cf709445ed412d4de99c06 Modified Files: exec/chat_llm.js exec/llm_index/dokuwiki.js Log Message: chat_llm: strip emoji as UTF-8 bytes, and bridge query vocabulary sanitize_reply() has never removed a supplementary-plane emoji. It matched UTF-16 surrogate pairs, and strings in this engine hold BYTES: U+1F310 is the four characters F0 9F 8C 90, so there is no code unit in the D83C-D83E range for the pattern to find. A literal emoji in a .js source file arrives the same way, which is why the pattern also could not be checked against one. Every emoji the model emitted went to the channel despite both the system prompt forbidding them and this backstop appearing to exist. Matching is done on the UTF-8 byte sequences instead. The E2 ranges are kept narrow deliberately: box drawing (U+2500, E2 94) and arrows (U+2190, E2 86) are ordinary text and must survive. The surrogate patterns are retained after them for an engine that decodes to UTF-16; they are inert here. Retrieval still could not reach a page whose vocabulary differs from the question's. BM25 matches literal terms, so "the name of the mail server recycle file" never reaches the semaphore-files page -- the documentation calls it a semaphore file, the question does not, and a section about a text string named "Recycle Servers" wins the literal match instead. A query_synonyms map adds bridging terms at retrieval time. Terms are ADDED, never substituted, so an unhelpful synonym costs ranking rather than correctness, and expansion is applied only to retrieval -- is_casual_input() also tokenizes queries, and judges small talk by how many content words it finds, so expanding there would make a greeting look substantive. Keys are single words: a multi-word key is split, and each word carries the expansion independently. Wiki subsections now carry a lead-in from their parent section. Splitting a page can separate an answer from the qualifier that makes it usable -- the recycle section says to touch ctrl/recycle while the "." suffix yielding recycle.mail is stated one level up, so chunked alone that section holds no "mail" token and a question naming the mail server cannot retrieve it. The appended "See for the official docs" citation is now scored by rarity among the retrieved candidates rather than plain overlap: a word carried by most of the retrieved chunks says nothing about which one the answer came from. It is also suppressed when the supporting chunk is not near the top of the ranking, because when two chunks support a reply about equally the pick is a guess, and a confidently wrong "official docs" link stapled to a correct answer reads as the bot not knowing its own documentation. index_url_append_rank tunes how strict that is, and 0 disables the append entirely. --- SBBSecho 3.37-Linux * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705) .