gopher://gopher.someodd.zip:70/1/catalog/tech/ General-purpose how-tos and technical notes: server setup, workstation tooling, gopher, and programming. 2026-06-12T00:00:00Z gopher.someodd.zip gopher://gopher.someodd.zip:70/0/tech/workstation/hibernation-nightmares.txt 2026-03-13T00:00:00Z 2026-06-12T00:00:00Z

The full saga, now CLOSED: resume was TWO bugs. The firmware-S4 reb...

amdgpu RDNA 3.5 hang; kernel version is the dial. Recover with SysRq, not power-off. (UPDATE 2026-06-12: caught it. Async device-resume race; pm_async=0 kills it. Kernel version was never the dial here either. Smoking-gun chapter below. closed.) * reboot to a fresh LUKS prompt -> firmware S4 flaking out (PM: Image not found, NO S4 wake). Fix is HibernateMode= shutdown PLUS aligning the initramfs resume target to the swapfile; shutdown alone wasn't enough. (closed 2026-06-03.) - Don't force-power-off a blanked resume; it invalidates the swap header. Use SysRq S-U-B. - Lid now does DIRECT hibernate on battery, nothing on AC. suspend-then-hibernate made resume worse, not better. - Bluetooth (MT7925) is dead on kernel 7.0.7, fine on 6.19.13. The hardware (so I stop misdiagnosing it) ----------------------------------------- Ryzen AI 7 350 / Radeon 860M, codename "Krackan", RDNA 3.5, gfx1152, PCI 1002:1114, Display Core 3.5. Wi-Fi and Bluetooth are one MediaTek MT7925 combo: Wi-Fi mt7925e on PCIe, BT btusb/btmtk on USB 0e8d:0717. Framework board FRANMGCP07, BIOS 03.05. It is NOT a Phoenix 7040 -- I assumed that for a while and it sent me chasing the wrong fixes. Also: this platform offers s2idle only, no "deep" S3, so mem_sleep_default=deep in my grub line is inert. The core nightmare: resume -------------------------- The image writes fine (battery barely moves overnight), but RESUME comes up to a black screen -- often with the machine alive underneath -- and sometimes reboots instead. The crash happens before journald persists anything, so there is no log of it. The only trace is on the next cold boot: PM: Image not found (code -22) A part-way resume crash invalidates the swap header (so it won't retry into a wedged state), which is exactly why the retry boot comes up clean and empty-handed. (UPDATE 2026-05-26: that story is right for the black-screen variant. For the REBOOT variant it's backwards -- the image was never restored at all, because the firmware never came back in S4. See "the lever I never pulled" below.) Cause of the black-screen variant: an upstream amdgpu regression on RDNA 3.5. Framework's own AI 300 threads put the failure rate at 20-50% and quote support saying hibernate-to- disk on these "is spotty at best ... wait for the bug to be patched." The lever that moves it is kernel version -- people report specific kernels working where neighbours fail. The setup (still valid) ----------------------- 1. Big swapfile to hold the image: fallocate -l 64G /swapfile-hibernate chmod 600 /swapfile-hibernate mkswap /swapfile-hibernate swapon /swapfile-hibernate /etc/fstab: /swapfile-hibernate none swap defaults 0 0 2. Resume from it. Find the offset: filefrag -v /swapfile-hibernate | grep " 0:" /etc/default/grub: resume=/dev/mapper/framework--vg-root resume_offset= then: update-grub && reboot Swap must be >= RAM or hibernate fails with "Not enough suitable swap space". Gotcha I left myself: the initramfs resume target DISAGREES with the cmdline -- cmdline: resume=/dev/mapper/framework--vg-root + offset initramfs: RESUME=/dev/mapper/framework--vg-swap_1 (1G LV) It works only because the cmdline wins. The 1G LV couldn't hold an image anyway. Should be aligned; left as a TODO. (UPDATE 2026-06-03: NOT harmless under shutdown mode -- this was the bug. See below.) Lid policy: the part that flipped --------------------------------- History: s2idle only -> overnigh]]> gopher://gopher.someodd.zip:70/0/tech/workstation/wireguard-nm.txt 2026-06-10T00:00:00Z 2026-06-10T00:00:00Z

WireGuard + NetworkManager (nm-applet) — setup notes ==============...

home server (friend0) over WireGuard on the LAN. DDNS (friend0.crabdance.com) for reaching it from outside. TOPOLOGY -------- server (friend0 @ 192.168.1.210) wg0 10.8.0.1/32 listen 51820 laptop friend0 10.8.0.2/32 Keys cross over: each side's [Peer] PublicKey = the OTHER side's PUBLIC key. A private key NEVER leaves its own box. SERVER /etc/wireguard/wg0.conf [Interface] Address = 10.8.0.1/32 ListenPort = 51820 PrivateKey = [Peer] # laptop PublicKey = AllowedIPs = 10.8.0.2/32 # /32 per peer LAPTOP /etc/wireguard/friend0.conf [Interface] Address = 10.8.0.2/32 PrivateKey = [Peer] # server PublicKey = Endpoint = 192.168.1.210:51820 AllowedIPs = 10.8.0.0/32 PersistentKeepalive = 25 NM-APPLET: SET THE WG IPv4 ADDRESS (the big trap) ------------------------------------------------- NM stores the tunnel IP in IPv4 settings, NOT the WireGuard section. New WG connections default IPv4 to Automatic (DHCP). WG has no DHCP -> you get NO ipv4 addr (only fe80:: link-local). GUI fix: nm-applet > Edit Connections > > IPv4 Settings Method: Manual Add Address 10.8.0.2 Netmask 32 Gateway (blank) Save, toggle the connection off/on. CLI: nmcli connection modify friend0 ipv4.method manual \ ipv4.addresses 10.8.0.2/32 nmcli connection up friend0 Best: import the conf, it sets IPv4 Manual for you: nmcli connection import type wireguard file friend0.conf Split tunnel: IPv4 > Routes > tick "use this connection only for resources on its network". MISTAKES I MADE (the derp log) ------------------------------ 1. Put the LAPTOP's private key into the SERVER [Interface]. -> server's identity became the laptop's; pubkey mismatch; no handshake. Each box uses ITS OWN private key. 2. AllowedIPs = x/24 on a peer. Must be /32 per peer (server side). /24 collides with the interface's own subnet route. 3. No IPv4 on the tunnel (the real blocker): NM "Automatic" left only fe80::. ping had no source -> server couldn't route the reply back. Set IPv4 Manual = 10.8.0.2/24. 4. Fought the wrong interface: laptop's wg0 is a DIFFERENT VPN. The friend0 tunnel is iface "friend0". Don't down/up wg0. 5. wg syncconf does NOT set Address (wg-quick-only directive, stripped by wg-quick strip). Use wg-quick up, or add the IP by hand. TROUBLESHOOTING (in order) -------------------------- Keys pair? on each box: echo "" | wg pubkey -> must equal what the OTHER box lists as [Peer] PublicKey. Server listening? sudo ss -ulnp | grep 51820 Firewall? (no ufw here; check nft/iptables) sudo nft list ruleset sudo iptables -L INPUT -n -v allow: sudo iptables -I INPUT -p udp --dport 51820 -j ACCEPT Handshake + transfer: sudo wg show - "latest handshake" present = crypto OK - X sent / 0 received = reply blocked (firewall, return route, wrong AllowedIPs, or no IP on the iface) - counters are cumulative; old failed retries inflate "sent" Does the iface actually have its IP? ip -br addr show - only fe80:: = NO ipv4 (the NM-Automatic trap) LAN reachable at all? ping Apply a config change: edit file, then sudo wg syncconf <(wg-quick strip ) # peers/keys only (NOT Address) — or a full sudo wg-quick up DDNS note: WG resolves Endpoint ONCE at bring-up. When the home IP changes the tunnel goes stale -> use the reresolve-dns timer (wireguard-tools examples) or just restart the tunnel. ]]> gopher://gopher.someodd.zip:70/1/tech/workstation/window_maker/rofi-omni 2026-06-09T00:00:00Z 2026-06-09T00:00:00Z

Companion to the rofi-windows switcher: a single rofi prompt that f...

Companion to the rofi-windows switcher: a single rofi prompt that fuzzy-filters across open windows (minimized ones included), installed apps, and recently-modified files under $HOME, ranked windows-then-apps-then-files with files newest-first. Bash + wmctrl + xdotool + fd, with background-refreshed caches so it stays instant -- files are mtime-sorted and capped to keep the prompt snappy. Bound to Mod4+s. gopher://gopher.someodd.zip:70/1/tech/gopher/gopher_tor_server 2026-05-22T00:00:00Z 2026-05-22T00:00:00Z

How this box serves the same gopherhole over both clearnet and a To...

How this box serves the same gopherhole over both clearnet and a Tor hidden service. Venusia's own package unit binds privileged port 70 as an unprivileged user (CAP_NET_BIND_SERVICE), while xinetd plus a tiny rewriting router expose it as an .onion that hands back .onion links. Notes the Tor gotcha that the client IP is always loopback. Descendant of the earlier reverse-proxy routing setup. gopher://gopher.someodd.zip:70/0/tech/programming/Verifying using Haskell in the Age of AI.md 2026-04-28T00:00:00Z 2026-04-28T00:00:00Z

Verifying using Haskell in the Age of AI

gopher://gopher.someodd.zip:70/0/tech/workstation/keepass/keepassxc-no-plaintext-secrets.txt 2026-03-31T00:00:00Z 2026-03-31T00:00:00Z

Stop storing API keys in plaintext by pulling them from KeepassXC t...

/usr/share/dbus-1/services/org.freedesktop.secrets.service << EOF [D-BUS Service] Name=org.freedesktop.secrets Exec=/usr/bin/false EOF' # ALSO stop D-Bus auto-activation via org.gnome.keyring # This is a separate service file that launches gnome-keyring-daemon # with --components=secrets. If anything pokes the org.gnome.keyring # D-Bus name, gnome-keyring starts and claims org.freedesktop.secrets # too. Blocking only org.freedesktop.secrets is not enough. sudo bash -c 'cat > /usr/share/dbus-1/services/org.gnome.keyring.service << EOF [D-BUS Service] Name=org.gnome.keyring Exec=/usr/bin/false EOF' # Hide the autostart entry (idempotent -- safe to run again) mkdir -p ~/.config/autostart cp /etc/xdg/autostart/gnome-keyring-secrets.desktop ~/.config/autostart/ grep -q '^Hidden=true' ~/.config/autostart/gnome-keyring-secrets.desktop \ || echo "Hidden=true" >> ~/.config/autostart/gnome-keyring-secrets.desktop # Kill any running instance killall gnome-keyring-daemon # KeepassXC won't automatically reclaim org.freedesktop.secrets -- # it tried at startup, lost to gnome-keyring, and gave up. # Restart it (or toggle Secret Service off/on in its settings). ``` Since apt upgrade will silently restore these service files (see caveats below), I keep a script that re-neuters both in one shot. Save as ~/fix-gnome-keyring-secrets.sh: ``` #!/bin/bash set -euo pipefail if [[ $EUID -ne 0 ]]; then echo "Run this with sudo." >&2 exit 1 fi REAL_USER="${SUDO_USER:?}" cat > /usr/share/dbus-1/services/org.freedesktop.secrets.service << 'EOF' [D-BUS Service] Name=org.freedesktop.secrets Exec=/usr/bin/false EOF cat > /usr/share/dbus-1/services/org.gnome.keyring.service << 'EOF' [D-BUS Service] Name=org.gnome.keyring Exec=/usr/bin/false EOF sudo -u "$REAL_USER" killall gnome-keyring-daemon 2>/dev/null || true if pgrep -u "$REAL_USER" keepassxc >/dev/null 2>&1; then echo "Restart KeePassXC to reclaim org.freedesktop.secrets." else echo "Start KeePassXC to claim org.freedesktop.secrets." fi ``` Run with `sudo ~/fix-gnome-keyring-secrets.sh` after any gnome-keyring package update. Then delete the old gnome-keyring data if you don't need it: ``` rm ~/.local/share/keyrings/login.keyring ``` ## 3. Enable KeepassXC Secret Servi]]> gopher://gopher.someodd.zip:70/0/tech/running_a_server/ipv6_ddns_afraid.md 2026-01-25T00:00:00Z 2026-01-25T00:00:00Z

Debugging session for a TLS chain bug where some clients could reac...

gopher://gopher.someodd.zip:70/0/tech/workstation/mcp-claude-habitica-trello.txt 2026-01-15T00:00:00Z 2026-01-15T00:00:00Z

Experiments with an MCP setup that uses Claude to merge redundant H...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/immich.txt 2025-12-31T00:00:00Z 2025-12-31T00:00:00Z

Setup guide for a self-hosted Immich photo and video system on a De...

Immich ---> Server Storage ---> Backups (RAID + tape) ``` Key principles: * One-way ingestion (no destructive sync) * Server never deletes data due to client actions * Immich handles indexing + UX * Filesystem handles durability + backups ## Storage Layout Root path: ``` /media/root/BackupRAID/baudrillard/photos/ ``` Structure: ``` immich/ -> main Immich media library immich-db/ -> postgres database storage staging/ -> temporary import area backups/ -> database dumps ``` ## Docker Setup Install Docker: ``` sudo apt update sudo apt install docker.io docker-compose-plugin sudo systemctl enable --now docker ``` Create Immich directory: ``` mkdir -p ~/immich cd ~/immich ``` Download compose file: ``` curl -LO https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml ``` Create .env file (based on example.env): ``` UPLOAD_LOCATION=/media/root/BackupRAID/baudrillard/photos/immich DB_DATA_LOCATION=/media/root/BackupRAID/baudrillard/photos/immich-db TZ=America/Los_Angeles IMMICH_VERSION=v2 DB_PASSWORD= DB_USERNAME=postgres DB_DATABASE_NAME=immich ``` Edit docker-compose.yml: 1. Ensure upload mount is correct: * ${UPLOAD_LOCATION}:/data 2. Enable HDD optimization: DB_STORAGE_TYPE: 'HDD' 3. (Optional) staging mount: * /media/root/BackupRAID/baudrillard/photos/staging:/staging:ro Create directories: ``` mkdir -p /media/root/BackupRAID/baudrillard/photos/immich mkdir -p /media/root/BackupRAID/baudrillard/photos/immich-db mkdir -p /media/root/BackupRAID/baudrillard/photos/staging mkdir -p /media/root/BackupRAID/baudrillard/photos/backups ``` Start Immich: ``` sudo docker compose up -d ``` Access: ``` http://SERVER_IP:2283 ``` Create admin account. ## WireGuard + Firewall Setup Goal: Only allow access via WireGuard (no public exposure) WireGuard subnet: ``` 10.1.0.0/24 ``` UFW rules: ``` sudo ufw allow 51820/udp comment 'WireGuard VPN' sudo ufw allow from 10.1.0.0/24 to any port 2283 comment 'Immich via WireGuard only' sudo ufw deny 2283 comment 'Block public Immich access' sudo ufw reload ``` Test: ``` http://10.1.0.1:2283 (over WireGuard) ``` ## Android Setup (GrapheneOS) Install Immich app. Configure: * Enable Backup * Select albums: * Camera (ONLY initially) * Enable: * Background backup * WiFi only (optional) * Charging only (optional) Test: * Take a photo * Verify upload appears in web UI Important: * Backup is one-way * Deleting from phone does NOT delete from server ## Storage Template Engine Leave OFF. Reason: * Simpler * More stable * Filesystem layout handled by Immich * Archive integrity prioritized over human-readable folders ## Importing External Media NEVER: * copy files directly into immich/ directory Use one of these methods instead: ## Method 1: Web Upload (simple) * Open web UI * Drag and drop files or folders * Create album afterward Best for: * small imports * manual curation ## Method 2: Immich CLI (recommended for server-side bulk import) 1. Create API key in Immich UI 2. Run CLI via Docker: sudo docker run --rm -it -v /media/root/BackupRAID/baudrillard/photos/staging/import1:/import:ro -e IMMICH_INSTANCE_URL=http://10.1.0.1:2283/api -e IMMICH_API_KEY='YOUR_API_KEY' ghcr.io/immich-app/immich-cli:latest upload --recursive /import Optional: * Use --album-name "Import Name" * Use --skip-hash for faster import * Use --dry-run for preview (slow on large datasets) Workflow: ``` server files -> staging -> CLI upload -> Immich library ``` ## Backups Media: * stored in RAID mirror]]> gopher://gopher.someodd.zip:70/0/tech/workstation/phone-files.txt 2025-12-31T00:00:00Z 2025-12-31T00:00:00Z

Troubleshooting notes on getting CalyxOS and Android files to show ...

/dev/null) if [[ -n "$exif_date" ]]; then echo "$exif_date" return fi fi # Fallback to modification date echo "$(date -r "$file" +"%Y:%m")" } # Function to copy files if they are missing or newer copy_files() { local file_type="$1" local target_dir="$2" shift 2 local extensions=("$@") for src_dir in "${SOURCE_DIRECTORIES[@]}"; do for ext in "${extensions[@]}"; do find "$src_dir" -type f -iname "*.$ext" | while read -r file; do # Extract year and month from EXIF or modification date year_month=$(get_file_date "$file" "$file_type") year=${year_month%%:*} month=${year_month##*:} # Determine target directory target_path="$target_dir/$year/$month" mkdir -p "$target_path" # Check if the file needs to be copied dest_file="$target_path/$(basename "$file")" if [[ ! -e "$dest_file" ]] || [[ "$file" -nt "$dest_file" ]]; then cp "$file" "$des]]> gopher://gopher.someodd.zip:70/1/tech/workstation/keepass/keepass-keyring-manager 2025-10-15T00:00:00Z 2025-10-15T00:00:00Z

Uses KeepassXC as an SSH agent and Secret Service provider for mini...

Uses KeepassXC as an SSH agent and Secret Service provider for minimal window managers like Window Maker. Targets setups where a full DE keyring (GNOME/KDE) isn't available. gopher://gopher.someodd.zip:70/0/tech/running_a_server/boot-full.txt 2025-09-29T00:00:00Z 2025-09-29T00:00:00Z

Recovery guide for when /boot fills up and apt upgrade gets stuck i...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/ii-logger.txt 2025-09-29T00:00:00Z 2025-09-29T00:00:00Z

Uses the ii IRC client as a minimal logger so gopher pages can surf...

"$BASE/$SRV/in" # keep the process in the foreground so systemd can supervise it wait "$pid" ``` then ``` chmod +x ~/.local/bin/ii-main-logger ``` create systemd `~/.config/systemd/user/ii-logger.service` ``` [Unit] Description=ii logger for %E{IRC_CHAN:-#main} on %E{IRC_SERVER:-127.0.0.1} After=network.target [Service] Type=simple Environment=II_DIR=%h/irc Environment=IRC_SERVER=127.0.0.1 Environment=IRC_PORT=6667 Environment=IRC_NICK=digestlog Environment=IRC_CHAN=#main ExecStart=%h/.local/bin/ii-main-logger Restart=always RestartSec=5 [Install] WantedBy=default.target ``` then ``` systemctl --user daemon-reload systemctl --user enable --now ii-logger.service ``` sanity check: ``` ls -R ~/irc tail -n3 ~/irc/127.0.0.1/#main/out ``` change nick/channel later (systemctl --user edit ii-logger.service and override the Environment= lines, then): ``` systemctl --user daemon-reload systemctl --user restart ii-logger.service ``` Also, logrotate so doesn't keep track everything forever: ``` sudo tee /etc/logrotate.d/ii-logger >/dev/null <<'ROT' # Rotate ii channel logs under your home. Adjust the path if needed. # Keeps 14 compressed rotations; trims when file >256k or daily, whichever first. # copytruncate avoids needing to restart ii. /home/*/irc/*/#*/out { daily size 256k rotate 14 compress delaycompress copytruncate missingok notifempty } ROT # test it (dry run shows what would rotate) sudo logrotate -d /etc/logrotate.conf ``` ]]> gopher://gopher.someodd.zip:70/0/tech/workstation/restic-on-my-laptop.txt 2025-09-16T00:00:00Z 2025-09-16T00:00:00Z

Adapting the server's restic-based backups for the laptop after mov...

gopher://gopher.someodd.zip:70/0/tech/gopher/gopher-http-interface.txt 2025-07-12T00:00:00Z 2025-07-12T00:00:00Z

Adds an HTTP interface to a gopherhole so web users can also browse...

gopher://gopher.someodd.zip:70/0/tech/workstation/projectm-audio-visualizations.txt 2025-03-13T00:00:00Z 2025-03-13T00:00:00Z

Notes on installing projectM, a modern port of classic Milkdrop/Win...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/music-server-mpd.txt 2025-03-11T00:00:00Z 2025-03-11T00:00:00Z

Sets up a music server with MPD to stream to an xbox, an Android at...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/xmpp-server-video-audio-calls.txt 2025-03-09T00:00:00Z 2025-03-09T00:00:00Z

Adds audio/video calling to a Prosody XMPP server on Debian by conf...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/mumble-server.txt 2025-03-08T00:00:00Z 2025-03-08T00:00:00Z

Sets up a Mumble voice-chat server — low-latency, strongly encrypte...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/letsencrypt-renewal-hook-mistake.txt 2025-03-02T00:00:00Z 2025-03-02T00:00:00Z

Catalogs a LetsEncrypt hook misconfiguration where the author used ...

gopher://gopher.someodd.zip:70/1/tech/gopher/gopher-routing 2025-02-25T00:00:00Z 2025-02-25T00:00:00Z

Sets up a Gopher reverse proxy so multiple services on different po...

Sets up a Gopher reverse proxy so multiple services on different ports are all reachable through port 70. Also covers exposing the same services over Tor. gopher://gopher.someodd.zip:70/0/tech/running_a_server/linux-counter-strike-16-server.txt 2025-02-25T00:00:00Z 2025-02-25T00:00:00Z

Counter-Strike 1.6 server setup on Linux (Debian), with nginx and L...

mapcycle.txt` * Add the map name to `mapcycle.txt` * i should make a script ## Resource server I'm not sure if this is working. ``` sv_allowupload 1 sv_allowdownload 1 sv_downloadurl "http://cs16.someodd.zip/cstrike/" ``` you could just serve the same cstrike dir you're already serving! and then hide the server.cfg to hide the rcon. just modify the nginx config i have here. it'll make it super fast. add to the server block: ``` location /cstrike/ { root /home/someuser/steamcmd/cs16/; # Specific root for the cstrike directory index index.html index.htm; location = /cstrike/server.cfg { deny all; # Deny access to the server.cfg file, returning 403 Forbidden } try_files $uri $uri/ =404; # Correctly handle the file paths within cstrike } ``` this is working super fast! ## Bots Back up the entire directory first. ### installing metamod `cd ~/steamcmd` Before installing Podbot, you need to install Metamod, which is a plugin interface for Half-Life modifications that allows you to use plugins like Podbot. **Download Metamod**: Visit the Metamod website and download the latest version appropriate for your server (P or V depending on your server's OS). http://metamod.org/ (i downloaded the regular linux, not x64 version). **IMPORTANT NOTE STARTING WITH DEBIAN TRIXIE (13)** just use Metamod-p like from https://sourceforge.net/projects/metamod-p/files/Metamod-P%20Binaries/. **Extract the Files**: Unzip the downloaded file into the `cstrike/addons/` directory of your server. Something like this: ``` mkdir -p cs16/cstrike/addons/metamod/dlls tar -xf metamod-1.20-linux.tar.gz -C cs16/cstrike/addons/metamod/dlls ``` edit `c16/cstrike/liblist.gam` and edit the value of `gameddl_linux`: ``` gamedll_linux "addons/metamod/dlls/metamod.so" ``` ### installing podbot Download podbot, I got `podbot_full_V]]> gopher://gopher.someodd.zip:70/1/tech/gopher/lagrange_gopher_ascii_art_tweaks 2024-12-30T00:00:00Z 2024-12-30T00:00:00Z

Lagrange client tweaks to better render ASCII-art-heavy gopherholes...

Lagrange client tweaks to better render ASCII-art-heavy gopherholes. Includes a screenshot of the fix in action. gopher://gopher.someodd.zip:70/0/tech/workstation/gpg_messages.txt 2024-12-30T00:00:00Z 2024-12-30T00:00:00Z

Step-by-step guide to encrypting and signing arbitrary text with GP...

``` I'm not going to get into keyservers. Set key expiration. You may also want to add your key for decrypting the message you sent as well. ]]> gopher://gopher.someodd.zip:70/0/tech/running_a_server/out-of-disk-space.txt 2024-12-04T00:00:00Z 2024-12-04T00:00:00Z

Steps to take when a Debian server runs out of disk space: diagnosi...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/upgrading-server-gpu.txt 2024-12-04T00:00:00Z 2024-12-04T00:00:00Z

Server upgrade writeup: quadrupled RAM plus a Quadro P620 GPU, incl...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/wireguard-simple.txt 2024-12-04T00:00:00Z 2024-12-04T00:00:00Z

Why the author prefers exposing a WireGuard VPN instead of SSH dire...

/etc/wireguard/publickey ``` ]]> gopher://gopher.someodd.zip:70/0/tech/programming/formal-verification-and-property-testing-of-an-algorithm.txt 2024-11-29T00:00:00Z 2024-11-29T00:00:00Z

How to confirm code does what it claims: introduces formal verifica...

1,000 test "cases" from just one actual property test. And this is because, I think, you're dealing with more abstract logic, which Haskell really lends itself to. ## more on the alonzo-church split ]]> gopher://gopher.someodd.zip:70/0/tech/programming/git-multi-user.txt 2024-11-28T00:00:00Z 2024-11-28T00:00:00Z

Configures git and SSH to cleanly separate multiple GitHub accounts...

username_public_key.asc gpg --armor --export-secret-keys somekeyhere > username_private_key.asc ``` Moving on, add the GPG key to GitHub (should be similar in the GitHub interface to adding a new SSH key): ``` gpg --armor --export somekeyhere ``` For these remaining configurations, I think there's a better way to do this, but configure `git` to use the GPG key for a specific repo: ``` git config user.signingkey somekeyhere git config commit.gpgsign true ``` Also, config `git` to use a specific username and email for this repo: ``` git config user.name "full name" git config user.email "you@example.org" ``` You can make signed commits like this: ``` git commit -S -m "Test signed commit" ``` ## Git configuration per domain This is a more maintainable approach to have defaults set per domain we have (matching our `~/.ssh/config` domains, used in repos): Create a `~/.gitconfig-username`: ``` [user] name = Full Name email = user@example.org signingkey = somekeyhere [commit] gpgsign = true ``` Update global git configuration (`~/.gitconfig`): ``` [includeIf "hasconfig:remote.*.url:git@github.com-username:*/**"] path = ~/.gitconfig-username ``` Check the applied settings in a repo that uses the domain `github.com-username`: ``` git config --get user.name git config --get user.email git config --get user.signingkey git config --get commit.gpgsign ``` ]]> gopher://gopher.someodd.zip:70/0/tech/running_a_server/bitcoin-server.txt 2024-11-26T00:00:00Z 2024-11-26T00:00:00Z

Notes on self-hosting Bitcoin by running bitcoind on a Debian serve...

User=youruser Group=youruser # Hardening measures #################### # Provide a private /tmp and /var/tmp. PrivateTmp=true # Use a new /dev namespace only populated with API pseudo devices # such as /dev/null, /dev/zero and /dev/random. PrivateDevices=true # Deny the creation of writable and executable memory mappings. MemoryDenyWriteExecute=true [Install] WantedBy=multi-user.target ``` Create script and set the permissions: ``` sudo vi /usr/local/bin/bitcoind-start.sh ``` The file: ``` #!/bin/bash # Just a simple wrapper to start bitcoind. # # If using systemd, simply create a file (e.g. /etc/systemd/system/bitcoind.service) # from example file below and add this script in ExecStart. # https://raw.githubusercontent.com-/bitcoin/bitcoin/76deb30550b2492f9c8d9f0302da32025166e0c5/contrib/init/bitcoind.service # # Then run following to always start: # systemctl enable bitcoind # # and the following to start immediately: # systemctl start bitcoind # If you are mounting a secondary disk, find the UUID of your # disk and a line entry in /etc/fstab e.g. # # UUID=foo-bar-1234 /path-to-dir/.bitcoin ext4 defaults 0 0 set -e # Let's wait for 30 seconds in case other processes need to come up first. sleep 30 echo "Starting bitcoind..." bitcoind --daemon --server -pid=/home/baudrillard/.bitcoin/bitcoind.pid -conf=/home/baudrillard/.bitcoin/bitcoin.conf echo "Done!" ``` ``` sudo chmod +x /usr/local/bin/bitcoind-start.sh ``` Enable: ]]> gopher://gopher.someodd.zip:70/0/tech/workstation/monero.txt 2024-11-26T00:00:00Z 2024-11-26T00:00:00Z

Cashing out Monero (XMR) from a long-neglected Debian server. Colle...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/ltfs-restic-backup-archives.txt 2024-11-18T00:00:00Z 2024-11-18T00:00:00Z

Using restic with LTFS to back up to LTO tape the easy and fast way...

> /mnt/ltfs/test.txt sudo umount /mnt/ltfs sudo mt -f /dev/st0 eject ls /mnt/ltfs ``` then try mounting again and checking the file. ## restic check out these tweaks: https://restic.readthedocs.io/en/latest/manual_rest.html from above i was sad to find that `--read-concurrency` is not available in the stable repo for debian as of right now (current restic verison is 0.14.x or whatever). Prepare the restic repository: ``` sudo RESTIC_PASSWORD_FILE=/etc/restic-password /usr/bin/restic init -r /mnt/ltfs ``` To find what you should exclude you should exclude firstly, the things in Linux that don't make much sense to me to backup (things like `/proc`), but also see what the largest files are, which may indicate which directories to ignore: ``` sudo find / -type f -exec du -h {} + | sort -rh | head -n 10 ``` Here's my command for backups, note the `pack-size` (chosen relative to LTFS block size), and be sure to `exclude` what you can (lowest hanging fruit, at least): ``` sudo RESTIC_PASSWORD_FILE=/etc/restic-password /usr/bin/restic backup / \ --exclude /home/baudrillard/.bitmonero \ --exclude /root/.bitmonero \ --exclude /nix \ --exclude /mnt \ --exclude /tmp \ --exclude /run \ --exclude /var/tmp \ --exclude /lost+found \ --exclude /media \ --exclude /sys \ --exclude /usr/share/ollama/.ollama/models/blobs \ --exclude /proc \ --exclude /dev \ -r /mnt/ltfs \ --compression max \ --pack-size 128 ``` Pretty impressive speed for me: ``` processed 414986 files, 64.833 GiB in 11:22 ``` Don't forget to keep `/etc/restic-password` backed up, like in a password manager! verify the backup: ``` sudo RESTIC_PASSWORD_FILE=/etc/restic-password /usr/bin/restic snapshots -r /mnt/ltfs ``` You can do a differential/incremental update by running the same backup command, but here I try adding over 1TB of data from my `/media` backup directories by simply running the backup command again, but removing `/media` from `exclude`. ## Things to think about handy debug: ``` sudo ltfsck /dev/sg4 ``` If you interrupt a backup it'll just totally mess up]]> gopher://gopher.someodd.zip:70/0/tech/running_a_server/lto_simple.txt 2024-11-18T00:00:00Z 2024-11-18T00:00:00Z

Simpler encrypted LTO6 tape backup workflow on Debian, meant as an ...

/etc/backup.passphrase' # now create the archive sudo sh -c ' tar --totals \ --checkpoint=100 \ --checkpoint-action=dot \ --use-compress-program="zstd" \ -cvf - /media/root/BackupRAID \ | gpg --symmetric --cipher-algo AES256 \ --batch --yes \ --pinentry-mode loopback \ --passphrase-file /etc/backup.passphrase \ | dd of=/dev/nst0 bs=1M status=progress ' ``` This is crazy fast. But if blocking factor is large you'll run out of space quickly. The solution is to perhaps place a single archive onto the tar. ## Test archive, restore See status: ``` sudo stenc -f /dev/st0 ``` Rewind and list contents: ``` sudo mt -f /dev/nst0 rewind sudo tar -tvf /dev/nst0 --use-compress-program=zstd ``` ### if you used pgp (best imo) Read test successful with: ``` sudo mt -f /dev/nst0 rewind sudo dd if=/dev/nst0 bs=64k count=1 | file - # Expect: "GPG symmetrically encrypted data" ``` and... ``` sudo mt -f /dev/nst0 rewind sudo dd if=/dev/nst0 bs=1M \ | g]]> gopher://gopher.someodd.zip:70/0/tech/running_a_server/server-too-noisy.txt 2024-11-18T00:00:00Z 2024-11-18T00:00:00Z

Making a home server quieter through BIOS fan curves and additional...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/dynamic-dns-linux-debian.txt 2024-11-16T00:00:00Z 2024-11-16T00:00:00Z

Dynamic DNS on Debian using afraid.org (freedns) with a cron-driven...

> /tmp/freedns_someodd_mooo_com.log 2>&1 & ``` Something like the above command, that you get from Afraid.org. You can crontab: ``` */5 * * * * ~/duckdns/duck.sh >/dev/null 2>&1 ``` You could also use systemd. Above does every five minutes. ]]> gopher://gopher.someodd.zip:70/0/tech/workstation/terminals_linux_vt320.txt 2024-10-20T00:00:00Z 2024-10-20T00:00:00Z

Connecting a VT320 terminal to a Debian server over serial via a US...

key` * `n key Tab setup: * clear all tabs * set 8 column tabs I think this is a reference to the tab setup: ``` T T T T T T T T 12345678901234567890123456789012345678901234567890123456789012345678901234567890 ``` ### Server setup Let's get the server to accept serial login/TTY. As you can see the device I'm dealing with is a USB/serial adapter/TTY. ``` systemctl enable serial-getty@ttyUSB0.service reboot systemctl start getty@ttyUSB0 ps -ef | grep agetty ``` I think I also edited `/etc/default/grub` (yikes!): ``` #GRUB_CMDLINE_LINUX_DEFAULT="quiet" GRUB_CMDLINE_LINUX_DEFAULT="console=ttyUSB0 console=tty0" GRUB_CMDLINE_LINUX="" ``` After editing grub, run `updategrub`. I think I may have set the speed of the TTY, but I'm not sure if that actually helped much: `stty -F /dev/ttyUSB0 speed 9600`. A handy test command: `cat /dev/ttyUSB0` and then try typing stuff on the terminal. I got a tip to set `$TERM` variable on the server. Try `echo $TERM`. I think it may help to set `TERM=vt220` or whatever you have a terminfo entry for. Try checking `/lib/terminfo/v/` and see the entries. You don't want to set that in your login script, since it'll be different if you're in an xterm versus on the terminal itself. Set it in the systemd script for the getty on the serial line. Edit `/lib/systemd/system/serial-getty@.service` and change: ``` ExecStart=-/sbin/agetty -o '-p -- \\u' --keep-baud 115200,38400,9600 %I $TERM ``` to: ``` ExecStart=-/sbin/agetty -o '-p -- \\u' --keep-baud 115200,38400,9600 %I vt220 ``` ### Updated instructions I did some extra stuff like adding to grub I'll document later... ## Terminal advice: living inside screen This advice comes from someone (sorry I forget their name, so no credit). Type 'screen', and then you get basically a fresh terminal instance. Now, you can run whatever programs you want. When you want a new "window", hit Control-a, then press c. Tapping control-a a will switch back and forth between this window and the window you were in. You can have lots of windows open, and switch between them with control-a #, where # is the window number you want to go to. (it starts at 0) There are a whole ton of other things you can do with screen, of course, but one of the more powerful things you can do is to type control-a d - which detaches the screen session. Then you can log out - and when you log back in, you can type 'screen -r' to reattach to the session you detached. And... it gets better. If you connect to this machine from another terminal, or from the network, you can type 'screen -x' on the other terminal, and attach to the same running screen session again. So you have two terminals, both interacting with the same set of "windows". And you can have different wi]]> gopher://gopher.someodd.zip:70/0/tech/programming/haskell-compiling-windows.hs 2024-10-14T00:00:00Z 2024-10-14T00:00:00Z

How to cross-compile Haskell binaries for Windows from Linux. Writt...

gopher://gopher.someodd.zip:70/0/tech/programming/liquidhaskell-verification-stack.txt 2024-10-14T00:00:00Z 2024-10-14T00:00:00Z

Getting started with LiquidHaskell and Stack for refinement-type ve...

gopher://gopher.someodd.zip:70/0/tech/gopher/liferea-gopher-rss.txt 2024-10-13T00:00:00Z 2024-10-13T00:00:00Z

How to subscribe to the author's Atom feed from gopherspace using L...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/restic_entire_server.txt 2024-10-13T00:00:00Z 2024-10-13T00:00:00Z

Restic backup recipe covering an entire Debian server. Chooses rest...

"$LOG_FILE" 2>&1 if [ $? -eq 0 ]; then DATE=$(date "+%Y-%m-%d %H:%M:%S") echo "Backup completed successfully at $DATE" >> "$LOG_FILE" else DATE=$(date "+%Y-%m-%d %H:%M:%S") echo "Backup failed at $DATE" >> "$LOG_FILE" fi ``` I also created a script to check the integrity and prune restic (`/usr/local/bin/restic-maintain.sh`): ``` #!/bin/bash LOG_FILE="/var/log/restic-maintain.log" export RESTIC_PASSWORD_FILE=/etc/restic-password export RESTIC_REPOSITORY=/media/root/BackupRAID/server-backups/restic-simulacra-entire-server-start-2024-10-13 # Check integrity and prune restic check > $LOG_FILE restic forget --prune --keep-daily=7 --keep-weekly=4 --keep-monthly=12 >> $LOG_FILE # Appends a timestamp to the log date "+%Y-%m-%d %H:%M:%S" >> $LOG_FILE ``` Let's simply add the script to root's crontab: ``` sudo crontab -e ``` Add these entries (backup daily at 2am, maintain weekly Saturday at 6am): ``` 0 2 * * * /usr/local/bin/restic-backup.sh 0 6 * * 6 /usr/local/bin/restic-maintain.sh ``` ## Restore from backup (when needed) ### List snapshots ``` restic snapshots -r /media/roo]]> gopher://gopher.someodd.zip:70/0/tech/workstation/lyx.txt 2024-10-02T00:00:00Z 2024-10-02T00:00:00Z

Quick notes on LyX, a WYSIWYM front end for LaTeX. Covers exporting...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/Server Torrent Create.txt 2024-10-01T00:00:00Z 2024-10-01T00:00:00Z

Creating torrents for data longevity on a Debian server using trans...

gopher://gopher.someodd.zip:70/0/tech/gopher/subscribing-gopher-feeds.txt 2024-09-05T00:00:00Z 2024-09-05T00:00:00Z

How to subscribe to gopherspace Atom feeds using Liferea's command-...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/diagnosing-server-latency.txt 2024-09-05T00:00:00Z 2024-09-05T00:00:00Z

Debugging persistent high-latency on a gopherhole, ultimately trace...

> DiG 9.20.1-1-Debian <<>> gopher.someodd.zip ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60485 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;gopher.someodd.zip. IN A ;; ANSWER SECTION: gopher.someodd.zip. 844 IN CNAME someodd.duckdns.org. someodd.duckdns.org. 60 IN A 73.93.14.18 ;; Query time: 3403 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Thu Sep 05 13:36:17 PDT 2024 ;; MSG SIZE rcvd: 96 $ gopherhole git:(master) nslookup gopher.someodd.zip Server: 127.0.0.53 Address: 127.0.0.53#53 Non-authoritative answer: gopher.someodd.zip canonical name = someodd.duckdns.org. Name: someodd.duckdns.org Address: 73.93.14.18 ;; communications error to 127.0.0.53#53: timed out ;; communications error to 127.0.0.53#53: timed out ;; communications error to 127.0.0.53#53: timed out ;; no servers could be reached ``` `dig`: I believe the results above show an unusually high response time for the DNS resolution process (while inside the network). `nslookup`: I think the above results show problems with the local DNS resolver. I then tested adding this line to my `/etc/hosts`, which made browsing my gopherhole practially instantaneous: ``` 192.168.1.100 gopher.someodd.zip ``` I removed the above entry after testing. At this point I became tasked with switching away from DuckDNS, especially after reading about other people's problems. ## Switch from DuckDNS to Afraid.org I updated my DNS records (CNAME) on the domain registrar service I use (for someodd.zip) to point to the new Dynamic DNS service I signed up for: afraid.org. Afraid.org has been around for a long time, people seem to like the service and think it's reliable. I also liked that it uses FreeBSD. I felt that after switching to Afraid.org that the issues were mostly resolved. However, now, if I'm on the local network it now works very fast. If I'm outside the server's network the gopherhole works very slow on port 70, yet port 7070 and 7071 are fast. ## NAT I started thinking that this was a NAT issue. For some reason I ran into the concept of NAT loopback/hairpin/reflect before when looking into some issue I think was related to XMPP. To just double-check DNS I tried accessing `gopher://[server ip here]` on the external network and it was still slow. I also used a DNS propagation checker and it looked like the CNAME records rolled over to `someodd.mooo.com` away from `someodd.duckdns.org`. I decided to test out the NAT loopback hypothesis. On the server's network router, I enabled NAT loopback/masquerading for my `LAN => WAN` zone. Once the router reloaded everything was working ver]]> gopher://gopher.someodd.zip:70/0/tech/workstation/using-gnu-screen.txt 2024-07-03T00:00:00Z 2024-07-03T00:00:00Z

Why the author prefers GNU screen for persistent sessions — especia...

gopher://gopher.someodd.zip:70/1/tech/workstation/window_maker/window_maker_megapost 2024-05-17T00:00:00Z 2024-05-17T00:00:00Z

Living inside the lightweight Window Maker window manager on Debian...

Living inside the lightweight Window Maker window manager on Debian Unstable, with tips for making it feel like a complete environment. Covers dockapps, app selection, and open TODOs the author still wants to solve. gopher://gopher.someodd.zip:70/0/tech/running_a_server/xmpp-server.txt 2024-04-22T00:00:00Z 2024-04-22T00:00:00Z

Sets up an XMPP server on Debian for hosting your own encrypted mes...

gopher://gopher.someodd.zip:70/0/tech/workstation/dillo.txt 2024-04-01T00:00:00Z 2024-04-01T00:00:00Z

Compiling the very lightweight Dillo browser (with Gemini and Gophe...

gopher://gopher.someodd.zip:70/0/tech/running_a_server/irc-server.txt 2024-03-30T00:00:00Z 2024-03-30T00:00:00Z

Journey setting up an IRC server on Debian 12 with ngircd plus athe...

``` ## ngircd + LetsEncrypt (SSL) Get SSL connections working on the IRC server by using LetsEncrypt to manage the certificate automatically. I was able to specify my icecast2 web root (selecting 2, place in web root when asked) since I already have that in order to do the ACME verification or whatever! Otherwise you might wanna do something like `sudo certbot certonly --standalone -d irc.someodd.zip` (which will start a web server for you). ``` sudo certbot certonly --webroot-path="/usr/share/icecast2/web" -d 'irc.someodd.zip' ``` Do a bunch of file management for ngircd's permissions sake, but also create a `dhparams.pem`: ``` sudo mkdir /etc/ngircd/ssl sudo openssl dhparam -out /etc/letsencrypt/live/irc.someodd.zip/dhparams.pem 2048 sudo cp /etc/letsencrypt/live/irc.someodd.zip/fullchain.pem /etc/ngircd/ssl/ sudo cp /etc/letsencrypt/live/irc.someodd.zip/dhparams.pem /etc/ngircd/ssl/ sudo cp /etc/letsencrypt/live/irc.someodd.zip/privkey.pem /etc/ngircd/ssl/ sudo chown -R irc:irc /etc/ngircd/ssl/ ``` Point to those files in `/etc/ngircd/ngircd.conf`, in the `[SSL]` section. In fact here's basically my entire `[SSL]` block, with comments removed: ``` [SSL] CertFile = /etc/ngircd/ssl/fullchain.pem CipherList = SECURE128:-VERS-SSL3.0 DHFile = /etc/ngircd/ssl/dhparams.pem KeyFile = /etc/ngircd/ssl/privkey.pem Ports = 6697 ``` Note I have SSL connections accepted on port 6697. Allow connections to the configured SSL port with: ``` sudo ufw allow 6697/tcp comment 'SSL IRC' ``` Of course if you want to test external connections, do some port forwarding on your router. Finally, test it out by first restarting `ngircd` (`sudo service ngircd restart`) and then connecting to your server on that port with SSL! ### Handling LetsEncrypt renewals Certificates expire or something, I guess. Luckily LetsEncrypt makes automation fairly simple. Configure this in `/etc/letsencrypt/renewal/irc.someodd.zip.conf` under `[renewalparams]`: ``` renew_hook = cp /etc/letsencrypt/live/irc.someodd.zip/fullchain.pem /etc/ngircd/ssl/ && cp /etc/letsencrypt/live/irc.someodd.zip/privkey.pem /etc/ngircd/ssl/ && chown -R irc:irc /etc/ngircd/s]]> gopher://gopher.someodd.zip:70/0/tech/programming/haskell-nix.txt 2024-03-12T00:00:00Z 2024-03-12T00:00:00Z

Notes on getting a Nix flake working for a Haskell project to reduc...

=4.7 && <5, ``` And Here's a list where you can tell which version of GHC you may want to use based off your `base` constraints: https://wiki.haskell.org/Base_package This tells me it can use anything from 7.8.1 to (as of now) 9.8.1. Based off of deduction of APIs I kenw I had to set the versions of a few dependencies in my cabal file, basically, too. However, one of the dependencies (`errata`) I had to pin wanted something between `4.12` (GHC 8.6.1) and `4.17` (GHC 9.4.1). I changed my `base` constraints around this in the hopes it'd make things easier. Things to pay attention to in the `flake.nix` generated: * Is the cache being used? * `nixpkgs.url`: this may come in handy: https://lazamar.co.uk/nix-versions/ * `basePackages` I also found out a package I was using is broken in nixpkgs right now, so I just added it myself and removed it from dependencies (a library to convert a number to a roman numeral). Another issue I encountered was I needed a specific version of errata, basically, based off the API. I also searched nixpkgs to see which version holds errata 0.3.0.0 so I could set `nixpkgs.url` basically, luckily the flake made this easy: ``` packages = { errata.source = "0.3.0.0"; # Hackage version override # shower.source = inputs.shower; }; ``` after searching for the right errata version in https://lazamar.co.uk/nix-versions/ -- I found it in nixos-22.11. After doing so I think it was clear that I should use something older and more stable than the nixpkgs `unstable`, so I did this: ``` nixpkgs.url = "github:nixos/nixpkgs/nixos-22.11"; ``` Even so I still have this left over: ``` hspec-golden >=0.1 && <0.2 ``` Which I found strange because of these results: https://lazamar.co.uk/nix-versions/?channel=nixos-22.11&package=hspec-golden So i just made sure to set to the latest version that comes before 0.2: ``` packages = { errata.source = "0.3.0.0"; # Hackage version overrid]]>