24bit color and ncurses S. Gilles 2017-03-19 Background My daily drive computer these days doesn't run X (or Wayland), partly because I decided to exile binary blobs from that particular machine, and partially because I found that I don't really need it. Insert standard inspirational story of personal workflow changes here. Anyway, I find myself caring a bit more about colors configuration for CLI/TUI these days. The current state of affairs may be interesting for those in the future, since a few people have recently decided that it needs to change. I'm not interested in politics, but the design problems that will need to be overcome are not unique. The problem Most terminals these days support a full 256 color palette. The first 16 are usually configurable, and aside from the convention “1 = red, 2 = green, etc.” there isn't much a program can expect. The last 240 colors, however, are about as tightly defined as unwritten rules can be. In 1999, Todd Larason's patch[0] to xterm added what I believe to be the origin of the algorithm used in the ubiquitous 256color2.pl script from the XFree86 tree. The 216 colors from 16 to 231 (inclusive) are meant to cover enough colors to be useful, and the last 32 spots give some greys. I don't like these colors, however. Some pairs, like 51 and 87 (#00ffff and #5fffff), 46 and 82 (#00ff00 and #5fff00), or 68 and 69 (#5f87d7 and #5f87ff) are indistinguishable to me. Meanwhile, CIEDE2000[1] says that the best approximation to #d4cdb8 (my preferred light background) is 224 (#ffd7d7) or 188 (#d7d7d7), which are completely wrong. Good luck trying to find a good (in the sense of CIEDE2000) approximation for #00260b, #312d44, or #402a01. My goal is to allow programs I use regularly to be colored to the extent sRGB allows, without causing problems for other programs, across all the terminals I use (st, yaft, tmux). I claim that the current state of affairs doesn't allow doing this in a “sane” way where “sane” is roughly defined as “reinventing no wheels and not introducing problems that have already been solved”. Solutions (Three Different Ones) I identified three possible solutions: o Come up with a better palette than Larason's, locally patch all my terminals, and configure with 16 through 255. o Patch desired software to use the non-standard ANSI-like 24bit sequence[2], then configure with #rrggbb. There is lots of discussion on this one, most of it already out of date. o Patch desired software to use rarely supported OSC 4 sequence to adjust the palette locally, then configure with #rrggbb. Each involves at least locally patching software, but such a thing isn't impossible. A bit of background Some underlying machinery has to be assumed (or patched in if needed). Wikipedia has[3] a pretty good article on the ANSI-style escape sequences, which I won't repeat, and more detail is provided by Thomas Dickey[4], from the perspective of XTerm. We care about the following: o SGR 38 and 48, which instruct the terminal to use an entry in the palette as foreground or background, respectively; o The 24bit version of SGR 38 and 48, which isn't quite as standard (and has some very irritating inconsistencies between use of ‘;’ or ‘:’ as the delimiter), which directly passes r, g, and b instead of referring to the palette; o OSC 4, a more rarely supported code, which changes the r, g, and b components of a palette entry; o OSC 104, another uncommon code, which resets the palette. The terminfo database exposes these capabilities, via (in part): o colors, an integer parameter which describes the maximum number of distinct colors the terminal supports; o pairs, which describes how many distinct foreground/background pairs the terminal supports; o ccc, a boolean which describes whether the terminal can change colors (e.g. via OSC 4); o initc, which encodes the color changing escape sequence. Finally, the 800 pound gorilla in the world of TUI programming, ncurses, exposes those via (in part): o COLORS, a macro corresponding to colors; o COLOR_PAIRS, a macro corresponding to pairs; o can_change_color(), which detects ccc; o init_color(), which wraps initc; o attron(), init_pair(), and COLOR_PAIR(), which together allow setting the current foreground and background, assuming the palette is already set up. The unfortunate truth will be that this pipeline isn't quite as connected (and the connections are not as seamless) as could be wished. Method one: hardcode a different palette Locally patching terminals would be a bit of a pain, as would adding new colors to my working set. The problem, however, is the “without causing problems for other programs” clause (this will be a recurring theme). This can be demonstrated by (in case your terminal supports the most common version of initc) running the following: #!/bin/sh for x in $(seq 16 255); do r=$(head -c 1 /dev/urandom | od -t u1 -A n | tr -d ' ') g=$(head -c 1 /dev/urandom | od -t u1 -A n | tr -d ' ') b=$(head -c 1 /dev/urandom | od -t u1 -A n | tr -d ' ') printf '\033]4;%s;rgb:%2x/%2x/%2x\033\\' ${x} ${r} ${g} ${b} done which will use OSC 4 to scramble the extended palette (simulating choosing some non-standard palette and recompiling the terminal). Something like ‘TERM=screen-256color emacs -nw -Q’ should demonstrate the problem, which is that lots of software is built with the assumption of a particular palette. I don't want to be in a situation where I need to use a piece of software and, without time to configure it, find it unusable. So that method's out. Method two: 24bit SGR codes This is probably the method which will succeed in the future. Thanks to some rather agressive feature requesting, most popular terminals now have support for the feature. The notable exception is xterm. And with xterm comes the terminfo database and ncurses itself. If ncurses were to support this, it's not immediately clear how the feature would work with COLOR_PAIR(), but that's not a serious obstacle. The real problem would lie in terminfo. The colors and pairs entries for a terminal that supports 24bit color would too large for a 16-bit integer, so terminfo would probably have to gain an extra flag - tmux uses ‘Tc’, which it interprets as “actually, ignore colors and pairs”. Furthermore, tmux has to hardcode the actual escape sequence, because there's no initc-like facility for describing it in terminfo. Emacs takes a completely different approach, and the general consensus is to wait for a general consensus[5], because: The terminfo maintainer has no interest[6] in modifying terminfo's internals to make this better. At present, then, this method runs afoul of the “introducing problems that have already been solved” clause. Hardcoding escape sequences is exactly what termcap was meant to fix back in 1978. So that method's out. Method three: Reinitialized palette Part of the reason terminfo will probably not be modified (in the near future, by the current maintainer) is that adjusting the palette is already possible. Assuming your terminal has ccc and initc capability, the following should display in a suitably #include #include void set_fgbg(uint8_t fr, uint8_t fg, uint8_t fb, uint8_t br, uint8_t bg, uint8_t bb) { static short next_clobber = 16; static short next_pair = 1; short fc = 0, bc = 0, pair = 0; if (COLORS <= 16) return; if (!can_change_color()) return; fc = next_clobber; init_color(fc, fr * 1000 / 0xff, fg * 1000 / 0xff, fb * 1000 / 0xff); if (++next_clobber >= COLORS) next_clobber = 16; bc = next_clobber; init_color(bc, br * 1000 / 0xff, bg * 1000 / 0xff, bb * 1000 / 0xff); if (++next_clobber >= COLORS) next_clobber = 16; pair = next_pair; init_pair(pair, fc, bc); if (++next_pair >= COLOR_PAIRS) next_pair = 1; attron(COLOR_PAIR(pair)); } int main(void) { initscr(); start_color(); clear(); set_fgbg(0xff, 0x39, 0x10, 0x91, 0xff, 0xa5); mvprintw(0, 0, "Hello"); set_fgbg(0x87, 0x80, 0x70, 0x67, 0x1c, 0x11); mvprintw(0, 7, "world"); refresh(); getch(); endwin(); return 0; } There's a problem, though. That program doesn't clean up the palette through OSC 104 (or any other means), So this program will either o leave part of the palette clobbered, as in method one, or o hardcode ‘puts("\033]104;\a")’ somewhere, spurning terminfo like method two. This method is almost fixable, though. A terminfo flag for detection of OSC 104 would not break anything or deprecate other flags. Adding a function to ncurses to invoke this (or doing it automatically on endwin()) would fix everything. In fact, ncurses could probably do this without any help from terminfo. o Inspect the terminal's palette (through a modified 256color.pl, or similar). It should be whatever the terminal's default is. o Run the above program, which should show 24bit colors. o Suspend the program via ‘^Z’ or similar, inspect the terminal's palette again. It should show some clobbered colors. (*) o Reset the terminal via ‘reset’ or similar, verify the palette is reset. o Resume the program via ‘fg’ or similar. (**) o Press a key to end the program. o Inspect the terminal's palette. The palette is probably still clobbered, which means that at (**) ncurses must have done something that involved holding information from (*). All ncurses would have to do would be to perform (*) on initscr() and (**) on endwin() for this technique to work. Conclusion As of this second, I don't believe it's possible to use 24bit color in an ncurses/terminfo program portably. This probably means that many programs which do this currently will break horribly at some unspecified point in the future. A small correction or two to ncurses would make method three work, but popular will seems more focused on method two. [0] http://invisible-island.net/xterm/xterm.log.html#xterm_111 [1] https://en.wikipedia.org/wiki/Color_difference#CIEDE2000 [2] https://gist.github.com/XVilka/8346728 [3] https://en.wikipedia.org/wiki/ANSI_escape_code [4] http://invisible-island.net/xterm/ctlseqs/ctlseqs.html [5] https://bugzilla.gnome.org/show_bug.cgi?id=778958 [6] http://lists.gnu.org/archive/html/bug-ncurses/2017-02/msg00019.html