ZIPPY ORG-AGENDA BUFFER BUILDING
-
-
Preamble
------------------------------------------------------------
`org-agenda', if you don't know about the Emacs package,
builds views out of `org-mode' notes. Usually the views will
show the current day or week, showing all notes with
timestamps landing within the date range. But you can also
see notes with todo keywords, specific tags, or other
properties if you want. It's pleasantly accommodating to a
variety of workflows.
I use `org-agenda' to show a chronological view into my note
taking. I like being able to look back on an arbitrary
period of time, like last week or last month, to see what
I've written. I find the act of review to build context and
confidence in my life.
,----
| Week-agenda (W24):
| Monday 10 June 2024 W24
| Tuesday 11 June 2024
| koreader: 19:42...... Developing
| github_cheatcodes: 20:33...... How to squash many patches
| Wednesday 12 June 2024
| Thursday 13 June 2024
| Friday 14 June 2024
| php: 8:14...... Setup on OpenBSD
| dsc_pc1555: 16:34...... Connecting
| Saturday 15 June 2024
| Sunday 16 June 2024
`----
Building and rebuilding my `org-agenda' is usually very
slow. I have 474 notes at the time of this writing. Each of
these notes needs to be opened and parsed to determine its
eligibility for inclusion in the agenda. It takes /minutes/
to build an agenda buffer, which is a huge deterrent on me
actually using the feature.
Many times I've searched the WWW for ways to speed up my
`org-agenda' views. There are some packages that purport to
speed things up, but I found their mechanisms difficult to
understand, and their integration into my existing
`org-mode' config excessive. (I like to maintain a pretty
simple set of configuration files for Emacs, about 400
lines.) So I decided to embark on the task of finding my own
way to faster `org-agenda' views. The remainder of this
writing traces my thoughts, development notes, and outcome.
Development notes
------------------------------------------------------------
To start, I ran the Emacs profiler while building a
`org-agenda' buffer build. I observed that the
`org-agenda-get-timestamps' function accounts for 33% of CPU
usage and 2682 samples. I figured the high samples count is
probably owed to the many agenda files; again, 474 such
notes. So with some confidence I assumed that `org-agenda'
buffers could be built faster if the agenda had fewer files
to look through. Searching the web, I found lots of ad-hoc
advice from other individuals facing a similar problem. I
read about hacks to dynamically populate the
`org-agenda-files' variable. Nobody's specific approach
appealed to me, but I got some good ideas. In short, I
decided I'd use `grep' to pre-sift my `org-mode' notes for
inclusion in my agenda.
I was concerned that my approach would get messy. Modifying
a variable crucial to building the agenda could break lots
of stuff? But because the damn thing was so slow and I
hardly used it anymore, better to restore one very-used
feature even if it risks breaking some amount of less-used
features.
Next, I looked into the advising functions of Emacs---a very
powerful feature! I had great success using such a function
before to fix a bug with `org-download'. For this task, I
imagined a similar approach: evaluating my function before
the `org-agenda' view forms evaluate. But I'd first need to
find the right function to advise. I experimented with a few
that seemed fortuitous.
,----
| (advice-add 'org-agenda-list :before #'roygbyte/org-agenda-list-before)
| (advice-add 'org-agenda-list :around #'roygbyte/org-agenda-list-around)
| (advice-add 'org-agenda-list :filter-args #'roygbyte/org-agenda-list-filter-args)
| (advice-add 'org-agenda-prepare :before #'roygbyte/org-agenda-prepare-before)
`----
I found that advising my function `:before'
`org-agenda-list' or `org-agenda-prepare' successful. At
either point, I could grab information related to the
generating agenda buffer. That information was contained in
some of the following variables:
,----
| (print org-agenda-redo-command)
| (print org-agenda-start-day)
| (print org-starting-day)
| (print org-agenda-span)
| (print org-agenda-overriding-arguments)
| (print org-agenda-buffer-name)
| (print org-keys)
`----
I observed the values of these variables in different agenda
contexts, like a week view, custom command, or existing view
rebuild. Trial and error led me to observe the variables
that would be helpful for my operation: `org-agenda-span'
and `org-agenda-overriding-arguments'. (By the way, I found
all candidates for observation by reading through the
`org-agenda-list' function, as well as the `defvar' and
`defcustom' variables declared throughout `org-agenda.el'.)
With this, I now had the key parts and could proceed to
writing the code.
I split the logic of the operation into two main parts:
identifying paths to the `org-mode' files of interest for
the given agenda; and, modifying `org-agenda-files' to only
contain files associated with the search query for a given
agenda. I began by building first part, which would in
effect be my invocation of `grep' from Emacs.
grep search for relevant files
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
To invoke `grep' from Emacs I use a subprocess. Subprocesses
can be invoked synchronously or asynchronously. I chose the
former, since it's easier to program and better accords with
the situation. I built and tested my `grep' command and its
argument in the command line, then translated it into
ELisp. Note that there are many ways to define a
subprocesses' command, and my method is maybe not the best.
,----
| (apply 'call-process "grep" nil t nil "-soE" regex-pattern files-to-search)
`----
I use `apply' to expand the list `files-to-search' into in
individual arguments that are passed to `call-process'. I
provide the files as individual arguments (and not just
their enclosing directory) because it offers more certainty
over how `grep' is invoked. Passing a directory and the `-R'
flag would be the alternative to this approach. But in my
trials recursing through a directory opened a host of
problems I decided to avoid.
,----
| (with-temp-buffer
| (setq files-to-search (directory-files
| directory t "\\.org\\'"))
| (apply 'call-process "grep" nil t nil "-soE" regex-pattern files-to-search)
| (goto-char (point-min))
| ;; Clean up buffer
| (while (search-forward ":" nil t)
| (delete-region (- (point) 1) (line-end-position)))
| ;; Translate buffer into list
| (seq-filter (lambda (a)
| (not (string-empty-p a)))
| (split-string (buffer-string) "\n")))
`----
The `grep' subprocess is invoked inside a temporary
buffer. When the subprocess finishes, that buffer contains
the results. Unfortunately, my distros version of `grep'
doesn't support only returning matching filenames, so I have
to clean up the buffer. Finally, the buffer is split by
newline into a list and completes by evaluating to that list
of org files matching the initial regular expression.
Ad-hoc modification of org-agenda-files
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The second part of my operation is building the regular
expression. In practice, this function is evaluated first. I
only wrote it second because it seemed the more difficult
bit of code to write. Indeed, it was.
Two values are crucial to building any agenda: start date
and span. The start date corresponds to when the agenda view
begins. The challenge is that `org-agenda' stores these
values in different variables depending on how the agenda is
being built. For instance, if the agenda is being built
"fresh", then `org-agenda-overriding-arguments' will be
`nil'. In that case, the start day is stored in
`org-agenda-start-day', unless that variable is `nil', in
which case the start day will be `(org-today)'. However, if
the agenda already exists and is being refreshed or moved to
the previous or next week then the values will all be in
`org-agenda-overriding-arguments'.
I also got tripped up on how `org-mode' represents
dates. After some digging, I learn that `org' represents
this value as "the number of days elapsed since the
imaginary Gregorian date Sunday, December 31, 1 BC." I find
this by tracing `(org-today)' back to `(date-to-day)'.
Another important detail to building an `org-agenda' view is
using the right start day. I wrote a small function to
translate a given day to the start of the week. For
instance: Thursday, October, 3, becomes Sunday, September
30. This function must also accommodate the possibility that
`org-agenda-start-on-weekday' is set to Monday, which it
could be for strange people who don't start their week on
Sunday.
Code
------------------------------------------------------------
Below is the finished code. By my eyes, it's not the best
looking Lisp. I still struggle with choosing the best place
for line breaks and indentation. It also seems really long
and complicated for such simple operations. About that:
there's a lot of logic required for ensuring dates are in
the correct format, and that edge cases are accommodated. I
don't know if it can be simplified? Maybe. Whatever.
A caveat: my extension doesn't handle non-standard todo
keywords or even tags. I don't use those much, if at all. I
provided a minimal amount of logic for the standard `TODO'
keyword, and that's it. I think extending the function to
support other stuff is easy, but I have no use for it right
now.
,----
| (advice-add 'org-agenda-list :before #'roygbyte/org-agenda-list-before)
| (defun roygbyte/org-agenda-list-before (&rest r)
| "Speed up agenda buffer generation by dynamically modifying
| `org-agenda-files`. variable before `org-agenda-list` function
| is evaluated. Performance gains for buffers are largely
| accomplished by using grep to search for org notes containing
| timestamps (or TODOs!) corresponding to the desired time range."
| (cond
| ((or (string-equal org-keys "t")
| (string-equal (car org-agenda-redo-command) "org-todo-list"))
| (setq org-agenda-files
| (delete-dups (roygbyte/files-in-directory-matching-pattern
| (file-truename
Happy helping ☃ here: You tried to output a spurious TAB character. This will break gopher. Please review your scripts. Have a nice day!
| (t (let ((start-day nil)
| (end-day nil)
| (span nil)
| (regex nil))
| (when org-agenda-overriding-arguments
| (setq start-day (roygbyte/day-to-start-of-week
| (nth 1 org-agenda-overriding-arguments)))
| (setq span (let ((span-ambiguous
| (nth 2 org-agenda-overriding-arguments)))
| (if (symbolp span-ambiguous)
| (org-agenda-span-to-ndays span-ambiguous)
| span-ambiguous)))
| (setq end-day (+ start-day span)))
| (when (not start-day)
| (setq start-day
| (cond
| (org-agenda-start-day
| (time-to-days (org-read-date nil t org-agenda-start-day)))
| (org-starting-day org-starting-day)
| (t (roygbyte/day-to-start-of-week (org-today))))))
| (when (not span)
| (setq span (if (numberp org-agenda-span)
| org-agenda-span
| (org-agenda-span-to-ndays org-agenda-span))))
| (when (not end-day)
| (setq end-day (+ start-day span)))
| (setq regex
| (string-join
| (cl-map 'list (lambda (day-number)
| (let ((l (calendar-gregorian-from-absolute day-number)))
| (format "%d-%02d-%02d"
| (nth 2 l) (nth 0 l) (nth 1 l))))
| (number-sequence start-day end-day)) "|"))
| (setq org-agenda-files
| (delete-dups (roygbyte/files-in-directory-matching-pattern
| (file-truename
| (concat org-roam-directory "/pages/")) regex)))))))
|
| (defun roygbyte/day-to-start-of-week (day)
| "Take a DAY, represented by number of days since December 31, 1 BC, and translate
| value so it becomes the start of week."
| (let* ((day-of-week (calendar-day-of-week
| (calendar-gregorian-from-absolute day)))
| (weekday-start org-agenda-start-on-weekday))
| (- day (abs (- weekday-start day-of-week)))))
|
| (defun roygbyte/files-in-directory-matching-pattern (directory regex-pattern)
| "Evaluates to a list of absolute file paths in DIRECTORY whose
| contents match REGEX-PATTERN."
| (with-temp-buffer
| (setq files-to-search (directory-files
| directory t "\\.org\\'"))
| (apply 'call-process
| "grep" nil t nil "-soE" regex-pattern files-to-search)
| (goto-char (point-min))
| (while (search-forward ":" nil t)
| (delete-region (- (point) 1) (line-end-position)))
| (seq-filter (lambda (a)
| (not (string-empty-p a)))
| (split-string (buffer-string) "\n"))))
`----
Footnotes & references
------------------------------------------------------------
- `org-roam' was once the cause for a brief crisis in my
life. One day, I rebooted my computer and setup my
environment in the usual way, by running `startx' in my
login shell after login. The computer appeared to "hang". My
i3 status bar and other window manager features did not
appear. I was terrified I broke my indispensible device!
Well, after a bit of investigation I uncovered the cause:
invokation of `emacs --daemon' in my Bash RC file, which
hanged /forever/ while `org-roam' attempted to sync files to
its database.
-