Org mode and Pandoc

· geek · software ·

In the intervening time since first setting up Spacemacs I have been getting more familiar with org-mode, and becoming increasingly impressed by its power and flexibility. Consequently, I’ve started writing by default in org-mode rather than Markdown. What I like about it is that it provides a lot of features for structuring, re-arranging and editing text. For example, text with headings, lists and so on is defined as a tree, and there are commands for moving nodes within this tree up and down, or promoting or demoting them. These nodes can also be shown or hidden selectively, and you can even ’narrow’ a buffer to a particular sub-tree, which means that this particular section of the document temporarily appears to be the only contents of the buffer. This is wonderful in a long document when you want to focus on a specific part, but still have access to the rest of the document when needed. Table editing is also superbly easy in Org files — it’s almost worth using org-mode for this alone if you often include tables in your documents. Of course, the point of writing in either Markdown or Org format is that it is human-readable and easy to edit, but can be transformed into many other final document formats using Pandoc. Originally, I used the Pandoc mode package with Spacemacs, which is excellent and works with any source file format, but for some reason it stopped working for me. I tried everything I could to get it working again, but had no luck, so I was looking around for another alternative.

I came across a very interesting article by Erik Clarke about using org-mode as a lab notebook. While I didn’t want to implement all of the details of his lab notebook setup1 he mentioned a Pandoc export mode for org-mode called ox-pandoc. The pandoc-mode package is certainly a more versatile solution since it works with files in any of Pandoc’s input formats, but since I have converted to org-mode, it doesn’t matter to me that ox-pandoc will only deal with Org format input files. In actual fact, it makes configuration a lot simpler, and means that it integrates with org-mode’s built-in export menu. This makes it easy to select an output format and either save to file (optionally opening that file automatically) or view the results in a buffer, all with quick access keystrokes. In Spacemacs, this means if I want to export to PDF and view the file in Preview, I hit , (my Spacemacs leader key) then epl (for Export Pandoc to LaTeX PDF). The keys are mostly nicely mnemonic, but the menu is there to jog your memory if you forget the precise keys needed.

I created a Spacemacs configuration layer (called ‘my-org’) in which I loaded the ox-pandoc package and configured the appropriate default settings. The package cleverly allows you to set common default settings that you want to apply to all files (or all files of a particular export format) in the configuration file, then use document settings at the top of your org file (like #+PANDOC_OPTIONS: toc-depth:3) to tweak particular settings for individual files. I have set up all files to be rendered as standalone files, parse raw markup and use my main *.bib file for the bibliography, then I have other settings for LaTeX and Beamer PDF files to use XeLaTeX, set fonts and so on. Similar settings for HTML output set up a nice CSS file to be included. This means the basic settings are already configured, and in the org file itself I can choose a CSL file to style the bibliography (which may be different for each file), and also set whether I want a table of contents, numbered sections or not, and so on.

It works very well. Most of the built-in org-mode features for adding metadata or defining structure (e.g. setting a title, author and date, or including figure or table captions) translate seamlessly to the output format file if appropriate. I haven’t managed to get cross-references working yet, but there is a pandoc-crossref filter that looks promising. I understand that converting from org mode is a relatively new feature in Pandoc, so I’m sure it will gain more features with time, but for now it seems very serviceable (for my needs, anyway: mainly converting to HTML, LaTeX/PDF and Word format).

As for a lab notebook, Erik’s article inspired me to set up a dedicated lab notebook Org file, and a set of capture templates so that I can quickly add notes, link to other Org files and so on by just hitting Ctrl-C c. The useful thing about the capture system is that you don’t have to open the lab notebook file and lose where you were in your current document. Instead, you hit the key binding which opens a mini buffer for you to enter your text. When you’ve finished, you hit Ctrl-c Ctrl-c, the mini buffer closes, and you’re back right where you started in your original document. In the lab notebook, I have it set up so that there’s a heading for each year, month and day (generated automatically), and each new entry is added with a time within the relevant day heading. If I want to browse to entries a few days, weeks or months ago, I can use the agenda window to find and move right to particular entries. I’m using this notebook to document random thoughts throughout the day, note papers I have read (this is where having Pandoc and pandoc-citeproc comes in very handy, as Erik pointed out), and record progress I have made. I tend to create separate Org files to work on more substantial notes or documents, but the capture system means that I can automatically capture snippets of text from them, with a link back to that point in the original document. I keep all these Org files in one place (an ‘org’ folder in my Dropbox) because I find it easier to have them all in one location, but once a particular document is finished and I am no longer working on it, I move it to the relevant folder for that project. I find this works well for me as it keeps things tidy, but I know where all my ‘working files’ are, and they are easy to switch between in Emacs.

I must say that Spacemacs has totally converted me to Emacs. The Spacemacs project is very active and the experience gets smoother and more powerful with each release. The best part is that it has allowed me to use the mighty org-mode without having to deal with Emacs’ text editing shortcuts, and use the power of Vim instead!

  1. I am a reflexive file saver, so committing a lab notebook to a repository on every file save means filling in a lot of very similar commit messages! It’s a clever idea though. ↩︎