Regular readers will know how much I enjoy a good tinker with my system. I have been playing with Tiddlywiki recently, partly as a result of admiring Jack Baty’s rudimentarylathe.wiki instance of Tiddlywiki, and partly because of Soren Bjornstad’s Grok Tiddlywiki book, which I think I also found out about via Jack. I had tried out Tiddlywiki before but never quite got it. Soren’s book helped me to see how flexible it could be and how I might be able to use it in a similar fashion to the way I have been using org-roam. While I still enjoy org-roam, things feel (to me anyway) a bit up in the air with it at the moment, as there are big changes coming in version 2 which will probably involve a bit of backwards incompatibility. I couldn’t decide whether to wait to make the changes, or transition to the new version now, and that indecision made me reluctant to add to my collection of notes. In addition — for reasons too long and boring to go into here — I have also moved (reluctantly) from Bookends to Zotero. I like the flexibility that Zotero offers to those of us having to live in a dual Word/Pandoc citations world, but I really miss Bookends’ speed and UI.

Anyway, this change in the tools I depend on left me with a puzzle: how could I export references (with metadata) from Zotero to Tiddlywiki so that I could make notes (known as ’tiddlers’) on each journal article of interest? There’s a vanishingly small possibility that anyone else might want to solve a similar problem in exactly the same way as me, but in case anyone is curious, this is how I made it work.

First, a health warning: this is big, messy hack, involving multiple bits of software and a lot of the coding equivalent of gaffer tape. It works for me, but there’s a lot which is specific to my setup. Nevertheless, I thought it might be useful to explain what I did, in case it inspires anyone else to set up their own system using bits of this solution. I’ll also indicate what you can do with each separate component, in case you want to pick and choose which parts you use.

The overview of this system is as follows:

• Zotero can export metadata of selected references in various forms using Javascript files called ’translators’. We can therefore export metadata in JSON format, making sure that we include the fields that Tiddlywiki expects to find in a JSON-format import.
• If you use the node multi-file setup of Tiddlywiki and run it as a server (locally), you can make use of the Tiddlywiki API to create tiddlers by passing some parameters plus a JSON body, which includes the content of the tiddler and its fields.
• The part we need to connect the two is a script of some kind to take the JSON exported from Zotero, construct a PUT request and submit it to the API. Since I have been learning Go recently, I decided to make a Go script to do this, and call that script with Keyboard Maestro to glue together a few other useful steps that I will explain below.

Zotero

First you need a translator file which gets placed in ~/Zotero/translators. I based mine (which I’ve put in this gist) on one I found (somewhere, unfortunately I can’t remember where) which also exported JSON. I then looked at the JSON format that Tiddlywiki needs for imports and used that to construct the correct keys and values. Zutilo is a very useful Zotero plugin that I was using anyway, but which offers a way to set up particular translators so that they are accessible from a contextual menu or keyboard shortcut. The goal I had with getting this pipeline to work was to export the JSON from Zotero to the clipboard, then use the macOS pbpaste command to pipe the clipboard contents into the Go script. To make this work, you add the translatorID UUID string (line 2 of the script) in Zutilo to identify the translator to call for the alt1 action. This is set up by going to Preferences > Advanced > Config Editor, then entering a value for extensions.zutilo.quickCopy_alt1 of export=3c2ea… with the full UUID string after the equals sign. Then in Tools > Zutlio Preferences you can set up a contextual menu item and a hotkey for QuickCopy Alt1 (I used Cmd+Ctrl+t).

At this point, you should be able to select one or more references in Zotero, use your hotkey to trigger the QuickCopy Alt1 command and find that your clipboard contains the JSON with the metadata of those references.

Go script

As I mentioned above, I’m only using Go here because I’m learning it and so it was a nice opportunity for a practical application. It does also enable you to build a self-contained binary that you can put somewhere in your path so that you don’t have to maintain a whole ecosystem of packages and their dependencies. However, if you are proficient in Ruby or Python (or shell scripting for that matter), it should be perfectly possible to use the Go script in my gist as model to write a version in your language of choice. I set it up so that the compiled binary (which I called twimport) takes the pasted JSON data exported using the translator as piped data. It parses this to extract the title and structure of each of the items (if more than one was selected), and then constructs a PUT request to the Tiddlywiki API to create the item.

The API requires you to specify the title of the tiddler as one of the request parameters. I decided to use the citekey field of the reference in Zotero as my title, as (the way I have it set up), it includes the first author’s name, the first few words of the title, and year, and I can guarantee that it is unique. The translator has this citekey as one of its fields, so the script just grabs that to construct the API call. It is also worth noting (as this tripped me up initially) that the translator stores the JSON items as an array, even if you only select one item. However, the Tiddlywiki API expects a single item and throws an error if it gets an array. The script must therefore iterate over the array and construct an API request for each item. If you are trying to work out how to do this in another language, I heartily recommend trying out Paw (which is available on Setapp). I’ve used it a lot recently when trying to use APIs, and it is fantastic. You can try out different parameters and see exactly what response you get back. I never know exactly how to format headers or bodies, so being able to click around and set it up in the Paw UI makes it so much easier. It also enables you to export a request once you have sorted out the correct setup (using File > Export Request) in a variety of different languages. This was how I got the skeleton of the Go version, which was a huge help.

If you get this stage working, you should be able to pipe your clipboard contents (containing the JSON exported by the translator) into this script to trigger Tiddlywiki to create a new tiddler. In my case, I can do this with pbpaste | twimport (as twimport is in my path).

Gluing the parts together

Now that we can export JSON containing the metadata to the clipboard, and pipe that clipboard data to our Go script to create a new tiddler, it would be handy to be able to trigger this thing with a keyboard shortcut rather than going to the command line. In other words, it’s time to get out the gaffer tape again and stick a few more components on to our beautiful mess of code!

As I own it anyway, and use it for everything, my method of choice here was Keyboard Maestro. This is fairly simple, as it just automates the steps we would otherwise perform manually.

1. We start with Zotero frontmost, with one or more references selected.

2. Keyboard Maestro then triggers the keyboard shortcut to QuickCopy the translated reference (in my case Cmd+Ctrl+t as explained above). That puts the JSON in the clipboard.

3. An ’execute shell script’ action then uses the pbpaste | twimport command to run the Go script and create the tiddler(s).

4. I use a single site app to view my Tiddlywiki (using Coherence X, again available on Setapp), to make it feel more like a real macOS app. The final step in my macro activates this app, then reloads the page. This is necessary because externally adding files to the server version of Tiddlywiki does not automatically trigger the interface to refresh and register those files. For completeness, I also use the macro to trigger a notification with the name of the tiddler we’ve added.

Of course, the node tiddlywiki server needs to be running for the API to work. Again, you can just start it at the command line, or you can use these instructions to create a .plist file to go in ~/Library/LaunchAgents so that the server starts when you log in.

Within Tiddlywiki, tiddlers tagged with the ‘Source’ tag (like the ones created with this workflow) have a template automatically applied which adds an HTML table to the top of the tiddler containing selected fields (such as authors, year, title and so on). I also copied the ‘Idea Explorer’ table from Soren’s own Zettelkasten, which automatically summarises and links to links and backlinks from the current tiddler.

I’m enjoying this setup. It started as an experiment driven by curiosity more than anything, but I’m enjoying taking notes and making links between ideas with Tiddlywiki, which means that I’m writing notes more often. Tiddlywiki also enables a lot of flexibility in the way that tiddlers can be searched, grouped and displayed, which means that if I decide in the future that I need to extract particular information from my notes, I can do so easily.