There has been an explosion in new note taking applications recently, spearheaded by Roam Research – a premium note taking application created by Conor White-Sullivan.
Many of these note taking apps are based on the belief that knowledge doesn’t exist in a vacuum – there is a good chance that as you take notes there are lots of opportunities for cross pollination of ideas. When taking a note, you can reference any of your other notes and eventually build up a graph of knowledge, with links between the ideas.
The key idea driving this personal-wiki revolution is hard to invent but easy to replicate. By that I mean it took many years for the zettelkaten to reach a mass audience but now that it exists we’ve see a new app on the market copying Roams features seemingly every week.
We now have Foam, Athens, Obsidian & others pushing in on the space created by Roam, but I don’t think Roam (the company) has anything to sweat – they have a fervent following and they have done fantastically well getting productivity gurus onside to help promote the product.
The vision of a tool for all your knowledge.
I however gave up on Roam within a few weeks of use. It’s not for me. I think the idea of linking notes digitally is a massive improvement over a little box of paper based notes.
Universal search, longevity of notes, the ability to backup and the ability to lose your device and still have access to all your notes is a marvel – but I think there are some philosophical choices in the product that don’t suit me.
Roam seems to be deeply inspired by emacs and org mode – and as such, feels like a relic of an older time. Everything in Roam is treated as a block of text, and, yes that text can be manipulated, searched and referenced in other blocks of text.
But it it feels unbearably limited. Attacking org-mode on the internet is never a good idea – its fans are rabid and swear by the tool. In every online forum thread about knowledge base software you will find the diehard org-mode fan telling you about their tower of lovingly crafted note taking magnificence. And I’m not really attacking it – It’s just not for me.
It seems like we should be striving to do more in the year 2020 than replicating a paper note book or reference cards in a filing cabinet. I find time and time again all these ‘affordances’ in our digital life that map real world objects to digital counterparts.
The emacs / VSCode divide.
The File Explorer makes heavy allusions to a real world manilla folder, your mail has ‘attachments’ like you are sending pieces of paper around with paperclips on them, you have a little trash bin where deleted files go.
All these little details mapping to the real world made an enormous amount of sense when the world didn’t understand computers.
But a modern computer has the capability for so much more – new ways of doing and thinking. When you start typing the double brackets in Roam to link another note it prompts the user to ‘Search for a Page’. I can link bits of text to other bits of text. And that’s it.
I can’t do anything with that graph above. I can move the nodes around, but it doesn’t save where I move them. I can’t drop files in there, I can’t see images. The graph in Roam is supposed to be the outcome of my connected thoughts, all the knowledge I’ve collected using the tool, but it can’t represent images or PDF file for example. It feels very much like emacs in that regard – it feels limited in the modern world.
Where is my VS Code of connected knowledge? I don’t want to mentally have to convert pieces of knowledge (images, files, etc) into text counterparts (hyperlinks). I want the tool to provide value and stay out of my way. I want the computer to watch over me with loving grace. Look what happens when I open a new file type in VS Code.
What does emacs do when it encounters an unseen file type? Nothing. It stays quiet. Doesn’t offer any help. This suits some people – they love that control. Not me – I wish computers did more things like the VS Code prompt. Be smart, help me when you can, and do what I think you should do!
I want to drop a file into Roam and have it show in the graph, and I want to be able to connect absolutely everything and anything to it.
Images to PDFs, Emails to Slack Messages, And everything to my personal notes. I might be proven wrong in the future about this – but it doesn’t seem like this is in Roams wheelhouse. The primitive is text – not the graph.
A hyper graph? What’s that?
Say you are a secondary school teacher. Let’s say you teach history, and you teach the numerous year groups. Let’s say from 1st year through to pre-university you teach Romans, The Vikings, Founding of America , World War 1 and World War 2.
How would this look on todays computers? You would maybe create a folder and have each topic in it split into each folder?
Each folder would have a number of files in it, and you would share the folder with your students and they would trudge through each folder in order.
But now let’s ask a question – what are you missing here? Well, aren’t you missing links between the events in each folder? How do you discover them at a glance? Can you? History is the study of the progression of time and events don’t happen in isolation.
So events that happen in one time period cause effects in the others. The events that lead up to World War 1 are incredibly important to the conditions in Germany that lead to World War 2.
But the aforementioned folder structure almost makes it seem as if those events exist in isolation.
Where would the Treaty of Versailles sit in your folders? If you say WW1 you’re modelling your lessons in units, and valuable context is lost. Sure you could copy the file into WW2 folder to represent it’s important to both. Lame!
This is a toy example, but the deeper I think about the promise of Roam and friends the more it frustrates me that no one has built something a bit more capable. A bicycle for the mind is great, but where is my electric bike?
Imagine instead the primitive isn’t files and folders, but nodes in a graph? That’s what I thought Roam was giving me, but the graph seems like an afterthought in that product. I think this is a mistake – the graph should be the tool I use to connect my thoughts.
In mathematics, a hypergraph is a generalization of a graph in which an edge can join any number of vertices.
While graph edges are 2-element subsets of nodes, hyperedges are arbitrary sets of nodes, and can therefore contain an arbitrary number of nodesHypergraph – Wikipedia
Keeping with the history example for a second – Imagine now each vertex (node, point) in above graph shows a different ‘event’ in history and the coloured sections are ‘periods’ or ‘topics’. This is so intuitive it barely needs any more explanation.
Anyone you show this to would get it without much effort. Doesn’t it seem that this structure is absolutely perfect for managing disparate, but conditionally interconnected knowledge?
You’ll notice that the image above shows some fairly organic looking “sub-graphs”, some live by themselves, some connect to nearby sub-graphs and then some are large all encompassing structures containing many subgraphs.
These structures are defined by connections, they aren’t categories to be filled up by your notes/files. That’s a huge issue in note taking apps – you need to categorise up front.
It can be solved by things like “reviews” where you dump all your notes into some temporary inbox, then once a week you go about categorising this into subfolders.
Fine, but I find I take a bunch of notes I never look at again, and wouldn’t it be great if they could just be dumped into the abyss and if I ever need what I wrote into them again it could be surfaced by a simple keyword search?
Think of what you do for your day job. If you manage multiple projects, or try to keep a bunch of knowledge about interconnected topics together – can it be modelled using a hyper graph?
Could you collect all the jobs documents together (Email, IM Messages, Notes, Contacts), link them all together and get a clearer picture of the whole?
A hyper graph sounds exotic and nerdy – but it’s so intuitive because it model the messiness of our real data, the messiness our applications and databases but an immense effort in to paper over.
Does it exist?
I know with a frightening degree of conviction a tool like this would help me. I am a software engineer by trade and the best way I can describe the job is that there are substrates that make up the role.
I need to know about general computer science to make my code run fast, I need to know programming languages that will run on the machine I want to build for, I work in a team made of multiple people that work on multiple projects and I have ideas for improving the products I work on.
So I had a look around for something like this. I found a few dead products, and a few research projects but the closest product that I can see that does *something* like this is Palintir.
Palintir allows a user to structure unstructured data. A note, text-document, image or file is treated as a data point, and Palintir gives you a graph tool to start placing nodes together, creating links between them and analysing how all of that data fits together.
Nodes can be connected to other clusters of nodes by just drawing a link between the two. This doesn’t really mean anything under the hood as far as I can tell – it just means they have some relationship that is meaningful to the analyst. The connection itself can have metadata attached to it if required (I.e this connection means a phone call took place between these two entities), but it’s not required.
A graph doesn’t have all the documents from the data sources you pump into it, you pick and choose which ones are relevant. You can then turn that graph into something – like a powerpoint, a shareable list of documents, or even just share the graph with your colleagues.
Unfortunately Palintir don’t seem like they are ever going to sell direct to consumers, and no matter how many times I put my gmail into their contact page, I never get a call back. Seems like I’m outside their target audience – a real shame.
Hand waving and platitudes
Okay so I’m sort of joking about using Palintir for mapping personal knowledge and files. As far as I can tell a Palintir installation involves an engineer going in and installing a bunch of integrations with exotic databases – which makes sense given their clients.
Web 2.0 however has brought us an opportunity on the consumer side for building something like this.
Most applications and services you use to create content come with an API or at least a bulk-export feature to get data in or out. What if we used those API’s as the integration layer (in Palintir-speak) to build our knowledge base?
Is there anyone building this?