JDex deep-dive: data and storage options

 /  [22.00.0182]

JDex deep-dive: data and storage options

This post is a deep-dive into two aspects of your JDex: whether or not you use it to store data, and the different ways you can store the JDex itself. This article is long, and isn't a tutorial; this is reference material that I can now link to when common questions arise.

Our analogy: the library

This analogy courtesy of Moriarty on Discord, who said:

…the revelation was realising that my filesystem, (long-form) notes folder, physical filing cabinet etc were the bookshelves in a library, and my JDex was the index card drawer. One (the card index) describes the shelves and what's on them, the other (the shelves) have the content.

That's brilliant. Let's make it really explicit. Put yourself in your local city library, some time in the 1960s. 🕺🏼

Index cards

Before computers, libraries kept records of books on index cards.

A photograph of a metal filing cabinet, one drawer open, full of cardboard index cards.
Figure 22.00.0182A. The good old days.

Crucially, you can think of the index card as a representation of the book itself. If a card doesn't exist, the book might as well not exist. To find a book in a library you don't start at a bookshelf. You start with the index of books. Here's the process.

  1. Go to the index.
  2. Using the structure of the index (Dewey, in the case of your library) find the book in question.
  3. The index card will tell you where the book is physically located.
  4. Assuming the book is in the building -- it might be out on loan -- walk to the location indicated on the card and get the book.1

Your JDex

Your Johnny.Decimal index (JDex) was designed to work exactly the same way. Your Johnny.Decimal IDs are the index cards. Each ID belongs to a category, which is analogous to the drawer. And -- not shown here because the diagrams are busy enough already -- we understand that each category belongs to an area just like each drawer is housed in a cabinet.

A line diagram. A purple box labelled 'Index card drawer = category e.g. 12' contains blue cards, one of which is labelled 'Index card = ID, e.g. 12.17'.
Figure 22.00.0182B. Categories hold IDs.

This is why I say the JDex is the most important thing in your system. A library without a catalogue is no longer a library: it's just a room full of books. Your life without a JDex is no longer a Johnny.Decimal system: it's just a bunch of files (even if they are neatly organised).

Metadata

Metadata is 'data that defines and describes the characteristics of other data' (Wikipedia). So if data is a book's contents, metadata is its publication date, ISBN number, current location, and so on.

We use this all the time without being conscious of it: the last modified time of a file is metadata. The fact that some piece of knowledge is in your email is metadata. (The knowledge itself being data.)

The thing about metadata is that it's typically tiny in comparison to the thing that it represents. As a result, we don't have to find some new place to store it. We already have one: the index entry.

For the library, this means writing this stuff directly on the index card. For your JDex, it means that your IDs' metadata is always stored directly with the JDex entry. Metadata is a fundamental component of the entry: it's not some separate thing to manage.

Here's how that might look with a handful of entries from the Life Admin System. Remember: purple box = category. Blue note = ID.

The previous diagram, but now we have 3x of them. Each purple box is labelled with the number of a Johnny.Decimal category, and each blue note that it contains is labelled with an ID from that category.
Figure 22.00.0182C. IDs from the Life Admin System with metadata.

Metadata should be standardised

You should decide on a common set of metadata 'keys' and use them consistently. This allows you to query it later: for example, finding all entries where Last updated: is after '2026-01-29', or all entries where Location: is 'fireproof lock box'.

This is only possible if you always call it Location. If you sometimes call it Where is it?, that's now a different thing.

Nerd note: we call these key/value pairs. Keys are standard. Their values change. As keys are special, I'll show them like this in this post.

This is an important distinction as we consider what's metadata vs. data, below.

In the near future, the Johnny.Decimal system will recommend a standard set of metadata keys. If I use it in this post, you may assume it will be in that standard.

'Above the line'

Note the horizontal line in these diagrams. That's deliberate: in my JDex, I use a line to separate metadata from data. Metadata is always 'above the line'. We'll see data 'below the line' shortly.

  • Above the line: the book's metadata.
  • Below the line: the book's contents.

Storing data in your JDex

In contrast to metadata, your library's index cards weren't designed to store the data that they represent. They aren't the book's contents.

But nor are computers 5×7" pieces of cardboard stored in a little metal box. So in this new world, you might choose to extend the usefulness of your index entries by using them to store data.

The blue ID card from our previous example now has a bunch of data in the form of notes. The ID is about your passport, and the data is about its renewal and cost.
Figure 22.00.0182D. An ID entry with data 'below the line'.

Here, we're still recording the location of your passport as metadata 'above the line'. But we're also storing a bunch of other data 'below the line'.

Nerd note: we call this 'structured' vs. 'unstructured' data. Here metadata is structured, with standard keys. Our below-the-line data is unstructured: it's different for each entry.

This is what I do and recommend. It's simple. Everything's in one place. It's very hard to lose information, and very quick to retrieve it.

To address the alternative scenario, we need to expand this model.

Storing data outside your JDex

Back to the physical library. There, index cards point at books, and books contain data.

Our block diagram now shows the representation of a book in purple/blue on the left, pointing to the contents of the book (Dickens' _A Tale of Two Cities_) on the right. _It was the best of times, it was the worst of times…_
Figure 22.00.0182E. A book's index card points to the book's contents.

In exactly the same way, we can move the data out of our JDex note and into its own file.

The same diagram now represents our JDex on the left in purple/blue, and where we previously had notes in the JDex entry, they've been moved out to .txt files on the right in orange. This mirrors the contents of the book from the previous diagram.
Figure 22.00.0182F. A JDex entry points to its data.

Here, 11.12 Passports, residency, &… is an ID folder in our filesystem (orange). We've created 2 files there (yellow): one for our partner's passport, and one for our own. And we've moved the data out of the index entry into these text files.

To be sure we don't forget about this data, we've reminded ourselves about it using a Data property. (This is optional. But if in doubt, it's worth doing. It takes no time.)

A note on Location

Remember that this metadata relates to our passports. As in, the little book you show at immigration: not the Johnny.Decimal ID that is the concept of your passport.

That's why Data is split from Location. We have data about the passports, which is in some place. And, separately, the (physical) location of the passports.

But let's not get distracted. I'll have more to say about Location in a future post.

Recap

To recap, we've introduced two scenarios.

  1. You store data in your JDex entry, below-the-line.
  2. You store data externally, and (optionally) reference it above-the-line.

It might be interesting to note that here at JDHQ we use both of these methods. Johnny tends to prefer the first. Lucy, perhaps because of her career as a copywriter and editor, leans towards the second. They happily co-exist.

Quadrant chart

Let's introduce the beginnings of what will become a quadrant chart. This will help us to orient these concepts relative to each other.

The left/right split represents the scenarios just discussed.

A square, split into two rectangles. On the left, with an axis label of 'Data in JDex', there's a data point labelled 'Data below-the-line'. On the right, axis labelled 'Data externally', a data point 'Data in filesystem'.
Figure 22.00.0182G. All quadrant charts are born as two rectangles. True fact.

This is a continuum

There's a hard line down the middle of this diagram, but life isn't like that. The reality is that this is a continuum, with most cases somewhere in the middle.

To the far left of this diagram is the situation where you exclusively store data in your JDex. This is unrealistic: will you never save a file? Files are data. So when I say that you 'store data below-the-line', I mean some data: usually textual data, as it's natural to add that to the existing JDex entry. As soon as you also save a file, or keep any artefact related to this ID outside the JDex entry, you've moved away from that left edge.

The right edge of this diagram is somewhere you might actually exist. In this situation, you never store any data in your JDex. For example, your JDex might be an Excel spreadsheet, with one row for each ID.2 Excel is a terrible place to try to write any sort of long-form text, so it would make sense for no data to be stored there.

This diagram is just useful for us to map out all the ways we can do it. But none of them is more correct than the others. It's all preference.

Where is your JDex stored?

There's another fundamental question to be addressed, and it relates to the storage of your JDex. We've established that you have a JDex. Where is it?

This is another continuum, the opposing sides being:

  1. Store your JDex as individual files in your filesystem.
  2. Store your JDex as some other artefact, not in your filesystem.

Let's explore these options.

JDex as individual files in your filesystem

What we mean here is simply that each JDex entry is an individual file, usually a text file, often formatted with Markdown.

You could manage these manually, but it's far more common to use an application, and the most common of those is Obsidian. For those who don't know it, you point it at a folder and it'll show you the structure of that folder and all of the Markdown files in it. Select a file from the left, edit it on the right. JDHQ provides downloads for the Life Admin and Small Business Systems that are designed to work with Obsidian.

A screenshot of Obsidian. The left pane shows a folder structure which mirrors Johnny.Decimal. The right pane show the content of the entry.
Figure 22.00.0182H. Obsidian.

People enjoy Obsidian because it sits over your files: it's a convenient utility. But if you get sick of Obsidian, or if they decide to charge $500/year for a licence, or if you want to give some other app a try, you can just stop using it. All of your files are still Markdown files in a folder that you control.

The vertical axis of our chart now represents where your files are stored. Obsidian sits in the top half, where your JDex is 'files in your filesystem'. It spans both top-left and top-right quadrants as you still have the choice of where to store data: in your JDex, or externally.

Our quadrant chart now has quadrants. The vertical axis has been labelled with 'JDex stored in filesystem' at the top, and 'JDex elsewhere' at the bottom.
Figure 22.00.0182J. Quadrant chart with Obsidian represented.

JDex stored elsewhere

There are a lot of options in this bottom half, so let's explore the simplest.

The other app that I love and support (with specifically-formatted downloads from JDHQ) is Bear. Superficially, Bear is identical to Obsidian: list on the left, editor on the right.

A screenshot of Bear. The left pane shows a list of JDex entries. The right pane shows the content of the entry.
Figure 22.00.0182K. Bear.

But there is a crucial difference: Bear manages these JDex entries entirely for you. There's no concept of 'text files on your disk' to manage. There's no option in Bear to change the location of these files. You can't, outside of Bear, go and look at them. Another app can't edit them. You can't put them in a shared folder and collaborate with your partner.

(Technically, in fact, they aren't even a collection of text files. Rather, Bear manages them using a database which they 'highly recommend' that you do not touch.)

This sounds limiting, and in many ways it is. Why would you want this? Well, it's a lot simpler. You load your JDex files into Bear once, and then never have to think about where they are. Want them on your iPhone? Just install Bear and they'll appear. They're harder to lose because Bear stores them in your iCloud Drive so if you drop your laptop in a lake, no worries.3

Many apps work like this: Apple Notes, Craft, Evernote, Google Keep, Notion, and OneNote to name a few.4

Let's keep it simple and add Bear to our quadrant chart.

The previous quadrant chart with 'Bear' added in the lower half.
Figure 22.00.0182L. Quadrant chart with Bear represented.

The point of this distinction is that when you're using a method in the bottom half of this chart -- more of which we'll see shortly -- the decision of where to store your JDex is made for you. This is one less thing for you to have to manage.

Where are your JDex files?

A question naturally arises: if we're talking about you storing your JDex as 'files in your filesystem', where are those files? As usual, there are a few options.

Ignoring your JDex for a moment, here's a simple representation of your filesystem. As in, the Johnny.Decimal structure, probably in your Documents folder or on a cloud drive, where you save all your stuff.

▓ 10-19 Life administration/
▓ └── 11 Me & other living things/
▓     ├── 11.11 Birth certificate & proof of name
▓     ├── 11.12 Passports, residency, & citizenship
▓     └── …and so on
__________________
Figure 22.00.0182M.

The dark blocks on the left will make sense shortly. These tree diagrams don't work well on mobile, sorry. I've made sure that they are clear and flow properly at larger sizes.

It's pretty obvious that if you had a PDF that you needed to save -- say your latest passport renewal -- you'd put it in 11.12. Because it's a file, that you're saving in your filesystem. No new concepts there.

So if we're storing our JDex as files in our filesystem, it must be in here somewhere. Where else would it be?

This is one of the IDs that are reserved by the Johnny.Decimal system, and I hope the ID is obvious. It's the very first thing you should encounter in your system, the top of the tree: 00.00. It lives inside the system-reserved category 00.

▓ 00-09 System-management area/
▓ └── 00 System-management category/
▓     └── 00.00 JDex for the system
▓ 10-19 Life administration/
▓ └── …and so on
__________________
Figure 22.00.0182N.

Happy with that? There's a lot going on, and it's about to get deep. Get comfortable with that before we move on.

What goes in 00.00?

Above, we noted how Obsidian just watches a folder full of files and lets you edit them. The screenshot (figure 22.00.0182K) shows those folders in the left pane, full of text files. Those folders look like your Johnny.Decimal system. As this is your JDex, they define your system. So they look something like this.

░ 00-09 System-management area/
░ └── 00 System-management category/
░     └── 00.00 JDex for the system.md
░ 10-19 Life administration/
░ └── 11 Me & other living things/
░     ├── 11.11 Birth certificate & proof of name.md
░     ├── 11.12 Passports, residency, & citizenship.md
░     └── …and so on
__________________
Figure 22.00.0182P.

No, I didn't duplicate the previous figure by accident. Note the subtle difference from figure 22.00.0182M: now the IDs are Markdown (.md) files, which is what Obsidian will display in the right pane.

So if we stitch these two diagrams together, this is what your filesystem looks like.5

▓ 00-09 System-management area/
▓ └── 00 System-management category/
▓     └── 00.00 JDex for the system/
░         ├── 00-09 System-management area/
░         │   └── 00 System-management category/
░         │       └── 00.00 JDex for the system.md
░         └── 10-19 Life administration/
░             └── 11 Me & other living things/
░                 ├── 11.11 Birth certificate
░                 │         & proof of name.md
░                 └── 11.12 Passports, residency,
░                           & citizenship.md
▓ 10-19 Life administration/
▓ └── 11 Me & other living things/
▓     ├── 11.11 Birth certificate & proof of name
▓     ├── 11.12 Passports, residency, & citizenship
▓     └── …and so on
__________________
Figure 22.00.0182R.

Be really comfortable with that before we continue. At first glance this looks bonkers. But when you understand that 00.00 is a Johnny.Decimal ID that just happens to contain a folder structure that mirrors the overall structure … it's fine. And you're never spending any time in there yourself: let Obsidian manage it.

This is the 'partially nested' pattern

When you download your JDex files from JDHQ, there are a handful of options for Obsidian. This is the 'partially nested' pattern, in that the ID files are nested within an area/category structure. Obsidian shows this structure in the left pane.

There's an alternative 'flat' structure, in which those Markdown files aren't nested within an area/category structure. Some people just prefer that. I'll leave it as an exercise to the reader to imagine how this looks in your filesystem. (You can download these files from JDHQ as many times as you like: try it out. Note that you'll need to select Obsidian as your JDex app first. Both links require a JDHQ account.)

Let's place a dot on the diagram that represents these patterns. We're not concerned with whether we're storing data in our JDex or not, so we'll stick it in the middle.

The quadrant chart now has a single data point, horizontally centered, vertically in the top half, labelled 'Obsidian files saved at 00.00'.
Figure 22.00.0182S. Obsidian on the quadrant chart.

The 'fully nested' pattern

In the scenario just described, your JDex files were separate from the rest of your files: they're all saved at 00.00 while the rest of your stuff is below.

What if you didn't want to do this? What if you wanted to merge these two structures -- JDex and filesystem -- so that there was only one? This is possible, of course, and we call it the 'fully nested' pattern.

In this pattern, folder 00.00 goes unused.6 Instead, each of the Markdown files that are your JDex entries are scattered through your filesystem. Again, I've used the dark and light blocks to represent which components come from the original diagrams.

▓ 00-09 System-management area/
▓ └── 00 System-management category/
▓     └── 00.00 JDex for this system/
░         └── 00.00 JDex for this system.md
▓ 10-19 Life administration/
▓ └── 11 Me & other living things/
▓     ├── 11.11 Birth certificate & proof of name/
░     │   ├── 11.11 Birth certificate
░     │   │         & proof of name.md
▓     │   └── Photograph of birth certificate.jpg
▓     ├── 11.12 Passports, residency, & citizenship/
░     │   ├── 11.12 Passports, residency,
░     │   │         & citizenship.md
▓     │   └── Passport application.pdf
▓     └── …and so on
__________________
Figure 22.00.0182T.

On the surface, this feels like a good idea. Why wouldn't you want it all integrated? But me and many others have tried this and the universal consensus is that this doesn't work.

The point of this article is to explain these methods, not to tell you why one of them is worse than the others. So, just briefly:

  • The clean separation of JDex from filesystem is a nice mental separation. It reinforces your JDex as being the more important thing. Merging them breaks that.
  • Previously, you were pointing Obsidian at a folder full of tiny text files. Now, you're pointing it at your entire filesystem. There might be terabytes of data in there. This can slow it down significantly.
  • You're no longer able to selectively synchronise folders because the folder is the JDex entry. This was the killer blow for me.

So you probably shouldn't do this. For completion, let's add it to the diagram.

We've added 'Obsidian files fully nested' _above_ 'Obsidian files saved at 00.00'.
Figure 22.00.0182U. Two Obsidian options.

I've placed this item nearer the top of the chart as your JDex files are more 'in your filesystem' using this method than they were with the previous method.

Core concepts finished

That's it for core concepts. In the next section, let's fill out the diagram with some examples.

Examples

There are a handful of scenarios that we already understand. The diagram's getting busy so I'll extract some meaning to a table.

There are a bunch more data points as described in the table below.
Figure 22.00.0182V. I honestly thought I was going to run out of letters for the captions on this one. Close call.
Data storedJDex stored
Obsidian 1in JDex'fully nested'
Obsidian 2in JDex'partially nested'
Obsidian 3externally'fully nested'
Obsidian 4externally'partially nested'
Bear 1in JDexmanaged by Bear
Bear 2externallymanaged by Bear
Excelexternallyas an .xlsx in your filesystem
Google Sheetsexternallyin your Google Drive
Notionwithin Notionby Notion in their cloud
SQLite databasewithin the databaseas a .db file in your filesystem
Airtableexternallyby Airtable in their cloud

Nit-picky points

We're getting in the weeds here, but I positioned those dots carefully so we might as well explain why.

The left/right split isn't so interesting: you either store data in your JDex, or you don't. Your choice here might be limited by the method you use to store your JDex. As we've already said, a spreadsheet is a terrible place to keep long-form notes; spreadsheets tend towards the right edge.

If you use a database like Notion, it's up to you. In this diagram I've designed a Notion where you do keep data there. Given Notion's power, this feels like a realistic scenario. But you could just have well designed a pure JDex-only database and stored all data externally.

Where the files are, on the vertical axis, is a touch more interesting. Google Sheets is in the lower half while Excel is in the upper because the Excel sheet is a file that you need to manage, and the Google Sheet exclusively lives in your Google Drive.

That said, you do still need to specify where in your Google Drive the file exists. You still need to file it somewhere, even if that somewhere happens to be in the cloud. Which is why Google Drive is above Airtable, an online database. With Airtable there is no consideration at all 'where' your database is stored. You don't get a folder structure that you have to 'file your database' in. (Other than a rudimentary dashboard that you can rearrange.) Your database is just at Airtable.

Similarly, Notion is at the bottom, because all of your stuff is in the Notion cloud. Conversely you might prefer to build yourself a SQLite database which is now a file that you need to manage. As we now know, that database should be stored at 00.00.

That'll do

I wrote this article to set up a framework that I can use in the future; a diagram into which I can slot more scenarios. This isn't meant to be comprehensive. There are a thousand scenarios that aren't included here.


100% human. 0% AI. Always.

Footnotes

  1. You might be thinking, 'that's not how I use my library'. But when you go to the library you're likely browsing the shelves. You don't know what you're looking for. That's a different behaviour from going to a reference library and recalling a specific book, which is the analogy here.

  2. A spreadsheet provides a nice way to think about your metadata. If each row is an ID, your columns store metadata. It's easy to see how you'd have one standard column for Location, and that adding another column for Where is it? is obviously silly. Columns define the keys of our key/value pairs.

  3. With the caveat that cloud storage is not a backup. See my mini-series on data, storage, and backups.

  4. If you use these apps, an important consideration becomes 'lock-in'. How easy is it to export your data, if you choose? Bear makes this easy. Others might make it harder, or it might not be practical. Craft and Notion, for example, are database apps. You can't just export a database to a bunch of text files and expect it to make sense.

  5. Actually, I add another folder below 00.00 that contains both of the folders 00-09 and 10-19. This is because Obsidian doesn't allow you to 'rename' a vault: the vault name is just the name of the folder of files that you have opened. In the situation shown, the vault name would be '00.00 JDex for the system'.

    I manage multiple vaults and want a more descriptive name. My business system's folder is named 'D25 JDex': D25 is the system identifier. This then appears as the vault name. In the screenshot above (figure 22.00.0182H) you can see the vault name is 'JDex - Life Admin System'. This is how it's downloaded from JDHQ. I left this detail out of the diagram as it's not important to the fundamental concept.

  6. Other than to store its own JDex entry, as shown.

'Powered by Johnny.Decimal' badge

 /  [22.00.0181]

'Powered by Johnny.Decimal' badge

Thanks to Karsten.Decimal on Discord for suggesting this idea, along with a proof-of-concept.

If you've ever wanted to show people how organised you are, now you can -- with an official 'Powered by Johnny.Decimal' badge!

A simple graphic. It has a border, and inside our logo and the words 'Powered by Johnny.Decimal'. It's very line-arty.

Full-res versions are available to download here. Don't save the image from above, it's super low-res.

FAQ

  • Q: Can I use these freely?

    • A: As you please.
  • Q: Would you like me to link back to your site when I use them?

    • A: I'd like that, but you don't have to.
  • Q: Do you expect me to use these?

    • A: This is just a bit of fun. I expect nothing.
  • Q: Can I tweak them for my needs by resizing them or changing their format?

    • A: As long as it still looks mostly like this, yes.

100% human. 0% AI. Always.

Backups aren't enough: create 'restore points'

 /  [22.00.0180]

Backups aren't enough: create 'restore points'

I almost lost a bunch of data this week. Here's how my backups saved me (just), and what I've changed.

Warning: thinking about backups can make you sleepy. But if you care about your data, you need to pay attention. There's a lot here: ask for help on the forum and Discord. This is important.

Note: now that I've finished this post, yeah, it's long, and complex. I might turn this into a short YouTube series. Let me know if that'd be helpful.

What happened

Actually this started months ago. I accidentally deleted my entire Documents folder! I was being too quick on the keyboard and I literally just Cmd+A then Cmd+Deleted everything. Turns out that's surprisingly easy to do.

I undid by pressing Cmd+Z a second later. But the damage was done. I store my Documents folder in iCloud Drive and I knew as soon as it happened that I'd likely triggered a cascade of network activity. My command to delete would have still been half-way to the cloud when I issued the command to undo (i.e. move those files out of my Trash back to Documents), and it would have been a more robust service than iCloud that handled that without issue.

What I should have done at that point was stop everything and spend a few hours making sure everything was okay. I didn't do that.

Lesson 1. If something like this happens you MUST drop everything and fix it, completely and properly, before doing anything else.

Discovering the issue

This week I finally found time to upgrade myself to Life Admin v2.1 I thought I'd take the opportunity to really clean up some old stuff, and in doing so I realised that a bunch of really old archive stuff was missing.2 The folders were there, but there was nothing in them.

Lesson 2. If it doesn't have a proper Johnny.Decimal ID, it might as well not exist. In my case, files buried in an 'archive' folder were, in fact, important to me. I was just too lazy to organise them properly. I've since given them their own IDs in a new area, 29 The Omnium.

Even if these files had their own ID, I might not have picked this up. Everything looked okay, in the moment.

These missing files were buried pretty deep -- this thing that I'm archiving is itself an old Johnny.Decimal structure.3 How would I have known to go looking for this file, way down at 41.14?

Lesson 3. I'm going to keep a short list of 'canary files'. They're files that I care about, but rarely look at. I will manually check them in case of a 'restore event'.

So now I need to restore from backup

Obviously I have backups. I wrote a whole series about it. But backups are complex, mine especially so. This is of my own making, but also hard to avoid given my situation. (I won't recap here: read that series.)

Here are a few of the problems I encountered.

  • As soon as you actually need to restore from a backup because you lost some data, shit gets real if you'll excuse my language.
    • Stress levels rise. Everything becomes difficult.
    • This is not a time to be confused.

Lesson 4. Good records are essential. This is what your JDex is for. See below for a template that you can download.

  • Not knowing which backup, as in from which date, to restore.
    • Because I can't remember exactly when this data-loss event occurred. I think it was around August? That's 4 months ago.
    • I use Arq and it allows you to search, really quickly, for a specific file across all of your backups. This made things much easier.
    • Its restore is also really fast, and this backup was local, i.e. a hard drive attached to the server, not cloud. So I could quickly do test restores to identify which was the right one.

Lesson 5. Know your backup software. Use good software: if yours isn't good, find another. I recommend Arq for Mac.

Lesson 6. Local backups are better than cloud backups when it comes to restoring data because they're much faster. (But you still need an offsite copy, in case your house burns down.)

  • Not knowing which backup, as in literally which one, to restore.
    • Because the way I'd configured them -- which I think is the way most people configure them -- didn't make it really obvious where things were.
    • So when I came to use them, there was confusion. Not what you want in a time of crisis.
    • More on this below.

Lesson 7. Change the way you structure backups. Previously, I backed up a hard drive or a computer. But what's actually important to me is the unit of data which is a Johnny.Decimal system.

  • My backups are encrypted.
    • Which they should be. And I use 1Password, and my vault is reasonably neat.
    • But when I tried the password I thought it should be, it failed. Shit. I tried another. Failed! Oh, shit. This is bad.
    • I must have mis-copy/pasted the first password. When I tried again, that worked.
    • But to have any possibility of doubt at all here is a total failure.

Ref: Lesson 4. Good records are essential.

'Set and forget'

This whole thing got me wondering if we're doing it all wrong. Backup services advertise themselves as 'set and forget': just install, then never think about it again!

That's what got me into this problem.

If you know about backups you're shouting at the computer: but Johnny, you should test your backups regularly and then you wouldn't be in this situation!

That's true. But you know what I think most people do when forced, begrudingly, to test their backups? They think of a file that they deleted from their Desktop yesterday and they restore that. This is not a realistic test. It doesn't stress you, or your system.

Lesson 8. Daily 'set and forget' backups are great and you should not stop doing them. But you should also have a subset of backups that are a lot more deliberate, and are manually activated. You must see these backups as an opportunity to be neat and tidy. You must cherish them. They are not a chore. They are your happy place.

Conscious backups aka 'restore points'

Let's address that last lesson 8 first. I think there's room in life for two types of backups. The first is the one I've already got: the 'set and forget'. 90% of the time this is going to do the job. Don't stop doing those.

But the other type is what I'm calling a 'restore point'. In the IT world, a restore point is a backup at a specific point in time. You're saying that you can restore to this point in time.

They are much more deliberate, which means that they take more work. But they give you the sort of 'comfortable awareness' of your system that only comes from giving it your clear attention.4

Today is a restore point

I finished my migration to LAS v2 today, and with it a bunch of tidying up and moving around of data.

Today -- now -- my system is in a really nice state. I've just looked at everything and I'm deeply familiar with it. I'm sure it's good. Today is a restore point.

So I'm configuring a new backup. Here's what makes it different from my daily backups.

'Restore point' rules

  1. They are manually triggered.

When I decide the time is right, I'll run this backup. I'll be adding something to my task system to prompt me to do this, perhaps quarterly.

  1. They have no retention rules.

Normal backups are smart about the data they retain. Your computer might perform a backup every hour, but you don't need to keep every one of those backups. That would be a waste of space.

A typical 'retention cycle' might be to keep:

  • Every hour for the last 72 hours.
  • Every day for the last 30 days.
  • Every week for the last 52 weeks.
  • Every month forever.

My 'restore points' have no retention cycle. Every one is saved forever.

  1. They have no exclusion rules.

My daily backup excludes a bunch of files and folders. Anything that looks temporary, or like a cached version, or like something I've downloaded, or like something that I know I could get from the cloud, is excluded.

For my daily rules, this keeps them lean and fast. This is particularly important for me as I live on the road. But that isn't relevant for 'restore point' backups. It is more important that they are complete, so they have no exclusions.

The structure of a 'restore point'

Per lesson 7, these backups don't just back up an entire drive, say, or my user folder. (That's what your daily backups probably do, which is appropriate in that context.)

Instead, they back up the specific folders and only those folders that make up the Johnny.Decimal system that they target. I have 2 restore point backups: one for my P76 Personal system, and another for D25 Johnny.Decimal, the business system.

The personal system has files split across 2 locations:

  1. My ~/Documents folder. This is stored in iCloud.
  2. An external hard drive. This contains ~500GB of media files that don't need to use cloud storage space.

So it only backs up these folders. Not my whole laptop. No other cruft. Nothing is excluded: if something is in one of those folders, it's because I put it there, and I want it backed up.

The business system is simpler as everything is in a single folder. So that folder is the only one in the restore point backup.5

Local backups using Arq

These backups all take place on an always-on Mac mini which is connected via Ethernet to a Synology. We learn from lessons 5 and 6: use good software that you know well to make local backups because they're fast.

Records-keeping

Lesson 4 taught us that good records-keeping is vital. You must have total confidence in these backups, and have good records that you can depend on in a crisis.

Of course, this is what your JDex is for. LAS and SBS both have a place for backups. At LAS that's 14.14 My data storage & backups, so in that note I have left myself copious notes. I've recorded:

  • The name of the restore point backup (in Arq).
  • Its data sources (the folders it backs up).
  • Its target (where it backs up to).
  • Its frequency (manual).
  • Any exclusion and retention rules (none).
  • Where its encryption keys are held (1Password: be explicit and name the entry).

Then, every time I manually trigger a backup, I'm noting:

  • The date & time of the backup.
  • How large each of the folders was at the time (right-click in Finder and Get Info).
  • That I checked my canary files.

You want this to be a story that you can return to. A place of total comfort.

Use my template

I've knocked up a template that I'm using myself. It's way far from perfect, but it does the job. Get them here.

Using the 3-2-1 method of data storage -- see the notes on pages 3 & 4 in the PDF, or search online -- it has a space for you to record:

  • Where the primary copy of your data is.

    • This is the folder that you back up.
  • Where the secondary copy of your data is.

  • Where the backup of your data is.

  • Which of the secondary or backup is your offsite copy.

Fill that in, and save it in your system.

  • LAS: 14.14 My data storage & backups
  • SBS: 14.23 Backups & recovery

Ask for help

I've created a dedicated forum post for this topic as it's a big one. Ask for help there, or in the thread on Discord.

In this series

This post is part of a series on storage, data, & backups.

  1. 22.00.0101 My data storage & backup strategy
  2. 22.00.0115 Storage, data, & backups [SBS.14.20]
  3. 22.00.0116 Data [SBS.14.22]
  4. 22.00.0119 Synchronisation [SBS.14.22]
  5. 22.00.0120 Backups [SBS.14.23]
  6. 22.00.0180 Backups aren't enough: create 'restore points'

100% human. 0% AI. Always.

Footnotes

  1. We added a handful of IDs to v1 and moved things around in the finance area to align it with the Small Business System.

  2. Specifically, the file that started Johnny.Decimal: a carefully-designed PDF of ticket prices that was stuck to our office wall in 2010. I wanted a way to reference this file that wasn't an ugly file path, so I came up with the idea of the numbers.

    That was 15 years ago, and I have no use for this file today. I think about it perhaps once a year. But I'd be devastated if I lost it.

    An A4 page, laid out with boxes, of ticket prices. At the top is the ID: 41.14.
  3. To make this clearer, here's a screenshot. You can right-click and Open in new tab to make bigger.

    A screenshot of Finder showing an entire old Johnny.Decimal structure saved at ID 29.11 in my current system.
  4. A concept taken from my task and project management methodology, where it works really well.

  5. This folder is on our server, an always-on Mac mini. I synchronise partial copies to our laptops using Syncthing.

The two-tea-towel method

 /  [22.00.0179]

The two-tea-towel method

It's a little slow around here at this time of year. I'm spending time with my family in the UK; my first Christmas here since 2002! The year after, I went to Australia as a backpacker and just never came home. I like to say that I moved to Australia by accident.

So here's something for the holidays. I have access to a kitchen again, and the first thing I did was go and buy fresh tea towels. If you care about cooking, this is how you need to tea towel. This is the two-tea-towel method.

Theory

When you're in the kitchen, you have two requirements in the cloth department.

First, you need one permanently slung over your shoulder that you can use to wipe your hands, the bench, to handle a pan, or mop a spill. This is your 'bench towel'.

Second, you need to dry things that you'll eat with, or off. You'll need to polish a glass or a spoon. This is your 'clean towel'.

Implementation

You need two (2) types of towel to meet these two needs.

Let's start with your clean towel. This needs to be a decent towel that can actually dry a thing. It needs a bit of fuzz to it, y'know? Your clean towel can't be one of those that feels like an old pair of underpants even when it's brand new.

Spend a tiny bit of money on your clean towels. For me. If you're on a budget, your bench towel doesn't need to be as fancy.

As a rule, simple cotton does the job. Avoid linen. And of course wash them before use to fluff them up a touch.

They must be visually distinct

It is vital that these two towels are different so that you can see, at a glance, which is which. If possible, don't choose a dark coloured 'clean towel': you want to be able to quickly intuit when it's no longer clean.

So find yourself two types of towels.

  1. Clean: light (in colour), heavy (in weight and drying power).
  2. Bench: dark (in colour), and can be cheaper.

We used these stalwarts from IKEA for many years. I used 'stripes' as clean and 'checks' as bench; while I wish they were more visually distinct, it worked. Importantly, the fabric suited both jobs perfectly and the price is right. Because...

Buy a dozen of each

That's twelve (12), six plus six. Two more than ten. Of each, not total. Eight more than you thought you needed before you read this article.

No, that is not too many. You must be able to use these towels freely. This is core to the method: you will never, ever use a dirty towel. You will use them with reckless abandon. You will be utterly free in your towel usage. This will change your life.

When we lived in a house and I prepared three meals a day I used at least one set of towels a day. On a normal day, one set is enough. It depends what you cook. But the instant you look at a towel and wonder as to its cleanliness: get a fresh one. Chuck yours in the wash.

You need hooks

This tip courtesy of my mam, who currently doesn't have hooks. Your tea towels must be on a hook, attached to an open wall. This removes all friction, literally.

Don't go hanging them over the oven handle or some other hack. Get a 3M hook and stick it on the wall. No excuses.

The weekly towel wash

We were lucky in having a house with a washing line (and generally sunny weather, in Canberra). As I sit and type this in a basement flat in the UK with my thermal top drying on a radiator I realise this situation is not universal. But do what you can.

At the end of the week, you've got 14 towels to wash. Chuck in some bathroom towels and that's a full load. Now you can hang them all out together and fold them together and that'll speed things right up.

Merry Christmas

Have a lovely Christmas. It's not too late to pop down the shops to get yourself a last minute gift. I'm sure IKEA is delightful today.


100% human. 0% AI. Always.

Lazy shortcuts lead to mistakes

 /  [22.00.0178]

Lazy shortcuts lead to mistakes

I just un-published post 22.00.0177 as it shouldn't have appeared. It's not finished and was published by mistake. Here's how.

Without getting too technical, I add things to this site using the ubiquitous software git. Normally this is a conscious action, and it takes two steps.

First:

git add 22.00.0177

– to say please add this file to the list of things I'd like to commit.1

And then:

git commit [some message to say what you did]

– to commit those changes to the site.

Well, why do in two steps what can be done in one? Long ago I created an alias -- a shortcut -- that merges these two steps. So now I just:

git addcommit [some message to say what I did]

The problem is that this git addcommit alias adds every new file to the commit. Which is very often what you want: but clearly not always. In this case, it added this unfinished post.

There's a reason this isn't a built-in command. It's a bad idea. I was being lazy.

I've deleted the shortcut. The post will be finished and re-published tomorrow. Apologies for the error.


100% human. 0% AI. Always.

Footnotes

  1. These git commands are what we call pseudocode. They're not the exact commands, but they vaguely resemble them. The idea is to give the layperson an idea of what's going on while not getting caught up in the details.

Standard zero 'AC.00 JDex for …' – what's it for?

There's a reason the standard zeros live in the site's 12 Advanced section. Including them all in our standard systems was a difficult decision.1

Let's look at one of these specifically, the AC.00 JDex for … entry.

00.00 JDex for the system

This one is the clearest. As the 'first' ID in any system, 00.00 serves as its introductory document.

If there's anything that you need to know about the system, it should go here. I think of it like, if I got amnesia, what's the one thing I'd need to know? It's the location of this note. From here, you should be able to find everything else.

Here's what I've written at 00.00 in my personal system:

00.00 JDex for P76 Johnny's personal system

  • JDex for this system is right here in Bear. You're reading it.
  • Filesystem for this system is primarily at iCloud Drive/Documents/P76 Johnny's personal system.
    • When folders live elsewhere, that location is specifically noted in the JDex entry.

Area-level entry

Let's go down to the area level, 10.00, 20.00 and so on. In exactly the same way, the JDex entries for the area serve as the documentation for that area.

What's in it? Why did you decide to put these things in it? What isn't in it?

I just did a bit of a tidy-up and I created myself a new area. (I use the Life Admin system but it's specifically not designed to handle all the other stuff we do: our hobbies and jobs and whatnot.)

I don't have a single hobby because I'm the sort of person who creates a bunch of stupid little 'projects'. These have accumulated over the years and I have an archive I'd like to save. And I'll need IDs for new projects from now on.

I also need a place to store stuff that I've downloaded. So I created a new area, 20-29 Create & collect.

An area is little more than the categories it contains, so the JDex entry 20.00 is very simple. I've just told myself:

This area is for all the crazy ideas. See the category JDex for more details.

– and then linked to 21.00, 22.00, and 29.00.

Category-level entry

Here are my categories:

  • 21 🧠 Imaginarium2
  • 22 😍 Humans are brilliant
  • 29 🏛️ The Omnium3

And here are their JDex entries:

21.00 JDex for Imaginarium

This is a place for all the crazy projects. But not for learning about, and documenting, technology and code. That's more useful in a work context, and is going to be way more useful for linking in Obsidian, so code learning and notes are at D25.4

22.00 JDex for Humans are brilliant

A place for 'collected works' but only things that are made by other humans.

29.00 JDex for The Omnium

This is a place for your historical collected works. You shouldn't be working in here day-to-day. Things here shouldn't ever change.

  • And don't move completed things from 21.00 JDex for Imaginarium here. They just stay there and age out per the Johnny.Decimal system design.
  • Only add entries here if, in the process of tidying up old hard drives/folders, you find more old things.

Summary

So in summary, these JDex notes are your documentation of your own system's structure and purpose, at each level.

  • 00.00 is your description of the entire system.
  • A0.00 is your description of this area.
  • AC.00 is your description of this category.

And if you don't need them, or that's confusing, you can just ignore or delete them.


100% human. 0% AI. Always.

Footnotes

  1. In the end, we included them. It's easier to delete what you don't want than it is to create what you do.

  2. A name gratefully borrowed from Lucy, who came up with it when she built her own system.

  3. Yes, I pushed this out to 29, further reinforcing its archive-ness.

  4. D25 is my Small Business System and it lives in Obsidian.

Commercial licence

 /  [22.00.0175]

Commercial licence

The Johnny.Decimal system will always be free for everyone, including for commercial use. The licence places no restrictions on use of the system.

But you might work for an enlightened company who recognises that ideas like this need to be sustained. So we just created a commercial licence with 3 tiers.

The employee numbers are total at your company, and the currency is USD:

Full details are at 01.02/Commercial licence.


100% human. 0% AI. Always.

Finder folders to Markdown links in Bear

 /  [22.00.0174]

Finder folders to Markdown links in Bear

I was just trying to get a list of Finder folders as a bullet list in a Bear note. (I'm finally upgrading my own LAS to v2 and am making it super neat as I go.)

I've known for a while that you can do this:

  1. Select some folders in the Finder. Typically I find myself shift-selecting a whole range, e.g. 20 dated subfolders within another folder.
  2. Cmd+C to copy.
  3. Now if you Cmd+V to paste, and you paste into a text field, you'll get the titles of the folders.

What you get depends where you're pasting. If I do that into this Visual Studio Code window, I get a long file path. If I do it in to a TextEdit window in the default rich text view, it tries to paste the contents of the folders!

I have TextEdit set to open a plain text window by default vs. RTF. That's Settings > New Document (tab) > Format > Plain text. And now when I paste there, I get a really simple list of the folder names.

If you paste into Bear (or, I presume, other Markdown-compatible apps), you get a Markdown link that, when clicked, opens the folder in Finder!

That's brilliant. Just what I want for my JDex: now it tells me what's in my filesystem folders, and I can just click to go there.

Note: only with 2+ folders

Robert points out on Mastodon that this only works if you have 2 or more folders selected. If you 'copy' a single folder, that will paste as a massive blue folder icon!

There's a lot more going on when you press 'copy' than you thought...

A video

Here this is a quick video.


100% human. 0% AI. Always.

Obsidian links (redux)

 /  [22.00.0173]

Obsidian links (redux)

Previously:
Things: cross-link to Obsidian and back
Obsidian tip: link directly to a header

I'm going to start creating 'permalinks' for each of my Obsidian notes using the ^block syntax.

I link to my notes prolifically from outside Obsidian. My primary use-case is Things, where every project has this block at the top (mirrored in the Obsidian note).1

The notes section at the top of a Things project. I have two Markdown links, one to Obsidian, one to Things.

If that Obsidian URL is the traditional format, where you link to the title of the note, it's very fragile. My note titles are fluid; I shouldn't be afraid to change them.

So I use the Advanced URI plugin which allows you to link to a ^block, and at the top of every note I create a permalink using the ID of the note.

Links as described in the post at the top of an Obsidian note.

Note that right-clicking on the block and selecting Copy URI for current block gives you a longer URI which contains the note's title. If you're strict about making your block identifiers unique – easy when they mirror the ID of the note they're in – you can remove that part of the URI, leaving it much neater.

obsidian://adv-uri?vault=D25%20JDex&block=50105

– because now I can leave that URI string in the templates for both Obsidian notes and Things projects, and all I need to do each time is change the last couple of digits of that block ID. So this only takes a few seconds and is low-friction.

A note on the experimental 50105~14.54 syntax

That's the note: it's experimental. There's a healthy debate over on Discord and there'll be a blog post shortly.


100% human. 0% AI. Always.

Footnotes

  1. Yes, Things links to itself, as does Obsidian when I copy it over. But now this block of links is copy/pastable anywhere as it's complete.

    This is why I use Markdown bullets and not Obsidian's properties: I prefer that my frontmatter be human-readable, and copy/pastable to other Markdown-compatible apps. I don't need to query it or use Bases.

There are more posts.


┌─────────────────────────────┐ 
      Life Admin System      │▒
                             │▒
 The fastest way to organise │▒
    your home life -- $33    │▒
├─────────────────────────────┤▒
  Johnny.Decimal University  │▒
                             │▒
 Learn how to build your own │▒
 system, & lots more -- $200 │▒
├─────────────────────────────┤▒
    Small Business System    │▒
                             │▒
 The fastest way to organise │▒
 your small business -- $300 │▒
├─────────────────────────────┤▒
│      No subscriptions       │▒
│      Updates for life       │▒
└─────────────────────────────┘▒
 ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒
  
Search ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────