I realized recently that I have become somewhat obsessed with the idea of logging and archiving the many little aspects of my life in one place. I'm already (somewhat cautiously) collecting a fair amount of data about myself:
- Oura ring (sleep and activity, mainly)
- Strava (longer workouts)
- Day One (semi-regular journaling and family timeline moments)
- Daily notes in Obsidian
- ... so much more, for better or worse
I don't want to just turn on the data firehose in all the apps and track everything but I am enamored with the idea of looking back through my life and seeing as complete of a picture as I can. There seem to be a number of terms in this arena like life logging, digital archiving (maybe different but related), quantified self (maybe more directional), etc.
This started a while back, when I started thinking about collecting the timeline of my life. I didn't (and still don't) have a great grasp of time in the first 2 decades of my life. I would have a memory and my wife would ask "how old were you" and I would reply "like 10? maybe 12, no ... 14?" I had a tough time piecing things together.
I have always been a bit of a hoarder so I started to comb though old emails in various formats, documents my mom saved, and old letters trying to put dates on certain memories. The more I figured out, the more I wanted to know about surrounding events and what other people in my life were doing at that time. I would ask my mom and dad about certain events and dates with varying success. The remaining gaps were frustrating.
When our first kid was born almost a decade ago, it was clear in my mind that I wanted to put together a very clear picture of her childhood. Mobile phones are, of course, the absolute best way to do that and the countless date and location tagged photos will do a huge amount of the legwork for us. But there's a more complete picture to be had. What was it like for her parents to be parents? Who were they at that time? How accurate will her memories be?
One of the best sources for this kind of thing, for a lot of people, is social media. I exported and deleted all my social media long ago so any updates I write are in a journal or in a daily note. I started to import a photo or two from events in the past into Day One, adding a bit of color around how the day went and what we did. It's quite hard to keep that up but I try to do it now and then. I also write my kids long letters each year during the holiday break with my favorite photos of them, who they were, what they liked, and all about the ups and downs.
A good portion of this is about my kids but there is something pulling me towards collecting more and more data (while being wary of what's stored in the cloud). I've started downloading my Google Calendar on a regular basis, bought software to export all my text messages, and have my eye on email now, my oldest and best record of communication (why did I delete my oldest account without exporting?!?). I have the export files from Facebook, Instagram, and Twitter before I nuked all the content there. I have Netflix history, contact list backups, iCloud data, old notes apps I used to use, Spotify plays, Amazon purchases. Just this all out makes me vibrate with nerdy excitement!
I also have a treasure trove of old photos, old documents, and genealogy work from my mom laying around that I haven't looked through. Transferring analog documents to digital ones is a whole other layer of excitement. One day I spent a whole day downloading PDF copies of the instruction manuals I had and recycling the paper version. I still get a smile on my face when I think about it and having it all at hand is quite useful.
Before you say anything ... I am perfectly aware that all of this life logging and digitizing and organization amounts to an actual value approaching nothing. If my daughter could know what movie we watched the day she was born (Pulp Fiction, seriously?) at some point in the future, that might be interesting for a second but then, who cares? This feels like the most navel-gazing use of time one could imagine.
But there is just something so appealing in it for me. Collecting the data that was collected on me and stitching it together to be able to paint a picture of specific moments in time. My life has been mostly mundane, especially for the last 10-15 years, but there is a story there and sometimes it's just nice to go back and relive it in whatever form you have it in.
The amount of data exhaust that we're all generating is pretty staggering. I'm fairly cautious about what I sign up for but I'm still finding more and more sources of data I can export and use. It's a bit of a bind, to be honest; I love having this data available to me but I don't love allowing companies to collect it. For example, I use Amazon as little as possible for buying things (buy local) but being able to see random purchases from many years ago is fascinating. I also turn off all the Google tracking I possibly can but seeing everywhere I've been over a period or time is so interesting to me. There's a catch 22 around every corner, it seems.
What I am truly cautious about, though, is how much time I spend recording and collecting and transforming. When I take photos, I try to take a few good ones rather than capture every moment. I'm cognizant of the "observer effect" where you take yourself out of a great moment just to capture it. I have felt this sense of anxiety/FOMO when I realize that I could have collected something but didn't or forgot to. Collecting for the sake of collecting is fine as long as it doesn't color my life in a negative way. There is a whole bunch of toil baked into this hobby and I'm wary of creating an endless source of digital chores for myself.
To that end, what I'm "working on" (in that, I'm writing and thinking about it on a somewhat regular basis) now is how to collect all this stuff in one place, keep it backed up, and access it without doing a bunch of manual work. I want something that just reads an Instagram export combined with a Netflix export combined with an email archive and just show me a little picture of the day/week/month in the past. I don't want metrics or a daily score or anything qualitative, I just want to see what I can see in whatever interesting way I choose.
I also want to make it as easy as possible to pull that data down and keep it safe (read: local or encrypted cloud backup), whether that's via API or scheduled backup downloads or just instructions on how to do it along with a date record of when it was last completed. I've been working on more projects involving data collection and transformation and storage in my day job and seeing how that relates to personal data. I want to make it easy to collect as much of my own raw data as possible, even if I don't know exactly what I want to do with it right now.
I'm realizing this is part of a bigger picture in my life. I recently wrote my own command line software for budgeting and one of the main joys there was that the software was my own and it did exactly what I wanted. That kind of software is, IMHO, the most fun kind software to write and I'm always hugely excited when I'm able to document and put it out there in the wild. And then, when I want something a little different, I go have fun changing it! I get what I want and I have fun in the process, what could be better?!
This feels like a somewhat similar thing to me. I have an idea of what I want and some of the skills to make it happen. It's not for other folks but maybe other folks will benefit. It's not a business idea but maybe it becomes one. Like the budgeting software, it combines two interests I have - digital archiving and software - into one fun project.
If you're doing something similar on your own or in public or know of any similar projects out there that I can learn from, I'd love to hear from you. I'll be working on this in the open, in both words and code, and it would be great to talk to and hear from folks thinking or working in the same vein.
Over a year later, I built a system that scratches the surface of what I'm describing here. I'm calling it a Personal Data Pipeline and I would be honored if you would take a look!
< References >
- Beautiful documentary on Life Logging from 2013
- Stephen Wolfram on taking things way further than I want to
< Take Action >
Comment via:
Subscribe via:
< Read More >
Tags
Newer
Jul 06, 2023
Conducting a Great Technical Interview
Technical interviews are a vital component of culture building and setting your team up for success. Here is how I make sure they go well and have a great time doing it.
Older
Mar 26, 2023
An Open Letter to My Mental Boardroom
A note to all the people that help make me better everyday.