What would you do with your data?
If you had the log of your entire life. Your mood, the foods you ate, the things you did and when you did them, everything. What would you do with it? Could you do anything cool, perhaps even incredible?
I’ve been thinking about this for the past couple of weeks, and it seems the answer is yes. We humans are flawed, we run on corrupted hardware, we are difficult to motivate, we are in almost every aspect at least a bit flawed. We go through our daily lives trying to fix our imperfections. Sometimes we make progress, but our flawed memory, our skewed images of ourselves and our remarkably limited understanding of the world makes me incredibly pessimistic about the pace at which we improve. But what if we had the log of every change we made that we thought would improve us, every philosophy, every skill, every Art. What if we could see how they actually changed our lives for better or worse?
We are not very good at being honest to ourselves, especially when we aren’t honest to others. “The part of yourself that distorts what you say to others also distorts your own thoughts” as Eliezer Yudkowsky put it in Twelve Virtues of Rationality. It’s not that we are inherently dishonest, our brains are just incredibly good at tricking ourselves, and the brain believes what we say whether we really think it is true, or not. Akrasia is a very real enemy, and it is incredibly difficult for us to know when we’re one step closer to defeating it, and what it was that caused our progress.
Allow me to give an example, regarding the remarkable industry of self-help books:
“Self-help books are written to sell, not to help. Most books are structured around a gimmick that will sell well. The authors usually show no interest what-so-ever in testing to see if their advice actually works. In fact, I sometimes suspected the book was being written to get people motivated without actually giving them good advice so that when they failed to achieve their goals six months later they would assume it was their fault but look back positively on their initial motivation, and then buy the next heavily-marketed self-help book to come out the pipe.”
― Luke Muehlhauser, The Best Self-Help Book of All Time
After reading this, you’re probably a little sceptic about self-help books in general (if you weren’t already). But some of them might actually work, how could we really know which books lead to the greatest improvement? Well, we could log the variable we tried to improve before we read the book, collect more than a few datapoints, and then read a self-help book and apply what is in it (if it even gets us that far). We keep logging, and after a while we have established a pre-state and post-state. If our records show we made progress since the book was read it would be in favor of the books efficacy, if not, well that’s too bad for the book (or rather, its author). There is a problem however, your dataset is now as biased as it can get. The proper way to do this would be to tell other people to do the same and report their results, you’ll find that your results vary, but the average speaks volumes. You might even take it to the next level and see what kind of person benefits the most from what book, but that’s a challenge I leave to the skilled data scientist.
The same applies to how much sleep you need (and how much is too much), the efficacy of diets/supplements/drugs, what music makes you more productive and so on… There are incredibly many unexplored avenues to take this further, and the value it would provide could be incredible.
I’ve been logging much of my own life for the past month. Mood, productivity and alertness 3x daily, the activities I’ve engaged in (coding, studying, training, reading, writing, etc.) and the insights I’ve gotten are many. Logging what activities I engaged in accidentally enabled the Seinfeld method which turned out to work better than expected. It also happened to have the same method as the non-zero days system introduced to me by ryans01 in this wonderful post. I’d encourage you to try it, it doesn’t take much time and is even surprisingly fun. I use Google Spreadsheets, and while it works well, there are limitations and possibilities that call for a specialized service. But for exploring, a spreadsheet works well.
I’d love to show you how I’m doing it and the discoveries I made, but that will have to wait for another time as I am still discovering the possibilities and the limitations of my (relatively) simple spreadsheet. In the meantime, cheap ways to gather data that would otherwise be expensive to gather includes services like RescueTime which logs your computer activities. I’m sure there are others out there, leave a comment if you find one!
Log on!