Analysis of email messages in 18 years: Unsuccessful time travel experiment

In our blog, we write a lot about creating emails and working with e-mail. We have already discussed the complexity of the fight against spam , the future of email, the protection of email correspondence , as well as techniques for working with email , and also figured out why over the course of time the topics of email messages are getting longer .

Many people have been using e-mail for decades - which means that the question arises that the analysis of these letters in a long time can tell about the evolution of a particular person. Journalist and creator of the Postlight web and mobile app development service, Paul Ford, on his Medium blog, published the results of an experiment in which he tried to create a dynamic email search system in his inbox that had more than 450,000 email messages in 18 years. posts. We present to your attention an adapted translation of this note.
')

Idea: time travel

The idea of ​​the researcher was as follows: if he could quickly look through all his old messages, this would allow him to trace how his own views and thoughts changed over time.

It should have allowed me to learn something important about myself, to see my height as a person — for example, comparing myself to a 20-year-old and myself to a 40-year-old.

However, the results of the experiments upset Ford.

Email and large corporations

First, he needed to resolve some technical issues. Ford uses a Gmail account, but Google's search engine is very specific. With it, you can easily find any of the recently received e-mails, like a needle in a haystack. But the researcher was going to jump into the stack from all over the place, and for that he needed a fast and efficient search engine that could scan tens of thousands of emails in an instant. And the Gmail search interface in such cases looks like this:



Considering that Ford also uses a Macintosh computer, he could download all his correspondence using Apple Mail and use Apple Spotlight's built-in search. But the point is this.

In 1996, the user could press the ⌘-F key, then enter the file name in the search bar, and the computer could find the file in a few seconds. Today, as a result of full-text search, a person in a similar situation receives 5,000 files that are not related to the one he was actually looking for.

You try again ... and you get the same result. My relationship with Mac OSX Spotlight is already ten years, and I can confidently say that this is one of the most difficult relationships in my life. I do not understand this program. Obviously, the problem is in me, so I have to move on.

Ford concluded that neither Google, a search corporation valued at $ 375 billion, nor Apple’s technology company, valued at $ 700 billion, could sort his emails.

And that's fine. Even if in a supermarket there are no pants of my size, I still go there for detergent.

Email and Free Software

Free software caused a problem with email, so Ford suggested that it could also be a solution. And, as it turned out, he was right. Using a tool called offlineimap, the researcher downloaded all correspondence from his Gmail account. It took him a few days, after which he had to deal with the processing of letters. This can be done in different ways, but Ford prefers to use the programs mairix and mu . He used to use mairix, but in mu there are more additional possibilities for compiling lists of letters and displaying them, so the final choice fell on him. To start a search in mu, you need to type:

mu find waffles

and the program creates a special folder in which all emails containing the word "waffles" are collected. In the case of Ford, there were 99 letters.

Email and person

So now the researcher had a laboratory to study his past. During the study, he found out that in 18 years he sent 82,865 emails, an average of 4,600 emails per year. A lot.

Then he began to search for letters on certain words in order to see how his attitude towards certain things changed over time. For example, when Ford just started this project, he finished an article about good manners and politeness. And to find out what he himself had thought about manners and politeness before, he entered into the search bar:

mu find from:ford@ftrain.com polite

On request, 196 letters were found, each of which contained a given word in the text (including the quoted part). Ford looked through all these letters.

For 18 years he wrote about manners and politeness about the same thing. In all the letters found, something was written like “I tried my best to be polite,” or “politeness is important to me,” or “I tried to behave politely and respectfully when I met people from the radio.” My opinion on this issue, my underlying beliefs, assumptions and manners have not changed.

Specifically, this topic is not very interesting. But what really changed in these 20 years is the network. Ford writes that he learned a lot about programming, about creating content management systems. His concept of technology had to change, right? So he decided to find out how much his understanding of the network has improved. Using the “HTML” query, Ford managed to discover a long-forgotten blogging tool that he wrote for his friends in 1999. He didn’t have a name.

It might as well be written yesterday. During this time, Ford learned a lot about programming and databases, and spent a lot of time studying computer science. And all this in order to do the same things, and then forget about it, and do them again. It's like a movie Groundhog Day about the movie Groundhog Day. He continued to read his letters and understood that all twenty years he had been talking about:

• Blogging;
• Content management;
• Writing letters;
• The future of magazines;
• The nature of technology.

And all twenty years arguing about:

• Policy;
• Racial discrimination;
• Self-determination;
• Sexism;
• Gender Differences.

Sometimes he told other people how to behave, sometimes they told him. The very content of these disputes has not changed much.

Before the researcher conducted this experiment, he was sure that he had previously taken a passive position and tried not to get involved in conflicts, and that it took him a lot of time to learn to prove his point - but now he is much more willing to defend his views. But that would be untrue. Judging by the archive, Ford was constantly involved in any dispute by e-mail, and, obviously, became skilled in this matter. And since he had long believed that in the past he had not firmly defended his views, in the present, he held the victim's passive-aggressive position more often than other people.

Realizing this, I no longer felt a sense of progress.

The word “hello” was met a thousand times in the headlines of letters and six thousand times in the content. Ford wrote a simple script, running which from the command line, you can find out how many times in 18 years letters with the word "coffee" were sent. Each "*" is equal to ten words "coffee".

 $ mu find from:ford@ftrain.com coffee|perl -ne '/(\d{4})/; print "$1\n";'|sort|uniq -c|perl -ne '/(\d+) (\d{4})/; print "$2 " . ("*" x ($1/10)) . "\n"' 2000 * 2001 *** 2002 ***** 2003 ******* 2004 2005 2006 * 2007 2008 2009 * 2010 **************** 2011 ******************** 2012 *********************************** 2013 ************************************** 2014 *********************** 2015 ******** 

And here you can trace some pattern. Until 2005, Ford went on dates, engaged in freelance work and drank a lot of coffee. Then he became an editor and met his wife - as a result, he lived without coffee for almost five years. In 2010, he quit and began to work for himself, so he began to drink coffee in liters. This is, of course, interesting information, but it does not carry any benefits and does not reflect any internal changes. It is just a consumption pattern.

It should also be noted that all these years, Ford longed for change. Of course, he is not the only one who sends emails. The Ford archives store letters from hundreds of other people, and he says that he still communicates with some of them.

It is very strange to re-read our correspondence, because many of us are still obsessed with the same ideas as ten or fifteen years ago. We grow up, get married and get a divorce. Some of us are rich, some are poor, some read comics, others write poems, and others write stories, and some wear the same T-shirts.

Children change us, for their sake we change throughout life. Divorce changes us. We try to catch the moment, to become better, but in the end we return to our usual way of life. Time is running like water. But as they say, water wears away a stone. Change comes from the outside.

That is why Ford believes that his experiment ended in failure. This is the era of introspection and radical transformation. He compiled tables, made calculations and searched for patterns. Now the researcher can name the 20 most frequently used words in each year or the number of letters in which he wrote about weight loss can tell when he first thought about becoming a father. By and large, all life can be presented in numbers - actuarial tables, bank statements, squares owned by a person, the number of children. But counting things does not change them.

In the process of writing this article, I came across one email 11 years ago. It turned out that it was this article - which you are reading at the moment - that I tried to sell to National Public Radio as an announcer on the text entitled “Collection and Analysis of Data about Myself”.
“The archive of my e-mail stores letters that I have received over the past 8 years,” I wrote in 2003, “and I have long wanted to write a search program and ...”

You know the rest.

They approved it.

“To figure this out,” I told them, “it will take time.”