This page presents the raw data behind the article E-mail users are individuals too - the lack of personalisation in email tools today posted on the Haystack blog on Nov 20, 2008.
If you have any questions or comments, please contact Max Van Kleek at emax at csail dot mit dot edu.
Dear CSAIL hackers,
Do you actively use custom code you've written/appropriated+modified to handle your e-mail inbox deluge? If so, please tell me about it!
Part of my research investigates ways for people to be able to exercise greater flexibility in their information management tools like e-mail. Back in the day when I was an undergrad, many of my friends prided themselves on the cool customizations they wrote for their Athena logins -- including procmail scripts to automatically filter incoming e-mail, page them when they got email from their girl/boyfriend(s), etc.
These days though, much of this hackery seems to have vanished. This seems relatively counterintuitive given the rise in hacker-friendly open platforms like GNU/Linux. Most of the people I know (even at CSAIL) use fairly standard features of mail client software (e.g. Thunderbird). Many even just use web-based email such as Gmail.
I'm trying to figure out why. Has the need decreased, e.g., do people just not need anything beyond what is offered by today's email clients? Do people use different tools? Are my friends just no longer hackers? Are people becoming increasingly lazy??
So I thought I'd poll one of the greatest hacker dens to see how much custom code-rolling still survived. I'm mainly focused on personal e-mail management -- and would like to know about hacks you've performed to your e-mail processing tools/mail client to improve its functionality.
If you've devised a hack that you actively use to manage your email and don't mind telling me why, could you please send me a sentence or two of what your hack does and why?
Thanks,
eMax
For the last decade or so I've been using a system based on spamsieve to auto file my email into the appropriate IMAP folder. This helps me focus only on important emails and file less urgent things into a delayed folder which i just glance through occassionly. It also forwards me important mail via text message. The filtering is at the server, so I can change my email client easily
I used to do all manner of filtering, like you said, and I've largely stopped. A lot of my procmail juju had to do with spam, and SpamAssassin has done pretty nicely with that.
A number of other bits, like backup and sorting some mailing lists into their own folders, are well taken care of by the "rules" in the WebSieve setup csail has, and it's easier to use than procmail. I have 9 rules.
The paging/etc has been taken over by better cell phone features and availability. If my girlfriend needs to reach me, she can more easily call or text. I mean, back then, I had an actual pager, and no cell phone.
Another task was vacation and forwarding rules -- the vacation program didn't deal with all mailing lists properly, etc. Again, where applicable, there are adequate, more task-specific solutions in place.
In sum, I think I used to hack because the standard features weren't standard yet. I don't lament it -- I've got more interesting problems.
Lots of people complain that spam is an intolerable burden. With Rmail I find that each spam message costs me a second or occasionally two, which means that around 200 spams a day cost me 4 minutes. Hardly even worth complaining about.
I wonder if other people's graphical mail interfaces are so slow that they make this problem far worse for them.
So here's what I do, and yes, I think I fall into the lazy category:
1) For my company email (in Outlook), I use rules to stuff incoming mail into the right folder, mostly based on email addresses -- a real pain when people start mailing from a new isp, like when gmail started.
For the most part, I file mail from family, friends, CSAIL into appropriate folders. Anything left from outside my company goes to Junk. 99% of that is really spam, but if I am expecting email confirmation from a conference or something, I check the junk folder more closely.
2) Personal mail goes to gmail. Their spam filter is pretty good, the rest I go through manually.
3) Anything I purchase online or any form that insists on an email address goes to my hotmail account. I look at it a few times a week. It's almost all useless, but not true spam.
So I guess I use multiple email addresses to sort my mail.
Back when I read all of my email in nmh on athena, I added a "procrastinate" command so I could make messages vanish from my inbox, and then show up as new mail x days later. This way if I didn't need or want to deal with a message when I received it, I could put it off without needing to actively remember to come back to it.
These days I read most of my email through webmail. I would still make use of a procrastinate command if I had one, but I don't know of an easy way to add it to the webmail system I currently use (not as easily as the 4 line shell script I used last time). And why do I use webmail?: mostly laziness.
I don't do anything too spectacular, but here it is anyway. I use the websieve management tool to filter all of the various email lists I'm on into different folders. I find this a good compromise between getting a digest of an email list and getting the emails in real time. I prefer to do server side filtering into folders as that way it is how I like it no matter which computer I check my mail on (I typically use thunderbird).
On my main machine I use gnubiff to notify me when new emails arrive into my inbox. That way I don't get notified every time a reuse mail comes (unless I'm looking for something in particular), but I do get notified of emails sent directly to me.
I occasionally tweak the rules so that messages from reuse (for example) related to particular items I might be interested in go directly to my inbox so I can see their arrival faster.
My hacks are mainly just filters setup on the CSAIL IMAP server. Essentially, I filter all mailing list mail (including csail-related) and non-personal mail into a single folder that I look at once a day. Everything else gets filed to my Inbox. What's nice about this is that I get only about 10ish messages a day in my Inbox, all of which are things I usually want to/need to read.
I've also hacked up my mail client with keybindings and the like so I can pretty much go through all of my mail without taking my hands off the keyboard. Of course with my primary client, mutt, that was easy. For my secondary client, Thunderbird, that required getting a few extensions.
As for your question as to why people hack less on their systems... I can't talk for the vast majority of people, but the reason why I’m slow to customize is that I have three machines that I use every day, and I need to find a solution that works on all three. This means I can only use IMAP server filtering for filtering, and I need a way to sync my configuration between all of the machines. With mutt, it’s easy — I just put the dotfiles in svn. I don’t know how to do that with Thunderbird yet, so I just use it on one machine (my laptop) because it has really good offline support.
One reason I don't hack my email much anymore is that I seem to migrate to a different client or server every couple of years, and it's hard to move the hacks over.
But I do have a couple of active hacks:
- a single email address for both me and my wife, "me+wife@mit.edu", which copies both of us on received messages. We use it for services that we use jointly, like our bank accounts.
- a script that lets me send "future emails" to myself. Basically I compose a message with a future date at the start of the subject line (like "(Mon)" if I want to see the message next Monday), save the message in my imap Drafts folder, and every day a cron job script visits my imap Drafts folder, looks for messages whose deadline has arrived, and simply moves them to my Inbox. I use this all the time for scheduling to-do items, reminders, and postponing handling email messages (by simply Forwarding the message and saving the forward as a dated draft). I looked at a few web services out there that send reminder emails, but they were ad-supported, annoying to use, and complex.
I used to spend a lot of time customizing the configuration of all sorts of things, including my email clients. However, over the past, say, 5-7 years, I've lost interest in the endless configuration. For the most part, the defaults are good enough, since the effort required to maintain these hacks and port them to new systems outweighs the benefits. (Same reason I now use a Mac, rather than Linux, for my personal computer).
I'm definitely in the camp of "I use a plain email client" now. I do have some hacks I've found useful for *work*, but not for my personal email:
1. Old fashioned email filters. I'm TAing a class, and having a separate folder for those emails based on the To: address is really useful. Actually, I do this for one mailing list I'm subscribed to for my personal email.
2. script to extract attachments from said class emails. This is critical for grading assignments.
3. Script to mass email students in the class.
For most of my personal email, the filtering/sorting that I get on CSAIL's IMAP server is good enough.
https://imap.csail.mit.edu/cgi-bin/websieve.pl
Most of my mail goes to The only real hack I use for email now is a vultures@csail -> zephyr
gateway. It's drastically improved the response time for me and my
officemates. :)
"Back in the day" I used to have customizations for all sorts of things. Then the APIs changed incompatibly. Then they changed again. And again. And again. It became easier to just adapt than to keep rewriting changes to track the API of the year (many of the changes weren't even possible when I moved from Lisp Machines to windows and unix). If the amount of customization seems to depend on age, this might have something to do with it.
I haven't done any coding, but I arrange my email folders so a few
letters will identify any one of them. Then any new email is filed
with: S
I use a bunch of procmail processing. I personally hate client size mail processing as I think it is stupid. Partially this is becasue I use clients (my cell phone's windows mobile mail reader, pine, and Windows Mail (Vista's Outlook express replacement), even sometimes webmail). Partially this is becasue I like the scriptablility that procmail + arbitrary unix commands give me.
Mostly I do boreing stuff with this, like some custom spam blacklists, some custom whitelists, sort all of my mailing lists etc. But I also have a few other uses, for one, I share netflix with my roomies so any e-mail I get about my roommates stupid movies, I just forward to them and don't even have it go to my inbox. Another example is that I have if you add +phone_only to my email, it only goes to my phone. Also, for a project I share with a roomate, I forward that mail to my roommate. Last, I used to SMS all of my e-mail to myself, before I had a smart phone. When I did this I wrote a custom script which texturally compresses the message with 31337-speak. It was a pretty sophisticated script as I wanted to get as much info in two text messages as was possible, but I didn't want to squish if necessary so it would drop vowels and spaces until it just fit in two messages and no more.
Here's the hack I've long wanted to implement, but haven't found the time to do. It could be implemented in a kind of kludgy way in a procmail script, but more properly belongs in a mail server.
What I want is a way to "mint" a custom email address that I can hand to any entity that _only_ accepts email from that entity. For example, if amazon.com wants my email address, I could hash together a secret passcode with "amazon.com" and hand that to them. If they use the address to write me, it would work but if they give the address to someone else, it bounces back and says that it's a domain-specific address. Strictly speaking, it doesn't have to be a hash, but that solution allows me to mint such an address without ever talking to the server (e.g. to update a whitelist).
As for email hacks, I think it's a lot of things that make them less common. Lots of people use email systems that are simply less hackable than old-style unix systems and also email has taken up less and less of the "electronic communication pie" as the web, IM, social networks, blogs, and the rest have each found their niche. Email is still vital and it's still a catch-all, but I recieve less email these days than I used to. Spam filters also seem to be really good.
"I'm trying to figure out why. Has the need decreased, e.g., do people just not need anything beyond what is offered by today's email clients? Do people use different tools? Are my friends just no longer hackers? Are people becoming increasingly lazy??"
No - people have to sit in front of too many different systems, and upgrade them on a regular basis in ways that break handwritten hacks.
Once upon a time I only used Athena because there wasn't anything else with net access, and I customized the heck out of it because I knew the Athena dev people would keep everything back-compatible so scripts didn't break. If an email program I liked became unpopular and stopped getting updates, no problem.
Now I have my Linux boxes, my home Mac, the windows laptop I borrow occasionally and other systems at friends' houses, so I use webmail, thunderbird, and evolution. None are particularly customizable, and if they were, the customizations would need to be repeated across platforms, in ways that I could access from different environments (secure and insecure). Even if I managed this, I know that upgrades in the mail UI or OS would break my customizations regularly. I've already migrated from exmh, eudora and netscape mail through mozilla and apple mail to what I use now. It's unsafe to continue using mail programs that are no longer being updated, because they develop security issues.
You mention "the rise in hacker-friendly open platforms like GNU/Linux", but what email UI can read HTML, handle attachments, work on Windows, Mac and Linux, and is customizable?
My current hacks are all infrastructure. I have scripts to dump my whole imap mailbox tree to local disk for backups, and grep the whole store when I want to find something from long ago. I've written scripts on the fly to check for batches of duplicate messages, look for mail tagged as spam that shouldn't be, and create mailing lists from batches of messages, but I don't maintain these when I don't need them.
Outlook has a reasonably flexible mechanism that covers all my needs. I've used it to filter messages to my old e-mail address, sent from a particular mailing list, to play a sound when a message comes from someone specific etc. No code, though.
i have hundreds of filters set up in tbird to auto- file my email into folders. it was a big pain, because tbird filters don't have enough expressiveness. for example i have to say "From X" or "To or Cc X" (two predicates) to catch everything to or from X.
there's also no easy way to express priority, for example that an individual message from a student should go to a different place than a group message. and there's no way to capture the manual filing i do, i.e., watch me file, infer a rule based on the headers and the filing action, and put that rule in place.
all of these would be nice to have in a better suite of email management tools.
I use a procmail script that invokes a custom python program to filter my email. There is a rule database that I maintain. The rules identify "good" email, not spam. A rule can allow email based on sender, words in content, and/or both. Email that doesn't pass gets sent a challenge message; replying to that adds a new rule for you. Bells and whistles, including the ability to update the rule database by sending email to myself, quarantining messages, etc...
I think there are a couple of straightforward factors:
- The default tools have gotten vastly better: e.g. mutt or gmail are both way better than pine or mh. - People attach a high value to standardization. It's nice to be able to sit down at an un-specialized terminal and just use your email. (Not that I do this myself.) People also value being able to hand their computer to another person and have them able to use it.
Hacker friends of mine have coalesced around the idea that good defaults are much better than excellent customizability. The latter is a timesink, a cost center. I think it's shortsighted to assume that hackers value customizability over all else. I'd rather not have to write my own mail reader, that's why I use mutt. It's also why I switched from emacs to vim a few years ago, and never looked back. My .vimrc is 54 lines and contains no code or macros.
Through a series of network steps, mail to all of my various email accounts ends up at my gmail account (I use gmail because because finding old mail by fast, reliable search knocks the socks off of any other interfaces I've found, and it was one of the first places that let me persist as much mail as I wanted in a network-accessible location.)
I use a custom script run as a daily cron job to automatically copy up any sent mail related to my company to my the sent mail folder on my company's IMAP server. My script is a a derivative of imapsync (http://freshmeat.net/projects/imapsync/) that I modified to allow me to include or exclude messages based on regexps applied to the full message (including all headers).
I don't do any custom email stuff other than filters, and an add-in for Thunderbird that lets me use pine keybindings to put mail in folders (saves me a lot of dragging and dropping). Reasons for not running custom scripts etc:
- I use thunderbird in the lab, apple mail at home, and webmail or alpine over ssh elsewhere, so there's no consistent platform for hackery
- Mail programs seem to do most things automatically these days
- Probably most importantly: I once had all these filters etc set up to do fancy things, with different folders for different ages of email etc, then I realized that automatically putting my mail in different folders all over the place before I've read it actually made me less efficient, and I kept forgetting to read the occasional important mail that got put in some folder somewhere. Now I've unsubscribed from the lists I don't bother reading, and have everything come into my inbox before I throw it into a folder for archiving after I've read it. I could probably do with having a couple of filters back again tho. One thing that would be cool is if my mail client could guess where I wanted to archive the mail, and then I could just hit a key to send it there.
I've been tweaking a sieve script over the years to delete clearly spam messages, file away possible spam, and automatically put much of my mail into the correct folder. This way I don't need to rely on a heavyweight desktop client to do the work, especially when I want quick web access.