Are Yahoo Mail users better customers than Gmail users?
I admit it: I judge you by your email address. Whenever I see a Hotmail, AOL, or Yahoo address, I can’t help but think that somebody didn’t get the memo in the mid-aughts about moving to Gmail.
It’s ridiculous. Somebody is no better or worse of a person for using Hotmail. All a non-Gmail, non-custom, address probably means is that the person is outside of my peer group. And yet the bias remains.
But what if that bias is justified? What if I can tell how good a customer will be based on the customer’s email address?
Let’s start with a look at the distribution of email address domain names. I queried the purchase logs for Blurity from October 2012 to find the percentage of purchases associated with each second-level domain name. Domain names that appeared in less than 1% of purchases were consolidated in an “other” group. From that, we can see that Gmail, Yahoo Mail, and Hotmail are the most popular email services for Blurity customers:
Not surprisingly, people also liked holding on to email addresses more commonly associated with ISPs, such as AOL, Comcast, Verizon, BT, and COX. (Time Warner fell just below the 1% threshold and is part of “other.”)
It’s tempting to say that Yahoo Mail is under-represented among customers, since in terms of US email service market share Yahoo Mail is roughly equal with Gmail. However, the share percentages are about equal for international users of the two services, and I couldn’t split my purchase data on international lines. Dead-end there.
So what can we look at? Maybe something to do with money? Well, how about return rates?
Blurity has a somewhat high return rate. Embarrassingly high. I attribute that to a combination of my poor UI design skills and the general misalignment between public perception of blur removal and the reality of blur removal. That, in turn, is complicated by the fact that many people don’t read directions, which is further compounded by people believing that the purchased version will somehow do better blur removal than the free trial version. As a result, I occasionally get emails asking for refunds. It’s the cost of doing business. I’d rather have a happy former customer than an angry customer.
I’d always felt that refund requests were more likely to come from the group of people using email services associated with lower degrees of technical prowess, such as Hotmail and AOL, so I was a bit surprised when I pulled the actual data. Turned out that the refund rate of Gmail users was not significantly different from that of Hotmail users, nor for Gmail versus AOL users.
Yahoo was a different story. The refund rate for Yahoo Mail users was higher than that for Gmail users at a statistically significant level (chi-squared, p<0.05). Found one!
A higher return rate is nothing to celebrate, but it’s always nice to discover data vindicating one’s intuition.
Overall, Gmail was pretty good, or at least average. The only group that had a non-zero return rate significantly lower than Gmail was the “other” bin (chi-sqaured, p<0.01). The rest of the domains, including Hotmail, were all roughly the same, ignoring a few small-sample-size zero counts.
After seeing those results, I wondered what other predictors for refunds I could find in the data. I suspected that customers paying with PayPal might have higher return rates than those paying with Stripe. Maybe it was a manifestation of the general ill-will towards PayPal in the tech community? Much to my surprise, the refund rate for PayPal is actually lower than for Stripe (6.4% vs. 9.5%), though not significantly so.
I’m not sure that this is very actionable. The vast majority of customers of all sorts, including those with Yahoo email addresses, have been fantastic. I might try an A/B test for users entering Yahoo email addresses, perhaps with a more explicit link to the user manual or a tutorial video. Of course, if I think that might work for Yahoo users, then I might as well try it with everybody. Yahoo might have more refund seekers than other services, but it by no means has a monopoly.
Update: Had another customer make a purchase today and then almost immediately (three minutes later) ask for a refund. The customer’s email service? You guessed it: Yahoo.
Update 2: My friend Luke pointed out that the Live.com, Hotmail, and MSN domains are really all just Microsoft services. I agree, they should probably be grouped. I ran the numbers again with that grouping and found that the refund rate for the Microsoft properties does not differ significantly from that of Gmail, and the confidence interval is very similar to that charted for Hotmail alone. Thus, the conclusions above remain unchanged. Regardless, a good catch!