FactoryCity

Even though I wasn’t able to attend the eighth Internet Identity Workshop this week in Mountain View (check out the latest episode of TheSocialWeb.tv for a glimpse), I wanted to do my part to contribute so I’m sharing the results of a study that Brynn Evans and I performed on Mechanical Turk a short while ago.

I’ll cut to the chase and then go into some background detail.

Of the 302 responses we received, we only rejected one, leaving us with 301 valid data points to work with. Of those 301:

• 19.3% had heard of OpenID (58 people)
• 9.0% knew what OpenID was used for (27) and 8.0% were unsure (24)
• 1.3% used OpenID (4) and 18.3% were unsure if they used it (55).
• 5.3% recognized the OpenID icon (16) and 7.0% were unsure (21).

Combining some of the results, we found that:

• of those who know what OpenID is, 14.81% use it.
• of those who have merely heard of it, 6.9% use it.

That’s what the data show.

Background

Several weeks ago, Yahoo released usability research and best practices for OpenID (PDF). This research was performed by Beverly Freeman in the Yahoo! Customer Insights division in July of this year and involved 9 female Yahoo! users age 32-39 with self-declared medium-to-high level of Internet savvy.

This research, along with Eric Sachs’ later contributions (Google), have taken us from virtually zero research on the usability of OpenID to having a much more robust pool of information to pull from. And though I’m sure many would agree that this research only points to opportunities for improvement, many people interpreted the results as an indication that “OpenID is too confusing” or that it “befuddles users“.

A lot of people also took cheap shots, using the Yahoo! results to bolster their long-held arguments against the protocol and its unfamiliar interaction flow. The problem with such criticism, as far as I’m concerned, is that generalizing from the experiences of nine female Yahoo! users in their thirties is not necessarily representative of the web at large, nor are the conditions favorable to such research. Y’know, Ford got a lot of flack too when he introduced the Model T because everyone loved their horse and carriages. Good thing Ford was right.

Now, some of the criticism of OpenID is valid, especially if it can be turned into productive outcomes, like making OpenID easier to use, or less awkward.

And it serves no one’s interests to make grandiose claims on the basis of minimal data, so given Brynn’s work using Mechanical Turk (with Ed Chi from PARC), I thought I’d ask her to help me set up a study to discover just what awareness of OpenID might be among a wider segment of the population, especially with Japanese awareness of OpenID topping out around 28% (with usage of OpenID at 15%, more than ten times what we saw with Turkers).

Mechanical Turk Demographics

First, it’s important to point out something about Turker demographics. Because Turkers must have either a US bank account or be willing to be paid in Amazon gift certificates, the quality of participants you get (especially if you design your HIT well) will actually be pretty good (compared with, say, a blog-based survey). Now, Mechanical Turk actually has rules against asking for demographic or personally identifying information, but some information has been gathered by Panos Ipeirotis to shed some light on who the Turkers are and why they participate. I’ll leave the bulk of the analysis up to him, but it’s worth noting that a survey put out on Mechanical Turk about OpenID will likely hit a fairly average segment of the internet-using population (or at least one that doesn’t differ greatly from college undergraduates).

Method

Over the course of a week (October 19 - 26), we fielded 302 responses to our survey, paying $0.02 for each valid reply (yes, we were essentially asking people for their “two cents”). We only rejected one response out of the batch, leaving us with 301 valid data points at a whooping cost of $6.02.

Findings

As I reported above, contrary to the 0% awareness demonstrated in the Yahoo! study of nine participants, we found that nearly 20% of respondents had at least heard of OpenID, though a much smaller percentage (1.3%) actually used it (or at least were consciously aware of using it — nearly everyone (18%) who’d heard of OpenID didn’t know if they used it or not).

There was also at least some familiarity with the OpenID logo/icon (5.3%).

What’s also interesting is that many respondents, upon hearing about “OpenID”, expressed an interest in finding out more: “What is it? LOL.”; “I’ve gotta look it up!”; “This survey has sparked my interest”; “Heading to Google to find out”. I can’t say that this shows clear interest in the concept, but at least some folks showed a curious disposition, as such:

How can I tell for sure whether I’ve used OpenID or not when I don’t know what it is? I’ve surely heard of it. That confuses me mainly in Magnolia {bookmarking service} where I want to sign up, but I can’t as it asks for OpenID. And until you mentioned above, it simply didn’t occur to me to just search it up. Hell, after submitting this hit, I’m going to do that first and foremost. Anyways, thanks a lot for indirectly suggesting a move!!!

Now, I won’t repeat the other findings, as they’ve already been reported above.

Thoughts and next steps

The results of this survey are interesting to me, but not unexpected. They’re not reassuring either, and they tell me that we’re doing well considering that we’ve only just begun.

Consider that 20% of a random sampling of 300 people on the internet had at least heard of OpenID, before Google, MySpace or Microsoft turned on their support for the protocol (MySpace announced their intention to support OpenID in July).

Consider that nearly a year ago Marshall Kirkpatrick sounded the deathknell of what seemed like the forgone conclusion about OpenID:

Big Players are Dragging Their Feet … Sharing User Info is a Whole Other Matter … Public Facing Profiles are Anemic … Ease of Use and Marketing Clarity Remain Low Priorities

Consider that no concerted effort has been made to date to inform or educate the general web population about OpenID, or about the problems with sharing your user credentials all over the web, and that many of the large providers have yet to turn on their OpenID support (despite all coming to the table and agreeing that it’s the way forward for identity on the web (save, as usual, Facebook, looking more Microsoftian by the day).

Consider also that momentum to rev the protocol to accommodate email addresses in OpenID is just now gaining traction.

In other words, with areas of user education becoming obvious, with provider adoption starting to happen (vis-a-via MySpace demonstrating the value and prevalence of URL-based identifiers) and necessary usability improvements starting to take shape (both in terms of the OpenID and OAuth flows being combined, and in terms of email addresses becoming valid in OpenID flows), we’re truly just getting started with making OpenID ready for mainstream audiences. It’s been a hard slog so far, and it’s bound to continue to be challenging, but the shared vision for where we’re going gets clearer every time there’s an Internet Identity Workshop.

I plan to re-run this study every 3-6 months from this point forward to keep track of our progress. I hope that these numbers will shed some much-needed balanced light on the subject of OpenID awareness and adoption — both to demonstrate how far we have to go, and how far we’ve come.