This past weekend, Gawker Media was hacked and its user account database was leaked online. The database contained about 1.3 million rows of information containing usernames, e-mail addresses, and passwords (encrypted via DES). This security breach is unfortunate for people whose information is contained within that database, but the silver lining is that it provides a rare opportunity for statistics nerds like me to analyze some otherwise completely unobtainable data.

Because the passwords were encrypted using such an out-of-date scheme (tsk, tsk, Gawker), about 200,000 of the passwords contained in the database have been decrypted. Of course, the passwords that were cracked were relatively weak. For example, all 2641 accounts that used some trivial modification of “password” or “querty” as their password were of course decrypted. In this post I will look at trends in which users’ passwords were cracked to gain insight into which users do and do not create strong passwords.

It should of course be made clear that, because this data comes from a single database, the results that follow may not be representative of the population as a whole, but rather may be skewed by the fact that people with Gawker accounts are generally more “techy” than the average internet user.

Preliminaries: Cleaning Up the Database

The database of course had to be significantly cleaned before it could be of too much use statistically, so some of the numbers here may differ slightly from the raw numbers you see from news outlets or if you download the raw database yourself. The numbers here are the result of removing any incomplete rows from the database (i.e., rows missing a password, e-mail address, or both) and removing any accounts that were clearly created by SPAMbots (I’m only interested in the password strength of real users).

Also, I will only look at accounts that contain an e-mail address with a domain that was registered in the database at least 50 times. This restriction is in place partly because it is extremely difficult to compute any sort of meaningful statistics on something with a sample size that is much smaller than 50, and it is partly due to the fact that Gawker doesn’t require verified e-mail addresses (so 46993 of the 52593 domain names listed in the database were used by exactly one person, many of which are clearly fake and/or for SPAM).

After making the aforementioned “fixes” to the database, there are 412670 accounts, 157794 (38.2%) of which had their password decrypted.

Password Strength by Domain Name

The following table displays the 10 most frequently-occurring domain names used for e-mail addresses in the database along with how many users of the domain had their password cracked.

The following table shows the z-values associated with the statistical test that the two given domains have the same proportion of users with strong passwords. Differences that are statistically significant at the α = 0.01 level are in bold. Click on a z-value to see a normal distribution showing the associated p-value. Notice in particular that gmail.com users have stronger passwords than users of any of the other top-10 domain names, while aol.com and hotmail.co.uk users have the weakest passwords.

Educational Institutions

Not surprisingly, users who entered an e-mail address from an educational institution typically had stronger passwords than the general population. Of the 2092 users who provided a college or university-based e-mail address, only 697 (33.3%) were decrypted. This proportion is significantly lower than the corresponding proportion for the general population (z = 4.64, p < 0.001).

However, two universities stood out as having particularly weak passwords: of the 56 users who used a University of Texas e-mail address, 27 (48.2%) had their password decrypted, and similarly 101 (45.1%) of 224 New York University passwords were decrypted.

ISP-Provided E-Mail Users

Users who used an e-mail address provided to them by their ISP (such as something@comcast.net) typically had weaker passwords than the general population, a fact that can perhaps be explained by the fact that tech-unsavvy folks are less likely to go out and get a new e-mail address for themselves at a place like GMail. Of the 31667 users who provided an ISP-based e-mail address, 13053 (41.2%) of them had their password decrypted. This proportion is significantly higher than the corresponding proportion for the general population (z = -11.36, p < 0.001).

E-Mail Addresses with Typos

Also unsurprisingly, users who entered an obvious typo in their e-mail address were much more likely to have a weak password than people who entered their e-mail address correctly (by “obvious typo” I basically mean an e-mail address containing a typo of a common domain name, such as “fred@yahoo,com” or “fred@hotmail”). Of the 530 users with a typo in their e-mail address, 280 (52.8%) had passwords that were decrypted. This proportion is significantly higher than the average (z = -6.87, p < 0.001).

Password Strength by Country

The following table shows the strength of user passwords based on the country associated with their e-mail address. Of course some e-mail addresses provide no information about the user’s country, so domains that serve a largely international market (such as gmail.com, mac.com and aim.com) are excluded from this analysis.

So Russia and Germany are the big winners when it comes to password strength, while India and the United Kingdom seem to have the weakest passwords. The following table shows the z-values associated with the statistical test that the two given countries have the same proportion of users with strong passwords. Differences that are statistically significant at the α = 0.01 level are in bold. Click on a z-value to see a normal distribution showing the associated p-value.

Attached below is an Excel Spreadsheet containing significantly more detailed information than the snippets contained in this post (though of course all passwords, e-mail addresses and personally-identifiable information has been removed).

Download: Gawker Database Statistics [Excel spreadsheet]

There are a lot of online tools out there for computing p-values and test statistics associated with common statistical distributions such as the normal or Student’s t-distributions. Unfortunately, most of them are either ad-ridden or powered by Java (and hence slow to initially load and finicky when it comes to which browsers they work with). So one of my summertime projects this year was to create a website that solves both of those problems:

The website computes p-values and test statistics in real-time via javascript (and thus does not need Java or any other plug-in). The computations themselves are fairly straightforward and are performed via the trapezoid rule. The graphic on the right is composed of a static PNG that displays the appropriate distribution. The distribution’s image is transparent under the graph and opaque above the graph, which makes it easy to display the p-value graphically – the light blue area is actually just a blue rectangle that is drawn beneath the distribution’s image.

Additionally, through the magic of PHP the tool automatically creates a URL that links to the current computation (and thus makes it much more citable). So, for example, if you want to know the T-value that corresponds to a right-tailed test with 12 degrees of freedom and a p-value of 0.1, you could simply click here.

Anyway, if you’re a nerd like me then enjoy it and of course feel free to leave any feedback/suggestions that you might have.

The term “infographic” is a ridiculous little buzzword that really took off on the internet sometime last year. It used to refer to genuinely useful things like subway maps and blueprints. Recently, however, the term has come to mean “an obnoxiously oversized image that has numbers on it”. My problem isn’t with infographics like these ones that just display some fun, meaningless information is a visual way, or this one that displays a phenomenon that is inherently visual. My beef is with infographics that reduce a variety of related statistics to an oversized mess of overlapping graphs and charts that are (purposely or otherwise) misleading.

This post will present four rules that infographic designers, if they decide that they absolutely must make an infographic, should always follow (but often don’t). To get the ball rolling, let’s consider an example that made its way around the internet just a couple of weeks ago (source):

American 2009 Season Premieres and Averages to Date (click to enlarge)

We are told that the above infographic depicts the US viewership for a variety of shows during their premiere (light red) and on average since they began their 2009 season (dark red). However, I have two main problems with the image, and they’re both problems that are prevalent throughout many infographics and can easily be solved by just using a simple bar graph.

1. Infographics should not require horizontal scrolling. The above infographic is 3133 pixels wide, which means there is no consumer-available monitor in the world capable of displaying the entire image on one screen without scrunching it down. This is apparently exactly what infographic makers want, since they all seem to subscribe to the school of thought that dictates their image deserves 45 inches of horizontal viewing space. This would be fine if infographics were readable when zoomed out, but by their very nature they almost never are.

Computer monitors were not meant to view posters. If you want to make the image high-resolution enough that it can be printed out as a poster then it should be created as a vector graphic, not a raster graphic. If you still insist that your infographic should be a monstrously large bitmap, make it readable from a zoom level that will fit on standard monitor resolutions.

Some other popular infographics that suffer from this problem are the new auto industry breakdown, weight of the world, and the first 100 days.

2. Two-dimensional figures should never be used to compare linear data. The above infographic compares the number of people watching different shows, so why are circles being used to represent the data? What represents the number of viewers — the radius of the circle or the area of the circle? The source doesn’t tell us, so we have no way of appropriately assessing how many more people are viewing NCIS: Los Angeles than The Good Wife. If it’s the radius of the circle, NCIS appears to have about 5% more viewers. If it’s the area of the circle then it’s probably over 10% (and the discrepancy gets much larger if you compare shows that are farther apart).

Furthermore, even if we were told whether it’s the radii of the circles or their areas that we should be looking at, there’s still a problem. If the radii are what are being compared, then the visual is misleading because the differences in areas cause the relative differences to appear larger than they actually are. If the areas are what are being compared, then it should be noted that people just plain suck at visually comparing areas. By looking at the above image (and not getting out a ruler or anything) can you tell which circles have about half as much area as the NCIS: Los Angeles circle? Can you tell how much higher the viewership of The Good Wife is than that of Glee? I certainly can’t, at least not quickly.

InfomationIsBeautiful.net is a particularly notorious violator of this rule, as these three examples show: deadliest drugs, how safe is the HPV vaccine?, reduce your chances of dying in a plane crash (scroll down to the “bad month” and “the odds” sections). What’s worse is they aren’t even consistent with whether it’s the areas of the circles or the radii of the circles they’re comparing.

Problems #1 and #2 can both be rectified by simply turning the data into a bar graph. A plain old-fashioned bar graph. Voila:

American 2009 Season Premieres and Averages to Date (easier to read)

The above bar graph doesn’t need to be zoomed in to be read, it makes it easier to compare the relative viewership of each show, and it actually contains more data than the previous infographic thanks to the labels on the vertical axis.

The next example (source) supposedly explains how and why low-cost airlines are able to offer flights that are so much cheaper than other airlines. It made its rounds this last spring during recession fever, when anything that had anything to do with something being cheap was instantly popular. While it does not suffer from problem #1 above (since it is readable when zoomed out), it suffers from two instances of problem #2 as well as multiple other problems.

How come cheap airlines are so cheap? (click to enlarge)

3. Infographics (and everything else) should be about substance over style. While there’s no denying that the above infographic is pretty, does it actually tell us anything? Beyond the myriad of small problems such as the average fare of Southwest flights including cents when none of the other numbers do, the misspelling of “Aer Lingus” and “maintenance”, and the mysterious 43% “total advantage” at the bottom that seems to pop out of nowhere, the infographic at its core doesn’t even make sense.

As the infographic itself says, low-cost airlines generally don’t do long-haul flights; they focus on short point-to-point routes. So why are their average fares being compared to the average fares of the likes of British Airways, who regularly do intercontinental flights? Doesn’t it make sense that travel distance makes more of a contribution to the price of the flight than whether or not tickets are sold primarily online? Average fare per kilometer travelled would make more sense to compare, though it would still be misleading because take-off and landing are disproportionately expensive.

Another recent offending infographic that just simply doesn’t say a thing is the $400 million club, which notes that Transformers: Revenge of the Fallen is only the ninth movie in history to gross more than $400 million at the box office in the US during its theatrical run. The infographic then compares the other eight movies, which of course are juggernauts like Star Wars and Titanic. The problem is that none of the figures are adjusted for inflation. If you scale the numbers properly, Transformers: Revenge of the Fallen actually comes in as about the 65th highest-grossing movie. Impressive, sure, but to say that the infographic is misleading is an understatement.

I will finish by presenting a graphic that ran on NewsWeek.com that shows obesity and “life evaluation” trends over the last year or two. It’s debatable whether or not it falls into the category of what most people would consider an “infographic”, but it perfectly illustrates a core problem with them.

4. Be careful with your data. Just making your graphic pretty doesn’t give you free reign to ignore basic statistical principles when presenting data. In the above graphic, the left graph shows two lines — one showing how many people have BMI less than or equal to 30 in a given month and one showing how many people have BMI over 30 in a given month. I have a news flash for you, NewsWeek: one of those lines is redundant. Not only that, but the redundant second line manipulates the reader by giving the false impression that the number of obese people is converging toward the number of non-obese people. Nevermind the fact that the vertical scale is completely out of whack and it jumps a vertical distance of 46.4% in the same amount of space that is used to represent about a 2.5% jump elsewhere.

I’m willing to bet that the vertical scale on the right graph is completely out of whack too, but it’s a little difficult to tell since they don’t tell you what percentages any of the intermediate y-values correspond to. On the blue “struggling” line, we are given a value of 48.4% on the left edge of the graph and a value of 49.6% at the right edge of the graph at a nearly identical height. Are we supposed to be able to tell how high and low the peaks in the middle of the graph are based on that? Does the blue line get as low as 40%? 35%? 30%? Would labels along the vertical axis (similar to the bar graph I showed above) really have detracted from the desired aesthetic too much?

So if you have a set of data that you wish to convey graphically, please first consider whether or not it can be presented by a simple bar graph or line graph. If it can, don’t try to make it more complicated than that. If it can’t, at least make sure that the information is the motivating factor in your decisions. If the layout ends up dictating how you present your data, you’ve got your priorities backward.

It’s time for a random dose of statistics courtesy of The Internet Movie Database. Let’s consider all movies that have been released theatrically over the last 60 years and see whether there is a trend in their perceived quality over time. That is, do new movies generally receive higher or lower scores on IMDb than old movies?

Before looking at the numbers though, we need some rules to clarify what types of movies we are considering:

• We only consider theatrically-released films — no straight-to-video movies or TV movies.
• Short films that were released theatrically (such as Pixar’s Presto) are included.
• We only consider movies that have received 1000 or more votes. This restriction is to prevent movies with only a handful of votes from skewing the results too much.
• The theatrical release date of the movie must have been at least as recent at 1950.

IMDb contains 10034 movies that satisfy the above criteria. The average score (on a scale of 1 to 10) of those movies is 6.38 and the median score is 6.6. The average score per release year is given by the following graph:

As you can see, older movies (1950 – 1975) have abnormally high scores, as do very recent movies (2000 – 2009). These differences are indeed statistically significant. For example, the p-value associated with the test that the mean score in 1950 is the same as the mean score in 1989 is less than 10^-19. The p-value associated with the test that the mean score in 2008 is the same as the mean score in 1989 is about 0.0021. Other nearby years give similar p-values.

So this tells us that, in general, particularly old movies receive the highest scores, followed by newly-released movies, followed by “semi-old” movies from the 1980’s and 1990’s. So why the differences? Were movies from the 1980’s really just that bad? Possibly, but the more likely explanation is that movies from the 1950’s  through 1970’s have artificially higher scores because people don’t generally go back and watch the crummy movies of the last generation, so they get forgotten and do not have 1000 votes on IMDb. Will people be watching Disaster Movie in forty years? I sure hope not.

On the other hand, particularly recent movies tend to draw a fair amount of hype and fanboyism. Remember when The Dark Knight had a score of 9.8 and was at #1 on the IMDb top 250? Now, one year later, it has a score of 8.9 and is located at #9 on the top 250. It will likely dwindle a little further down over the coming years as well.

The Best and Worst of Each Year

While we’re looking at ratings of movies over the years, I suppose I might as well provide a list of the best and worst movie of each year (based on the votes of IMDb users), since such a list is not available on the IMDb website itself to my knowledge. Keep in mind that, as before, only movies with 1000 or more votes are considered. Enjoy!

Downloads:

• IMDb rating data [.zip of an Excel spreadsheet — 341KB]
• IMDb rating data [tab-delimited plaintext — 0.97MB]