Rating The Crowdsourced Beer Rating Apps

On Tuesday I regaled you with my wholly uninformed thoughts on a European soccer tournament, so I feel obligated to follow that up today by explaining the current Pokémon craze, despite the fact that the only things I know about Pokémon are that it involves the word “Pikachu” and cranky people hate it.

Some people are, of course, natural born haters, electing to train their shit-ray vision on a goofy, harmless Pokémon game that is bringing joy to millions of people while slightly inconveniencing a handful. Yes, it seems a bit tacky to hunt for them at the Holocaust Museum. And it’s not cool to linger at bars all damn day waiting for some Poke to show up, unless you order enough to earn your keep. And I guess be careful playing in traffic and things like that. But on the whole: good for the Pokenerds. They ain’t hurting nobody.

When I defended the Pokemovement on Twitter the other day, several people pointed out that we beer geeks have our own version: Untappd, the beer-rating app that awards you with badges for checking in certain styles of beer— “Congratulations, you’ve unlocked the ‘I believe in IPA’ badge!”—or the badge for a sponsoring brewery’s beer—Lagunitas Stoopid Wit is currently badgeable—and sometimes, it seems, a badge just for hanging in there. I believe I’ve gotten at least one “Congratulations, you’ve unlocked the ‘What the boss doesn’t know won’t hurt you!’ badge for drinking Coors Light out of your coffee mug at 10:30am!”, which wasn’t even technically accurate but a fair enough guess that I let it slide. What I’m saying is people like collecting these badges, which strikes me as silly and fine.

Get the latest in beer, wine, and cocktail culture sent straight to your inbox.

I don’t use Untappd much, but I do have an account and I’ll check in a couple beers a month because it’s a handy way to keep track of what I’ve tried and what I thought of it in the moment. It can be used in a more social way, though I don’t think I’ve ever interacted with any fellow Tapprs. But if some people get a kick out of that feature, all the better. Untappd is fine by me.

That said, Untappd and it’s less-mobile peers Beer Advocate and RateBeer have plenty of detractors, some of whom are just cranks who hate everything, or beer-industry people, media included, who resent that the unbearded masses now have a platform to express their plebeian opinions. But they also make some valid points that transcend simple haterdom.

Beer is about the moment. Throughout history beer has brought people together in communal settings. When you are rating beer you are isolating the experience.

The biggest issue to me, a relative outsider, is the way a citizen army of beer-scorers contributes to the joy-killing gamification of every damn one of life’s simple pleasures. I realize I feed that beast—I write a column called BeerFight!, for Christ’s sake—but my thin justification is that it’s my job. Sometimes I think all of you people blessed with actual skills that open up other avenues of gainful employment should do more drinking and less quantifying, and spend more time talking to your fellow humans rather than tapping on your phones. Chris Lohring of Massachusetts-based Notch Brewing agrees and says: “Beer is about the moment. Throughout history beer has brought people together in communal settings. When you are rating beer you are isolating the experience.”

Lohring also cites the problem of style bias. When I asked if he thought certain breeds of beer tend to be favored by the rating masses—which was a total softball, because of course they are, and of course Lohring, a staunch champion of pilsner who only brews beer below 4.5-percent ABV, has noticed. His response was succinct: “My lowest rated beer has been my number-one seller for six straight years.” He was referring to Notch Pils, which has a score of 81 from the BeerAdvocate community despite being one of the very favorite beers of America’s foremost authority on Czech-style pilsner (by which I mean: I really dig Notch Pils, and I understand pilsner marginally more than I do soccer or Pokémon).

Beer Advocate’s ratings bias toward big IPAs and stouts is in no way unjust or corrupt—people are allowed to like what they like and score it accordingly—but it does paint an incomplete picture. This is why I vastly prefer RateBeer, which includes both raw scores and contextualized, style-adjusted numbers. The same Notch Pils that gets an 81 from BeerAdvocate scores a raw 72 on RateBeer, but a style-adjusted 96. This clearly communicates the following pertinent information: “To the extent that the self-selecting cohort of people who opt into using beer rating sites are willing to consider a pilsner’s merits, they have conceded that this is an excellent one.” This is an enormously useful distinction for a pilsner fan.

Jesse Friedman, brewmaster and cofounder of San Francisco’s esteemed Almanac Beer Company, is more sanguine than Lohring when it comes to ratings sites. When contacted for this story, Friedman expressed no philosophical misgivings about beer-measuring contests, and said, “Taken at scale [rating sites are] a good indicator of how a beer was received.” It’s worth noting here that Almanac specializes in the pricey, challenging sour ales that are catnip to a large subset of beer raters. Simply put, it makes sense for him to monitor them for the exact same reason it doesn’t for Lohring. But that doesn’t make one brewer’s position more valid than the other’s. Friedman’s right when he says the sites can provide useful feedback, and Lohring’s not wrong when he says the opposite.

One of Jeremy Danner’s duties as Boulevard’s Ambassador Brewer is to monitor social media and ratings sites. He views them in a generally positive light, appreciating that they represent the passionate consumer base that allows craft beer to thrive. But the market’s zeal can drive weird behavior, too: He admits to having to correct inaccurate listings created by people whose mania to be first! compels them to rate beers that haven’t been brewed yet.

Which reminds me of another potential pitfall. At least twice in the past month, I have definitely been served the wrong beer. At the Field in Cambridge, I was brought a pale ale (Dale’s, I’d guess) when I’d ordered a Sixpoint Sweet Action, and then a stand in Coney Island sold me an amber lager masquerading as a pilsner. Since I write about beer for something approximating a living, I didn’t complain to Untappd that my Sweet Action was too bitter and my Mermaid Pilsner too caramel-forward, but I wouldn’t blame anyone who had. Thinking about this sort of honest mistake, combined with storage and freshness concerns that can affect a beer’s perceived quality, I worry about how much bad data must be floating around out there.

An email exchange with Eno Sarris of BeerGraphs set my mind back at ease (somewhat). BeerGraphs, an offshoot of Sarris’s successful FanGraphs baseball-data site, uses Untappd reviews to tell beer’s story with numbers. That very sentence is likely to turn off some people, and I admit that I don’t delve too deeply into the site’s more complicated statistical analysis, but I like BeerGraphs and appreciate the way it presents beer raters’ preferences and predilections.

The signature BeerGraphs stat is BAR, which is defined as “a style-indexed rating adjusted for popularity.” What the hell does that mean? Well, among other, mathier things, it means Untappd users’ five favorite beers are Heady Topper, Zombie Dust, Pliny the Elder, Pliny the Younger, and Bell’s Hopslam. This is likely the sort of thing Sarris was referring to when he told me, of the rating sites, “I think they’re generally good at rating beer!” Hard point to argue when confronted with that data.

Small, cult brewers’ scores are aided by loyal fan bases that are willing to wait in line half of Saturday morning for the privilege of buying their beer, and unwilling to drink it more than three weeks after its been brewed.

But beer rating apps and sites have their flaws. In addition to the ones detailed above, older beers tend to be treated more gently than newer ones, because a beer that debuts in July of 2016 is facing much stiffer competition than one that came around when Beer Advocate started accepting community-generated reviews and ratings 15 years ago. You’re never going to catch me badmouthing Dogfish Head 90 Minute, but I will speculate that it benefitted from a significant head start on the way to its current 94-point rating, as it racked up thousands of near-perfect scores before the current crop of IPA savants were glimmers in a hop-farmer’s eyes.

But then, Dogfish 90 might be adversely affected by its broad distribution. Small, cult brewers’ scores are aided by loyal fan bases that are willing to wait in line half of Saturday morning for the privilege of buying their beer, and unwilling to drink it more than three weeks after its been brewed. The guarantee that your beer will almost always be drank fresh and by people heavily predisposed to like it can’t hurt when the Untappd report cards come out.

But despite these many and significant qualms, I still come down on the side of thinking beer rating sites ultimately do more good than harm because they entertain tons of people without taking too much off anyone else’s table, and they’re useful databases of what’s out there these days. That said, if I ever score my dream cat-walking job, the first thing I’m doing is wiping them all the hell off my phone.