Getting Blind With Crafty: How We Judge

March 30, 2015, by Crafty Pint
Getting Blind With Crafty: How We Judge

Roughly once every couple of months, we host a blind tasting of beers – usually matching the season or a current hot topic. Below is an explanation of why we started doing them, how we carry out the judging and the thinking behind our methods.

Initially, the first tasting was a bit of fun, inspired by Chuck Hahn sending us a case of new recipe Hahn Premium with which he was particularly pleased. When his beer came out comfortably top of the pile, it was a great finale to a fun evening and led us to develop the concept into something closer to an official judging competition, albeit tailored to be more relevant to consumers. We did this, in the main, by looking to source as many of the beers as possible from retail outlets. In judging competitions, brewers get to choose what beer to submit; here, we are assessing beers off the shelf that any of our readers could pick up.

Over time, we've tweaked the process to the point that we think it's pretty set in stone. And this is that process:

  • Once we've selected a style, we try to buy as many Australian versions of that beer from a selection of bottleshops and bars. In cases where particular beers aren't available in Victoria (where Crafty Towers is based) and we really want to include them, we'll ask breweries to send us stock.
  • These are chilled to the appropriate temperature for the style.
  • A panel of beer experts from different areas of the industry – for example, brewers with competition judging experience, brewery reps, beer writers, retail staff, those with Cicerone certification, homebrewers – is invited to take part. We look to keep some consistency in panelists but there is variation from one to the next.
  • Steward present the beers in flights of three, having decided the order in which they are to be served. Panelists have no knowledge of the lineup.
  • Beers are assessed and marked in four areas: appearance (scored out of 3); aroma (out of 6); taste (out of 6); and overall (a mark out of 10).
  • The overall score is probably the other area in which the process differs most from official judging competitions. Because we're not awarding medals but trying to offer something of meaning to readers based on what they can buy in stores, while appearance, aroma and taste are judged tight to style, we don't want to overly punish a good beer. Ultimately, your choice when buying a beer or going back to one again is whether you like it, not whether it's a wonderful example of a style. So we encourage judges to use this – if they like – to mark up a great beer that's not quite to style or mark down a beer that meets style guidelines but isn't one they think is a great beer.
  • In terms of other advice to judges, again because we're not awarding medals we don't expect everyone to judge with the same generosity or harshness in their scoring as each other. All we ask is that they personally are equally harsh or generous across all the beers.
  • Each flight is judged in silence (at least there is no discussion of the beers) until all judges have completed their scoring for that flight. In this way, no one person can try to influence the others.
  • We ensure we know which bottle or can has been served to each judge in case there is discrepancy in scoring over a particular beer. We have found two beers from the same brewery from the same store to be presenting very differently in the past.
  • Once the judging is finished, we total the scores and begin the best part of the day: the reveal from last to first (usually with a glass of a beer that's as far from the style we've spent the previous couple of hours assessing in our hands!).

How We Source The Beers

In order to differentiate these tastings from official beer competitions, at which brewers can choose which beers to enter and, potentially, submit their freshest or best batches – and to make the lineup more representative of the beers our readers can pick up – we aim to buy from retailers where possible. In some cases, we're unable to find what we're after but, were we to restrict the potential entrants to beers available from bottleshops in and around Melbourne, we'd instantly be excluding heaps of small breweries with limited distribution.

So, while it's not perfect, here's how we try to involve as many brewers in the fairest way possible:

  1. We contact well known and highly rated independent bottleshops with the full list of beers we hope to feature and buy all we can from them.
  2. We contact breweries whose beers aren't available at those stores direct and ask where can find them in retail in and around Melbourne.
  3. Where there's no stock in Victoria, we allow breweries to send beer direct.

It's a situation that can favour breweries sending direct if the request happens to coincide with a fresh packaging run but it's that or excluding smaller interstate operations. 


Once the judging is completed, we usually report on the tasting in two parts: a look at the particular style by our Resident Beer Scholar Chris Brady (pictured, hazily, above left) followed by a write up of the results.

Having initially listed the beers in order right down to last place, from 2015 onwards we took the decision to list in order only the top half or so in order, with the remaining beers that were tasted named but not put in order. Why? We felt that it was unnecessary as it could be that we picked up a bad bottle of an otherwise good beer or one that had been handled badly in trade. We'd rather focus on those that did well while providing feedback to brewers on those that didn't in the hope it proves useful to them for future brews.

When we list the results, we state from where the beer was purchased as well as its brewed on, packaged on, best before or best after date – as long as these are provided by the brewery on the bottle or can.

If requested, we also send feedback to the brewers on every beer tasted. According to some brewers, this is unique as they don't get such feedback on their beers – tasted blind by a panel – once they've been sent into retail. Hopefully, they appreciate the feedback – even when it is bad; we've only had a handful of breweries complain so far!

Safe to say, while these are fun exercises (we'd recommend anyone with an interest in beer to set up tastings like these of their own – it's fascinating to see how beers perform once you take away the branding) we take them seriously and carry them out as genuinely as we can. Taken together, the compiling of panel and beers, sourcing the beers, setting up and running the panels, writing up the results and then providing feedback to the brewers takes up at least two full days. Hopefully, you find the articles worth the effort!

You can read Chris' past Drinking In Style articles here and past blind tasting results here.

Discover more Beer Styles articles

If you enjoy The Crafty Pint, you can become a supporter of our independent journalism.

You can make a donation or sign up for our beer club, The Crafty Cabal, and gain access to exclusive events, giveaways and special deals.

WA Beer & Brewing Conference 2024 2
Gypsy Hub
Bintani- Updated Behind The Brew- E
Cryer E