Wine Spectator reveals Top 100 Wines of 2009, but... are all wine rating systems flawed?
This article was originally on a blog post platform and may be missing photos, graphics or links. See About archive blog posts.
Now that Wine Spectator has finished dragging out the reveal of its Top 100 Wines of 2009 -- a 2005 Columbia Crest Cabernet Sauvignon was ranked No. 1 -- over a yawn-inducing three days, we have to ask: Are wine ratings an accurate and useful guide for consumers? Or are they merely a series of wildly subjective impressions based more on context and expectation than the actual qualities of the wines? That’s the question Leonard Mlodinow explores in a recent Wall Street Journal story, ‘A Hint of Hype, A Taste of Illusion.’
Given the high price of wine and the enormous number of choices, a system in which industry experts comb through the forest of wines, judge them, and offer consumers the meaningful shortcut of medals and ratings makes sense. But what if the successive judgments of the same wine, by the same wine expert, vary so widely that the ratings and medals on which wines base their reputations are merely a powerful illusion? That is the conclusion reached in two recent papers in the Journal of Wine Economics.
He’s referring to findings published by Robert Hodgson, a retired statistics professor and the proprietor of Fieldbrook Winery. A few years ago, Hodgson joined the California State Fair wine competition advisory board, which allowed him to run a controlled scientific study of its tastings.
The results, published in the Journal of Wine Economics, showed that the judges’ ratings varied by ±4 points on a standard 100-point rating scale. And ‘only about one in 10 [judges] regularly rated the same wine within a range of ±2 points.’
In September in the private wine newsletter the California Grapevine, Hodgson discussed his analysis of the complete records of several wine competitions. ‘The distribution of medals,’ he wrote, ‘mirrors what might be expected should a gold medal be awarded by chance alone.’’ Ouch. Many winemakers feel vindicated by Hodgson’s findings, while wine experts generally dismiss them.
If you want to try a similar, albeit much less scientific, experiment on your own, serve a group of friends the same wine in different bottles: first as a humble table wine and then as an expensive, award-winning wine. You’ll likely get a very different set of responses. When we know that something is highly valued by others, we tend to value it more highly. When we know that something is expensive, we’re more inclined to enjoy it. Almost nobody is immune from that. Apply that knowledge to the powerful tastemakers of the multi-billion-dollar wine industry, and where does that leave wine buyers?
At recently opened wine shop Domaine LA on Melrose near Highland Ave., owner Jill Bernheimer, who has run an online wine store for years, holds an annual contest for consumers to guess Wine Spectator’s No. 1 zinfandel. The winner receives a six-pack of wines that are rated 85 points or less, ‘just to prove a point,’ she says. Namely, that there are plenty of underrated wines that are worth drinking. For Bernheimer, it’s about figuring out whose palate you trust.
‘You have to start somewhere and the easiest place is reading wine publications and scores,’ she says. ‘But as people gain more experience and confidence in themselves, the best thing consumers can do is find a sommelier, a retailer or a critic who they feel they share a palate with. That can often be a better guide than scores on their own.’
--Elina Shatkin
--