Spector: journalists and players may be "just mediocre" at assessing success
In a lengthy musing on how we should measure success, Deus Ex and Epic Mickey creator Warren Spector has cast aspersions on the ability of critics and gamers to judge whether games are any good or not.
When we put our faith in Metacritic as an impartial, scientific measure of quality, we should probably ask ourselves whether the crowd – the crowd of journalists as well as players – is really wise or just mediocre, incapable of recognizing and rewarding the new and different."
"Gamers have every right to evaluate games themselves, and to share their opinions with other gamers. It would be insane to argue against that, or even to think about it too much," Spctor wriote in his now-regular column on GamesIndustry.
"Reviews and reviewers? That's a very different story. We - gamers, developers, publishers and reviewers themselves - need to think more about the purpose of reviews. Only by doing this will we get better reviews and understand how best to interpret them, as business people, creators and consumers."
Spector said that reviewers assess games to say whether they're good or bad, but fail to address any of the criteria which he considers important to measuring a game's success - something he believes reviewers in other media - like film - do.
"Great critics don't focus on questions of good or bad. Their assessments of a film's success or failure are supported by an underlying philosophy they apply to all films. It doesn't matter if you agree or disagree with their assessment of an individual film - heck, they often disagreed with one another. What matters is that, reading them, you can weigh your own likes and dislikes to determine if you will like a film, based on their review," he said.
"An enthusiastic 'This game sucks!' or 'The AI is bad' or 'This game gets a 4 out of 5' tells me next to nothing I need to know," he added.
Moving on to Metacritic, Spector argued that by aggregating critical voices and reducing them to a number, the resulting average often fails to account for the kinds of games which reach niche audience, or innovate substantially.
"By this logic, Metacritic, at best, rewards games that are conventional and well understood by players and critics alike. New and challenging things are, by their very nature, disruptive and easily misunderstood," he said.
"When we put our faith in Metacritic as an impartial, scientific measure of quality, we should probably ask ourselves whether the crowd - the crowd of journalists as well as players - is really wise or just mediocre, incapable of recognizing and rewarding the new and different," he continued.
Ouch. Spector also took issue with Metacritic's policies, saying it cherry picks specific reviews, the lack of transparency of its weighting system, Metacritic's conversion of all scores to a 100 point scale. He also said that Disney's Epic Mickey, one of his lowest-rated games, sold substantially better than his highest-rated efforts, Thief and Deus Ex.
"Reviewers and some gamers may have preferred Deus Ex and Thief (and I have the fan mail to prove it!), but I received substantially more fan mail - and more heartfelt messages of thanks, and more fan art, and more everything - on the Disney Epic Mickey games than on both Deus Ex games I worked on combined," he said.
"Metacritic be damned - I'll take an emotional connection with players and the praise of Disney fans any day of the week"
Spector ended with an exhortation fo developers to measure their success by other means.