Sections

Metacritic weighting system report “wholly, wildly inaccurate”, site says

Thursday, 28th March 2013 00:02 GMT By Brenna Hillier

A presentation at GDC 2013 gave a list of weightings purportedly used by Metacritic to determine how much a publication’s review scores affect a title’s average rating, but the site has vehemently denied the report.

The list does not come from Metacritic itself, but was produced by a team of researchers led by Adams Greenwood-Ericksen of Full Sail University. During a a talk titled ‘A Scientific Assessment of the Validity and Value of Metacritic’, Greenwood-Ericksen said Metacritic has confirmed it applies different weightings to various critics.

Looking at existing averages, the team or researchers created a hypothetical model of how Metacritic applies this weighting, and than checked it against the next six months of results, finding it “almost entirely accurate”, according to Gamasutra.

The list of weightings has been published in full and you can check it out below, but Metacritic has responded by saying the information is “wildly, wholly inaccurate”.

“They are simply their best guesses based on research. And here’s the most important thing: their guesses are wildly, wholly inaccurate,” a representative for the site wrote on its Facebook page.

“We use far fewer tiers than listed in the article. The disparity between tiers listed in the article is far more extreme than what we actually use on Metacritic; our publication weights are much closer together and have much less of an impact on the score calculation. Our placement of publications in each tier differs from what is displayed in the article. The article overvalues some publications and undervalues others. In addition, our weights are periodically adjusted as needed if, over time, a publication demonstrates an increase or decrease in overall quality.”

    HIGHEST

  • Dark Zero
  • Digital Chumps
  • Digital Entertainment News
  • Extreme Gamer
  • Firing Squad
  • Game Almighty
  • Game Informer
  • GamePro
  • Gamers Europe
  • GameTrailers
  • GotNext
  • IGN
  • IGN AU
  • IGN UK
  • Just Adventure
  • Machinima
  • Planet Xbox 360
  • PlayStation Official Magazine UK
  • PlayStation Official Magazine US
  • Telegraph
  • The New York Times
  • TheSixthAxis
  • TotalPlayStation
  • VGPub
  • Videogameszone.de
  • Wired
  • Xboxic
  • Yahoo Games
  • ZTGames Domain
    HIGH

  • Absolute Games
  • ActionTrip
  • Adventure Gamers
  • Computer & Video Games
  • Console Gameworld
  • Da GameBoyz
  • Darkstation
  • Edge Magazine
  • EGM
  • EuroGamer Italy
  • EuroGamer Spain
  • G4 TV
  • Game Chronicles
  • GameDaily
  • Gameplayer
  • Gamer 2.0
  • Gamervision
  • Games Master UK
  • Gamespot
  • GameSpy
  • Gaming Age
  • Gaming Nexus
  • Maxi Consoles (Portugal)
  • Pelit
  • Play.tm
  • PlayStation Universe
  • PlayStation Official AU
  • PSM3 Magazine UK
  • PS Extreme
  • RPG Fan
  • Strategy Informer
  • Team Xbox
  • The Onion (AV Club)
  • Totally 360
  • WonderwallWeb
  • XGN
    MEDIUM

  • 1Up
  • CPU Gamer
  • Cubed3
  • Cynamite
  • D+Pad Magazine
  • DailyGame
  • Destructoid
  • Eurogamer
  • Everyeye.it
  • Game Revolution
  • Game Shark
  • Gameblog.fr
  • GameKult
  • Gamereactor Denmark
  • Gamers’ Temple
  • GameShark
  • Gameblog.fr
  • GamesNation
  • GameStar
  • GameTap
  • Gaming Target
  • Gamereactor Sweden
  • The Guardian
  • Hardcore Gamer Magazine
  • HellBored
  • NiceGamers
  • Joystiq
  • Just RPG
  • Level 7.nu
  • Modojo
  • MondoXbox
  • Multiplayer.it
  • N-Europe
  • Netjak
  • NGamer Magazine
  • Nintendo Life
  • Nintendo Power
  • Nintendojo
  • Nintendo World Report
  • NZGamer
  • Official Nintendo Magazine UK
  • Official Xbox 360 Magazine UK
  • Official Xbox Magazine
  • Official Xbox Magazine UK
  • PALGN
  • PC Format
  • PC Gamer (Germany)
  • PC Gamer UK
  • PC Gamer
  • PC Powerplay
  • PGNx Media
  • Play Magazine
  • PlayStation LifeStyle
  • Pocketgamer UK
  • PT Games
  • Real Gamer
  • SpazioGames
  • Talk Xbox
  • The Escapist
  • Thunderbolt
  • Total VideoGames
  • Worth Playing
  • X360 Magazine UK
  • Xbox World 360 Magazine UK
  • Xbox World Australia
  • Xbox360 Achievements
  • Xbox Addict
    LOW

  • 360 Gamer Magazine UK
  • 3DJuegos
  • Ace Gamez
  • Atomic Gamer
  • BigPond GameArena
  • Console Monster
  • Deeko
  • Eurogamer Portugal
  • Game Focus
  • Gameplanet
  • Gamer Limit
  • Gamer.nl
  • Games Radar (in-house)
  • Games TM
  • Gamestyle
  • GameZone
  • Gaming Excellence
  • Gaming Trend
  • Impulse gamer
  • Kombo
  • MEGamers
  • Metro Game Central
  • MS Xbox World
  • NTSC-uk
  • PS Focus
  • PSW Magazine UK
  • Video Game Talk
  • VideoGamer
    LOWER

  • Armchair Empire
  • Cheat Code Central
  • Game Over Online
  • Game Positive
  • Gamer’s Hell
  • Gamereactor Sweden
  • Gamers.at
  • Giant Bomb
  • PS3bloggen.se
  • RPGamer
  • Vandal Online
    LOWEST

  • 9Lives
  • Boomtown
  • Computer Games Online RO
  • GamerNode
  • GamingXP
  • IC-Games
  • Insidegamer.nl
  • Jolt Online Gaming
  • Kikizo
  • LEVEL
  • Meritstation
  • My Gamer
  • Official PlayStation 2 Magazine UK
  • Play UK
  • WHAM! Gaming

Latest

19 Comments

  1. DSB

    The fact that they actually feel like they have to apply weightings pretty much undermines the whole idea of that aggregate.

    Hopefully someone will make one similar to Rotten Tomatoes one day.

    #1 1 year ago
  2. NightCrawler1970

    WHAT VG24/7 not in the list????

    It’s time that Brenna Hillier start promoting this site in bikini maybe VG become on top of everyone!!!!!

    #2 1 year ago
  3. Brenna Hillier

    VG247 is not on the list because we don’t do review scores.

    Also: fuck off. That’s not funny.

    #3 1 year ago
  4. Dragon246

    @Brenna,
    Ignore or ban him. Seriously, was that meant to be a joke? Horrible one at that.

    #4 1 year ago
  5. NightCrawler1970

    @Brenna, sorry :(

    #5 1 year ago
  6. Brenna Hillier

    Alright. Let’s never speak of this again.

    #6 1 year ago
  7. Telepathic.Geometry

    It makes sense to weight sites with more credibility, and take weight away from sites known to have taken money or be biased in some way, but Jesus, that’d be a hell of an undertaking. I think you might be better served in giving it up to a large number of reviews and hoping that the size of the pool sorts it out naturally.

    In that way, I think a metacritic of reader reviews would have more value.

    #7 1 year ago
  8. kart_racer3

    Is it wrong that I immediately checked the importance of the publications that I typically read? Granted, the list is supposedly inaccurate.

    #8 1 year ago
  9. DSB

    @7 It’s really pretty simple. Do you have a valid opinion or not? If you do, then you deserve to be counted. If not, then why would you be.

    Once again, Rotten Tomatoes is well ahead of Metacritic. Not that it makes any sense to try and create an average based on scales that don’t have parity in the first place. People should be smarter than that.

    I think it’s another case where videogames, or the press itself, is resigned to be a second rate product.

    #9 1 year ago
  10. Telepathic.Geometry

    Well, I’m just coming at this from a statistical point of view. Considering the fact that there cannot be an absolute, correct answer, weighted averages is the best way in theory. I don’t think that you can discount it out of hand.

    People do it all the time of course. If a small number of experts tell me something, it would take a very large number of people who don’t know shit to persuade me otherwise. The number of people would depend on the disparity between their ignorance and the expertise of the opposing side.

    So I think that you could argue that publications manned by veterans of the gaming world would have a weightier opinion than some new site who might be inexperienced.

    Naturally, I don’t do it this way. I tend to weight the reviewers/publications that see things my way over those who, say, love games I hate and hate games I love. So I tend to ignore Edge reviews, and put a lot of weight in EG reviews, with a few exceptions (the dude who gave Dead Space 1 a 7 for example).

    But yeah, it’s all just a theoretical discussion for me. I don’t even know what the metacritic website looks like, nor have I any real interest in finding out…

    #10 1 year ago
  11. salarta

    Personally, I think what this site really needs to get it on top is Dave Cook in a bikini.

    (Potentially bad) joke aside, I’m not entirely sure how I feel about this matter. Regardless of whether the claims about Metacritic are accurate or unfounded, it does point out that relying so much on only one bar of measurement when it comes to critical response is a terrible idea. If you rely on only one source, there are many ways things could go wrong with that source, and you don’t have anything to compare for validity.

    It doesn’t affect me entirely too much because I really only check Metacritic to see critical response of games with premises that I loathe; I usually don’t bother if it’s a game I’m looking forward to playing, since I don’t need reviews to tell me what I already know.

    #11 1 year ago
  12. The_Red

    So, fanboy / always-high-score sites like IGN are among HIGHEST while more decent and believable sites like Giant Bomb are in LOW tier?

    #12 1 year ago
  13. Telepathic.Geometry

    Pat and Dave in bikinis! Sam Cook, make that shit happen!

    #13 1 year ago
  14. OlderGamer

    I really doubt the impact of review scores and reviews in general.

    Brand names and marketing pushes(which admitidly sometimes will included reviews/scores) have a far greater impact. Walking into gamestop and seing a cut out near the door or a new poster on the window will get people noticing and talking about the game.

    The media is important, but it a tool for the pubs. And it has to be, reviews/scores and advertising are a symbiotic dependency. One of the reasons I am glad pat doesn’t do out and out reviews. Sometimes it feels like things get “pitched” once in awhile. But when you wade into the cess pool that has become reviews you are asking for trouble. And ultimatly lose creed.

    #14 1 year ago
  15. roadkill

    @2 Shame on you because you wanted to see a girl in bikini. How could you!?

    @4 Are you gay?

    #15 1 year ago
  16. salarta

    @15: … I hope you’re just joking/trolling and I’m fail at picking up on it. The problem wasn’t wanting to see a girl in a bikini, it was the joke being about one of the site’s female writers (the only one as far as I’ve seen) using her body and sex appeal to market the site. Regardless of any intentions, it sets a tone of her value coming more from her sex appeal than her credentials, and when that picks up, it’s hard to get attention back to the credentials.

    I’m going to refrain from saying anything more about the topic after this comment in hopes that the whole thing dies out, but I felt like I needed to speak up. Sorry to Brenna if this comment and my prior one cause or caused any problems for her or upset her.

    #16 1 year ago
  17. Telepathic.Geometry

    Jesus, what age are we all in here?

    #17 1 year ago
  18. DSB

    @10 I don’t agree. A weighted average isn’t scientific any more than a review is scientific. You’re just applying your subjectivity to someone elses subjectivity, and demeaning it in the process.

    None of that makes any sense. You’re either at the mercy of the guy picking the score, or the guy assigning a value to that score.

    @12 Why the surprise, isn’t it obvious who Metacritic really serves? It’s not critics, and it’s not its users.

    @14 It doesn’t have to be OG, that’s just the way things have panned out for videogames. There’s still every reason why it could and should improve.

    I don’t know how VG247 shows up in a standard browser, but on mobile I don’t see any videogame advertisements in the banners etc. It’s all music and other products.

    I think that’s definitely the way to go.

    #18 1 year ago
  19. roadkill

    @16 He was making a joke you moron. If I’d have been in Brenna’s place I would have took it as a compliment. Because it was a joke! Wow! Just wow! You people need to lighten up.

    #19 1 year ago

Comments are now closed on this article.