Video Game Review Grading Scale

[Wolf Kanno]

Am I the only person who looks at the numbered grade scale on review sites like Metacritic like you do for school grading curves. You know you see 50/100 and think "I'll give that a pass" or do you view them differently?

[Jiro]

I wrote a paper on this.

(SPOILER)
High scores: How game scores affect and damage the industry
Barret Wallace
ABSTRACT: Video game journalism is developing alongside the industry it covers and critiques. Unlike film or literature reviews, an enormous amount of weight is given to the single score in a game review, as opposed to the actual content and experiences of the reviewer. The disproportionate value given to scores has led to a toxic environment for reviewers who buck the trend and has seen financial pressures placed upon developers by publishers. Problems stemming from the reliance on a single number, which is commonly and mistakenly thought to be an objective and accurate representation of a game’s true worth, are increased by a lack of uniformity or common standards, as well as an overly passionate and outspoken fan base. A solution to the problem, based on thoughts from gaming journalists, and the consideration of developer and publisher interests, is suggested.

Introduction
As the video game market is flooded with more titles than ever before, thanks in part to a healthy independent (indie) development community, the job of the reviewer has become increasingly important. Gamers, adult gamers especially, do not have the time to sift through the countless new releases in order to find a game that appeals to their tastes or is worth their investment of time and money. The task falls to the games reviewer to experience, summarise, and then grade these games, giving their personal appraisal in order to guide gamers towards their next adventure.

Game reviewers, like their counterparts in film, literature, music, and so on, are viewed as authorities in their field, and are expected to provide honest comments and critiques with as much objectivity as possible. Subjectivity is unavoidable, and personal biases are easily explained within the review itself. However, the numerical score seems to be given the most weight and is seen by parts of the gaming community and gaming industry as the most important factor in a review: high scores are everything.

This disproportionate value given to review scores has created an unhealthy environment in the games industry. The personal experiences of reviewers are overlooked and stripped away in favour of the ‘objective’ scoring system—despite the score itself being a subjective and non-standardised summation of the reviewer’s thoughts about the game. Reviewers are criticised by gamers for giving a score considered ‘too low’ or ‘unfair’, and development teams have bonuses withheld for not meeting a predetermined aggregated score, regardless of other factors. Abolishing scores from reviews would remove the positive aspects of the system and make a general consensus on a game’s ‘success’ harder to calculate, but an alternative to this lopsided and anger-inducing system needs to be introduced so reviews can continue to inform.

Critics criticised over ‘low’ scores
Disagreeing with critics is not a new phenomenon. Personal preferences factor into decisions about every aspect of life—art and artistic pursuits are no different. But the ferocity and frequency of attacks on reviewers who dare to differ from the mainstream is cause for concern. Different viewpoints are integral to creating a positive and healthy discourse about a subject, video games included (Humprecht & Büchel, 2013). If everybody shared the same opinion, the entire industry—from developers to critics—would stagnate.

Therefore, attacking reviewers, in addition to being completely unnecessary, damages and weakens the discourse surrounding the industry, which is counterintuitive to the intent: defending the ‘good’ in video games. Gamespot reviewer Carolyn Petit received varying degrees of hate, including a petition to have her fired, for her ‘prejudiced’ and apparently ‘extremely feminist’ score of 9 out of 10 for satirical crime simulator Grand Theft Auto V, with her comments about the misogynistic portrayal of women—in spite of her positive comments about the complex and flawed male characters—singled out as evidence of a feminist agenda, or lack of knowledge about the game and franchise (Parfitt, 2013).

Petit is not the only victim of the angry gamer mob: feminist blogger Anita Sarkeesian received threats of violence and rape—in addition to the creation of a game where players could punch her in the face—over her YouTube series exploring sexism and tropes in video games (Rosenberg, 2012; Pinchefsky, 2012), and former Gamespot reviewer Jeff Gerstmann received death threats after giving The Legend of Zelda: Twilight Princess an apparently low score of 8.8 out of 10 (System Wars Magazine, 2013). Countless other examples of outrage over review scores exist, but every case boils down to the same problem: there is a fundamental disagreement with the rating a game is given.

With such backlash against reviewers and their opinions, often before a game has even been released to the public, it seems that a game’s score is at least partially predetermined. Critic Jim Sterling’s 8 out of 10 score for Gears of War 3 was met with confusion and outrage by gamers and developers alike. Sterling responded, saying that it is ‘utterly, completely, intellectually insulting’ that the video game industry ‘has decided 9/10 is the manifest destiny of certain games’ (2011). If the expectation is that reviewers should simply follow the crowd, then their critical role has become defunct.

The cost of a ‘bad’ score
Gamers are not the only stakeholder when it comes to review scores. Publishers want to see their game do well critically, as critical success often leads to commercial success (Chen & Xie, 2008). Developers also want something they created to be well received, not just to validate their hard work, but also because thousands of dollars could ride on the outcome.

Obsidian Entertainment designer Chris Avellone revealed via Twitter that his team missed out on bonuses totalling around $1 million because Fallout: New Vegas failed to achieve the required score of 85 on review aggregator Metacritic (Schreier, 2012). Both the Xbox and PC versions of Fallout: New Vegas hold an average score of 84, a score maintained since its 2010 release (CBS Interactive, 2013). A single missed point, based on the averaged opinions of 81 reviewers, cost each member of the Obsidian team $14,000 in bonuses.

Metacritic scores can determine whether a development studio even has their game picked up by a publisher. Developer Kim Swift, notable for working on the wildly successful game Portal, has had pitches renegotiated or turned down based entirely on Metacritic scores, costing both her and her company money and contracts (Schreier, 2012). Metacritic weights each review in order to calculate the average, but the exact formula remains a secret. While Metacritic founder Marc Doyle claims the website is there purely to educate gamers, it is clear that publishers are using it as leverage against development studios (Schreier, 2012). IGN games journalist Keza MacDonald (2012) suspects the Metacritic influence has gone too far, and could even be ruining the gaming industry.

Financial interests can also affect what score a game receives, with dire consequences for any reviewer who puts advertising revenue in jeopardy. Jeff Gerstmann was fired on the spot after his critical review of Kane & Lynch: Dead Men caused publisher Eidos Interactive to reconsider its advertising on the Gamespot website (Cavalli, 2012).

Games are both expected and required to meet high scores, with pressure placed on both the reviewers and the developers to make it a reality. The industry can only suffer if creative teams are forced to stifle new and untested ideas in favour of proven successes. Likewise, journalistic integrity is destroyed if reviewers are forced to give a dishonest but favourable review in order to keep their job. The current system is failing every stakeholder, and a solution must be found.

The four point scale
Much of the confusion and anger comes from the lack of a universally recognised metric (Ivory, 2006). Publications and reviewers use their own scales, including 1-5, 1-10, and 1-100. While these could be equated mathematically—a 4 out of 5 is the same as 8 out of 10 and 80 out of 100—the fact is that these scores are entirely subjective; games within different genres have different expectations, and a reviewer’s personal preferences and experiences undoubtedly factor into the outcome, so even two scores of 8 out of 10 are not necessarily the same (Jenkins, 2010).

This problem is exacerbated by the fact that a lot of reviewers and publications tend to use only the top half of their scale: a scale of 1-10 functions as 5-10, or even 6-10, which results in inflated scores and confusion. Instead of reflecting the scale it is using, reviews tend to favour a system similar to that of the United States school system: anything below 60% is a fail, meaning an ‘average’ game would score around 75% (Kate, 2011; Kain, 2013). With half of the scale devoted to failure, each point comes to represent a greater range of quality. Cases where a reviewer or publication uses the full range on the scale—such as X-Play, Good Game, and Hyper—are the exception, even if the scale is defined within the publication itself.

Games journalist Daniel Wilks (in Carr, 2013) says he ‘hates’ review scores, because there is no standardised way to calculate them. Each game has different criteria: music might not be an important factor in an indie puzzle game, but could have a huge impact on a blockbuster action game (Jenkins, 2010). Reviews can also be affected by the skill, experience, and time investment of the reviewer (Ivory, 2006). A single number cannot hope to convey the complexity or subjectivity of the process or criteria used to calculate the score.

Doing away with review scores is not the solution. Sterling (2012) cites several drawbacks to removing them: readers who want a simple score to gauge the general quality of a game might boycott his reviews, diminishing the influence Sterling has as a reviewer. He says the fault lies with readers who take review scores ‘too seriously’. Review scores are intended to give readers the ‘essence’ of whether or not a game can be considered ‘good’ (Pattison, in Carr 2013). Even so, what constitutes ‘good’ is still subjective.

Review scores themselves are harmless, and an easy tool to communicate a general impression, not an inflexible, scientifically calculated, objective rating—which seems to be the point of most confusion. If review scores are an important and inevitable part of games journalism, what can be done to ensure scores are being interpreted correctly?

Saving the games industry from itself
A simple solution does not exist. Review scores are entrenched in contemporary games journalism, and are seen as the authoritative and objective mark of a game’s worth and success. Gamers have become invested in the success and failure of their favourite game series, regardless of the actual quality of each instalment, to the point where reviewers are vilified for going against expectations.

A universal metric would remove discrepancy, and give standardised criteria for which to judge games. However, a single metric is impossible to apply to such a diverse field; even a scoring system unique to each genre would encounter problems with games that combine multiple genres, create a new genre, or are simply unclassifiable. Critics would also be further burdened with trying to adapt their personal experiences and biases into a rigid system, attempting—however impossibly—to strip away one of the most important aspects and benefits of a diverse reviewing environment: subjectivity.

The responsibility lies with the individual. Publishers must understand that Metacritic scores are not the ultimate marker of a game’s success. Gamers must understand that review scores—both individual scores and the Metacritic average—are not objective and are not a personal slight against the game or its fans. Reviewers should be able to give their honest opinions, and the diversity of reviewers should represent the diversity of the community they serve. Readers should find a reviewer that entertains, informs, and aligns closely with the reader’s tastes and preferences, and realise that it is impossible to make the perfect game.

The adventure continues
Compared with literature and film, gaming is still an industry in its youth. As such, it is still growing and developing as technology advances and gaming itself becomes a mainstream pursuit. The heavy reliance on review scores—by both gamers and publishers—to determine the ‘objective’ worth of a game is damaging to the entire industry. Only by taking pause and remembering that everybody is entitled to an opinion, and that respectful disagreements are healthy for discussion, can gaming present itself as a mature industry worthy of respect.

References
Carr, J. (Producer). (2013, October 22). Good Game [Television broadcast]. Sydney, NSW: Australian Broadcasting Company.
Cavalli, E. (2012, March 16). Jeff Gerstmann explains his departure from Gamespot. The Escapist. Retrieved from Jeff Gerstmann Explains His Departure From Gamespot | The Escapist.
CBS Interactive. (2013). Fallout: New Vegas for Xbox 360 reviews. Metacritic. Retrieved from Fallout: New Vegas for Xbox 360 Reviews - Metacritic.
Chen, Y. & Xie, J. (2008). Online consumer review: Word-of-mouth as a new element of marketing communication mix. Management Science. 54(3). 477-491.
Humprecht, E. & Büchel, F. (2013). More of the same or marketplace of opinions? A cross-national comparison of diversity in online news reporting. The International Journal of Press/Politics. 18(4). 436-461.
Ivory, J. (2006). Still a man's game: Gender representation in online reviews of video games. Mass Communication and Society. 9(1). 103-114.DOI: 10.1207/s15327825mcs0901_6.
Jenkins, B. (2010). Staying objective: The effect of corporate public relations on video game journalists. (Masters dissertation, Louisiana State University, 2010). Retrieved from http://etd.lsu.edu/docs/available/et...ted/Thesis.pdf.
Kain, E. (2013, June 14). Video game reviews: A discussion of the ten-point scale and inflated scores. Forbes. Retrieved from Video Game Reviews: A Discussion Of The Ten-Point Scale And Inflated Scores - Forbes.
MacDonald, K. (2012, July 16). Is Metacritic ruining the games industry? IGN. Retrieved from Is Metacritic Ruining The Games Industry? - IGN.
Parfitt, B. (2013, September 18). Gamers petition for sacking of GameSpot writer who criticised GTAV for misogyny. MCV. Retrieved from Gamers petition for sacking of GameSpot writer who criticised GTAV for misogyny | Games industry news | MCV.
Pinchefsky, C. (2012, September 7). Feminist blogger is a victim of a vicious videogame retaliation. Forbes. Retrieved from Feminist Blogger Is a Victim of a Vicious Videogame Retaliation - Forbes.
Rosenberg, A. (2012, July 10). Anita Sarkeesian, Stephanie Guthrie, and the strategic failures of trolls. Think Progress. Retrieved from Anita Sarkeesian, Stephanie Guthrie, And The Strategic Failures of Trolls | ThinkProgress.
Schreier, J. (2012, March 15). Why are game developer bonuses based on review scores? Kotaku. Retrieved from Why Are Game Developer Bonuses Based On Review Scores?.
———— (2012, April 11). Metacritic matters: How review scores hurt video games. Kotaku. Retrieved from Metacritic Matters: How Review Scores Hurt Video Games.
Sterling, J. (2011, September 15). Gears of War 3: Perfect review scores by divine right. Game Front. Retrieved from Gears of War 3: Perfect Review Scores By Divine Right | Game Front.
———— (2012, December 3). Jimquisition: Review scores are not evil. The Escapist. Retrieved from Review Scores Are Not Evil | Jimquisition Video Gallery | The Escapist.
System Wars Magazine. (2013, September 16). Gamespot reviewer gives GTA V a generous score, readers want her fired. System Wars Magazine. Retrieved from REPORT: Gamespot reviewer gives GTA V a generous score, readers want her fired | System Wars Magazine.

[Scotty_ffgamer]

This is why I never put a score on my reviews I write. Of course, I understand why bigger review sites and such have scores.

[Jiro]

I always find personal comments to be more enlightening than a simple score.

[Scotty_ffgamer]

I do too, but scores are quick and easy to see. If what you want are people just clicking on your pages, you are more likely to get views or whatever with scores. That's why I understand the need to have scores if you want to generate a lot of traffic. I think it would be better if everyone did away with scores though.

[Jiro]

Oh, yeah. Scores are a necessary evil, I think. It's not standardised which leads to a lot of misunderstanding or score inflation, but scores bring in the traffic and hopefully the comments actually convince people one way or the other.

[Ayen]

I seldom pay them attention ironically enough, but if a game was to have multiple low scores from various sources I'd be less likely to check it out. I treat them like news outlets since I know those within the industry can be bought.

[Miss Mae]

I've read Jiro's essay before, so I know a lot of my thoughts are covered in that, but in short I consider the way games scores are currently being used to be incredibly flawed. For a lot of reviewers, the scale runs at a 7-10 system unless the game is absolutely terrible, at which point it might receive a 5. Because this is such a common way of using a 1-10 scale, it introduces a lot of inconsistencies when other other reviewers use all the points in their 1-10 scale (e.g. if they consider a 5/10 to be a pass rather than a dismal fail). This makes it nearly impossible to use reviewer scores to judge whether a game is 'good' or not, because it's difficult to tell where mark-0 is for the particular reviewer you are reading unless you are familiar with their work.

As said in other comments, I pay a lot more attention to personal comments than the number a game is given. The only time a score will pique my interest is if it is from a reviewer that I know well, if understand how they use their scores and generally relate to the personal preferences of. Or if a game received 10s across the board, because that usually means it's worth looking at.

[The Man]

As with everything else in life, TV Tropes has articles on this:

8.8 - Television Tropes & Idioms
Four Point Scale - Television Tropes & Idioms

And, of course, there's an xkcd for it too:



Scores are ok if the reviewer explains how they grade things on their site, but they probably get too much attention on the whole. I haven't read all of Jiro's article yet but I suspect I'll agree with a lot of it.

[Slothy]

On my phone right before bed so I'll read Jiro's article later. I'm sure it probably says a lot of things I'll agree with though.

But I will say there's a reason why there's a general consensus that game reviews run on a 5-10 scale at best. It's because it's largely true. Unless you're familiar with how a specific reviewer grades a game the score is useless, and most don't know the numbers 0-4 even exist.

[Dr Unne]

What about this theory: There are a lot of games that would get 1-5 scores on the 1-10 scale, but no one bothers to post reviews about them because they're so bad that they aren't worth the time. Maybe the 1's are all Flash games made by 10 year olds.

[Jiro]

A lot of reviewers are getting paid per review or per traffic, so not posting a review about a game you've played would be counter productive. Might as well monetise everything you do.

[NeoCracker]

I liked Bilstered thumbs set up, where they would both rank the game 1-10 and qualify it with a 'For Fans Only' or 'Try it out'.

[Shauna]

I don't pay attention to reviews anyway.

[Madonna]

Game review scores? What? I do not look for potential games to own in review sites/magazines, but rather keep an open ear for potentially cool games. I wait for gameplay trailers and Let's Play videos. I am cool with game previews to hype up a game, and am down with consumers' gameplay videos to help confirm desire. I prefer to go with a "Is it fun/good?" angle than an arbitrary number metric.