Reviews in the Media - Flawed and Inaccurate?

[Shicky256]

I remember watching a video about a month ago with a good point: Since review scores are arbitrary and vary by the individual, why does MetaCritic exist, and why do developers get fired/paid less for poor MetaCritic scores? This and the fact that MetaCritic doesn't allow for updates to the reviews mean that I don't read most reviews anymore. I just watch CGR! Mark just looks to see if a game is fun or not, and doesn't give it a score based on this.

 

[gamefan5]

You always seem to write the most interesting articles.

I know right?
Ryukouki is always writing some interesting articles and sparking some debates. kudos to u and keep it up. XD

 

[Ryukouki]

Thanks guys Keep the comments flowing!

At this point I stopped reading. Why wouldn't it be ok to give a 7.5 score to a Zelda game? You state that's "by the standard that score is appalling". If that's true, we might as well have automated scripts in place that rate every Zelda game between 9 and 10.

You can't expect every single reviewer to score a game in the exact same way, even if their reviews aren't subjective. This doesn't change the fact that I myself am not a fan of scores, since it's far too difficult to accurately convert all of the game's aspects into a number. Reviews shouldn't have scores at all, but then again, the majority of gamers only cares about the scores, not the justifications behind them. You often hear people complaining about a score, but you rarely hear someone argument against a certain point brought up in a review. Then again, since almost everything is subjective, what's the point? One person might consider a given gameplay aspect to be the best thing ever, while another might be sick and tired of it, since they've played far too many games utilizing similar or identical gameplay features.

Reviews are losing their relevance mainly due to the fact that gaming community is changing. We want scores, and we want them now. That's why things like Metacritic exist. I'm not surprised video reviews took off like they did, though they're still combined with a score.

As for developers and publishers trying to influence review scores, it's nothing new, and the Kane & Lynch example isn't the only one. Publishers will often submit suggestions and guidelines on how their game should be scored. This has been brought up on several occasions, but happens far more often. It's a "I'll scratch your back, you'll scratch mine" type of deal. After all, both the developers and video game websites are trying to make money.

In the end, it all comes down to one's own judgement. I'm not going to miss out on a game simply because someone didn't like it. Games aren't that expensive these days, and if a game appears interesting enough, I want to check it out on my own. I've bought my share of games with bad reviews, and I don't regret it. Why? Because they weren't bad to me.

I should totally address the Skyward Sword point, as I feel I was a little off in the article. The reason why I was a bit irritated by the score was because the review itself was fundamentally flawed. There were plenty of factual mistakes in the review, and it essentially boiled down to the guy playing the game wrong. Very wrong. This could very much fall into the category of inexperience. The review is valid, don't get me wrong, I'm not saying it isn't by any means, but a lot of the criticisms were incorrect, and at this point have since been corrected. No score adjustments, however, which I find just a little bit odd.

I dont think I agree with that, at least the sites I check have a wide range of scores (GS for example). I kind of like the review score, it tells me at a glance if I should even bother with the game, so when the game scores 7+ I read the review and do some research to see if the game deserves my time and money.
But for me, it has become harder and harder to have a go-to source for reading review. It used to be that I trusted GS reviews completely and very rarely I was disappointed but ever since the main guy (Jeff Gretsmann, Vinny Caravella, Brad Shoemaker and Alex Navaro who I think is one the best the reviewers around) left GS, it just turned to crap. The reviews became a place for people to just force their political and social views on people , reviews became widely contradictory and other stuff.

I never believed the conspiracy that sometimes the review are bought, but unfortunately I have seen more than enough examples that are really main me believe that. The prime example that comes to my mind is this:
http://www.egmnow.com/articles/reviews/egm-review-aliens-colonial-marines/#/
Who in their right mind can rate that game a 9?!
And lately one incident let me to lose all my respect for GS after 11 years of being a loyal fan: The 6 score for Batman Arkham Origins and and later, the 8 for COD:Ghosts. For Batman review all they did was moan that the game is too much like the other entries in the series and how the lack of the innovation has led to the game becoming stale...fast forward to the COD review and no where in the review they complained about the repetitiveness of the game. or when it came to the graphics, in which the game is seriously lacking, in a desperate attempt to find something, anything positive to say about the game they came up with this:
""
Or IGN for example, they are basically Nintendo fanboys and rate Nintendo games, ALWAYS, higher than other sites. They went as far as rating the Legend of Zelda: Windwaker HD higher than the original game on GC! They gave Skyward Sword a perfect 10!

Overall it has become harder and harder for us gamers to find a outlet we can trust to make our game buying decisions which is very sad...no wonder many pirate their games first to try them out and then decide on buying them.

The review score tidbit was more of a broad offhand generalization, which may have ended up confusing you. I think it was definitely a little broad of me to say that. But yeah, I totally hear you. I find that going to a bunch of review outlets lets me sum up the experience well enough as a sort of "Averaging" system.

 

[Ryukouki]

Crap, I ran out of space to address in the previous post, so I'll be doing a rare double post.

The scoring is a huge problem and week after week we hear of big name developers bashing metacritic and even having their pay bonus based on how they average on metacritic after release. In some cases I've read of how games are developed just to check off review points.
Imo Kotaku have it right with the YES/NO review system, but even that is flawed when it comes to MP games at launch, which are often not working as they should day 1.

Then we have embargos which 99% of the time give the review to one huge site like IGN which will come a day early and never bash a game as they have the exclusive.
I also hate it when someone reviews a game and marks it down because they don't like the genre, like gamespots recent Batman Backgate review, the reviewer expected a port of the console game, that shouldn't even factor or be mentioned. Same with gametrailers review of Killzone Mercenary, they gave it a 7.7 but based that score on in being a FPS and comparable to console FPS. Imo it should be rated as a FPS on a handheld, because thats what it is!

I could talk about this all day but the only way things will change is if you don't go to the sites that are guilty of biased numeric review scores, which for a lot of people is the first port of call before buying a game.

And people wonder why I hate giving numeric scores. One of the things I would like to change in the review center sometimes...

Something else worth mentioning is the merit of reviewers. I mean, just compare reviewers for film and games. Film had people like Siskel and Ebert (whatever you may think of his opinion on other subjects). These were guys deeply versed in the art of filmmaking, with a passion for the art. They were able to make insightful analyses of films while keeping things accessible for a general audience; everyone, from the average joe to fellow film buffs, could appreciate what they had to say.

Gaming, though? Not even close. There's dirty money and tricks getting passed around, sure, but I think it's more than that. I feel like most game reviewers aren't really versed in the language of games the same way folks like Siskel and Ebert were versed in the language of cinema. These reviewers only scratch the surface and stop there. I also think we're being stunted by a weird, industry-wide inferiority complex. It feels like a lot of game reviewers are insecure about their profession, and so they try to justify gaming as a medium at every turn. Bioshock: Infinite? "It's the Citizen Kane of gaming, guys, you can take us seriously now!" Gone Home? "10/10, see, we're topical, really!" etc. etc.

You know game reviewing is downright farcical when Yahtzee is probably one of the best out there, and his "reviews" are made for comedy.

(Now, don't get me wrong - I know film reviewing has plenty of hacks all its own. Believe me, it does. Still, they seem to be quite the majority when it comes to game reviews.)


Also: Video Game "Journalism"

I sometimes wonder how Kotaku even gets some of its articles out. Page view journalism at its best. One of the reasons why I left my old gig and loved it here. I absolutely hated writing for the page counts. I get the freedom to write whenever I feel comfortable and I never have to worry about page counts, which is something that GBAtemp draws in droves considering its large user base.

 

[sporkonomix]

I've blogged about this before. I eschewed review scores entirely and focused on the real reasons people read reviews: decision making. It's turned my reviews into more of a conversation than a bleak description with meaningless numbers attached to it. I don't get many visitors, but I'd like to think that my change in approach has helped a few people make better decisions.

 

[The Real Jdbye]

I don't think so. Reviews are more of a guideline than anything else, and it has always been. It shouldn't affect whether you purchase a game or not. The main factor there should be how much the gameplay, genre, storyline, and such appeal to you personally. The best way to find out is to try a demo if there is one. Otherwise watch gameplay videos.
If the game has a terrible rating it's probably best to stay away, but as long as the review is at least decent you might enjoy it a lot more than the review would make it seem.

 

[Ryukouki]

I've blogged about this before. I eschewed review scores entirely and focused on the real reasons people read reviews: decision making. It's turned my reviews into more of a conversation than a bleak description with meaningless numbers attached to it. I don't get many visitors, but I'd like to think that my change in approach has helped a few people make better decisions.

I've actually done that system before as well in the form of a "Should you buy it" section. I find the number is an awful indicator and it makes me feel "restricted" like I'm being forced to tie my voice around the score I give.

 

[sporkonomix]

I've actually done that system before as well in the form of a "Should you buy it" section. I find the number is an awful indicator and it makes me feel "restricted" like I'm being forced to tie my voice around the score I give.

Yeah, I felt the same way; like I had to think of a score before I used my review to "defend" it instead of discussing the merits and faults of the game in question. Doing it in reverse didn't help, either. Overall I think scoring systems serve as an enabler of pissing contests and statistics instead of honest opinions on media.

 

[sporkonomix]

I don't think so. Reviews are more of a guideline than anything else, and it has always been. It shouldn't affect whether you purchase a game or not. The main factor there should be how much the gameplay, genre, storyline, and such appeal to you personally. The best way to find out is to try a demo if there is one. Otherwise watch gameplay videos.
If the game has a terrible rating it's probably best to stay away, but as long as the review is at least decent you might enjoy it a lot more than the review would make it seem.

Interesting that you say "It shouldn't affect whether you purchase a game", but contradict that with "if it has a terrible rating you should probably stay away from it". In that case, aren't the poor scores influencing one's hypothetical purchasing behavior?

It mostly depends on the review style, imo. I try to frame my final opinion with a specific taste, like "You'll probably like this game if you like X or Y game", or "If you're into X kind of game or enjoy Y concept, this game will be a good match for you", etc. It's pretty much impossible to form an absolute, concrete, objective opinion on something that is inseparably subjective and dependent upon taste.

You're right that nothing can match a demo and actually spending time with something, but that's not always practical or even possible. I also agree that reviews aren't something to be taken as gospel; it's just some person's opinion.

 

[EgoTrip]

This is nothing new. It's been standard practice since the beginning of gaming really. Game companies offer reviewers an incentive to give them good reviews, get their games headlined. All too often the main game each month in old magazines was not the best that month had to offer yet it had the most coverage and highest score. Some games were never even reviewed properly, and were just rewordings of the game instructions.

Read up on stories of 8 and 16 bit developers, they all have stories to tell about how corrupt the people they worked for were, and obviously, still are. Some things will never change.

 

[stanleyopar2000]

*goes back to imfamous GameSpot Kane and Lynch conspiracy*

fucking flawed as fuck

 

[grossaffe]

That's another problem right there. If a score of 6.3, which by definition is above average on a 10 point scale, is considered too low to pick up a game, we might just as well not have scores at all, since people clearly have no idea how to operate within such scoring systems. If someone asks: "What do you think, is this game good, poor, playable?", and I answer with "it's not great, but definitely above the average", it effectively means it's a 6 ~ 7 point game. That's why expressing an opinion is much more effective than simply coming up with a number, especially today, when people aren't able to score games below 5 out of 10.

I agree with you there. Reviewers should make use of the full scale of review scores, but unfortunately with today's standards of review, a score that goes outside of that 7.5-10 range means "do not play". If this were some independent reviewer, I'd give benefit of the doubt of how his grading scale worked (or if the reviewer bothered leaving us with closing thoughts and what kind of gamer might enjoy the game), but this is not one of those guys, and that's not how games get graded on IGN.

 

[aiat_gamer]

Some reviewer site did that to Bioshock Infinite.

Can you mention some examples? As far as I know that game got 8+ across!

 

[Heran Bago]

 

[DinohScene]

Can you mention some examples? As far as I know that game got 8+ across!

Yar I been trying to find that article again but can't seem to find it.
Some reviewer gave it a lower grade cause it didn't had multiplayer.

 

[aiat_gamer]

Yar I been trying to find that article again but can't seem to find it.
Some reviewer gave it a lower grade cause it didn't had multiplayer.

I am pretty sure that it must have been a pretty obscure review site. As far as I know everyone fell in love with that game, which is very strange to me considering how problematic that game was.

 

[aiat_gamer]

Thanks guys Keep the comments flowing!



I should totally address the Skyward Sword point, as I feel I was a little off in the article. The reason why I was a bit irritated by the score was because the review itself was fundamentally flawed. There were plenty of factual mistakes in the review, and it essentially boiled down to the guy playing the game wrong. Very wrong. This could very much fall into the category of inexperience. The review is valid, don't get me wrong, I'm not saying it isn't by any means, but a lot of the criticisms were incorrect, and at this point have since been corrected. No score adjustments, however, which I find just a little bit odd.

So you are saying the score was too low? How would have rate it?

 

[DinohScene]

I am pretty sure that it must have been a pretty obscure review site. As far as I know everyone fell in love with that game, which is very strange to me considering how problematic that game was.

Can't find it anymore unfortunately.

Game wasn't problematic at all.
Loved it from the moment it got leaked.
Well that and with previous BS installments.
<3 the Bioshock series <3

 

[Ryukouki]

So you are saying the score was too low? How would have rate it?

That's a good question. His review was factually proven to be flawed at some point if I recall correctly. When I played it I thought the controls were a huge step up from the waggle era of Twilight Princess. If you play the game the way it is meant to be played (again, this ideal is dependent on a number of factors) it plays fantastic. If I had to give an on the spot score I would award it an 8.5, subjective to a number of factors that I account in an as-general-as-possible approach. This score is in comparison to other games in the Zelda franchise. It also is subjective to improvements over mechanisms that needed improvement from titles past. Some major strengths could include the stronger story, the unique puzzles, the strategy that is involved in fighting monsters. Some of the point deductions could be subject to the linearity that some gamers may have faced. Other issues could include the forced inclusion of Wii Motion Plus. I can go into the nitty gritty all day but I might as well write a full review on it.

There really is no such thing as a decent "score" to apply, as it's so subjective it takes far too long to delve into the deep and nitty gritty. As a reviewer, it's difficult to apply an objective approach. There really is not a one size fits all model here. If I had to apply a "Thumbs up, thumbs down" approach, it'd get a thumbs up, but that's too broad. People may have hated the game, and I acknowledge those arguments as well for the whole Motion Plus thing.

 

[The Real Jdbye]

Interesting that you say "It shouldn't affect whether you purchase a game", but contradict that with "if it has a terrible rating you should probably stay away from it". In that case, aren't the poor scores influencing one's hypothetical purchasing behavior?

It mostly depends on the review style, imo. I try to frame my final opinion with a specific taste, like "You'll probably like this game if you like X or Y game", or "If you're into X kind of game or enjoy Y concept, this game will be a good match for you", etc. It's pretty much impossible to form an absolute, concrete, objective opinion on something that is inseparably subjective and dependent upon taste.

You're right that nothing can match a demo and actually spending time with something, but that's not always practical or even possible. I also agree that reviews aren't something to be taken as gospel; it's just some person's opinion.

That is the only case where it should affect your decision. I just included it as a footnote.