Polygon's SimCity review is busted policy in action

I'm not a professional games writer. I've wanted to be one since I first picked up a compendium of reviews for games on the Sega Master System and Mega Drive (Genesis), but I've made peace with the fact that I'll always be a commentator more than a genuine contributor. That, and my current occupation - providing policy advice and administration services in a university context - has done a good job of putting food on my table and a roof over my head. You might see me pop up on a few different gaming sites, but rest-assured my money has and will pretty much always be earned elsewhere. I hope that with this introduction, you'll be satisfied that this examination of relatively-new games website, Polygon's review process and policy isn't sour grapes; it's me looking at games writing in the context of my day job. 

Polygon's review policy, titled Polygon's Review Scores and "The Bump" is about as toothless and cosmetic as policy documents come. There's a pretty lengthy preamble to give the reader context as to why an outlet would have an actual policy to begin with, and this primarily relates to the evolving nature of the game as a product. There's no definitive purpose spelled out in the document, and by that I mean I couldn't tell you what purpose a Polygon review is supposed to serve other than to establish and maintain the site's "legacy and credibility." There is an acknowledgement that reviews are valuable to the site's readership, but also a puzzling commitment to "being as fair as possible to the developers and publishers who pour time and money into the games you play." Is a Polygon review supposed to be a purchase recommendation or an evaluation of a game under ideal conditions? From the policy document, it's difficult to say for sure.

There is one statement in the policy for which I intend to take Polygon to task for though, and it reads as follows:

At Polygon, we operate on the supposition that reviews are a critical evaluation of a game on the day of its release, which the score we assign reflects.

If the review is supposed to be a "critical evaluation of a game on the day of its release," then this should imply a few things that would put it at odds with the typical games website milieu of assessing review code in the weeks leading up to release, sometimes at events held at the publisher's offices. Ideally, a review under this supposition would be conducted using a retail copy of the game. I wouldn't expect them to buy it - publisher supplied would be above board - but the publication's recent review of SimCity along its with subsequent revisions shows just how meaningless the policy and the concept of "The Bump" really is.

To be clear, I'm not alleging that there's any impropriety on the part of the reviewer, Russ Pitts or Polygon's Reviews Editor, Arthur Gies: it is stated in the site's forums, publicly on Twitter, and in the review itself that the game was trialled on "development servers". I am, however, implying that Polygon's review policy is for show only, with its writers being able to point to it as proof of the site's supposed integrity rather than it guiding the site's process and practices.

The flippant references to the testing conditions in the cut-away headed On "Always-Online" effectively highlights how the initial review contravenes the review policy:  "on the day of its release," readers won't be able to play the game "on one of their (EA's) reserved servers," or "offline in a private server environment with no other invited players." A quick Google search reveals a multitude of reports on how readers (and anyone else who purchased SimCity for that matter) endured great trouble playing the game that they paid for on the release date and afterwards.

The employment of The Bump seemed to be at odds with the grand scheme of the broken policy as well. To acknowledge the trouble that most consumers have endured trying to connect to EA's servers and actually play SimCity on release (March 5), Gies employed The Bump and reduced the game's review score from a near-perfect 9.5 to a still respectable 8.0. Just two days later, the score dropped significantly to 4.0 to recognise the withdrawal of certain features that apparently affected Pitts' enjoyment of SimCity. Once again, to quote the policy, review updates are supposed to chart the "timeline of that game's evolution." I would argue that it's not to document launch hiccups.

Polygon's review of another always-online PC title that endured issues with access at launch, Diablo III is proof of my assumptions regarding the intent of the policy. In another cut-away, this time titled Server Issues, Arthur Gies explains why the dungeon crawler's score remained at a perfect 10 with the following:

Ordinarily, my position as Reviews Editor at Polygon is that we review a game as it exists on release day, because our responsibility is to our audience. While we do all we can to maintain due diligence with regards to giving a game every opportunity to deliver, we choose your wallet and your time before the benefit of the doubt. 

But Diablo 3 is different. It's different because Blizzard has a track record spanning almost two decades of games that have become institutions, and they've also run the most popular MMO around for almost eight years. Put simply, Blizzard, more than any developer around, has earned that benefit of the doubt. I believe that the server issues will be resolved. With that in mind, it does both our audience and Diablo 3 a disservice to dwell on that aspect in this review.

So in this case, Blizzard, a high-profile developer that has experience delivering games with online infrastructure is spared the wrath of The Bump as "they've earned the benefit of the doubt." I can't attest to developer Maxis' experience with online-only products, but it's publisher, EA has released a vast catalogue of online-dependent or otherwise enabled titles in a variety of genres. For what reason does the Polygon review team believe that EA is unable to deliver a stable experience? How have they not earned the benefit of the doubt? At least in the case SimCity, Maxis delivered the product in full and withdrew features (and will presumably restore them once the servers become stable), as opposed to Diablo III where the real money auction house and Player versus Player features weren't ready for the release date and took varying amounts of time to be delivered.

This lack of consistency again raises the question of what purpose the review is supposed to serve - not just at Polygon, but at games writing outlets in general? Is the review supposed to be a purchase recommendation for readers? If so, Polygon did a disservice to their audience by trialing the game in conditions different to those that would be available to consumers and rendering a near-perfect score. Besides, it's not like The Bump will carry over to Metacritic - whose ratings are the only thing arguably close to an objective measure in terms of games reviews. Surely, with the collective experience of Diablo III's muddled launch, the Polygon team must've thought it prudent to test the retail servers before publishing a review for another always-online game. Then again, if the review is supposed to assess the game in ideal conditions, then The Bump should never have been applied as Pitts got to see the game at its best.

I approached both Pitts and Gies with my concerns via Twitter, and received a reply from the reviewer (Pitts) who stated (across multiple tweets) that:

We did our best to ensure we were reviewing the game in conditions as approximating the user experience as closely as possible. That's not always going to be 100%. And in this case, launch conditions were far  more terrible than anyone assumed. Hence, the repeated updates. TL;DR: This is an extreme case of almost everything.

I responded with the allegation that, as per the above, that Polygon's policy is more for show and highlighted both the breach of policy and the inconsistency with regards to the outlet's Diablo III review and was told that:

It's not that simple, but I appreciate your feedback. Definitely something to consider.

Look, I'm not naive. I know that most gaming sites live and die by their reviews, and that waiting to trial the game on retail servers would have cost Polygon no small amount of hits. Further to that, it's possible that non-disclosure agreements were signed and delivery dates may have been promised. I don't know all the factors that were at play, but I know that Pitts was being genuine with his response. When we're talking about "legacy and credibility" and integrity, however, sacrifices need to be made. I mean, why have a policy if you have no intention of honouring it? I'd point to Gamespot, and their Senior Editor, Kevin Van Ord's review of SimCity as an example of best practice: it acknowledges the broken state of the game as well as highlighting promising aspects which just weren't functional at launch. I think if you're going to apply a bump to the review score, you should start with the product that was delivered to consumers and work up (or down) from there.

I work with well-written policies and I work with dated, ineffective ones as well. A good policy accounts for most situations and variables while allowing sufficient flexibility through interpretation for the benefit of stakeholders. Polygon's review policy is not a well-written policy, and while not old, has already been rendered ineffective by the business of video games writing. If you want that audience -- or better yet, to expand it -- you've got to go to press early, first if you can. Promising to evaluate something close to what your readership will have in its collective hands is at odds with this business model, so Polygon has to make a choice: do you stand with your audience, or do you postulate to them? Whatever the answers to this question and the others published above, I think the site's management need to revise the policy or consider withdrawing it altogether.

Update (14/03/13):

Polygon's Diablo III review was published prior to the website's official launch (it was first published on The Verge) and before the review policy was actually written. Still, I think given the flippant way in which Diablo III's server issues were addressed, it's unfair and inconsistent for SimCity's score to have been reduced so drastically given that server capacity and features disabled will more than likely return in due time.

The fact remains that Polygon have previously reviewed an "always-online" title  and the server issues didn't even factor into the original score awarded because of the aforementioned "benefit of the doubt". Further to that, the review policy would've been written after the Diablo III review and the ruling regarding launch woes and "benefit of the doubt" should've factored into the outlet's approach to policy development.