I’m In Your MMO, Reviewing Your Beta

Last time there was any talk about how to (or how not to) review MMOs was around the whole Eurogamer vs Aventurine debacle (often called “Zitrongate” after the reviewer). This was obviously a deep and traumatic event for some people, especially Darkfall fans that still can’t stop taking cheap shots at Eurogamer (despite them doing a re-review of the game). To me, though, the question has crept back into the front of my mind, as I’m currently reviewing two MMOs at the same time.

Ensign Squid

Ensign 418 of 666, codename "Squid", at your service.

As I mentioned in my last entry, this is an incredibly stupid thing to do. Last time I had two MMOs on my hands, I decided to skip one of them because the other had more priority (Aion over the first Runes of Magic expansion), this time both felt important enough to warrant my attention. It’s Global Agenda and Star Trek Online, two completely different games that both feel interesting enough to cover. I kind of wish I had stopped myself, but it’s too late now. They shall be reviewed.

So how do I go about reviewing a MMO? Thing is, I’ve had access to both of these games’ betas, Star Trek Online all the way back since closed beta and Global Agenda since some point during open. If I just had gone down the same route as a lot of other games journalists, I would have played both extensively over a long period of time, then tried them out for a bit after launch and then getting my reviews done a few days after they were released. Job done, I can sleep at night, my editors are happy, our readers are happy, everyone is happy (except perhaps the developer/publisher, if I had decided to give the game low scores).

My problem is that I refuse to review a MMO based on beta. No matter what people say, a beta is not a finished product. It might be as close as you can get, and most of the time the end of open beta will look exactly like the launch candidate, but by calling it “beta” and not “head start” or “early access”, the developers themselves are saying that the game is not done. So I won’t review it based on my experiences during beta, just like I would not review a game based on a late preview build (which are almost always like the finished product, except for perhaps a couple of bugs). I am not going to play the beta for anything except for an early look at what the game might become, just so I can get a review in a few days after launch.

Also, more often than not, the developer adds some form of patch close to launch that changes some fundamental things to the game. It might not warrant the title of “miracle patch”, but it’s often enough to make the game at least a bit better (or, in the case of Champions Online’s launch day, worse according to a lot of people). It’s kinda sad that this doesn’t happen earlier during the beta process, but the reason is probably that the developer still thinks beta is beta, while the marketing department thinks it’s free marketing. I’ve seen way too many reviews, some even printed in a magazine and passed off as a “real” review, that have even taken beta rumors as facts.

Global Agenda

This is my Global Agenda recon character. She dies a lot.

Do I blame the journalists themselves for this? No, of course not. They have deadlines, we all do, and have to work towards them. Also, there’s often pressure from the readers, who are dying to try out the game but want to know what their magazine/website of choice thinks. The only way to review a MMO and get a review out quickly is to play beta. I am just lucky enough to have editors that allow me to take the time I believe is needed, up to a certain degree of course. And despite this, I never feel fully satisfied, always having to add a disclaimer that there is no way I have seen or experienced everything and that some players will always have had more time than me and is bound to disagree. That’s why we have comment fields, as long as the discussion can be kept civil.

What I would like to see is a civil discussion about how to review MMOs. Last time it happened, the discussion was filled with so much anger at Eurogamer, or disdain towards how Aventurine handled the situation, that it was almost impossible to catch the good stuff amongst the constant din. So I’m going to try here – do you want your MMO-reviews early, or are you comfortable with waiting for them? I do realize that I’m probably talking to the wrong crowd, since most of my readers here are probably just as fanatic about MMOs as I am, but I still want to hear your thoughts about one of the hardest genres a games journalist can tackle.

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

31 comments

  1. Stargrace says:

    I love the fact that you don’t review based on beta. It’s a smart way to do things, and I envy the fact that you do reviews at all – that’s one thing I don’t tackle in my job. I’ll do interviews, play games, write guides, but when it comes to reviews I just don’t have the thick skin you need for it.

  2. I think all reviews should wait. I really wonder how accurate a game review is when it comes out too quickly, especially when it’s about MMOs, and it often makes me doubt the authenticity of it.

    I can understand early reviews of single player games somewhat more because they go “gold” at some point and preview copies can be sent out to reviewers a short while before they hit the shops but for MMOs the development continues right up to the last whistle. And you’re right, miracle patches *do* happen – Age of Conan had one.

    Frankly, I think this whole culture of open betas with MMOs really should stop. I don’t think it’s fair on the developers to judge a game before it’s even had it’s proper release and how can we even judge a game that’s still in development?

    I admire you taking your time, Petter and I will trust your reviews a lot more than some others I’ve read when they come out.

    • Petter says:

      The problem, though, is the readers – a lot of people want quick reviews, even of MMOs. I guess that’s why a forum like this isn’t the best place to ask, since we are all MMO gamers and we know how the genre works. Most people don’t really care about the same things we do.

      Thanks for the nice words, even though I am not sure – I always feel they lack something. It’s also hard, since I don’t always have the time to go the extra mile, even if my reviews can be quite late.

  3. Ysharros says:

    I’m happy to wait for reviews too. But then, I’m not one to think that just because news is “hot off the press” it has to be the *best* news, especially when it comes to something that really isn’t time-sensitive, like a game review.

    In fact, it’s the need to scoop/be at the news forefront versus the need to provide a decent review that causes — to my mind — most of the problems I have with game reviews that aren’t much more than a quick look at character creation and a description of the first playable zone.

    I’d much rather wait till someone had a chance to get a real look and a good feel for a game — its flavour as well as its mechanics.

    But then, I’m also not a content locust. Content locusts MUST play a game as soon as it comes out so that they can a) devour all the content and b) devour it before anyone else. But that’s a whole nother kettle of fish. Or locusts.

    • Petter says:

      “A kettle of locusts”? Nice one, sounds like a good new term for “WoW tourist” (if one supports that theory, that is). ;)

      Right, I hate that kind of review as well – but a lot of journos go past that by just playing beta, giving the starter areas another shot after launch to see what has been improved. Is that a bad way of doing it?

  4. Spinks says:

    I do enjoy reading beta reviews, but I think as a reader you have to understand what sort of things can change between beta and live (ie. levelling curves can be tweaked easily, but there won’t be a whole new graphics engine or total class redesign).

    So I’d like beta reviews to focus on the actual gameplay that is unlikely to change. And then note how stable the beta was, were there any other issues, and so on.

    I’m not really fond of reviews that consist entirely of the chargen and introductory tutorial. Sure, it’s fun to know what you’re going to see in game — I like to plan characters ahead and I know I’m not alone — but if it doesn’t answer my basic gameplay questions then it isn’t what i want to know.

    But in the end, the initial review only needs to convince me that it’s worth my time to try the game for a month or not. It doesn’t need to make me buy a lifetime sub, I’ll decide that when I’ve tried it myself.

    • Petter says:

      The problem, I suppose, is that everyone really understand that difference when they read an early review.

      But I agree on what you expect from a MMO review. It’s kinda hard to go back to that mind-set though, considering a lot of us try out every MMO that is released. The only one I can remember that I didn’t play was Darkfall. A lot of us also play most betas that come our way, so we more or less know what to expect when the first reviews start coming out.

  5. pasmith says:

    You can do a nice Preview of a game based on beta but like everyone else here I think doing a Review based on beta is a bad choice. The problem is, most gamers are too ignorant to understand the difference between a Preview and a Review.

    And it’s Americans rewarding of “fastest news” vs “quality news” that killed off all the good print mags over here, and it now destroying any kind of thoughtful online gaming journalism as well. The only decent gaming mag you can buy now comes from Europe.

    Ack, don’t mind me, I’m a real curmudgeon when it comes to this topic.

    • Petter says:

      Mind mentioning what mag that is?

      I absolutely agree with people not understanding, or seeing, the difference. I see way too many out there throwing around the word “review”, as if every article written about a game is a review. Just see what happened when Aionsource caught wind of Sera’s articles about Aion. That is hardly limited to people not writing either…it happens on a lot of gaming blogs too.

  6. Moxy says:

    I want both early beta-based reviews, and deep, well-thought-out reviews. I just want the reviewer to be up front about how far into the game they went, how much time they spent, and whether beta builds were used in the review process. I’m an intelligent reader that can take those factors into account while reading your review. With MMOs, we all *know* the game will evolve and adapt to player feedback. There’s no use trying to hide that fact and pretend that there is one definitive point in the state of the game (e.g. launch) or in your gameplay (e.g. 20 hours) where your opinion is optimal. The Zitrongate incident might have been less extreme if he had put caveats on his review about having spent X hours in the game (whatever X actually was!) from the beginning.

  7. Dusty says:

    First, I laud you for reviewing only the launched game as well. I am of the exact same mind. In fact it has often been a policy of mine to not even *play* a game in beta, especially if it’s one I’m very interested in, because it can so shade my perceptions. But I’ve found that $60.00 is too high of a price tag anymore to “try” a game for 30 days to see if I’ll keep it, so I’ve revised my own “no beta” policy.

    As for actually reviewing an MMO, honestly I would focus on certain aspects of the game. There has been much what-to-do about “how long is long enough” to review an entire MMO, and it’s a fair question, because well gosh darn it they’re big games. But for instance, it’s perfectly legitimate to evaluate the “new player experience” after 1 hour of play, because that’s what it is. And by the same token, it’s fair to evaluate the levelling curve of the first ten levels after you’ve reached level 10. To write an honest “personal experience” type of review for an MMO, it’s only fair to talk about those aspects of the game you’ve personally experienced. But a whole lot of opinion can be formed after even just 10 hours of play, and it’s absolutely within a reveiwer’s rights to express that opinion, and to a certain extent speculate on the game as a whole based on that opinion.

    Something else I think is legitimate in reviewing MMO’s is to look at the metadata. That is, to not just write about personal experience, but to listen to, and observe what the playerbase and delopment teams are saying about the game. Because MMO’s are such big games, your own personal viewpoint is quite likely to be highly skewed in one aspect or another. For MMO’s more than any other game, I think it’s important to try to gather a composite view of the game as a whole. Forums are one source of that information, though certainly those have to be filtered, as forum opinions tend to lean towards the negative. But what are the developers focusing on? What areas are people mostly complaining about? Is the userbase growing after a month even, or are people leaving in droves? As a reviewer, the fundamental question you’re trying to answer is “is this a good game”. And for that, I really do think it’s important that you look at more than just your own personal experience in the game.

    Oh and good luck.. :)

    Dusty

    • Petter says:

      I absolutely get what you are saying about meta-data and I agree, but that is one thing that really got Sera into trouble – “reviewing” (which she didn’t do, since Massively don’t do reviews) Aion not only on her own experience, but also drawing from what she had read elsewhere. While I think that’s a perfectly good way of doing things, at least to add more depth to the review, do you think that most people would appreciate that?

      Then again, would there be any other way to report that there is no end-game in place, for example? It’s tricky. I am just trying to sort all of this in my head, since most people that will read the final reviews are not people that think about these things in the same ways we do.

      And thanks. ;)

      • Dusty says:

        It’s a good question – and it may be that many people might not appreciated it per-se, but it might still be a better review. That’s not to say that you should rely on forum-trolls for your review — not at all. I still think your own personal experience should be the primary source of inspiration for any review. But if I had by pure chance rolled up a Bright Wizard on day 1 of Warhammer Online, and played 25 levels and written about what a rich and satisfying experience the PvP was, I’m not sure my review would be better for not acknowledging that there were legions of players complaining about how unbalanced the Bright Wizard was.

  8. Dickie says:

    I’m interested in your thoughts on Global Agenda. Sadly, I have no interest in Star Trek Online, but that’s just me… I’ll give that title some time to cook before I try it out.

    I’m really enjoying GA right now, but the difficulty of the higher level PvE missions is really killing me!

  9. [...] thanks to pingback technology, this post from “Don’t Fear the Mutant” came to my attention (btw: there is no such thing as a [...]

  10. Billy Hicks says:

    Reviews are easy. No reviewer can ever fully grasp an MMO, all they can do is try to give an overview of how the game currently works IN THEIR OPINION.

    All reviewers should make a little background available for the reader. How long did you play for? How far did you progress? What is your gaming background? If your a HelloKitty fanboi, you may see things very differently from a UO/EVE player.

    Let the reader get a good idea of those things and they can then add weight to what you say. If you claim to be a HelloKitty Fan then other HelloKitty fans or people who liked that game will give your review more weight than they would someone who only ever played boardgames before.

    Its all about the context. Just don’t pull an Ed Zitron and not play the game, claim you did and base your review on stuff you read on a forum.

  11. Lyrics says:

    That was a real to read your notes, I’m gonna check the other soon, as well. I’m going to visit the blog once more next week so I could check the rest of entries. Wish that’s okay…

  12. Wow, marvelous blog layout! How lengthy have you ever been blogging for?

    you made running a blog look easy. The entire look of your website is
    wonderful, let alone the content!

    My web blog – tank domination hack

  13. Prettƴ! This was an extremely wonderful article.

    Thank you forr providing tɦis іոfo.

  14. Its like you read my mind! You appear to know so much approximately this, like you wrote the e book in it or something.
    I believe that you simply can do with some p.c.
    to power the message home a bit, however instead of that,
    that is magnificent blog. A great read. I will definitely be back.

  15. Apart from enjoyment and comforts you will also be capable of save huge money from.
    As if your equipment weren’t already expensive enough, Blu-Ray movies still cost considerably more than regular DVD’s.
    Effectively in case you have a top pace web connection and a laptop or computer you will be capable to watch all of the newest movies online.

  16. If that’s not your style, Runic also promises LAN capabilities for the game.
    Crysis 3: Again, you can play with every setting switched
    to high quality. However, the Radeon 5770 has new features which
    the Radeon 4870 does not.

    Feel free to visit my weblog :: Crysis 3 free keygen

  17. Hmm it looks like your blog ate my first comment (it was extremely long) so I guess I’ll just sum it up what I
    had written and say, I’m thoroughly enjoying your blog. I too
    am an aspiring blog writer but I’m still new to the
    whole thing. Do you have any helpful hints for newbie blog
    writers? I’d certainly appreciate it.

  18. I’ve learn several just right stuff here. Definitely value bookmarking for revisiting.
    I wonder how a lot attempt you put to make this kind of great informative site.

  19. Hey fantastic website! Does running a blog similar to
    this require a massive amount work? I have no expertise in coding but I was hoping to start my own blog
    soon. Anyway, should you have any ideas or techniques for new blog owners
    please share. I know this is off subject however I simply needed to ask.
    Appreciate it!

  20. Unquestionably believe that which you stated. Your favorite reason seemed to be on the net the simplest thing to
    be aware of. I say to you, I certainly get annoyed while people think about
    worries that they plainly do not know about. You managed to hit the nail upon the top and defined out the whole thing without
    having side-effects , people could take a signal.
    Will probably be back to get more. Thanks

    Feel free to surf to my website office 2013 keygen (http://msoffice2013keys.blogspot.com/2014/02/microsoft-office-key-generator.html)

  21. home says:

    I am truly grateful to the holder of this web site who has shared this fantastic post at here.

  22. that became our national symbol.ONSet the cost of a $188watch up to $92 and shoppers will flip absent. While you are it valid that the simple gladness on the bestThe interactive showcase is filled

  23. Jimmie says:

    whoah this blog is magnificent i love studying your articles.
    Keep up the great work! You already know, many individuals
    are hunting round for this info, you can help them greatly.

Copyright © Don’t Fear the Mutant
Virtual worlds, massive multiplayer games and assorted ramblings

Built on Notes Blog Core
Powered by WordPress