XIV:ARR Halts Sales to Help Fix Server Issues

Originally Posted by Rostum

Yeah you can try and white knight this all you want, but they should ​have been a lot more prepared than this. People spending money on a game, and a week later still can't log in to play.

They're an incredibly large global company, dealing with a global market using one of the most prominent franchises in a multi-billion dollar industry. That's why they should have been prepared. Let's not even get into over a decade's worth of real-world data for these situations.

The problem was that everyone and their uncle were "undecided" until Phase Four. As pointed out earlier, the demand didn't jump up three fold, it jumped up five fold. Everyone was saying this relaunch would be fairly doomed because of the tainting that came about from 1.x's launch, and that the game would lose out significantly because of that alone. It doesn't look like any of these predictions (which were happening until Phase Four) came true. They made a game that is better than they thought, if anything.

Also, they overestimated Tomb Raider's sales and many others, so it makes sense that they would opt towards being conservative on this occasion.

I'm not really against Square at the moment, they're trying to fix things and that's fine. I just don't understand how they can say that, in Early Access, they weren't expecting there to be that many people logging on. Surely they knew exactly how many people would be logging on because you only got in by pre-ordering, so the pre-order figures would've shown them the number of people ready to play..

Originally Posted by I Don't Need A Name

I'm not really against Square at the moment, they're trying to fix things and that's fine. I just don't understand how they can say that, in Early Access, they weren't expecting there to be that many people logging on. Surely they knew exactly how many people would be logging on because you only got in by pre-ordering, so the pre-order figures would've shown them the number of people ready to play..

Not exactly. Legacy players were automatically given early access and early access was still procurable via pre-order until about two days before early access launched (I'm also hear that people could actually pre-order and get early access codes while early access was going on) and while Japan got a limited number of pre-orders available to them, NA & EU had unlimited pre-order codes. You predict based on pre-orders up to a certain time (and based on marketing). But when everyone orders just a day or two before early access starts, then you certainly can't magic up the additional servers at the click of your fingers. You have to pay for them, you have to wait for delivery, you have to set them up, you have to stress test them, etc. etc.

Some people might suggest (going by what I'm reading around the rest of the internet, not here, so this is a slightly unprovoked rant within this thread ) that you can always get servers at the drop of a hat, too, but that's not the case - there are lead times for such things, especially when you want the exact specification as your others servers which at a launch time they will want. There is a global queue at the server manufacturers and they will be waiting on parts from other parts manufacturers on occasion. Sometimes you simply can't get the servers on a next day delivery, even when you're Square Enix. I work for Computacenter who sell to some of the largest banks in the world including HSBC, Lloyds TSB, Halifax Bank of Scotland, Royal Bank of Scotland. I think we spend high hundreds of millions of pounds at Lenovo ever year. And sometimes we have to tell the banks they will have to wait for their servers because there are none and there is a queue. We can escalate at Lenovo or HP or wherever and do everything we can to make things happen, but this stuff simply doesn't happen overnight. This is something I've been dealing with since the day I got my job over eleven years ago. I wish we could get things turned around that fast!

Sorry, the IT worker in me just gets a little annoyed when people assume things are that easy. SE is a major company in this world and they do their marketing. Sometimes it's not right, but I'm pretty confident they will be doing a far better job at it than the likes of us. And when it goes wrong because of inaccurate marketing (and a very unpredictably high market at that which couldn't have been predicted based on what was being said around the internet by users before Open Beta) then the IT side of things will not be solved at the drop of a hat.

Originally Posted by Loony BoB

The problem was that everyone and their uncle were "undecided" until Phase Four. As pointed out earlier, the demand didn't jump up three fold, it jumped up five fold. Everyone was saying this relaunch would be fairly doomed because of the tainting that came about from 1.x's launch, and that the game would lose out significantly because of that alone. It doesn't look like any of these predictions (which were happening until Phase Four) came true. They made a game that is better than they thought, if anything.

Who's "everyone"? Also companies like these have teams dedicated to market researching and forecasting sales, they get paid lots to do it too...

Seriously, you were looking around the internet and seeing rave reviews and lots of excitement before Phase Four? Because for the majority of things I was reading, a lot of people were very cautiously about any optimism placed, many places were citing "But 1.x may have tainted the name too much", stuff like that. And as pointed out, pre-orders shot up very, very late.

I'm personally surprised by the level of interest - I spent a lot of time nagging people to try to play the game and now people are arriving by the e-truckload trying to get in. Going by how extremely cautious the optimism was just a month or two ago, I struggle to see how they could have looked at the EU/NA market and have thought "yeah, our MMO is gonna rock the MMO community and we're going to need so many servers!" I mean, even the diehard reporter of FFXIV over at Joystiq was always cautious and wary when discussing the game. People weren't willing to get hyped and with no hype, the game was probably not expected to do so well.

They get paid lots to do it and they still didn't anticipate this. That could be seen as bad marketing, absolutely - and it probably was to some extent - but I trust people who are paid that much to do a better job than I would have done.

A box sale != A CD Key activation

A CD Key activation != +1 concurrency 24/7

1,000,000 boxes sold != 1,000,000 concurrency at any one point in time

Sales forecasts and figures alone aren't reliable indicators of peak concurrency. There are a myriad of other factors.

Really as someone who's worked with MMOs for the past 6 years I could go into a lot more detail about this but I get the feeling it'd just be a fruitless endeavour.

We can't login, it sucks balls and made me very sad last night when I couldn't play. But, they're working on it to the best of their ability. Not really much more that can be done beyond that and no end of bashing Square Enix on an internet forum will get things fixed any faster than they possibly can be fixed.

Originally Posted by Aulayna

Really as someone who's worked with MMOs for the past 6 years I could go into a lot more detail about this but I get the feeling it'd just be a fruitless endeavour.

But it would be insightful! If you'd even want to, anyway.

Originally Posted by Aulayna

Really as someone who's worked with MMOs for the past 6 years I could go into a lot more detail about this but I get the feeling it'd just be a fruitless endeavour.

Not really much more that can be done beyond that and no end of bashing Square Enix on an internet forum will get things fixed any faster than they possibly can be fixed.

Or you could cut the passive aggressiveness, please... Or maybe I can join in! "I'm sorry, we could just discuss and/or use a public forum for things you want to discuss and/or use a public forum for in the future, perhaps."

I honestly doubt Square Enix pays their forecast and research departments a lot of money to get their consumer interest from eye-balling news websites, or count up box sales and do some munted math. It goes far beyond that. You need to keep in mind here that they are a large global company dealing with an extremely large global market with one of the most prominent franchises in the industry. They should have been prepared, they weren't and now unfortunately they could pay for it (and so will their loyal consumer base like (unfortunately) myself). In any other industry this would just be unacceptable, and sometimes I wonder why gamers bend over and take it.

People keep saying that this is like every other MMO launch. It isn't. If it were this wouldn't be as big news as it is right now. I've been following a lot of major MMO releases over the last decade, and in this later half of the decade either none of them had any issues at all, or it only took a few days to fix those issues - not over a week or two!

In the end, they should and could have been prepared to deal with this issue days after not weeks. However, I'm willing to bet this is just more internal issues in the company that have dragged on from previous management (like not communicating with the consumers at all, or just general lack of foresight and stubbornness (remember that Tanaka fellow?).

I also just wanted to say that whilst the "AFK kick" feature won't relieve server stress, it'd still allow for more players to actually play the game than those that would just leave their characters idle for 8+ hours at a time. Sure, it's a short-term feature easy enough to implement, but it might have helped settle down angry consumers and angry reviewers. I'm also completely dumbfounded they don't have a working queue system, which would also help alleviate people's current perceptions (probably more so than the AFK kick feature).

TL;DR - I like discussing and bickering about stuff that is affecting me right now, deal with it. 8)

Originally Posted by Rostum

Originally Posted by Aulayna

Really as someone who's worked with MMOs for the past 6 years I could go into a lot more detail about this but I get the feeling it'd just be a fruitless endeavour.

Not really much more that can be done beyond that and no end of bashing Square Enix on an internet forum will get things fixed any faster than they possibly can be fixed.

Or you could cut the passive aggressiveness, please... Or maybe I can join in! "I'm sorry, we could just discuss and/or use a public forum for things you want to discuss and/or use a public forum for in the future, perhaps."

Please, Rostum, drop the personal attacks and sarcasm. I'm sure Aul wasn't meaning for any offence to be made.

I honestly doubt Square Enix pays their forecast and research departments a lot of money to get their consumer interest from eye-balling news websites, or count up box sales and do some munted math. It goes far beyond that. You need to keep in mind here that they are a large global company dealing with an extremely large global market with one of the most prominent franchises in the industry.

Do tell us at which point they went wrong, then! You clearly have more of a mind for it than the professionals. Seriously, reviews and comments do matter, and sales do matter. If there is an unpredented and unanticpated and unpredictable increase in demand very late in the pre-order process, they can't magic up additional infrastructure at the drop of a hat. They openly stated that they got it wrong, but that was down to unexpected demand. They simply did not expect this many people to be playing the game. I trust that they did their absolute best to ensure they could get maximum profit, it's their business to.

They should have been prepared, they weren't and now unfortunately they could pay for it (and so will their loyal consumer base like (unfortunately) myself). In any other industry this would just be unacceptable, and sometimes I wonder why gamers bend over and take it.

...have you ever played an MMO launch before? O_o Every launch has an issue. In FFXIV ARR, it just happens to be the issue of "too many people like the game". When you have a game that people are willing to spend hours clicking buttons just to play, you know it's a good game. I don't think they will pay for it and I think you're just enraged because the problem is affecting your ability to play. I kind of expect a bit more level headedness out of you given your experience with MMO's. I'm told this is pretty standard across the board for all successful MMO's in, well, MMO history.

People keep saying that this is like every other MMO launch. It isn't. If it were this wouldn't be as big news as it is right now. I've been following a lot of major MMO releases over the last decade, and in this later half of the decade either none of them had any issues at all, or it only took a few days to fix those issues - not over a week or two!

It hasn't been a week since launch. And I'm genuinely interested in which of these successful MMO's had a successful major launch without bugs or whatever. From what I'm reading, the game itself is one of the most bug-free launches ever. Unfortunately the demand is huge, so they are adapting. They have got their servers, they are getting them prepared, it won't be 24 hours before they're up. Which, by the way, is a week since launch. Not "over a week or two".

In the end, they should and could have been prepared to deal with this issue days after not weeks. However, I'm willing to bet this is just more internal issues in the company that have dragged on from previous management (like not communicating with the consumers at all, or just general lack of foresight and stubbornness (remember that Tanaka fellow?).

Sigh. It's easy to come up with scapegoats when you're on the outside. It seems to me that the game is a huge success, people are enjoying it and there are few bugs of note. Because of this, the demand to play the game is huge, because they know the game is good. Because of this, we have the issues we are facing. As for why they couldn't get it all up within days - they are getting it fixed within days. They had to buy the servers, they had to stress test them. Servers are not available at your local Tesco, and Square Enix is not going to bump up the queue for servers ahead of any of the other global corporations that buy them every day. I can't stress this part enough because it's my job to buy these things for companies and I speak with Lenovo to try to jump ahead of the queue on a regular basis. You simply can't. If the world's largest bank can't, why do you think SE can? Servers are not delivered on a sameday service. They have already increased the number of worlds, I'm sure at least twice, and they are having to do it again. This means they are getting servers on top of their backup servers. This is high demand. This could be construed as poor marketing, yes, but it is not a quick fix on the lines of "Oh, we'll just pop down to the local and whip out a server from the bartender's butt".

I also just wanted to say that whilst the "AFK kick" feature won't relieve server stress, it'd still allow for more players to actually play the game than those that would just leave their characters idle for 8+ hours at a time.

Again, this would increase stress and thus they would have to lower the headcount once more, making it relatively redundant. From what I've read from the more technically knowledgeable people out there and also going by my own knowledge of servers, the afk crowd probably have such a small impact that it wouldn't be worth their time to implement an AFK autokick, and if they do it would be better off as an extremely temporary measure until the subscription period begins and the populations suddenly drop. And, again, it may be worth simply forcing people to go to their inn to idle so people don't have to constantly load up the afk'ing characters. In 1.x, you could only get rested bonus by going to your inn, and very few people AFK'ed around aetheryte.

In a more positive note : FINAL FANTASY XIV, The Lodestone | Server Congestion and Trial Period Extensions (09/03/2013)

Am I the only one who found the amusement in reading
Technical Difficulties
>> Endless
on the forum top? xD

In all seriousness, I just can't wait until SE finish their maintenance and sort out the congestion and quieten all the people who are crying about their consumer rights. I know people already who are screaming blue murder and are saying they're inches away from emailing SE and asking for a refund. That's the problem when it comes to the retail business. The consumer always jumps onto the hate-wagon, cries blue murder and tries to sully the name of the company and those who support them and don't just wait patiently for the inevitable recovery and fix. From what I hear from 1.x, the game was truly enjoyable once the initial problems were sorted out and everything was back on track again. People just need a little patience..

I preordered the game like 48 hours before early access started. The following day, 5 of my friends got the game on my recommendation. You can't always prepare for people that come out of the blue. Mind you, I haven't been able to get into the game for the past couple days so the problems are persistent and that's annoying, but I had homework to do so I guess it's not all bad

Just an a

Originally Posted by Rostum

I honestly doubt Square Enix pays their forecast and research departments a lot of money to get their consumer interest from eye-balling news websites, or count up box sales and do some munted math. It goes far beyond that. You need to keep in mind here that they are a large global company dealing with an extremely large global market with one of the most prominent franchises in the industry. They should have been prepared, they weren't and now unfortunately they could pay for it (and so will their loyal consumer base like (unfortunately) myself). In any other industry this would just be unacceptable, and sometimes I wonder why gamers bend over and take it.

Analysts can (and often do) get it wrong. I really wish I could give you numbers here but I have to respect my employment NDA so all I can say is when one of the biggest brands in the industry can exceed even it's own extremely lofty "best case" expectations that they thought there was no chance in hell of meeting - then it can and will happen to others.

As for other industries there really aren't that many online services that will be hammered by a few hundred thousand people the minute the switch is flicked. The very comparable thing I can think of right now is ticket sales for major sporting events and concerts which have the exact same problems when they get hammered at the same time. Even things like the Google App store run incredibly slowly when a hot "killer app" gets released.

Let's not forget that Square Enix isn't exactly the best of gauging these numbers either when something like Tomb Raider which sold metric buckloads was considered to have underperformed.

You can even see in Yoshi P's posts the concurrency figures they were planning for, which were by no means modest in relation to 1.x and they still surpassed them. The gaming press argument may feel superfluous but it is very true that there had been a massive surge of interest in ARR between Phases 3 and 4 of the beta once the NDA got lifted - by which time a lot of the hardware and contingency plans were probably in place.

They've openly admitted that they got it wrong, the producer has even written a heartfelt apology for it, they are giving people an additional 7 days of free game time (which comes at a loss to their bottom line) in response to these issues. They are doing the best they can in response to the situation and to their credit aren't fobbing people off with bulltrout like EA and Maxis did in the wake of the Sim City launch. It's also a lot more than what Gameforge DE gave European TERA players in the wake of ongoing issues (to the point that TERA EU was at one point several patches behind the KR version and 2 or 3 patches behind the NA one)

People keep saying that this is like every other MMO launch. It isn't. If it were this wouldn't be as big news as it is right now. I've been following a lot of major MMO releases over the last decade, and in this later half of the decade either none of them had any issues at all, or it only took a few days to fix those issues - not over a week or two!

I already addressed this in a response to your last post where you said this. ARR has only been out for a week. The GW2 in Europe example I gave it took them over two weeks to sort it. SWTOR at launch it was practically impossible for me and my friends to get online at the same time for around the first two weeks. Neverwinter I lost an entire days worth of progress due to server rollbacks they had to preform in response to an economy breaking exploit they hadn't found out about during testing.

Firefall is STILL plagued by problems and has been in open beta and willingly taking peoples money for far longer than A Realm Reborn has been on shelves.

Even WoW got hit by unexpected demand related issues when Cataclysm was first released and had a couple of very major bugs that had to be rectified post launch.

Then there's also other recent always-online client/server games like Diablo III and Sim City, that whilst not MMORPGs, have had similar incredibly bad launches.

Acting like ARR is a travesty the likes of which haven't been heard of in years is just frankly far from the truth.

In the end, they should and could have been prepared to deal with this issue days after not weeks. However, I'm willing to bet this is just more internal issues in the company that have dragged on from previous management (like not communicating with the consumers at all, or just general lack of foresight and stubbornness (remember that Tanaka fellow?).

Again I addressed your claim of there being "no communication" in my previous post.

Server hardware doesn't suddenly appear out of thin air overnight. Especially when that hardware is being housed in a data center whereby things like contracts sometimes have to be re-negotiated or their technicians may actually also be busy dealing with other problems at the same time. It's not a case of pressing a magic button on a desk and everything is resolved.

Even the entire process of getting the hardware has to go through budgeting and approval first.

I also just wanted to say that whilst the "AFK kick" feature won't relieve server stress, it'd still allow for more players to actually play the game than those that would just leave their characters idle for 8+ hours at a time. Sure, it's a short-term feature easy enough to implement, but it might have helped settle down angry consumers and angry reviewers. I'm also completely dumbfounded they don't have a working queue system, which would also help alleviate people's current perceptions (probably more so than the AFK kick feature).

The thing is, there is a queue. I have been in it had to wait for it from position 561 before. However if the realm is at capacity and it already has a set number of people in the queue it will simply reject any further additional requests to join that queue. However due to the ongoing issues with lobby servers it can sometimes cause the queue to get dropped as well as the lobby and login servers themselves are struggling to keep up with the demand of people spamming them.

A lot of reviewers are actually holding off on publishing reviews until these issues are alleviated. Even with all the issues the game is still getting largely favourable user reviews on sites like MetaCritic which normally get negative review bombed as soon as a situation like this has opened up.

The situation isn't great, no. It's frustrating, it sucks. However SE aren't burying their heads in the sand over it and waiting for it to blow over. They've communicated it, Yoshi-P himself has been very honest and open about it and they've already taken steps to compensate people for it. The issues are annoying and in an ideal world they wouldn't happen, but sometimes people get things wrong and rather than spurt out hyperbole and making sweeping negative assumptions I'll give them the benefit of the doubt. For now.

No-one's trying to argue that the situation isn't unacceptable or ideal - however a lot of people are trying to provide reasoned responses to your claims of "I don't understand how" or "I'm dumbfounded as to why."

Guild Wars 2 was broken for at least a week [1]. For those who managed to log in, people were often kicked into overflow servers and disconnected, forming parties didn't work, etc. The in-game Trading Post was shut down for weeks. It got sorted out. Same for Diablo 3 [2], the servers for that game were going up and down like a yo-yo at launch. If Blizzard of all companies can't make an online game that works for 100% of users on day 1, what chance does SE have?

Has anyone else here ever played an MMO with real issues? Like days-long downtimes, weekly rollbacks, databases lost without backups, lag to the point of unplayability? I'm often shocked at how smoothly modern MMOs manage to run.

EDIT: ...

[1]: Guild Wars 2 Is Off to a Bumpy Start - Forbes
[2]: Diablo III woes plague launch day players | Crave - CNET

Originally Posted by Aulayna

Good stuff.

Thank you!

Maintenance in thirty.

Originally Posted by Loony BoB

Seriously, you were looking around the internet and seeing rave reviews and lots of excitement before Phase Four?

Absolutely. I feel like it kind of blew up during Phase 3. There was a lot of hype, a lot of talk about how much better it was, a naysayer here and there, but the overwhelming majority thought it was going in the right direction and was a very solid game. It totally ramped up, and I wasn't surprised that things came to a head during Phase 4 due to the good stuff people had been saying everywhere.

Well a free week of play isn't a bad way to make up for the troubles.

Originally Posted by Jiro

Well a free week of play isn't a bad way to make up for the troubles.

*cough*

Originally Posted by Endless

In a more positive note : FINAL FANTASY XIV, The Lodestone | Server Congestion and Trial Period Extensions (09/03/2013)

Originally Posted by Jiro

Well a free week of play isn't a bad way to make up for the troubles.

Cool, thanks for the news, Jiro.