Yoshi-P On Final Fantasy XIV's Capacity Issues

As producer, the responsibility for being unable to predict the current influx of new players falls to me. The days before an expansion’s launch are supposed to be ones of excitement, but instead they’ve brought many players grief, and for that I, personally, am extremely sorry.

Naoki Yoshida, Final Fantasy XIV Producer & Director

I play two MMORPGs. In EVE Online, the "EVE is dying" meme is back in style with the recent drop in player activity. Final Fantasy XIV faces an entirely different issue. Over the past weekend (17-18 July), login queue times in North America averaged 10-20 minutes with players unable to create new characters for several hours. The situation in Europe was worse, with queue times up to 40 minutes. Yesterday, FFXIV producer and director Naoki Yoshida (aka Yoshi-P) issued a letter giving players an update on the situation.

Yoshida called the recent influx of players over the last two weeks onto the North American and European data centers "staggering and unexpected." The problems are not contained to the western worlds though. The global version of FFXIV in the Japanese region as well as the localized Chinese and Korean versions. 

The immediate steps Square Enix has taken recently include the implementation of an AFK timer with Patch 5.58 and hardware upgrades to the North American data centers. Yesterday's patch implemented a feature usually reserved for expansion launches. Players who are AFK for 30 minutes will automatically be logged out of the game.

The hardware upgrade gets a little complicated. Yoshida provided a brief explanation about the game's server infrastructure:

The population that can be logged in to each World server at one time is capped; however, as some of you have noticed, this cap is not the same for all servers, and differs somewhat between data centers (Mana, Aether, Chaos, etc.). The main reason for this has to do with a data center’s “matching server.” A matching server can only handle so much information, so to ensure that it functions properly, a capacity limit must be set. This limit determines the overall maximum population of a data center, which is then divided by the number of Worlds in the data center to calculate how many players can be logged in to each World. If a matching server were to overload, it would result in the Duty Finder ceasing to function properly and instability when entering instanced content, hence the importance of login caps.

Differences in login caps occur due to differences in the number of Worlds per data center, as well as differences in matching server performance.

The North American data centers received upgraded matching servers that have increased the maximum number of logged in account by 18,000, or 750 additional accounts per world server. If I have perform the math correctly, the maximum population on the world servers on the two European data centers should increase by 1000 once the matching servers are upgraded. Yoshida did not give a date for when Chaos and Light will receive the new equipment.

What about creating new world servers and/or data centers? Yoshida wrote:

Though the obstacles that still face us are many, we believe that the best, most effective method for providing our players a stress-free gameplay experience is through the introduction of new servers and data centers. To see this goal realized as soon as reasonably possible, we on the development, operations, and infrastructure teams pledge to continue to do our utmost.

At Fanfest 2021, Square Enix revealed the creation of a new data center for Oceania with the launch of 6.1. Yoshida revealed in the letter that the original play was to evaluate user numbers and planned content after the release of Endwalker and then build new servers and/or data centers for the release of 7.0.

Unfortunately the global semiconductor shortage is impacting Square Enix' plans. The company is experiencing "a prolonged delay in the delivery of the servers" already ordered. Even offering above market rates for equipment has failed to speed up delivery. Instead of quickly adding capacity, the letter gives the impression Square Enix may not have the new Oceania data server online by the launch of patch 6.1.

Adding to the difficulty of spinning up new game servers are travel restrictions related to the COVID-19 pandemic. Working remotely has slowed down the work on creating the Oceania data center. The same challenges face the infrastructure team on adding capacity in North America and Europe. The team is getting their COVID vaccinations and will travel overseas once the situation is deemed safe for the staff. Even then, members of the infrastructure team will not travel outside Japan if they feel the situation is not safe.

As for Yoshida's concluding thoughts, I'll just copy them below in their entirety. 

As producer, the responsibility for being unable to predict the current influx of new players falls to me. The days before an expansion’s launch are supposed to be ones of excitement, but instead they’ve brought many players grief, and for that I, personally, am extremely sorry.

For years, we have watched devoted players from around the world stream their time with FINAL FANTASY XIV, and now, we sit in awe while new fans join their ranks. As we enter our final push in the development of Endwalker, it is this constant flow of positive energy that motivates all of us on the development and operations team to carry on. Nothing can compare to the feeling we get watching you enjoy playing the game we’ve created.

We are aware of the burden we have placed on our player base, and are working diligently to lighten it in the coming weeks and months. We will continue to develop measures to combat server congestion, and will keep you up to date with any and all developments in this area.

Thank you once again for your continued support!