Following the launch of early access for Final Fantasy XIV: Endwalker, many players have endured various issues like long queue lines and log-in errors. This is due to the unprecedented amount of active players in the game. The official release is still plagued with these issues as even more players try to play the new expansion. The future of the game is still looking brighter than ever and producer/director Naoki Yoshida addresses the issues below:
Hello, this is Naoki Yoshida, Producer & Director of FINAL FANTASY XIV.
Thank you all for participating in the Early Access and official launch of Endwalker.
Since the start of Early Access, we have been seeing an unprecedented number of active players, the highest ever seen since the launch of FFXIV, as well as record numbers of simultaneous logins, but along with this, the number of connections has far surpassed the scale of our services which has resulted in long wait times when attempting to log in.
Even during this congestion, we’ve received words of support and encouragement, for which we’re truly grateful. Thank you very much. On the other hand, we sincerely apologize for the long waiting times and the resulting frustration experienced by players attempting to log in.
I’d like to share some information regarding the ongoing congestion, the reasons behind errors that have been appearing while waiting in queues, and the current status of our fixes. For information on error codes not listed below, please see please see my previous announcement on December 5. https://sqex.to/QDFMi
■ Lobby server maintenance to address errors
Allow me to share some information regarding specific errors. We are planning for a full maintenance on the lobby servers at the earliest timing possible, in order to resolve these errors. Although the maintenance time isn’t fully solidified, it is estimated to take about 30 minutes. This will not affect any players who are already logged into the game, but during these 30 minutes, players will not be able to make new login attempts, and the login queue will be unavailable.
We deeply apologize for this but ask for your cooperation as we address these server errors. The exact maintenance times will be shared in an upcoming announcement.
■ All Worlds Maintenance
On Tuesday, December 7, we will be conducting maintenance across all Worlds to address server errors aside from those mentioned above, as well as a few issues that can be resolved on the server side. This requires all servers to go offline and is estimated to take about four hours to complete. Details on maintenance timing will be announced in the future. I’d like to ask for your understanding and cooperation regarding this matter as well.
Below are some details regarding errors that have been occurring as a result of congestion.
■ Disconnects by Error 4004 right before logging in
Currently, I believe the most frustrating cases are when you’ve been waiting in the queue and get disconnected by an error when your turn comes around, forcing you join the queue again from the end. These disconnects are often happening as a result of Error 4004 and 3001, and here I’d like to provide the details and statuses of countermeasures against Error 4004.
・ How the Process Works
To prevent servers from going down as a result of mass logins or overloading, the process of logging into a World that has reached the login cap involves waiting until the World can admit 100 players, then admitting the first 100 players in the login queue, a process that is being repeated constantly.
・ The Issue
We have confirmed an issue during the process of admitting these 100 players, which can occur if certain conditions coincide while the servers are experiencing extremely high loads, which will cause up to 25% of these players to receive an error. This issue is not guaranteed to occur even when the servers are experiencing extremely high loads, and the number of players who experience the issue varies. However, considering login queues can take up to several hours and this error can cause players to have to return to the back of the queue, we currently recognize this as the most frustrating issue.
* Added: Dec. 7, 06:20 a.m. PST
This fix has been implemented, but I would like to clarify some things in relation to the on-screen display. Worlds that reach the login limit will be checked for 100 available positions, and 100 new logins will be executed sequentially. However, since the maximum number of characters that can log in to a World in the space of one minute sequentially is limited to 75 in order to reduce the load, the number of people in the queue will often appear to decrease by 75 in the on-screen display. (This is how it will appear when there is a clean overlap in the update timing of the displayed number of players waiting.) This has been the way the system operates from before, and even if the processing is carried out properly with the fix this time, the maximum count will decrease by 75 each in the dialogue, making it look as if the aforementioned fix has failed. Due to the coincidental overlap in the numbers, I apologize for any confusion my choice of words may cause.・ Resolving the Issue
We have been able to locate the cause of this error, which occurs due to extremely high server loads, and are currently fixing the source code for the lobby servers. Although the fix itself is finished, the process of checking whether the fix has actually resolved this issue requires time as it is an extremely complex process. In addition, due to the complexity of the conditions of occurrence itself, it is possible that we have not yet addressed more specific conditions, but we will implement the fixes and continue to monitor the situation. As soon as our preparations are ready, we will carry out the work through the aforementioned lobby server maintenance. We ask that you wait a little longer until the fix is made.
■ Disconnects by Error 3001 right before logging in
I’d also like to provide the details and countermeasures for Error 3001.
Error 3001 Cause #1
A login cap is in place on every World in order to prevent them from going down due to high loads. Despite this, a number of extremely overloaded servers have allowed players to log in beyond this cap, resulting in the number of permitted lobby server connections to fail and cause Error 3001 and disconnect players in the queue.
・ Resolution for Cause #1
We currently have a general idea of what is causing this issue with extremely overloaded Worlds, and a fix is underway. However, these are areas where complex processing is carried out and careful fixing is required for this process. As such, fixing/confirming/implementation into the lobby server will require time. This will be implemented in the maintenance scheduled for December 8, and we are putting our full effort into it, but haven’t been able to determine the full details. Further details regarding this will be announced at a later timing as well.
・ Error 3001 Cause #2
This is an issue that is occurring only on the Tonberry World on the Elemental data center. The Tonberry World has the largest number of login requests of all the game Worlds, and players are helping to mitigate this issue by using the World Visit System and logging into other Worlds. However, due to the way chat connections work when using the World Visit System, we cannot completely cut off a player’s connection with their character’s Home World, causing the total number of active connections to become enormous, and we have confirmed that as a result, a situation similar to Cause #1 above can occur.
・ Resolution for Cause #2
Although we are aware of the relative risk of the Tonberry World going down, we’ve increased the number of network connections to its absolute maximum. Your patience would be greatly appreciated as we monitor the situation based on this tentative fix for the time being.
■ Error 2002
Now I would like to explain about the details related to Error 2002 and our countermeasures.
・ Details of the Process
In order to prevent a major login server outage, FFXIV has a limit of “no more than 17,000 total players waiting to log in per logical data center”. This cap is set uniformly regardless of how many Worlds exist in a logical data center.
・ The Issue
If the total number of players waiting in the logical data center exceeds 17,000, Error 2002 will be displayed and you will not be able to queue for login.
・ The Resolution for this Issue
We are now ready to deploy the backup development servers to the public lobby servers, which will improve the capacity management of the lobby server and raise the maximum number of players in the queue to around 21,000. Since this will not require any maintenance, we will err on the side of caution and proceed with the deployment outside of peak hours. It is scheduled to be completed late in the night of December 7, Japan time.
■ Regarding Measures Towards a Fundamental Resolution for Congestion
When it comes to fundamental measures to prevent this type of large-scale congestion—and this is something that was brought up in the https://sqex.to/4IkYs we made ahead of early access—the most effective method would be a “large-scale addition of new Worlds in all regions.”
On the one hand, following the explosive rise in players coming to FFXIV beginning in spring 2021, we spent over half a year scrambling to address the need to add new Worlds. However, due to policies surrounding COVID-19, which have had an effect on the entire world, even now we have been unable to procure the required hardware for adding new Worlds.
■ Regarding the Global Semiconductor Shortage (Supply Chain Issues)
When it comes to adding new Worlds, we need tens of “server machines” for every World that we add. Server machines are high-performance computers, which utilize numerous semiconductors. However, due to the COVID-19 countermeasures currently in place, many factories across the globe which produce semiconductors have halted production, or have faced labor shortages. This has ultimately led to a decrease in the number of semiconductors being produced, and has resulted in a worldwide semiconductor shortage.
Furthermore, this does not only pertain to the production of semiconductors, but from a logistics standpoint as well, COVID-19 countermeasures continue to explosively push up demand and create this turbulent situation. We have made considerable investment—even more so than usual—to secure the required hardware, but even so, a long lead time will be needed to prepare the server hardware.
If we try to look to when this semiconductor shortage may be solved, unfortunately given the state of things, we aren’t able to estimate when we might reach a resolution. The FFXIV Server and Infrastructure teams accelerated work to implement the additional high-performance machines we had secured, as well as optimizing the server source code and increasing the number of maximum logins per World. I promise that we will continue to put in this type of effort.
Given the current situation, we anticipate that it may take several months or more to add new Worlds. I want to express my sincerest apologies to all of our players for the inconvenience this is causing. I appreciate your continued understanding and support regarding this matter.
Hopefully, more servers and worlds will be introduced in the near future to help the congestion of players in the game. Final Fantasy XIV: Endwalker is out now for PS4, PS5 and PC. Stay tuned at Final Weapon for the latest Japanese gaming news and updates!
