The Next Step To EVE's Holy Grail

When I first started playing EVE Online in 2009, downtime lasted an hour every day, from 1100-1200 UTC. On 1 November 2010, downtime officially was reduced to 30 minutes. On 11 May 2016, the daily downtime for the Tranquility server was halved again to 15 minutes. Today, downtime typically lasts 4-5 minutes. And tomorrow, EVE will have no downtime.

Downtime today is not like downtime was a few years ago when it was usually up to an hour, enough time for a long lunch, including dessert. These days Tranquility's auto-reboot on weekends takes approximately 4 minutes and 20-40 seconds, just enough for a quick cup of tea. But it is still an inconvenience since you must time your activities to be at a safe place at 11:00 UTC. Downtime waits for no one.

Therefore, for the first time in the more than six thousand days of EVE Online, there will be no downtime on Wednesday, 4 December! We will begin the first 48 hour run ever of EVE at 11 o'clock on Tuesday 3 December and end it on Thursday, 5 December.

In a time honored IT practice, CCP plans to shut off a feature (the daily reboot) and see what breaks.

First, let’s take a step back and look at the reasons why we have downtime in the first place:

• We have excessive memory consumption and lack of clean-up in certain areas, and we don't necessarily refresh cache since the daily reboot will take care of it all.

• We still have daily database jobs that run during downtime.

• There are certain things that must be done regularly, and it is most convenient to do them during startup when there are no players online.

This is our dirty laundry. We have documented all the things we know that can go wrong. Then there are the things we don't know about and testing is the only way to find out…

Eliminating downtime is a goal CCP has pursued for a long time. Back in the 2010 dev blog announcing the reduction of downtime down to 15 minutes, CCP Hunter (yes, that CCP Hunter), wrote:

What has been done to reduce downtime?

In the old days, systems in EVE Online were built on the fact that there was daily downtime. In the last few years no new code has been produced that relies on downtime and a great deal of work has been done in removing old dependencies on downtime. You could say that we are still paying for past sins.

In addition to this we have worked on the cluster shutdown procedure and startup procedure so that the cluster goes down and up faster.

What does the future hold, when will the daily downtime go away?

As a part of the Carbon initiative, cluster management is being re-architected. It is our goal that sometime in the not too distant future, EVE Online will have no daily downtime. How awesome will that be!

I put the above in to show that Pearl Abyss is not responsible for the concern about downtime. I can't help but point out that downtime currently occurs at 2000 (8pm) in Seoul. Also, with the launch of the Korean language client in November, the Australian/East Asia time zone is noticeably growing. However, CCP isn't totally comfortable with moving downtime to before the AUTZ begins their playtime.

The first instinct would be to say that downtime should be at 7 o'clock when the online population is the lowest. This is the "Pacific Downtime"; after the American play session and before the Asian play session...

There is more than can be read from these graphs such as that CPU usage overnight and at 7 o'clock, during the lowest population, is quite high, so activity per player is high during the American play session. Americans also play longer into the night than, say, Europeans. Shortening the American play session at the tail end with the "Pacific Downtime" is, therefore, not the right choice.

The dev blog suggests that downtime could move from 1100 UTC to 0930 UTC (1830 in Seoul), which might play into some laws about excessive playing in South Korea. But for now, CCP wants to try to move to only having downtime once every 48 hours. Tomorrow's cancelled downtime is the first step toward accomplishing a long dreamed of goal.