With the latest update to the Trinity graphics engine going live on EVE Online's Tranquility server on Tuesday, I thought the time was right to bring up another significant update to CCP Games' Carbon Engine: the move from Stackless Python to Python 3 back in mid-February. But unlike other major technical changes in the EVE Evolved series, the change came from a highly reviled source by some quarters of the gaming media. That's right, the change came from the team making EVE Frontier.
The developers of the Carbon Engine ran into a serious problem: the lack of support for Stackless Python. According to a dev blog from the Frontier team, Stackless Python would not progress any farther than Python 3.8. A real problem as a look at the Python developers guide shows the end of life for Python 3.8 was October 2024.
 |
| The end of life of Python's major releases |
CCP has a long history with Stackless Python but had reached a fork in the road. Either maintain Stackless Python 3.8 themselves or migrate to the main development line of Python 3. This, of course, presented a major challenge.
Over the years, CCP has been a major contributor to the Stackless Python fork but it has been stuck on a maximum of Python 3.8 for some time now, which gave us two options if we were to continue the development of EVE Frontier as a game for the next 20 years (and beyond):
1.) Maintain Stackless Python fork ourselves
2.) Move towards vanilla Python 3
At this point I should mention the same challenge for development also faced the team of developers working on EVE Online. The upgrade was not an easy, or quick, task.
Around 4 million lines of Python code and 1 million lines of C++ code make up the EVE Frontier client and server. Many of these needed to be updated or completely changed.
Python 3 is a major upgrade but contained a number of backwards-incompatible changes. Many of these have a very simple migration path, such as changes to print statements which transforms to print "hello world!" to print("hello world!") . However others were more complicated, such as division changing from being an integer division to a floating-point division (in this case we can, at the cost of code readability and a small performance cost, do a safe transformation).
Veteran EVE players probably winced at the thought of the effect those small changes potentially would have on the game if not performed properly. Or, at least on EVE Online as those playing in Frontier wouldn't feel the effects as they didn't have experience with changing outcomes of formulas.
In addition, other challenges arose: some things no longer work. For example you can’t compare two types that aren’t comparable. This had historical reasoning but in our development we have come across code where sorting list of tuples sees the second element being not comparable. We also have C++ code that is exposed to Python and that interface changed quite a bit between Python versions. For example any code written and compiled for Python 2 does not work for Python 3.
These and similar issues have already been found and fixed, however there may be more lurking in code that is seldomly executed or only executed in a production environment.
I think with the relaxation of the NDA I can mention I was involved in the test phase in which the developers flipped the switch to run the version of the Carbon Engine using Python 3. Trust me, I'd rather the testing continue on the Frontier servers.
The amount of work was staggering:
- 4 million lines of Python code made Python 3 compatible
- 5840 files modified
- 80,564 lines changed
- 18,461 lines added
- 16,305 lines deleted
- All C++ interfaces to Python now compatible with a Python 3 interpreter
- The developers rewrote a large portion of the network stack
- Implemented a new scheduler to work with greenlets
- Upgraded all of of the developers tools to use Python 3
Estimating performance improvements in a complex product such as Frontier is difficult. When we do measurements on certain code paths we can see improvements in some and regression in others. However, what the net effect it has depends on how often those code paths are executed.You could get some numbers if we had consistent load before and after the Python 3 upgrade but that is not the case for Frontier. That being said, what we can clearly see in our numbers is that when we measure more complicated code paths, such as multiple users logging in or weapons being fired we are seeing between 10% to 30% performance increase.An impressive result, especially when considering we are still developing EVE Frontier itself in a live environment! You could say we have been replacing the engine while the car is running. A massive undertaking the payoff is huge: EVE Frontier is now running smoothly on Python 3.12, setting the stage for all future development.
