Thursday, October 7, 2021

Quasar: Updating EVE Online's Network Layer

Despite providing features players wanted, sometimes for years, a lot of complaining about the new skills UI in EVE Online occurred on places like the EVE sub-Reddit.  A frequent request was to make the new user interface optional. If the latest dev blog is any indication, CCP will not provide a toggle setting. The developers have a new tool, Quasar, and they intend on proceeding full speed ahead.

Quasar is the newest iteration of the networking layer of Tranquility. CCP has been hampered by the use of Stackless Python and its limit of performing one task at a time. EVE initially used. CCP explained the issue when the company switched from using StacklessIO to CarbonIO in 2011.

Bottom line: Code written in Stackless Python can only execute as fast as your fastest CPU core can go. A 4 or 8 CPU big-iron server will burn up a single CPU and leave the others idle, unless we span more nodes to harness the other CPUs, which works well for a lot of the logic in EVE which is stateless or only mildly reliant on shared state but presents challenges in logic which is heavily reliant on shared state, such as space simulation and walking around space stations.

This is not a problem as long as a single CPU can handle your load, and for sane uses of Python this is true. I shouldn't have to tell you that we are hitting the point at which a single CPU can't process a large fleet fight, despite the work of teams like Gridlock in squeezing out every drop of optimization that they can. CPUs are not getting any faster. They are getting more cache, fatter busses and better pipelines, but not really any faster at doing what EVE needs done. The way of the near (and perhaps far) future is to 'go wide' and run on multiple CPUs simultaneously.

The introduction of CarbonIO allowed for the implementation of time dilation which allows for the world record fights for which EVE is famous. The situation, while improved, still needed work. CCP needed a better way to work around Python's Global Interpreter Lock (GIL), the gatekeeper that only allows Python to perform one task at a time.

CCP's effort was codenamed Project Sanguine. This week's dev blog gave an example of one of the events the developers need to consider and improve upon.

The first form of Project Sanguine emerged with ESI and the first iteration of EVE Portal in late 2016. Through these projects, a new paradigm was established within the server architecture of EVE Online: a message bus. From this new escape hatch, the bottlenecks associated with the GIL were rediscovered, but with a clearer picture of their expensive manifestations: message routing, serialization, and transmission. If one ship fires one laser in the middle of 1000 ships, that’s 1000 messages which need to be sent immediately all over the globe. The simulation must address that message to 1000 destinations as a copy (message routing), convert that data to a wire format (serialization), and then send the data over the wire (transmission). In most cases, CarbonIO has been addressing each of those issues, but within the custody of the GIL. CarbonIO has served EVE Online well for quite some time, but much has changed on the turbulent seas of the internets since 2011.

CCP has introduced features in the last five years designed to have a smaller footprint on the Tranquility infrastructure and databases. Features like the Activity Tracker, Opportunities, and Abyssal Proving Grounds leaderboards fell under the Project Sanguine framework. But the features still needed to interact with Tranquility through the CarbonIO and the GIL. With Quasar, Skill Plans do not have to interact with the GIL at all.

Over the years players have noted the single-threaded nature of Python and wanted the developers to rewrite the EVE code base to take advantage of the growth of cores on servers. A total rewrite is not going to happen. But with the introduction of Quasar, CCP has begun the process to update EVE to run into a third decade. As the dev blog noted:

Work continues on clearing the table by refurbishing many old services which provides two vectors of momentum: build up more foundational capabilities for Quasar in terms of manipulating more than just Skill Plans in the universe, and normalize ancient systems to pave the way for faster iteration.

What does that mean for the average Capsuleer? More opportunities to expose more powerful features across more mediums.

The idea of publishing simulation data through Quasar has also been toyed with…but that might take a minute.

No comments:

Post a Comment