Why CCP is still using Python 2

We at CCP are maybe the heaviest users of Python in videogames, though I have no data to back that up. (I’ll also use this opportunity to say this is a personal blog post, I am in no official capacity here) We use it in the client and server of both EVE Online (PC) and DUST 514 (PS3), and nearly all of our internal infrastructure uses it. What’s stopping us from upgrading at least some portions to Python 3?

Everything. Even if Python 2 weren’t good enough (it is), even if a hundred compelling features were added to Python 3 (there aren’t), even if Stackless was available for Python 3 (it isn’t), we still probably wouldn’t switch. Because literally everything conspires against enterprise employees who want to upgrade any significant codebase. Let’s go over some of those things.

We have our own localization solution inside EVE and the unicode/str bugs have been worked out. Oh, the solution is a nightmare, and our string handling is often a mess, but that just means changing it would be even more difficult. So there’s no real external product need, and internal products and tools usually aren’t localized. But wouldn’t it be great to change it and get rid of that technical debt and simplify things?

Sure it would be great to get rid of that tech debt! As it would for literally every area of our 11+ year old product. There is a limited amount of technical debt we can clean up, and none of it has to do with string handling or any Python 3 features. We just removed a custom importer we’ve wanted to remove for years, which paves the way for other technical debt cleanup. But we’re at least a year from another codebase-wide cleanup, of which there are many to do (let’s remove our dozen remaining builtins, please!). When low-level or non-value-adding work involves convincing people all the way up to the corporate/business level, there are very few people who can organize that sort of thing. I generally prefer that energy is spent on activities that general more value to our engineers.

We have relatively few automated tests. We’ve made some great progress on testing in the past couple years, especially for new or refactored/rewritten code, but there’s absolutely no way to uncover and fix Python 3 upgrade bugs easily. We have “extensive” manual regression tests, but it takes time to get builds to those testers and I imagine the regression test cycle would take months to get everything worked out. It would be a hard sell to QA and the turnaround time on bugs would mean the upgrade process would take months, not weeks.

The only place a Python 3 upgrade gets traction is with the core of the Python community here. And unfortunately we spend most of our capital by trying to improve our own systems, training our Python programmers, and even keeping other developers using Python.*

We can get backports of libraries developed for or in the stdlib of Python 3. If they weren’t provided in a sustainable way by others, we’d just copy them in and keep them locally.

We keep a critical eye on performance. While Python 3 seems fast enough for us now, that’s a pretty new development, and we absolutely will not regress on performance. So that could mean upgrading our codebase and still not be able to use Python 3, since we won’t fully know until we do it. Lovely!

Middleware, the scourge of OSS! We use Autodesk Maya, which uses 2.6 (soon 2.7) for our art pipeline. We would need to write a large chunk of our code to support Python 2 and 3 (an ongoing inconvenience), but also need infrastructure to test them in both (added work but not a huge deal, since we already do this in some cases for 2.6/2.7). But middleware is an easy excuse to stop an upgrade. Even upgrading middleware like Maya, which carries little risk, can take over a year.

Ultimately we’ve been too sloppy and monolithic** (like a large number of enterprise users, I’d imagine). Our products make money. Our developers are supposed to be working on those products and thus making money. We didn’t do a good job balancing technical and business needs, and besides our codebase is really old (EVE Online’s codebase is still an ancestor of the original codebase that EVE released with in 2001).

All of this means the “default” Python for me is Python 2. It is my “go to” language when I’m working on something where I have the option. I haven’t used Python 3 for any real work, so there’s still a threshold to cross. Familiarity and comfort. I don’t even have Python 3 installed at work. And most of the people I know are working in Python 2, so there’s a synergy of inertia. I have no problem with Python 3*** and plan to roll it out at work somewhere (it is a 2014 goal of mine), though I am intrigued with all the Python 2.8 (both vanilla and Stackless) discussions going on recently.

---

* There are some defections to other languages, and causing inconvenience to these customers on the fence is only more likely to push them into other languages. I have nothing against using other languages inhouse (in fact I wholly support it), I just want to make sure they’re used for the right reason. There’s value in being able to work in other people’s code without having to learn a new language and environment, and there’s value to doing as much as possible in high level languages.

** This has in many ways changed, with lots better automated testing and code quality overall, but legacy code is still the majority. We’re just producing less new legacy code.

*** I also understand that upgrading applications like EVE Online was not part of the Python 3 adoption strategy.