In march 2015, we started to think that code refactoring and architecture improvements, will not be the only way to optimize the response time on dailymotion.com. This is the core problem of websites with high load : “how to scale without investing too much in people/servers”.
HHVM experimentation
We’ve heard about HHVM, an open source interpreter developed by Facebook that can replace PHP interpreter, which was old and stalling for a long time (PHP6 fail). We started by installing HHVM on a development environment. At the beginning, a simple configuration of HHVM and 50 lines of code commented are enough to make our website work. We made a quick benchmark with ab and results were really surprising. All pages had a response time divided by two and same for memory / cpu.
We had a look on why we have errors on code we had to disable. Two reasons :
- There is a lot of code incompatibilities between PHP and HHVM ( some functions doesn’t return the same results…, send extra warnings…) (list here)
- We use custom extensions which are not compatible with HHVM.
So we developed our custom extensions in HHVM style following advices of this guide : http://blog.golemon.com/2015/01/hhvm-extension-writing-part-i.html. And we made some pull requests to correct some HHVM incompatibilities (list here).
We decided that before going further, we had to test HHVM on one production server. After the deployment, we discovered more and more incompatibilities that we fixed with pull requests on HHVM project until everything seemed good. HHVM has a short release cycle, which is great to quickly correct problems. They accept easily pull requests if you explain what you’ve done and write a unit tests for what you changed. During this short test, we had several times HHVM stuck with all cpu core at 100% during this test (issue link). We didn’t used the Repo authoritative mode with better performance because it implied that we had to change our release process to build the bytecode file and we had some dynamic configurations exported by zkfarmer in PHP. Zkfarmer is a set of tools that easily manage our distributed server farms using Apache ZooKeeper.
Firsts results in production with HHVM :
We were really excited about results. But we still have some questions :
- did we have hidden incompatibilities ?
- how to go back to PHP interpreter if we are bored by HHVM
- how to explain to developers incompatibilities with PHP (a lot of time wasted to explain / understand why this is broken)
- how much we will spend time to maintain this solution in production
- how to modify our release process to warm JIT cache
During few months, this project wasn’t the priority, so we decided to wait the release of PHP 7 to compare performances .
PHP7 : easiest migration, performance and stability
In december 2015 PHP7 was released, and some big platforms like Badoo migrated to PHP7 in production with a good results (https://techblog.badoo.com/blog/2016/03/14/how-badoo-saved-one-million-dollars-switching-to-php7/). So we tried to deploy 2 servers, results were almost equal to HHVM performances in term of response time, cpu and memory usage. Moreover it appears to be more stable on servers and extensions are already developed (i.e.: extension like pinba, php-redis, custom Dailymotion extensions).
Graph comparison between PHP 5.5 / PHP 7 / HHVM on two servers to compare performances : 
Phan developed by Etsy is a good tool to help you in your first effort, it’s a static analyser an it can find incompatibilities between PHP 5/7 in your codebase and more… It took less than a week to migrate our codebase (a 10 years old PHP monolith) and most of errors were warnings about E_STRICT that haven’t been fixed since a long time (reclassified in notice / warning). And it took 4 hours to migrate our custom extensions.
So we decided to choose PHP7 because its easier to deploy, there is no regressions except backward incompatibilities listed on php manual, it will have (maybe) a better long time support, and same gain in term of response time / cpu / memory.
tl;dr: we can handle twice more traffic with same infrastructure. Most of big web companies that use PHP have to do this migration to improve their platform costs. And I’m happy to had the time to test different solutions before making the final jump .
Graph of response time / cpu / after PHP7 migration (datadog graph):
Graph comparison between PHP 5.5 / PHP 7 / HHVM on two servers to compare performances :
Pinba : memory footprint comparison.
PHP 7 :
PHP 5.5 :