Parallelised run-tests.php (patch)

Hi there, Do you spend HOURS every day AGONISING over how long run-tests.php takes to complete? Have you long since ABANDONED every test directory besides Zend/tests? Does it feel like your eight CPU cores are pointlessly WASTED by single-threaded PHP test execution? Are you SADDENED every day at how high-quality the run-tests.php code is, DREAMING of an even worse mess? Then I've got just the trick for you! …*ahem*. Okay, enough terrible salesmanship. I felt like parallelising run-tests.php, so I did it. If you give it the flag -jX, it'll spawn X worker processes and throw batches at tests at them, and those worker processes will send back the results of tests for the parent process to display and collate. To avoid potential problems with tests that step on eachother's toes (i.e. access the same file or port or whatever), it gives each worker a particular directory of tests and it then executes those in parallel. Obviously, this means things aren't as parallel as they could be for very large test directories (looking at you, Zend/tests/…), though that's a solvable problem. :) Per a suggestion by Rasmus in an earlier thread, the parent hands out large directories to children first to reduce the time spent on children occupied with huge directories. And what a difference it makes. On my Ubuntu 16.04 VirtualBox VM running inside Windows 10 Pro x64 on my Ryzen 1700 8-core 16-thread PC, here's vanilla run-tests.php against master on a --disable-all build: ===================================================================== TEST RESULT SUMMARY --------------------------------------------------------------------- Exts skipped : 68 Exts tested : 7 --------------------------------------------------------------------- Number of tests : 15613 8481 Tests skipped : 7132 ( 45.7%) -------- Tests warned : 0 ( 0.0%) ( 0.0%) Tests failed : 30 ( 0.2%) ( 0.4%) Expected fail : 31 ( 0.2%) ( 0.4%) Tests passed : 8420 ( 53.9%) ( 99.3%) --------------------------------------------------------------------- Time taken : 317 seconds ===================================================================== And here's run-tests.php -j16: ===================================================================== TEST RESULT SUMMARY --------------------------------------------------------------------- Exts skipped : 68 Exts tested : 7 --------------------------------------------------------------------- Number of tests : 15613 8481 Tests skipped : 7132 ( 45.7%) -------- Tests warned : 0 ( 0.0%) ( 0.0%) Tests failed : 35 ( 0.2%) ( 0.4%) Expected fail : 31 ( 0.2%) ( 0.4%) Tests passed : 8415 ( 53.9%) ( 99.2%) --------------------------------------------------------------------- Time taken : 80 seconds ===================================================================== That's about a quarter of the single-threaded time. It could get even better still if there were more test directories to work with; currently the number of alive workers declines as the test run goes on, because all the small directories get finished quickly and you're left waiting on the Zend/tests-like behemoths. One idea might be to add some sort of special file you could add to a directory which tells run-tests.php that it is safe to parallelise, in which case it would break it into chunks at runtime for you. Though I don't think we should have huge numbers of files in a single directory anyway, if possible. Okay, but what's the catch? Well, the code isn't the most… elegant thing. I did make run-tests.php faster, but I didn't refactor it to be less of a global-variables-dependant mess than it is at present, except for moving all the top-level code into a single function for the sake of my own sanity. Therefore, the worker child processes get their own copy of the global state to initialise them. It's a bit hacky, but this is basically what UNIX fork() does, so maybe I shouldn't feel so bad. ;) On the other hand, the remainder of the design is quite elegant! It's all clean message-passing over STDIN and STDOUT. This isn't fully tested yet. I haven't actually investigated those 5 test failures, and my implementation probably breaks some run-tests.php features right now. I also haven't bothered to try this on a non --disable-all build yet. That said, I'm impressed how well it actually works right now and how effectively it tolerates errors. The child processes will even happily kill themselves if the parent does. The code is here: https://github.com/php/php-src/pull/2822 Happy Halloween.