Everything

php

[RFC] keeping unit tests green

php.internals

Stas Malyshev

12 years ago

Hi!

As promised, I have created an RFC on unit test policy, here:
https://wiki.php.net/rfc/travis_ci

Please discuss. Any comments/suggestions/corrections welcome.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

Sebastian Bergmann

12 years ago

Am 14.04.2014 19:56, schrieb Stas Malyshev:

> As promised, I have created an RFC on unit test policy, here:
> https://wiki.php.net/rfc/travis_ci

Nitpicking: To the best of my knowledge, PHP does not have a single
unit test. PHPT tests, of which we have many, are integration tests.

Stas Malyshev

12 years ago

Hi!

>  Nitpicking: To the best of my knowledge, PHP does not have a single
>  unit test. PHPT tests, of which we have many, are integration tests.

OK, I'll change the RFC to reflect that.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

Florian Anderiasch

12 years ago

On 04/14/2014 09:26 PM, Sebastian Bergmann wrote:

> Am 14.04.2014 19:56, schrieb Stas Malyshev:
>> As promised, I have created an RFC on unit test policy, here:
>> https://wiki.php.net/rfc/travis_ci
>
>  Nitpicking: To the best of my knowledge, PHP does not have a single
>  unit test. PHPT tests, of which we have many, are integration tests.

If
https://github.com/php/php-src/blob/master/ext/standard/tests/strings/bin2hex.phpt
(and most of the other function tests) is not a unit test I'm not sure
how you would set the requirements.

Greetings,
Florian

Ferenc Kovacs

12 years ago

On Tue, Apr 15, 2014 at 10:46 AM, Florian Anderiasch <ml@anderiasch.de>wrote:

>
>
> On 04/14/2014 09:26 PM, Sebastian Bergmann wrote:
> > Am 14.04.2014 19:56, schrieb Stas Malyshev:
> >> As promised, I have created an RFC on unit test policy, here:
> >> https://wiki.php.net/rfc/travis_ci
> >
> >  Nitpicking: To the best of my knowledge, PHP does not have a single
> >  unit test. PHPT tests, of which we have many, are integration tests.
>
> If
>
> https://github.com/php/php-src/blob/master/ext/standard/tests/strings/bin2hex.phpt
> (and most of the other function tests) is not a unit test I'm not sure
> how you would set the requirements.
>
> Greetings,
> Florian
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

they are still testing our PHP_FUNCTION() macro, and
zend_parse_parameters(), etc. instead of testing the php_bin2hex() function.
from the userland POV, they could be seen as unit tests, but they still
test the given unit from a mile away through multiple layers of cruft.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Sebastian Bergmann

12 years ago

Am 15.04.2014 10:46, schrieb Florian Anderiasch:

> If
> https://github.com/php/php-src/blob/master/ext/standard/tests/strings/bin2hex.phpt
> (and most of the other function tests) is not a unit test I'm not sure
> how you would set the requirements.

A PHPT test is a PHP script that is executed using the PHP interpreter.
This constitutes an integration test. A unit test would be a test that
is written in C that exercises a single C unit of code in isolation
from the rest of the interpreter.

Julien Pauli

12 years ago

On Mon, Apr 14, 2014 at 7:56 PM, Stas Malyshev <smalyshev@sugarcrm.com> wrote:

> Hi!
>
> As promised, I have created an RFC on unit test policy, here:
> https://wiki.php.net/rfc/travis_ci
>
> Please discuss. Any comments/suggestions/corrections welcome.

Hi,

This looks like something we already do nowadays, except we (well, I)
don't go to watch for the CI but look after the tests in the branch
I'm as RM managing.

So I'm OK, as soon as it is simple.
For example, I dont know Travis. So when opening the Travis page about
PHP, I see many things, red or green, and feel like lost in all those
informations.
I don't know where to look to find quickly that something went wrong, and why.
So if we could have on the wiki a page explaining how to manage Travis
builds, I would be more than happy :-)

Thx.


Julien.P

Crypto Compress

12 years ago

Hi,

> So I'm OK, as soon as it is simple.
> For example, I dont know Travis. So when opening the Travis page about
> PHP, I see many things, red or green, and feel like lost in all those
> informations.

Travis-CI is as trivial as this:
1. a (github) notification hook queue a testrun
2. as resources are free a preconfigured vm starts and
3. all the things in .travis.yml are executed

As a RM you can let the VM execute whatever you want and how you want.

https://github.com/php/php-src/blob/master/.travis.yml
http://docs.travis-ci.com/user/languages/cpp/

Crypto Compress

12 years ago

Am 15.04.2014 14:02, schrieb Crypto Compress:

> Hi,
>
>> So I'm OK, as soon as it is simple.
>> For example, I dont know Travis. So when opening the Travis page about
>> PHP, I see many things, red or green, and feel like lost in all those
>> informations.
>
> Travis-CI is as trivial as this:
> 1. a (github) notification hook queue a testrun
> 2. as resources are free a preconfigured vm starts and
> 3. all the things in .travis.yml are executed
>
> As a RM you can let the VM execute whatever you want and how you want.
>
> https://github.com/php/php-src/blob/master/.travis.yml
> http://docs.travis-ci.com/user/languages/cpp/

ops: http://docs.travis-ci.com/user/languages/c/

Stas Malyshev

12 years ago

Hi!

> So I'm OK, as soon as it is simple.
> For example, I dont know Travis. So when opening the Travis page about
> PHP, I see many things, red or green, and feel like lost in all those
> informations.

Travis is pretty simple as such - you go to
https://travis-ci.org/php/php-src/builds and check out your branch
(branch name is in Commit column). If it's not green, you click on the
build number and get the log which says what went wrong. The failed
tests are usually at the very end, and once you know the names of the
tests failed, look them up in the log to find the details.

With pulls on github, you just click on "Details" link and it gets you
to the same log page.

Setting up a fork requires some configs, but Travis docs are good
enough. I might also add a wiki page on how to set up the fork in Travis
a bit later.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

Julien Pauli

12 years ago

On Tue, Apr 15, 2014 at 6:34 PM, Stas Malyshev <smalyshev@sugarcrm.com> wrote:

> Hi!
>
>> So I'm OK, as soon as it is simple.
>> For example, I dont know Travis. So when opening the Travis page about
>> PHP, I see many things, red or green, and feel like lost in all those
>> informations.
>
> Travis is pretty simple as such - you go to
> https://travis-ci.org/php/php-src/builds and check out your branch
> (branch name is in Commit column). If it's not green, you click on the
> build number and get the log which says what went wrong. The failed
> tests are usually at the very end, and once you know the names of the
> tests failed, look them up in the log to find the details.
>
> With pulls on github, you just click on "Details" link and it gets you
> to the same log page.
>
> Setting up a fork requires some configs, but Travis docs are good
> enough. I might also add a wiki page on how to set up the fork in Travis
> a bit later.

Yes I spent some time to see how it works, it seems pretty easy.

In fact, if we could add more analyze tools, it would be nice, because
actually, it just acts as an autommated "make test".

Julien

Stas Malyshev

12 years ago

Hi!

> In fact, if we could add more analyze tools, it would be nice, because
> actually, it just acts as an autommated "make test".

Yes, for now. We can definitely expand it. Only downside of simplicity
is that we have basic yes/no trigger and build artifacts are only
available as a log AFAIK. So for some tools it may not be the best and
Jenkins (which has more extensive reporting and build artifacts support)
could be better. But suggestions of course are most welcome, and we can
put anything we want in .travis.yml if Travis supports it.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

Pierre Joye

12 years ago

Hi Stas,

On Mon, Apr 14, 2014 at 7:56 PM, Stas Malyshev <smalyshev@sugarcrm.com> wrote:

> Hi!
>
> As promised, I have created an RFC on unit test policy, here:
> https://wiki.php.net/rfc/travis_ci
>
> Please discuss. Any comments/suggestions/corrections welcome.

I very much like this idea, a must do.

I also like to extend it more:

- for stable branches, I would like a staging branche where all of us
can commit, RM can then cherry pick stable patches for a given branche

- add static analyzers to the staging branch (gcc, clang, VC and PVS,
that will covers much issues including vicious 64bit issues
. it could be possible to do it per commit

- try as much as possible to get warning free commits, linux and
partially windows. Windows may have false positive while it will be
drastically reduced once we have the int64 patch in

Doing so we will always be able to do security releases very
efficiently and safely

I do not care much about the wording, unit tests vs whatever the name is :)

Cheers,

--
Pierre

@pierrejoye | http://www.libgd.org

Stas Malyshev

12 years ago

Hi!

> - for stable branches, I would like a staging branche where all of us
> can commit, RM can then cherry pick stable patches for a given branche

Travis does pull requests already, and you can hook up your fork to it
easily so it would test any branch you push into your fork. E.g. see
https://travis-ci.org/smalyshev/php-src/builds

> - add static analyzers to the staging branch (gcc, clang, VC and PVS,
> that will covers much issues including vicious 64bit issues
>    . it could be possible to do it per commit

That would be most welcome, not sure how easy/hard it is to do it. If we
get our own Jenkins running and stable, that would be of course possible.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

Stas Malyshev

12 years ago

Hi!

Speaking on Windows, Travis does not support it - so what could be use
instead?

As for warnings, I was planning to take a look into this soon, clang has a
lot of things to complain about our code :)