PRNG: Raise warning and/or provide better pseudo random generator?

php.internals

Yasuo Ohgaki

10 years ago
Hi all, We have issue on pseudo random generators generates only odd/even numbers. https://bugs.php.net/bug.php?id=63174 https://news.ycombinator.com/item?id=9941364 We should raise E_WARNING/E_NOTICE if user supplies random number range that generated random number cannot be random at least. Patch for rand/mt_rand. https://gist.github.com/yohgaki/1519f65dffd66735bafe It seems we need more reliable(fool proof) pseudo random generator. Anyone working on this? We may extends rand()/mt_rand() so that they work with larger range by calling random generators multiple times. If this is implemented, the patch raises errors is not required. mt_rand() extension breaks compatibility with other MT rand implementations, but we already broke it. Therefore, it should not matter. (This was the reason why mt_rand() wasn't made to support 64bit int, IIRC) IMO, we should provide better pseudo random generators than now. Any comments? P.S. I think we should fix mt_rand() to be true MT rand. Function named mt_rand() is not being MT rand is not a nice thing to keep _forever_.
-- Yasuo Ohgaki yohgaki@ohgaki.net

Andrey Andreev

10 years ago
Hi, On Mon, Feb 22, 2016 at 11:30 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> > IMO, we should provide better pseudo random generators than now. >
We already have a better PRNG in random_int(). Cheers, Andrey.

Yasuo Ohgaki

10 years ago
Hi Andrey, On Tue, Feb 23, 2016 at 6:38 AM, Andrey Andreev <narf@devilix.net> wrote:
> On Mon, Feb 22, 2016 at 11:30 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: >> >> >> IMO, we should provide better pseudo random generators than now. > > > We already have a better PRNG in random_int().
randon_int() is _not_ a alternative of rand()/mt_rand() as they cannot produce the same pseudo random numbers. e.g. Games etc requires the same pseudo random numbers. Regards,
-- Yasuo Ohgaki yohgaki@ohgaki.net

Scott Arciszewski

10 years ago
On Mon, Feb 22, 2016 at 4:42 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> Hi Andrey, > > On Tue, Feb 23, 2016 at 6:38 AM, Andrey Andreev <narf@devilix.net> wrote: >> On Mon, Feb 22, 2016 at 11:30 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: >>> >>> >>> IMO, we should provide better pseudo random generators than now. >> >> >> We already have a better PRNG in random_int(). > > randon_int() is _not_ a alternative of rand()/mt_rand() as they cannot > produce the same pseudo random numbers. e.g. Games etc requires > the same pseudo random numbers. > > Regards, > > -- > Yasuo Ohgaki > yohgaki@ohgaki.net > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php >
Have I shared this here? https://github.com/paragonie/seedspring Warning: May summon Cthulhu if deployed to production. Scott Arciszewski Chief Development Officer Paragon Initiative Enterprises <https://paragonie.com>

Yasuo Ohgaki

10 years ago
Hi Scott, On Tue, Feb 23, 2016 at 7:14 AM, Scott Arciszewski <scott@paragonie.com> wrote:
> Have I shared this here? > > https://github.com/paragonie/seedspring > > Warning: May summon Cthulhu if deployed to production.
I didn't know at least. Nice class!
-- Yasuo Ohgaki yohgaki@ohgaki.net

Yasuo Ohgaki

10 years ago
Hi all, On Tue, Feb 23, 2016 at 6:30 AM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> > We have issue on pseudo random generators generates only odd/even > numbers. > > https://bugs.php.net/bug.php?id=63174 > https://news.ycombinator.com/item?id=9941364 > > We should raise E_WARNING/E_NOTICE if user supplies random number > range that generated random number cannot be random at least. > Patch for rand/mt_rand. > https://gist.github.com/yohgaki/1519f65dffd66735bafe > > It seems we need more reliable(fool proof) pseudo random generator. > Anyone working on this? > > We may extends rand()/mt_rand() so that they work with larger range by > calling random generators multiple times. If this is implemented, the > patch raises errors is not required. mt_rand() extension breaks compatibility > with other MT rand implementations, but we already broke it. Therefore, it > should not matter. (This was the reason why mt_rand() wasn't made to support > 64bit int, IIRC) > > IMO, we should provide better pseudo random generators than now. > > Any comments?
This is edge case that produces odd/even numbers only. https://3v4l.org/kYpAF This is the worst case. Current implementation uses 32bit int for generating random numbers and any number exceeds the range could be biased because the result is computed by RAND_RANGE() which uses double for arithmetic. PHP allows huge min/max without any warning/error under 64bit OS. Limiting range can prevent this and we can be sure rand()/mt_rand() produce the same random numbers on both 32/64 bit platform. (If rand() uses the same algorithm, of course) https://gist.github.com/yohgaki/1519f65dffd66735bafe Valid range is limited to 2^31 according to current implementation. Actual range could be determined by PHP_RAND_MAX/PHP_MT_RAND_MAX, but I heard Windows' PHP_RAND_MAX is only 2^15. Is this correct? I don't prefer to have strict range error for these systems. I'll write patch that does not raise warning for smaller PHP_RAND_MAX. It's unreliable pseudo random generator anyway. It should not matter much. Any comments for adding out of range warnings to rand()/mt_rand()? If nobody has comment on this, I'll write RFC for additional warnings. Anyone prefer to extend rand()/mt_rand() for 64bit OSes? Regards
-- Yasuo Ohgaki yohgaki@ohgaki.net

Christoph Becker

10 years ago
On 23.02.2016 at 11:41, Yasuo Ohgaki wrote:
> Actual range could be determined by PHP_RAND_MAX/PHP_MT_RAND_MAX, but > I heard Windows' PHP_RAND_MAX is only 2^15. Is this correct?
Yes, see <https://msdn.microsoft.com/en-us/library/398ax69y.aspx>.
> Any comments for adding out of range warnings to rand()/mt_rand()? If > nobody has comment on this, I'll write RFC for additional warnings. > Anyone prefer to extend rand()/mt_rand() for 64bit OSes?
See also the discussion for <https://github.com/php/php-src/pull/1416>; it might make sense to disallow such ranges in the long run (i.e. for PHP 8), in which case we might want to raise deprecated notices instead of warnings now.
-- Christoph M. Becker

Pierre Joye

10 years ago
hi, On Tue, Feb 23, 2016 at 5:41 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> Hi all, > > On Tue, Feb 23, 2016 at 6:30 AM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: >> >> We have issue on pseudo random generators generates only odd/even >> numbers. >> >> https://bugs.php.net/bug.php?id=63174 >> https://news.ycombinator.com/item?id=9941364 >> >> We should raise E_WARNING/E_NOTICE if user supplies random number >> range that generated random number cannot be random at least. >> Patch for rand/mt_rand. >> https://gist.github.com/yohgaki/1519f65dffd66735bafe >> >> It seems we need more reliable(fool proof) pseudo random generator. >> Anyone working on this? >> >> We may extends rand()/mt_rand() so that they work with larger range by >> calling random generators multiple times. If this is implemented, the >> patch raises errors is not required. mt_rand() extension breaks compatibility >> with other MT rand implementations, but we already broke it. Therefore, it >> should not matter. (This was the reason why mt_rand() wasn't made to support >> 64bit int, IIRC) >> >> IMO, we should provide better pseudo random generators than now. >> >> Any comments? > > This is edge case that produces odd/even numbers only. > https://3v4l.org/kYpAF > This is the worst case. Current implementation uses 32bit int for > generating random numbers and any number exceeds the range could be > biased because the result is computed by RAND_RANGE() which uses > double for arithmetic. PHP allows huge min/max without any > warning/error under 64bit OS. > > Limiting range can prevent this and we can be sure rand()/mt_rand() > produce the same random numbers on both 32/64 bit platform. (If rand() > uses the same algorithm, of course) > https://gist.github.com/yohgaki/1519f65dffd66735bafe > Valid range is limited to 2^31 according to current implementation. > > Actual range could be determined by PHP_RAND_MAX/PHP_MT_RAND_MAX, but > I heard Windows' PHP_RAND_MAX is only 2^15. Is this correct? I don't > prefer to have strict range error for these systems. I'll write patch > that does not raise warning for smaller PHP_RAND_MAX. It's unreliable > pseudo random generator anyway. It should not matter much. > > > Any comments for adding out of range warnings to rand()/mt_rand()? If > nobody has comment on this, I'll write RFC for additional warnings. > Anyone prefer to extend rand()/mt_rand() for 64bit OSes?
Thing is the MT algorithm may not be design to do that, at all but was designed for 32-bit integers. I won't be in favor of changing (again) the implementation without any safety about the results (safety means compliance or be even more different from the MT algorithms). Adding warning when the given ranges are out of bounds sound good, and reduce them within the maximum range. I joined the other person proposing not to change anything else in our MT implementation as there is little to no benefit. If we need pure implementation of one pseudo RNG or another, we can provide new implementations. But changing again this one may bring more troubles than what we are trying to solve. Cheers,
-- Pierre @pierrejoye | http://www.libgd.org

Scott Arciszewski

10 years ago
On Tue, Feb 23, 2016 at 7:46 AM, Pierre Joye <pierre.php@gmail.com> wrote:
> hi, > > On Tue, Feb 23, 2016 at 5:41 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: >> Hi all, >> >> On Tue, Feb 23, 2016 at 6:30 AM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: >>> >>> We have issue on pseudo random generators generates only odd/even >>> numbers. >>> >>> https://bugs.php.net/bug.php?id=63174 >>> https://news.ycombinator.com/item?id=9941364 >>> >>> We should raise E_WARNING/E_NOTICE if user supplies random number >>> range that generated random number cannot be random at least. >>> Patch for rand/mt_rand. >>> https://gist.github.com/yohgaki/1519f65dffd66735bafe >>> >>> It seems we need more reliable(fool proof) pseudo random generator. >>> Anyone working on this? >>> >>> We may extends rand()/mt_rand() so that they work with larger range by >>> calling random generators multiple times. If this is implemented, the >>> patch raises errors is not required. mt_rand() extension breaks compatibility >>> with other MT rand implementations, but we already broke it. Therefore, it >>> should not matter. (This was the reason why mt_rand() wasn't made to support >>> 64bit int, IIRC) >>> >>> IMO, we should provide better pseudo random generators than now. >>> >>> Any comments? >> >> This is edge case that produces odd/even numbers only. >> https://3v4l.org/kYpAF >> This is the worst case. Current implementation uses 32bit int for >> generating random numbers and any number exceeds the range could be >> biased because the result is computed by RAND_RANGE() which uses >> double for arithmetic. PHP allows huge min/max without any >> warning/error under 64bit OS. >> >> Limiting range can prevent this and we can be sure rand()/mt_rand() >> produce the same random numbers on both 32/64 bit platform. (If rand() >> uses the same algorithm, of course) >> https://gist.github.com/yohgaki/1519f65dffd66735bafe >> Valid range is limited to 2^31 according to current implementation. >> >> Actual range could be determined by PHP_RAND_MAX/PHP_MT_RAND_MAX, but >> I heard Windows' PHP_RAND_MAX is only 2^15. Is this correct? I don't >> prefer to have strict range error for these systems. I'll write patch >> that does not raise warning for smaller PHP_RAND_MAX. It's unreliable >> pseudo random generator anyway. It should not matter much. >> >> >> Any comments for adding out of range warnings to rand()/mt_rand()? If >> nobody has comment on this, I'll write RFC for additional warnings. >> Anyone prefer to extend rand()/mt_rand() for 64bit OSes? > > Thing is the MT algorithm may not be design to do that, at all but was > designed for 32-bit integers. I won't be in favor of changing (again) > the implementation without any safety about the results (safety means > compliance or be even more different from the MT algorithms). > > Adding warning when the given ranges are out of bounds sound good, and > reduce them within the maximum range. > > I joined the other person proposing not to change anything else in our > MT implementation as there is little to no benefit. > > If we need pure implementation of one pseudo RNG or another, we can > provide new implementations. But changing again this one may bring > more troubles than what we are trying to solve. > > > Cheers, > -- > Pierre > > @pierrejoye | http://www.libgd.org > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php >
If we're going to consider new non-cryptographic random number generators, PCG is worth considering. ;) http://www.pcg-random.org/ Scott Arciszewski Chief Development Officer Paragon Initiative Enterprises <https://paragonie.com>

Yasuo Ohgaki

10 years ago
Hi Scott, On Tue, Feb 23, 2016 at 11:36 PM, Scott Arciszewski <scott@paragonie.com> wrote:
> > If we're going to consider new non-cryptographic random number > generators, PCG is worth considering. ;) > > http://www.pcg-random.org/
It's simple and supports 64bit int out of the box. Looks great!
-- Yasuo Ohgaki yohgaki@ohgaki.net

Tom Worster

10 years ago
On 2/23/16 7:13 PM, Yasuo Ohgaki wrote:
>> >http://www.pcg-random.org/ > It's simple and supports 64bit int out of the box. > Looks great!
PSG is very interesting. But it's new and hasn't been peer reviewed yet. It's in the "experimental" stage while others are more "well known". xorshift+ seems fairly popular. Tom

Yasuo Ohgaki

10 years ago
Hi Tom, On Wed, Feb 24, 2016 at 9:57 AM, Tom Worster <fsb@thefsb.org> wrote:
> On 2/23/16 7:13 PM, Yasuo Ohgaki wrote: > >>> >http://www.pcg-random.org/ >> >> It's simple and supports 64bit int out of the box. >> Looks great! > > > PSG is very interesting. But it's new and hasn't been peer reviewed yet. > It's in the "experimental" stage while others are more "well known". > > xorshift+ seems fairly popular.
It looks excellent. We may be better to provide better repeatable pseudo random generators and algorithm may be select-able like hash. Regards,
-- Yasuo Ohgaki yohgaki@ohgaki.net

Tom Worster

10 years ago
I agree that mt_rand() should warn before delivering bogus outputs. But when it works, it works ok: https://gist.github.com/tom--/a12175047578b3ae9ef8 Given that it hasn't been MT19937 or many years, it probably doesn't need to be. If there is really a need for fast repeatable RNGs (the kind popular in monte-carlo sims) then MT is no longer the most likely candidate. A new API could allow the user to select a generator -- there are a lot to chose from (see below)! Tom These are the built-in generators in dieharder 3.31.1. Note the xorshift family isn't present. Melissa O’'Neill's new PCG family is interesting too. 000 borosh13 001 cmrg 002 coveyou 003 fishman18 004 fishman20 005 fishman2x 006 gfsr4 007 knuthran 008 knuthran2 009 knuthran2002 010 lecuyer21 011 minstd 012 mrg 013 mt19937 014 mt19937_1999 015 mt19937_1998 016 r250 017 ran0 018 ran1 019 ran2 020 ran3 021 rand 022 rand48 023 random128-bsd 024 random128-glibc2 025 random128-libc5 026 random256-bsd 027 random256-glibc2 028 random256-libc5 029 random32-bsd 030 random32-glibc2 031 random32-libc5 032 random64-bsd 033 random64-glibc2 034 random64-libc5 035 random8-bsd 036 random8-glibc2 037 random8-libc5 038 random-bsd 039 random-glibc2 040 random-libc5 041 randu 042 ranf 043 ranlux 044 ranlux389 045 ranlxd1 046 ranlxd2 047 ranlxs0 048 ranlxs1 049 ranlxs2 050 ranmar 051 slatec 052 taus 053 taus2 054 taus113 055 transputer 056 tt800 057 uni 058 uni32 059 vax 060 waterman14 061 zuf 203 ca 204 uvag 205 AES_OFB 206 Threefish_OFB 207 XOR (supergenerator) 208 kiss 209 superkiss 400 R_wichmann_hill 401 R_marsaglia_multic. 402 R_super_duper 403 R_mersenne_twister 404 R_knuth_taocp 405 R_knuth_taocp2

Yasuo Ohgaki

10 years ago
Hi all, On Tue, Feb 23, 2016 at 7:41 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> Any comments for adding out of range warnings to rand()/mt_rand()? If > nobody has comment on this, I'll write RFC for additional warnings. > Anyone prefer to extend rand()/mt_rand() for 64bit OSes?
I'll write RFC to raise errors for invalid range, hopefully this weekend. Thank you for feedback!
-- Yasuo Ohgaki yohgaki@ohgaki.net

Nikita Popov

10 years ago
On Wed, Feb 24, 2016 at 1:27 AM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> Hi all, > > On Tue, Feb 23, 2016 at 7:41 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: > > Any comments for adding out of range warnings to rand()/mt_rand()? If > > nobody has comment on this, I'll write RFC for additional warnings. > > Anyone prefer to extend rand()/mt_rand() for 64bit OSes? > > I'll write RFC to raise errors for invalid range, hopefully this weekend. > > Thank you for feedback! >
I think it would be preferable to wait a bit -- there's clearly interest in replacing rand/mt_rand etc with a higher quality PRNG, so maybe lets give people time to create a proposal for that before we start throwing warnings. Nikita

Yasuo Ohgaki

10 years ago
Hi Nikita, On Wed, Feb 24, 2016 at 9:00 PM, Nikita Popov <nikita.ppv@gmail.com> wrote:
> On Wed, Feb 24, 2016 at 1:27 AM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: >> >> Hi all, >> >> On Tue, Feb 23, 2016 at 7:41 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: >> > Any comments for adding out of range warnings to rand()/mt_rand()? If >> > nobody has comment on this, I'll write RFC for additional warnings. >> > Anyone prefer to extend rand()/mt_rand() for 64bit OSes? >> >> I'll write RFC to raise errors for invalid range, hopefully this weekend. >> >> Thank you for feedback! > > > I think it would be preferable to wait a bit -- there's clearly interest in > replacing rand/mt_rand etc with a higher quality PRNG, so maybe lets give > people time to create a proposal for that before we start throwing warnings.
Reasonable suggestion! Adding warnings is trivial. It could done before RC. I'll wait until then. Regards,
-- Yasuo Ohgaki yohgaki@ohgaki.net

Yasuo Ohgaki

10 years ago
Hi all, On Thu, Feb 25, 2016 at 6:31 AM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
>> I think it would be preferable to wait a bit -- there's clearly interest in >> replacing rand/mt_rand etc with a higher quality PRNG, so maybe lets give >> people time to create a proposal for that before we start throwing warnings. > > Reasonable suggestion! > Adding warnings is trivial. It could done before RC. > I'll wait until then.
I updated the doc for the time being. http://svn.php.net/viewvc?view=revision&revision=338662 MT rand range is 2^32, but our mt_rand() implementation is 2^31. http://lxr.php.net/xref/PHP_5_6/ext/standard/rand.c#331 Because RAND_RANGE() macro simply does arithmetic on supplied parameters, out of range values result in biased numbers. Therefore, I replaced mt_rand() caution to my warning. Regards,
-- Yasuo Ohgaki yohgaki@ohgaki.net