Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Unbundle libsqlite3? »

Unbundle libsqlite3?

php.internals

Christoph Becker

7 years ago
Hi! We bundle an unmodified libsqlite3 for at least two years. Since then all updates go into any dev, alpha and beta releases, while security patches (usually backports from libsqlite3) go into stable branches. ext/sqlite3 requires libsqlite ≥ 3.3.9[1] which has been released on 2007-01-04[2] (i.e. more than eleven years ago!). ext/pdo_sqlite has no specific version requirement; there is a check for sqlite3_open()[3] which I believe is available since 3.0.0. So is there any particular reason why we still bundle libsqlite3? [1] <https://github.com/php/php-src/blob/php-7.3.0RC1/ext/sqlite3/config0.m4#L37-L47> [2] <https://sqlite.org/oldnews.html#2007_01_04> [3] <https://github.com/php/php-src/blob/php-7.3.0RC1/ext/pdo_sqlite/config.m4#L55-L58>
-- Christoph M. Becker

Kalle Sommer Nielsen

7 years ago
Hi Den man. 17. sep. 2018 kl. 15.16 skrev Christoph M. Becker <cmbecker69@gmx.de>:
> > Hi! > > We bundle an unmodified libsqlite3 for at least two years. Since then > all updates go into any dev, alpha and beta releases, while security > patches (usually backports from libsqlite3) go into stable branches. > > ext/sqlite3 requires libsqlite ≥ 3.3.9[1] which has been released on > 2007-01-04[2] (i.e. more than eleven years ago!). ext/pdo_sqlite has no > specific version requirement; there is a check for sqlite3_open()[3] > which I believe is available since 3.0.0. > > So is there any particular reason why we still bundle libsqlite3? > > [1] > <https://github.com/php/php-src/blob/php-7.3.0RC1/ext/sqlite3/config0.m4#L37-L47> > [2] <https://sqlite.org/oldnews.html#2007_01_04> > [3] > <https://github.com/php/php-src/blob/php-7.3.0RC1/ext/pdo_sqlite/config.m4#L55-L58>
I don't think there is any paticular reason for bundling it anymore, at least not from the Windows side of things where we should be easily able to adapt the build system to link to it instead of building the bundled version. (Anatol cc'ed)
-- regards, Kalle Sommer Nielsen kalle@php.net

Johannes Schlueter

7 years ago
On Mo, 2018-09-17 at 15:16 +0200, Christoph M. Becker wrote:
> So is there any particular reason why we still bundle libsqlite3?
An issue is that users often  don't control the version being used and SQLite adds features to it's SQL dialect from time to time. At the moment, with the bundled version you can be quite certain that a PHP version maps to a SQLite version, so when migrating to a different host you're going to have the same feature set available.  (Or is that not true since Linux distros don't like using bundled libs, but prefer system installed shared libs so they just upgrade that on security issues?) Most other libraries don't impact in regards to features as much as newer library features would often have to be exported to userspace explicitly. johannes

Christoph Becker

7 years ago
On 17.09.2018 at 16:44, Johannes Schlüter wrote:
> On Mo, 2018-09-17 at 15:16 +0200, Christoph M. Becker wrote: > >> So is there any particular reason why we still bundle libsqlite3? > > An issue is that users often  don't control the version being used and > SQLite adds features to it's SQL dialect from time to time. At the > moment, with the bundled version you can be quite certain that a PHP > version maps to a SQLite version, so when migrating to a different host > you're going to have the same feature set available.  (Or is that not > true since Linux distros don't like using bundled libs, but prefer > system installed shared libs so they just upgrade that on security > issues?)
I suppose that (most) distros use shared system libs instead of our bundled stuff.
-- Christoph M. Becker

Nikita Popov

7 years ago
On Mon, Sep 17, 2018 at 3:16 PM Christoph M. Becker <cmbecker69@gmx.de> wrote:
> Hi! > > We bundle an unmodified libsqlite3 for at least two years. Since then > all updates go into any dev, alpha and beta releases, while security > patches (usually backports from libsqlite3) go into stable branches. > > ext/sqlite3 requires libsqlite ≥ 3.3.9[1] which has been released on > 2007-01-04[2] (i.e. more than eleven years ago!). ext/pdo_sqlite has no > specific version requirement; there is a check for sqlite3_open()[3] > which I believe is available since 3.0.0. > > So is there any particular reason why we still bundle libsqlite3? > > [1] > < > https://github.com/php/php-src/blob/php-7.3.0RC1/ext/sqlite3/config0.m4#L37-L47 > > > [2] <https://sqlite.org/oldnews.html#2007_01_04> > [3] > < > https://github.com/php/php-src/blob/php-7.3.0RC1/ext/pdo_sqlite/config.m4#L55-L58 > > >
+1 on unbundling libsqlite. If we a) don't patch a library and b) it is widely available in distros, then we have no business bundling the library. Bundled libraries are extra maintenance work for us and a security risk for users (we are slower at updating bundled libraries than distros). Three more libraries that we (as far as I know) bundle without modification are: * libzip: Unbundling was already planned for 7.3 but held up due to some CI issues. Next try at https://github.com/php/php-src/pull/3532. * oniguruma * pcre I'm not sure how widely available libonig and libpcre2 are. Nikita

Anatol Belski

7 years ago
Hi,
> -----Original Message----- > From: Nikita Popov <nikita.ppv@gmail.com> > Sent: Monday, September 17, 2018 7:07 PM > To: Christoph Becker <cmbecker69@gmx.de> > Cc: PHP internals <internals@lists.php.net> > Subject: Re: [PHP-DEV] Unbundle libsqlite3? > > On Mon, Sep 17, 2018 at 3:16 PM Christoph M. Becker <cmbecker69@gmx.de> > wrote: > > > Hi! > > > > We bundle an unmodified libsqlite3 for at least two years. Since then > > all updates go into any dev, alpha and beta releases, while security > > patches (usually backports from libsqlite3) go into stable branches. > > > > ext/sqlite3 requires libsqlite ≥ 3.3.9[1] which has been released on > > 2007-01-04[2] (i.e. more than eleven years ago!). ext/pdo_sqlite has > > no specific version requirement; there is a check for > > sqlite3_open()[3] which I believe is available since 3.0.0. > > > > So is there any particular reason why we still bundle libsqlite3? > > > > [1] > > < > > https://github.com/php/php-src/blob/php-7.3.0RC1/ext/sqlite3/config0.m > > 4#L37-L47 > > > > > [2] <https://sqlite.org/oldnews.html#2007_01_04> > > [3] > > < > > https://github.com/php/php-src/blob/php-7.3.0RC1/ext/pdo_sqlite/config > > .m4#L55-L58 > > > > > > > +1 on unbundling libsqlite. If we a) don't patch a library and b) it is > widely available in distros, then we have no business bundling the library. > Bundled libraries are extra maintenance work for us and a security risk for users > (we are slower at updating bundled libraries than distros). > > Three more libraries that we (as far as I know) bundle without modification > are: > * libzip: Unbundling was already planned for 7.3 but held up due to some CI > issues. Next try at https://github.com/php/php-src/pull/3532. > * oniguruma > * pcre > > I'm not sure how widely available libonig and libpcre2 are. >
For SQLite3 it should be fine. For PCRE2 - a case we require the bundled lib badly is for debugging and Valgrind integration for JIT. Sure some debug symbols can be available from a distribution, but I never tried it that way and it might differ from one Linux to another especially regarding Valgrind. Non-Linux environments like BSD based are even not to mention. An alternative to the distribution supplied builds might be to always have custom Valgrind enabled builds of libpcre2 just for the PHP development, whereby different versions might collide with some system provided libraries. IMO it is more convenient for us to keep the bundled PCRE libs for this reasons. Some libonig versions seem to be available on even old stable Debian and current Fedora, but further targets need to be investigated. Windows builds for all these should be doable. I'd need to check closer for PCRE2 and Oniguruma, but SQLite3 is for sure good. Regards Anatol

Remi Collet

7 years ago
Le 17/09/2018 à 15:16, Christoph M. Becker a écrit :
> Hi! > > We bundle an unmodified libsqlite3 for at least two years. Since then > all updates go into any dev, alpha and beta releases, while security > patches (usually backports from libsqlite3) go into stable branches. > > ext/sqlite3 requires libsqlite ≥ 3.3.9[1] which has been released on > 2007-01-04[2] (i.e. more than eleven years ago!). ext/pdo_sqlite has no > specific version requirement; there is a check for sqlite3_open()[3] > which I believe is available since 3.0.0. > > So is there any particular reason why we still bundle libsqlite3?
+1 to drop the bundled library Remi

BohwaZ/PHP

7 years ago
On Mon, 17 Sep 2018 15:16:13 +0200 / "Christoph M. Becker" <cmbecker69@gmx.de> said :
> Hi! > > We bundle an unmodified libsqlite3 for at least two years. Since then > all updates go into any dev, alpha and beta releases, while security > patches (usually backports from libsqlite3) go into stable branches. > > ext/sqlite3 requires libsqlite ≥ 3.3.9[1] which has been released on > 2007-01-04[2] (i.e. more than eleven years ago!). ext/pdo_sqlite has > no specific version requirement; there is a check for > sqlite3_open()[3] which I believe is available since 3.0.0.
I believe that SQLite3Stmt::readOnly requires SQLite 3.7.4 released in 2010, bundled since PHP 5.3.6 (!). There is also a PR patch (from me) that should make it in PHP 7.4 that implements this feature in pdo_sqlite too. I believe this is the most recent feature that we are using. But it should return false if the SQLite version installed is older than that (IMHO it should throw an exception instead, but that's the current behaviour anyway). As for bundling or not, I'm on Devuan and it is using the system version of SQLite as far as I know so I don't see the need to bundle it with PHP, but I don't have a good view of whether it might be needed on other platforms.

Christoph Becker

7 years ago
On 19.09.2018 at 18:53, BohwaZ wrote:
> On Mon, 17 Sep 2018 15:16:13 +0200 / "Christoph M. Becker" > <cmbecker69@gmx.de> said : > >> ext/sqlite3 requires libsqlite ≥ 3.3.9[1] which has been released on >> 2007-01-04[2] (i.e. more than eleven years ago!). ext/pdo_sqlite has >> no specific version requirement; there is a check for >> sqlite3_open()[3] which I believe is available since 3.0.0. > > I believe that SQLite3Stmt::readOnly requires SQLite 3.7.4 released in > 2010, bundled since PHP 5.3.6 (!).
Ah, indeed[1]!
> There is also a PR patch (from me) that should make it in PHP 7.4 that > implements this feature in pdo_sqlite too.
PR #2698[2] seems to require an RFC. IMHO, it's best to present RFCs and voting on them as early as possible (instead of rushing them at the last moment). :)
> I believe this is the most recent feature that we are using. But it > should return false if the SQLite version installed is older than that > (IMHO it should throw an exception instead, but that's the current > behaviour anyway).
I agree that an exception would be better. A sensible alternative might be to not even include the method for older libsqlite3. Either way, a PR targeting master would be welcome. [1] <https://github.com/php/php-src/blob/php-7.3.0RC1/ext/sqlite3/sqlite3.c#L1393-L1397> [2] <https://github.com/php/php-src/pull/2698>
-- Christoph M. Becker

BohwaZ/PHP

7 years ago
On Wed, 19 Sep 2018 19:41:11 +0200 / "Christoph M. Becker" <cmbecker69@gmx.de> said :
> PR #2698[2] seems to require an RFC. IMHO, it's best to present RFCs > and voting on them as early as possible (instead of rushing them at > the last moment). :)
Yeah there was an RFC and it was voted no: https://wiki.php.net/rfc/implement_sqlite_openblob_in_pdo Discussion was that current PDO way of handling DB-specific features is not the best. Unfortunately no one volunteered to change that fact, and as my patch wasn't approved PDO_SQLite is still behind SQLite3 and missing features. I was actually talking about https://github.com/php/php-src/pull/2760 which was approved just after feature freeze of PHP 7.3.
> I agree that an exception would be better. A sensible alternative > might be to not even include the method for older libsqlite3. Either > way, a PR targeting master would be welcome.
Yeah unfortunately due to the result of the previous RFC I stopped my efforts trying to bring up pdo_sqlite and SQLite3 up to date. I have very little free time, so spending it for useless patches is not the best use of it, sorry.

Christoph Becker

7 years ago
On 19.09.2018 at 19:57, BohwaZ wrote:
> On Wed, 19 Sep 2018 19:41:11 +0200 / "Christoph M. Becker" > <cmbecker69@gmx.de> said : > >> PR #2698[2] seems to require an RFC. IMHO, it's best to present RFCs >> and voting on them as early as possible (instead of rushing them at >> the last moment). :) > > Yeah there was an RFC and it was voted no: > https://wiki.php.net/rfc/implement_sqlite_openblob_in_pdo
Ah, thanks! Since the RFC has been declined, I've closed the PR.
> Discussion was that current PDO way of handling DB-specific features is > not the best. Unfortunately no one volunteered to change that fact, and > as my patch wasn't approved PDO_SQLite is still behind SQLite3 and > missing features.
It's unfortunate, but unless *someone* invests the time to come up with a solution, things will likely not change.
> I was actually talking about https://github.com/php/php-src/pull/2760 > which was approved just after feature freeze of PHP 7.3.
Ah, thanks. I've update NEWS and UPGRADING accordingly[1]. [1] <http://git.php.net/?p=php-src.git;a=commit;h=7b9b398daad24d8829df487131bdf5d266e34f74>
-- Christoph M. Becker

Christoph Becker

7 years ago
On 17.09.2018 at 15:16, Christoph M. Becker wrote:
> We bundle an unmodified libsqlite3 for at least two years. Since then > all updates go into any dev, alpha and beta releases, while security > patches (usually backports from libsqlite3) go into stable branches. > > ext/sqlite3 requires libsqlite ≥ 3.3.9[1] which has been released on > 2007-01-04[2] (i.e. more than eleven years ago!). ext/pdo_sqlite has no > specific version requirement; there is a check for sqlite3_open()[3] > which I believe is available since 3.0.0. > > So is there any particular reason why we still bundle libsqlite3? > > [1] > <https://github.com/php/php-src/blob/php-7.3.0RC1/ext/sqlite3/config0.m4#L37-L47> > [2] <https://sqlite.org/oldnews.html#2007_01_04> > [3] > <https://github.com/php/php-src/blob/php-7.3.0RC1/ext/pdo_sqlite/config.m4#L55-L58>
I have submitted <https://github.com/php/php-src/pull/3548>.
-- Christoph M. Becker