Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Add support for HTTP/1.1 »

Add support for HTTP/1.1

php.internals

Rowan Collins

9 years ago
Hi internallers! No, you haven't misread the subject line, I'm not talking about the exciting new HTTP/2, but HTTP/1.1, 20 years old this year. To my surprise, PHP's HTTP stream wrapper (e.g. file_get_contents('http://example.com');) defaults to sending HTTP/1.0 requests. You can tell it to use HTTP/1.1 with an appropriate "stream context", but it's fiddly [1], and the implementation is incomplete. There's actually already an exception to this: some SOAP servers were discovered to only talk HTTP/1.1, so ext/soap forces the wrapper to upgrade by default [bug-43069]. This in turn revealed some of the bugs in the wrapper's protocol support, one of which I have recently created a patch for [bug-47021]. It's reasonable to suppose that 9 years on from that bug those SOAP servers aren't alone in rejecting (or mishandling) HTTP/1.0 requests, and moves to implement HTTP/2.0 will only make that more likely. I think it would be good to get this support into a better state, make it easier to switch on - e.g. with an INI setting, or some new stream URL syntax - and possibly make HTTP/1.1 the default in PHP 8.0. As I understand it, supporting HTTP/1.1 as a client requires the following mandatory features on top of HTTP/1.0: a) Send a "Host" header with every request. (RFC 7230 Section 5.4) b) Support persistent connections, or send "Connection: Close" with each request. (RFC 7230 Section 6.1) c) Ignore 1xx status lines (notably, "100 Continue") "even if the client does not expect one" (RFC 7231 Section 6.2) d) Support "chunked" transfer encoding (RFC 7230 Section 4.1) Let me know if there are any I've missed. I believe this is the current state of the PHP implementation: a) Host header always sent regardless of protocol version. b) The SOAP wrapper explicitly generates a "Connection: Close" header as well as forcing HTTP/1.1; users need to set it as a further option in stream_context_create(). It would be better if this was implicit with the protocol version. c) I can't find a bug report for this, but 100 Continue appears to be treated as an error status. This should be fixed. d) Chunked encoding itself was implemented, but the code for parsing headers was broken. If my PR [pr-1902] is approved, this will hopefully work correctly. What do people think? Would this be a worthwhile effort? [1]: Sample HTTP/1.1 context https://gist.github.com/IMSoP/8c62c97afc8765f418468b28a501409e [bug-43069]: https://bugs.php.net/bug.php?id=43069 [bug-47021]: https://bugs.php.net/bug.php?id=47021 [pr-1902]: https://github.com/php/php-src/pull/1902 Regards,
-- Rowan Collins [IMSoP]

Niklas Keller

9 years ago
Rowan Collins <rowan.collins@gmail.com> schrieb am So., 2. Okt. 2016, 23:12:
> Hi internallers! > > No, you haven't misread the subject line, I'm not talking about the > exciting new HTTP/2, but HTTP/1.1, 20 years old this year. > > To my surprise, PHP's HTTP stream wrapper (e.g. > file_get_contents('http://example.com');) defaults to sending HTTP/1.0 > requests. You can tell it to use HTTP/1.1 with an appropriate "stream > context", but it's fiddly [1], and the implementation is incomplete. > > There's actually already an exception to this: some SOAP servers were > discovered to only talk HTTP/1.1, so ext/soap forces the wrapper to > upgrade by default [bug-43069]. This in turn revealed some of the bugs > in the wrapper's protocol support, one of which I have recently created > a patch for [bug-47021]. > > It's reasonable to suppose that 9 years on from that bug those SOAP > servers aren't alone in rejecting (or mishandling) HTTP/1.0 requests, > and moves to implement HTTP/2.0 will only make that more likely. > > I think it would be good to get this support into a better state, make > it easier to switch on - e.g. with an INI setting, or some new stream > URL syntax - and possibly make HTTP/1.1 the default in PHP 8.0. > > > As I understand it, supporting HTTP/1.1 as a client requires the > following mandatory features on top of HTTP/1.0: > > a) Send a "Host" header with every request. (RFC 7230 Section 5.4) > b) Support persistent connections, or send "Connection: Close" with each > request. (RFC 7230 Section 6.1) > c) Ignore 1xx status lines (notably, "100 Continue") "even if the client > does not expect one" (RFC 7231 Section 6.2) > d) Support "chunked" transfer encoding (RFC 7230 Section 4.1) > > Let me know if there are any I've missed. > > I believe this is the current state of the PHP implementation: > > a) Host header always sent regardless of protocol version. > b) The SOAP wrapper explicitly generates a "Connection: Close" header as > well as forcing HTTP/1.1; users need to set it as a further option in > stream_context_create(). It would be better if this was implicit with > the protocol version. >
I think we already send it starting with 7.0, maybe just for 1.0, because there are some servers that respond correctly with an HTTP/1.1 response to an HTTP/1.0 request, but fail to close the connection and imply "Connection: close" for those requests. c) I can't find a bug report for this, but 100 Continue appears to be

Rowan Collins

9 years ago
Hi all, It sounds like the feeling so far is quite positive. I'll have another look for a dupe, then file a bug for the missing 100-continue support, and try to work on a patch. Once that's in place, and unless anyone comes up with other major features we need to support, we can discuss whether we should make it the default, or at least easier to opt into. On 03/10/2016 09:43, Niklas Keller wrote ...:
> I think we already send it starting with 7.0, maybe just for 1.0, > because there are some servers that respond correctly with an HTTP/1.1 > response to an HTTP/1.0 request, but fail to close the connection and > imply "Connection: close" for those requests.
... and on 05/10/2016 17:14, Andrea Faulds wrote:
> In order to connect to many modern websites, you need a Host: header, > and in order to use a Host: header, you need to use HTTP/1.1 (if you > don't, servers will sometimes send you back a 1.1 response anyway!).
Indeed, checking 7.0 suggests that we are unconditionally sending both a Host: header and Connection: Close with every request, while still claiming "HTTP/1.0". I believe both headers are technically backwards compatible with servers (i.e. ignored by) that genuinely speak HTTP/1.0, so this is not technically "wrong". However, the fact that it's necessary does strengthen the case for implementing HTTP/1.1 in full, rather than backporting features bit by bit. On 04/10/2016 09:55, Björn Larsson wrote:
> Would fixing > this behaviour also be applicable for HTTPS?
Yes, I'd be very surprised if this code wasn't shared between the two contexts. (It's a little more complicated than just piping an "http" connection through a "tls" one, because of things like SNI, but there'd be no reason to have the actual header processing code twice.) Regards,
-- Rowan Collins [IMSoP]

Kalle Sommer Nielsen

9 years ago
Hi Rowan On Oct 5, 2016 20:21, "Rowan Collins" <rowan.collins@gmail.com> wrote:
> > Hi all, > > It sounds like the feeling so far is quite positive. I'll have another
look for a dupe, then file a bug for the missing 100-continue support, and try to work on a patch. Once that's in place, and unless anyone comes up with other major features we need to support, we can discuss whether we should make it the default, or at least easier to opt into.
> > > > On 03/10/2016 09:43, Niklas Keller wrote ...: >> >> I think we already send it starting with 7.0, maybe just for 1.0,
because there are some servers that respond correctly with an HTTP/1.1 response to an HTTP/1.0 request, but fail to close the connection and imply "Connection: close" for those requests.
> > > ... and on 05/10/2016 17:14, Andrea Faulds wrote: >> >> In order to connect to many modern websites, you need a Host: header,
and in order to use a Host: header, you need to use HTTP/1.1 (if you don't, servers will sometimes send you back a 1.1 response anyway!).
> > > Indeed, checking 7.0 suggests that we are unconditionally sending both a
Host: header and Connection: Close with every request, while still claiming "HTTP/1.0". I believe both headers are technically backwards compatible with servers (i.e. ignored by) that genuinely speak HTTP/1.0, so this is not technically "wrong". However, the fact that it's necessary does strengthen the case for implementing HTTP/1.1 in full, rather than backporting features bit by bit.
> > > > > On 04/10/2016 09:55, Björn Larsson wrote: >> >> Would fixing >> this behaviour also be applicable for HTTPS? > > > Yes, I'd be very surprised if this code wasn't shared between the two
contexts. (It's a little more complicated than just piping an "http" connection through a "tls" one, because of things like SNI, but there'd be no reason to have the actual header processing code twice.) Afair then the code is separate in ext/standard/fopen_http_wrapper.c

Björn Larsson

9 years ago
Den 2016-10-02 kl. 23:12, skrev Rowan Collins:
> Hi internallers! > > No, you haven't misread the subject line, I'm not talking about the > exciting new HTTP/2, but HTTP/1.1, 20 years old this year. > > To my surprise, PHP's HTTP stream wrapper (e.g. > file_get_contents('http://example.com');) defaults to sending HTTP/1.0 > requests. You can tell it to use HTTP/1.1 with an appropriate "stream > context", but it's fiddly [1], and the implementation is incomplete. > > There's actually already an exception to this: some SOAP servers were > discovered to only talk HTTP/1.1, so ext/soap forces the wrapper to > upgrade by default [bug-43069]. This in turn revealed some of the bugs > in the wrapper's protocol support, one of which I have recently > created a patch for [bug-47021]. > > It's reasonable to suppose that 9 years on from that bug those SOAP > servers aren't alone in rejecting (or mishandling) HTTP/1.0 requests, > and moves to implement HTTP/2.0 will only make that more likely. > > I think it would be good to get this support into a better state, make > it easier to switch on - e.g. with an INI setting, or some new stream > URL syntax - and possibly make HTTP/1.1 the default in PHP 8.0. > > > As I understand it, supporting HTTP/1.1 as a client requires the > following mandatory features on top of HTTP/1.0: > > a) Send a "Host" header with every request. (RFC 7230 Section 5.4) > b) Support persistent connections, or send "Connection: Close" with > each request. (RFC 7230 Section 6.1) > c) Ignore 1xx status lines (notably, "100 Continue") "even if the > client does not expect one" (RFC 7231 Section 6.2) > d) Support "chunked" transfer encoding (RFC 7230 Section 4.1) > > Let me know if there are any I've missed. > > I believe this is the current state of the PHP implementation: > > a) Host header always sent regardless of protocol version. > b) The SOAP wrapper explicitly generates a "Connection: Close" header > as well as forcing HTTP/1.1; users need to set it as a further option > in stream_context_create(). It would be better if this was implicit > with the protocol version. > c) I can't find a bug report for this, but 100 Continue appears to be > treated as an error status. This should be fixed. > d) Chunked encoding itself was implemented, but the code for parsing > headers was broken. If my PR [pr-1902] is approved, this will > hopefully work correctly. > > > What do people think? Would this be a worthwhile effort? > > [1]: Sample HTTP/1.1 context > https://gist.github.com/IMSoP/8c62c97afc8765f418468b28a501409e > [bug-43069]: https://bugs.php.net/bug.php?id=43069 > [bug-47021]: https://bugs.php.net/bug.php?id=47021 > [pr-1902]: https://github.com/php/php-src/pull/1902 > > Regards, >
Yup, personally I do find this a worthwhile effort! Would fixing this behaviour also be applicable for HTTPS? Regards //Björn Larsson

Andrew Faulds

9 years ago
Hi Rowan, Rowan Collins wrote:
> I think it would be good to get this support into a better state, make > it easier to switch on - e.g. with an INI setting, or some new stream > URL syntax - and possibly make HTTP/1.1 the default in PHP 8.0.
HTTP/1.1 ought to be the default now, I think. The lack of the Host: header in HTTP/1.0 is pretty serious given how much of the web runs on vhosts now.
> As I understand it, supporting HTTP/1.1 as a client requires the > following mandatory features on top of HTTP/1.0: > > a) Send a "Host" header with every request. (RFC 7230 Section 5.4) > b) Support persistent connections, or send "Connection: Close" with each > request. (RFC 7230 Section 6.1) > c) Ignore 1xx status lines (notably, "100 Continue") "even if the client > does not expect one" (RFC 7231 Section 6.2) > d) Support "chunked" transfer encoding (RFC 7230 Section 4.1) > > Let me know if there are any I've missed.
I don't know about ones you've missed, but the Host header, persistent connections and chunked transfer were the main ones that caught me out when I wrote my own minimal HTTP client a while ago. In order to connect to many modern websites, you need a Host: header, and in order to use a Host: header, you need to use HTTP/1.1 (if you don't, servers will sometimes send you back a 1.1 response anyway!). In order to understand the output of those servers, you may need chunked transfer encoding, particularly if they're running dynamic code (say, PHP), and you need to specify the connection is non-persistent, lest you wait forever.
> What do people think? Would this be a worthwhile effort?
I think it's the absolute minimum we need to do to have it work properly at all. Using HTTP/1.0 alone probably means PHP's HTTP stream wrapper support is effectively broken at present, if my experiences are anything to go by. Thanks for bringing this up!
-- Andrea Faulds https://ajf.me/