What's up with emails?

Deleu

2 years ago

On Thu, Feb 22, 2024 at 10:55 AM Robert Landers <landers.robert@gmail.com>
wrote:

> I have noticed quite a number of emails appearing on externals but not
> appearing in my inbox. I randomly get an email saying:
>
> Hi, this is the Mlmmj program managing the <internals@lists.php.net>
> mailing list.
>
> Some messages to you could not be delivered. If you're seeing this
> message it means things are back to normal, and it's merely for your
> information.
>
> Here is the list of the bounced messages:
> - 122446
> - 122448
>
> But I can clearly see far more than that in externals that are never
> received.
>
> IIRC from the email spec and internet customs, soft-bounces should be
> retried for a number of days (though I have no idea why gmail
> soft-bounced the messages, maybe rate-limiting?).
>
> If I'm using gmail and not getting emails, how many other people are
> also not getting emails on this list?
>
> Robert Landers
> Software Engineer
> Utrecht NL
>

I'm also affected by this exact same thing.

[image: image.png]

--
Marco Deleu

Erick de Azevedo Lima

2 years ago

> If I'm using gmail and not getting emails, how many other people are
> also not getting emails on this list?

Same here.

Derick Rethans

2 years ago

On 22 February 2024 08:52:49 GMT-05:00, Robert Landers <landers.robert@gmail.com> wrote:

>I have noticed quite a number of emails appearing on externals but not
>appearing in my inbox. I randomly get an email saying:
>
>Hi, this is the Mlmmj program managing the <internals@lists.php.net>
>mailing list.
>
>Some messages to you could not be delivered. If you're seeing this
>message it means things are back to normal, and it's merely for your
>information.
>
>Here is the list of the bounced messages:
>- 122446
>- 122448
>
>But I can clearly see far more than that in externals that are never received.
>
>IIRC from the email spec and internet customs, soft-bounces should be
>retried for a number of days (though I have no idea why gmail
>soft-bounced the messages, maybe rate-limiting?).
>
>If I'm using gmail and not getting emails, how many other people are
>also not getting emails on this list?

Lots.

Gmail is rejecting emails after we moved the servers without telling us why, in enough detail to do anything about it.

We can put a delay in delivering to Gmail, but that means mails get delayed for days.

So my advice would for now to not rely on Gmail while we figure out how to improve this.

cheers
Derick

Jeffrey Dafoe

2 years ago

> Gmail is rejecting emails after we moved the servers without telling us why, in
> enough detail to do anything about it.

It's not just gmail.

-Jeff

Michael Kliewe

2 years ago

Am 22.02.2024 um 15:14 schrieb Jeffrey Dafoe:

>> Gmail is rejecting emails after we moved the servers without telling us why, in
>> enough detail to do anything about it.
> It's not just gmail.

I guess it's because of a wrong setting in the mailinglist server.

The content of the original email is changed (the subject is prepended
by "[PHP-DEV]"), which breaks the original DKIM signature by the sender.

A new DKIM-signature is added by php.net, but it's not aligned to the
From:-header domain (which is still the original sender).

So the email doesn't have a valid ALIGNED DKIM signature.

SPF is valid for the envelope from, but is not aligned to the From:-header.

The result is:

There is no aligned SPF nor aligned DKIM, which results in a dmarc=fail.

Providers which honor DMARC will quarantine or reject these
unauthorized/forged emails.

There are 2 solutions:

1. Don't change the content or the DKIM-signed headers of the email (do
not prepend something in the subject). Then the original DKIM signature
stays valid and the From:-header can stay untouched

2. If you change the the email and break the original DKIM signature,
also change the From:-header to a domain which matches the new DKIM
signature, in this case set the From:-header to internals@lists.php.net.
This is called "munging" in mailinglist software.

Michael

Tim Düsterhus

2 years ago

Hi

On 2/22/24 15:44, Michael Kliewe wrote:

> 1. Don't change the content or the DKIM-signed headers of the email (do
> not prepend something in the subject). Then the original DKIM signature
> stays valid and the From:-header can stay untouched
>

Strong +1 for that one. I've suggested the same in the past,
unfortunately without much success so far.

Best regards
Tim Düsterhus

tag Knife

2 years ago

On Thu, 22 Feb 2024 at 14:12, Derick Rethans <derick@php.net> wrote:

> On 22 February 2024 08:52:49 GMT-05:00, Robert Landers <
> landers.robert@gmail.com> wrote:
> >I have noticed quite a number of emails appearing on externals but not
> >appearing in my inbox. I randomly get an email saying:
> >
> >Hi, this is the Mlmmj program managing the <internals@lists.php.net>
> >mailing list.
> >
> >Some messages to you could not be delivered. If you're seeing this
> >message it means things are back to normal, and it's merely for your
> >information.
> >
> >Here is the list of the bounced messages:
> >- 122446
> >- 122448
> >
> >But I can clearly see far more than that in externals that are never
> received.
> >
> >IIRC from the email spec and internet customs, soft-bounces should be
> >retried for a number of days (though I have no idea why gmail
> >soft-bounced the messages, maybe rate-limiting?).
> >
> >If I'm using gmail and not getting emails, how many other people are
> >also not getting emails on this list?
>
>
> Lots.
>
> Gmail is rejecting emails after we moved the servers without telling us
> why, in enough detail to do anything about it.
>
> We can put a delay in delivering to Gmail, but that means mails get
> delayed for days.
>
> So my advice would for now to not rely on Gmail while we figure out how to
> improve this.
>
> cheers
> Derick
>

This could be related to the new servers new ip not being trusted. Did you
do any sort of IP Warming
<https://docs.sendgrid.com/ui/sending-email/warming-up-an-ip-address> for
the new server?
As without this, Google sees a new IP sending thousands of emails out of
nowhere and it triggers spam filters. We have
to do this at work when we onboard a new client, since we give all clients
apps there own IP, we have to run a 14 day warming
system before we can let the client go live, since without it, their emails
will most likely be blocked for spam from various inbox
providers.

Alain Williams

2 years ago

On Thu, Feb 22, 2024 at 09:05:22AM -0500, Derick Rethans wrote:

> Gmail is rejecting emails after we moved the servers without telling us why, in enough detail to do anything about it.

I run other mail lists and have had similar problems with gmail recently.

The problem seems to be SPF. This says which IP addresses can be used for
sending email for particular domains. gmail has recently become much stricter.

List emails are sent from php-smtp4.php.net (45.112.84.5).

However the 'From' address is still that of the original sender -- which is OK
for the email that I am replying to as it is
From: Derick Rethans <derick@php.net>

and the MX for php.net is php-smtp4-ip4.php.net. which has address 45.112.84.5

However I suspect that *this* email (the one that I am typing now) will get
bounced by gmail as my address is: addw@phcomp.co.uk and 45.112.84.5 is not
allowed (by SPF) to send email for phcomp.co.uk.

So: how to fix ?

Make the From: address internals@lists.php.net

The envelope address is already, something like
internals+bounces-122457-addw=phcomp.co.uk@lists.php.net

This means a fix to whatever list processor that you are using.

Other recommendations:

• MX for lists.php.net is php-smtp4-ip4.php.net.
Change to php-smtp4.php.net.

• SPF for lists.php.net is "v=spf1 a mx a:osu1php.osuosl.org. ip4:45.112.84.5 ip6:2a02:cb43:8000::1102"
Change to "v=spf1 a mx -all"

•• osu1php.osuosl.org. does not seem to exist - so zap it
•• If you make the MX change that I suggest then the
following are redundant: ip4:45.112.84.5 ip6:2a02:cb43:8000::1102
•• -all means no other addresses accepted. gmail started making that implicit in
the last few months, it is this that is causing the current problems.

Once that has settled down you should enable sending email from (& thus to)
IPv6 addresses. This is the way that the world is moving. It is not hard.

Regards

--
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256  https://www.phcomp.co.uk/
Parliament Hill Computers. Registration Information: https://www.phcomp.co.uk/Contact.html
#include <std_disclaimer.h>

Michael Cordover

2 years ago

On Thu, 2024-02-22 at 10:29 -05:00, Alain D D Williams <addw@phcomp.co.uk> wrote:

> On Thu, Feb 22, 2024 at 09:05:22AM -0500, Derick Rethans wrote:
>
>> Gmail is rejecting emails after we moved the servers without telling us why, in enough detail to do anything about it.
>
> I run other mail lists and have had similar problems with gmail recently.

Gmail explicitly tightened their requirements for senders this month, so while it's likely the server change is related, it may not be the entire cause.

Specifically, they require:

- valid spf, with specific rules for quarantining email that has @gmail.com From but doesn't pass spf
- valid dkim
- dmarc configured if the server sends more than 5k emails to Gmail

They also equivocally indicate a requirement for ARC and a List-Id header.

This is all at https://support.google.com/a/answer/81126.

I think ARC is a must; without it we'll never get to passing dkim at least, even if we rewrite from so spf passes.

I'm happy to pitch in to help getting this configured if that's helpful, though I'm also very conscious that too many cooks is often a greater harm than good when it comes to administration. But reach out if you think I can be helpful.

mjec

Derick Rethans

2 years ago

On Thu, 22 Feb 2024, mjec wrote:

> On Thu, 2024-02-22 at 10:29 -05:00, Alain D D Williams <addw@phcomp.co.uk> wrote:
>> On Thu, Feb 22, 2024 at 09:05:22AM -0500, Derick Rethans wrote:
>>
>>> Gmail is rejecting emails after we moved the servers without telling
>>> us why, in enough detail to do anything about it.
>>
>> I run other mail lists and have had similar problems with gmail
>> recently.
>
> Gmail explicitly tightened their requirements for senders this month,
> so while it's likely the server change is related, it may not be the
> entire cause.
>
> Specifically, they require:
>
> - valid spf, with specific rules for quarantining email that has @gmail.com From but doesn't pass spf
> - valid dkim
> - dmarc configured if the server sends more than 5k emails to Gmail

We should have all of that though?

>
> They also equivocally indicate a requirement for ARC and a List-Id header.

We have a List-Id header, but I've not even heard of ARC. It is on our
list to investigate though. Our SMTP logs definitely don't say anything
about this though.

> This is all at https://support.google.com/a/answer/81126.
>
> I think ARC is a must; without it we'll never get to passing dkim at
> least, even if we rewrite from so spf passes.

But SPF is now passing? The postmaster tools show this:

Date	DKIM success rate	SPF success rate	DMARC success
rate
15 Feb 2024	100.0%	100.0%	0.0%
16 Feb 2024	100.0%	100.0%	0.0%
17 Feb 2024	100.0%	100.0%	0.0%
18 Feb 2024	100.0%	100.0%	0.0%
19 Feb 2024	100.0%	100.0%	100.0%
20 Feb 2024	100.0%	100.0%	100.0%

> I'm happy to pitch in to help getting this configured if that's
> helpful, though I'm also very conscious that too many cooks is often a
> greater harm than good when it comes to administration. But reach out
> if you think I can be helpful.

We could definitely use some people that now email delivery.

cheers,
Derick

Andreas Heigl

2 years ago

On 22 February 2024 17:49:10 CET, Derick Rethans <derick@php.net> wrote:

> On Thu, 22 Feb 2024, mjec wrote:
>
> > On Thu, 2024-02-22 at 10:29 -05:00, Alain D D Williams <addw@phcomp.co.uk> wrote:
> >> On Thu, Feb 22, 2024 at 09:05:22AM -0500, Derick Rethans wrote:
> >>
> >>> Gmail is rejecting emails after we moved the servers without telling us why, in enough detail to do anything about it.
> >>
> >> I run other mail lists and have had similar problems with gmail recently.
> >
> > Gmail explicitly tightened their requirements for senders this month, so while it's likely the server change is related, it may not be the entire cause.
> >
> > Specifically, they require:
> >
> > - valid spf, with specific rules for quarantining email that has @gmail.com From but doesn't pass spf
> > - valid dkim
> > - dmarc configured if the server sends more than 5k emails to Gmail
>
> We should have all of that though?
>
> >
> > They also equivocally indicate a requirement for ARC and a List-Id header.
>
> We have a List-Id header, but I've not even heard of ARC. It is on our list to investigate though. Our SMTP logs definitely don't say anything about this though.
>
> > This is all at https://support.google.com/a/answer/81126.
> >
> > I think ARC is a must; without it we'll never get to passing dkim at least, even if we rewrite from so spf passes.
>
> But SPF is now passing? The postmaster tools show this:
>
> Date	DKIM success rate	SPF success rate	DMARC success rate
> 15 Feb 2024	100.0%	100.0%	0.0%
> 16 Feb 2024	100.0%	100.0%	0.0%
> 17 Feb 2024	100.0%	100.0%	0.0%
> 18 Feb 2024	100.0%	100.0%	0.0%
> 19 Feb 2024	100.0%	100.0%	100.0%
> 20 Feb 2024	100.0%	100.0%	100.0%
>
> > I'm happy to pitch in to help getting this configured if that's helpful, though I'm also very conscious that too many cooks is often a greater harm than good when it comes to administration. But reach out if you think I can be helpful.
>
> We could definitely use some people that now email delivery.

The main issue is that the things came together:

Us moving to a new listserver and
Gmail introducing new ways of fighting spam.

The later is what makes life hars now and would also have caused trouble on the old server.

We are currently in the process of implementig all of the stuff that gmail expects us to have.

The last ones are ARC and one-click unsubscribe.

If you want to know more about the hoops we have to jump in addition to moving the list to a new server check out https://blog.google/products/gmail/gmail-security-authentication-spam-protection/

Cheers

Andreas

--
Andreas Heigl

Derick Rethans

2 years ago

On Thu, 22 Feb 2024, Alain D D Williams wrote:

> On Thu, Feb 22, 2024 at 09:05:22AM -0500, Derick Rethans wrote:
>
>> Gmail is rejecting emails after we moved the servers without telling us why, in enough detail to do anything about it.
>
> I run other mail lists and have had similar problems with gmail recently.
>
> The problem seems to be SPF. This says which IP addresses can be used for
> sending email for particular domains. gmail has recently become much stricter.
>
> List emails are sent from php-smtp4.php.net (45.112.84.5).
>
> However the 'From' address is still that of the original sender -- which is OK
> for the email that I am replying to as it is
> 	From: Derick Rethans <derick@php.net>
>
> and the MX for php.net is php-smtp4-ip4.php.net. which has address 45.112.84.5
>
> However I suspect that *this* email (the one that I am typing now) will get
> bounced by gmail as my address is: addw@phcomp.co.uk and 45.112.84.5 is not
> allowed (by SPF) to send email for phcomp.co.uk.
>
> So: how to fix ?
>
> Make the From: address internals@lists.php.net

We don't want to change the From address, as that means that replies go
to the list, rather than the original sender. We never had to do any of
that before.

> The envelope address is already, something like
> internals+bounces-122457-addw=phcomp.co.uk@lists.php.net
>
> This means a fix to whatever list processor that you are using.
>
> Other recommendations:
>
> • MX for lists.php.net is php-smtp4-ip4.php.net.
> Change to php-smtp4.php.net.

Done

> • SPF for lists.php.net is "v=spf1 a mx a:osu1php.osuosl.org. ip4:45.112.84.5 ip6:2a02:cb43:8000::1102"
> Change to "v=spf1 a mx -all"

Done, but I kept it as ~all.

> •• osu1php.osuosl.org. does not seem to exist - so zap it

Done

> •• If you make the MX change that I suggest then the
> following are redundant: ip4:45.112.84.5 ip6:2a02:cb43:8000::1102

Done

> •• -all means no other addresses accepted. gmail started making that implicit in
> the last few months, it is this that is causing the current problems.

I've added ~all instead.

> Once that has settled down you should enable sending email from (&
> thus to) IPv6 addresses. This is the way that the world is moving. It
> is not hard.

Done that now too.

Fingers crossed to see if any of this is helping.

cheers,
Derick

Alain Williams

2 years ago

On Thu, Feb 22, 2024 at 04:46:04PM +0000, Derick Rethans wrote:

> > Make the From: address internals@lists.php.net
>
> We don't want to change the From address, as that means that replies go to
> the list, rather than the original sender. We never had to do any of that
> before.

I understand that, it also means that someone looking at a list of emails in a
mail box sees who (ie individuals) they are from rather than an email list.

> Fingers crossed to see if any of this is helping.

If things do not improve - change the From address - just for a day to see if
that makes things better.

--
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256  https://www.phcomp.co.uk/
Parliament Hill Computers. Registration Information: https://www.phcomp.co.uk/Contact.html
#include <std_disclaimer.h>

Tim Düsterhus

2 years ago

Hi

On 2/22/24 16:29, Alain D D Williams wrote:

> • SPF for lists.php.net is "v=spf1 a mx a:osu1php.osuosl.org. ip4:45.112.84.5 ip6:2a02:cb43:8000::1102"
> Change to "v=spf1 a mx -all"
>
> •• If you make the MX change that I suggest then the
> following are redundant: ip4:45.112.84.5 ip6:2a02:cb43:8000::1102

I disagree with that. An MX is the receiver for inbound mail. It does
not follow that emails are sent out via the MX. Removing the explicit
listing of the IP addresses will just cause accidents in the future, if
the MX no longer happens to also send email.

If anything the 'a' within the SPF record could be removed, because
that's the HTTP box on DigitalOcean.

Best regards
Tim Düsterhus

Alain Williams

2 years ago

On Thu, Feb 22, 2024 at 07:45:41PM +0100, Tim Düsterhus wrote:

> Hi
>
> On 2/22/24 16:29, Alain D D Williams wrote:
> > • SPF for lists.php.net is "v=spf1 a mx a:osu1php.osuosl.org. ip4:45.112.84.5 ip6:2a02:cb43:8000::1102"
> > Change to "v=spf1 a mx -all"
> >
> > •• If you make the MX change that I suggest then the
> > following are redundant: ip4:45.112.84.5 ip6:2a02:cb43:8000::1102
>
> I disagree with that. An MX is the receiver for inbound mail. It does not
> follow that emails are sent out via the MX. Removing the explicit listing of
> the IP addresses will just cause accidents in the future, if the MX no
> longer happens to also send email.

If the setup changes and the MX no longer send email then the SPF record can be
updated. For now it is a quick way of saying a:php-smtp4.php.net.

> If anything the 'a' within the SPF record could be removed, because that's
> the HTTP box on DigitalOcean.

"a" allows lists.php.net. to send email for the domain. I do not know if it
does but could easily believe that administrative/error/... type messages come
from there.

Regards

--
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256  https://www.phcomp.co.uk/
Parliament Hill Computers. Registration Information: https://www.phcomp.co.uk/Contact.html
#include <std_disclaimer.h>

Jeffrey Dafoe

2 years ago

I had an email to the list bounce due to SpamCop adding a sizeable portion of Outlook's IP space into a block (as they've been erroneously doing for 25 years now). Emailed systems@php.net about it, never heard back. I'm also receiving next-to-no list traffic and receiving similar bounce messages as what is shown below.

-Jeff

Daniil Gentili

2 years ago

> If I'm using gmail and not getting emails, how many other people are
> also not getting emails on this list?
>

Same here, with both gmail and zoho mail.

Personally I would switch to github issues and discussions, even ignoring all the deliverability issues which I had even before the switch to the new mailing list server on my zoho mail daniil@daniil.it account, I don't think using a mailing list is a good idea for a modern programming language, seeking new contributors and new ideas in 2024.

VCS was already moved to github after the recent hack of the php VCS, a lot of technical internals-related discussion is already using exclusively github issues and pull request discussions, I believe that the mailing list is nothing more than a redundant relic of the past.

Regards,
Daniil Gentili

Rowan Collins

2 years ago

This is a test e-mail from a subscribed GMail address, to see if the
"451: Temporary lookup failure" errors are now resolved.

Thanks to those working on it!

--
Rowan Tommins
[IMSoP]

Tim Düsterhus

2 years ago

Hi

On 2/27/24 18:49, Rowan Tommins wrote:

> This is a test e-mail from a subscribed GMail address, to see if the
> "451: Temporary lookup failure" errors are now resolved.
>

Email successfully received, let's see whether my response is delivered
back to you via the list.

Best regards
Tim Düsterhus