2

CRITICAL UPDATE

While working on this, there'd been no spam relayed for over a single day so I left it up and overnight, TWO emails got relayed! ACK!

Here they are, from my tail / grep commands (specifics below):

Feb 13 02:30:01 fs1 postfix/smtp[749779]: 1442CBE4C12: to=<[email protected]>, relay=mail.s1.alexdf.ru[82.202.199.173]:25, delay=3.4, delays=0.03/0.06/2.4/0.84, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A8D104D1D)
Feb 13 06:22:24 fs1 postfix/smtp[793411]: 8702BBE4C12: to=<[email protected]>, relay=mail.s2.alexdf.ru[188.68.206.204]:25, delay=3, delays=0.03/0.06/2.2/0.8, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as DAD424D20)

So, I turned it off again. Further updates below.

Background

This question has a longer background with two other questions here as the troubled server was long established and I won't get into it AT ALL in this question, but how a working system got all screwed up and some of what's been done so far can be found here.

It's critical to note that at the time the spam crackers convinced the system to relay mail, we THOUGHT it was configured roughly as before, and nmap reported it wasn't an open relay! YET, as documented on that other page, some days later, the system was sending hundreds of thousands of relayed spam!

Regarding the other questions already asked, the first was asking about how to stop it from being an open relay, which is, briefly put, to just configure Postfix thus:

default_transport = error: This server is presently not sending email

(Choose your own message, of course - this is not actually ours, but here, I'm being polite.)

The second asks How do I know for sure it's fixed, and, if not, fix it fast?, and it has an answer that may be correct but I don't feel I can trust. Here, it's not about testing the site; the question is regarding the interaction between the two packages as it relates to being used as an open relay, closing that, and returning service to valid users, even though the standard (albeit simple / simplistic) tests (such as from nmap) say it isn't an open relay - and never have that we've seen.

OF COURSE, the postfix and dovecot security related settings are all interrelated, but, well, we've been using these two packages on this system (updated as needed, of course) since the late 1990s, I personally set them up and have maintained them through all these years, updating, making tweaks, etc. But that doesn't mean I've kept up on all the changes over time and know what all the vulnerabilities are, and so forth. For example, I was unaware until recently of the bifurcation in v 2.1 of smtpd_recipient_restrictions which split relay issues to smtpd_relay_restrictions. But, I have surely read the vast majority of the official materials that are presently current. (I find some of it seems contradictory and / or unclear, but that comes with the territory.)

It may help to know the original installation had to be rebuilt from scratch in late January (less than a month ago), to the same OS version, and that shortly before that, it had gotten an update but hadn't yet been rebooted. So, at the time the saga began on Jan 19, 2023, we were running on our old settings and following a restoration, we accidentally got the default Dovecot configuration, and were puzzled why we had to hack on it enough to get our Dovecot-using users (mostly external) connecting in (though I'm not even sure that worked completely before the spammers got us).

This is a good spot to point out that the spammers seemed to crack us about a day after we hacked Dovecot's Auth section to get our remote users connecting, which was also about two days after the system came back up receiving email. We dropped the remote users ASAP in an effort to stop the spamming and valid remote connections have never worked since, though we thought that had been better configured later. A KEY GOAL now is getting our remote users connections working again WITHOUT making this an open relay again! (That's presuming, of course, it really isn't an open relay presently, but we thought we fixed this a few times already, only to have them return!)

The current versions are:

  1. Fedora Server v 37 with updates as of Jan 20 or so, 2023
  2. Kernel 6.0.7-301.fc37.x86_64
  3. postfix-3.7.3-1.fc37.x86_64
  4. dovecot-2.3.20-1.fc37.x86_64

Related But Ancillary Questions

While not strictly a part of the stated goals, I've been a little too distracted to do more than think about these: I would sure love to know how to restrict the outbound error messages to not send any reason for disconnection - the local logs are enough, thanks! They don't need to know there's "no such user", and so forth, but mailbox full is OK. And, I'd like to also know how I can permanently ban some IP addresses, just on my own observations of the IP addresses trying to crack my/our systems. For example, no IP address originating outside the USA has any validity connecting as a user, and in fact, the valid user list is short (though I don't know the exact IPs, and indeed CANNOT know the exact IPs of a number of valid users because of how most ISPs change IP addresses).

Present Configuration

In the already cited question page, someone said they thought it was "a stolen credential," and so ALL the credentials were replaced since we errantly hadn't been logging the auth (now we log ALL of it!), and had NO idea what credential may have been used. So, a brute force, trust none policy was implemented. The contents of said authorization materials are obviously beyond what should be shared here.

Before prattling off all the values, it's worth saying that at the time of this writing, NONE of the imap/imaps/pop3/pop3s connections are working. They're up, accepting connections but valid users can't get through and that has to be fixed ASAP.

Regarding what the two packages say about the config now, I used: postconf -f > file postconf -Ff > file and doveconf -n > file

The total contents are some 1404 lines! Too much to post here, of course. Just postfix's "main" entries comprise 1043 lines, however I found the heretofore unknown to me command LANG=C comm -23 <(postconf -n) <(postconf -d) > file which did an amazing job and returned only the postconf main entries that are not at the "compiled in default.", which are only 42 lines! So...

I went through it all and selected what seems appropriate here - if more is desired, just let me know. First Postfix's main config in order:

milter_default_action = accept
mydestination = $myhostname,
 localhost.$mydomain,
 localhost,
 ...60 more...
mynetworks = explicit list of IPs, no ranges
relay_domains = $mydestination,
 ...and most of the internal box IPs...
smtp_tls_security_level = may
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks,
 check_helo_access hash:/etc/postfix/helo_access,
 reject_invalid_helo_hostname,
 reject_non_fqdn_helo_hostname
smtpd_recipient_restrictions = permit_mynetworks,
 permit_sasl_authenticated,
 reject_unauth_pipelining,
 reject_non_fqdn_recipient,
 reject_unknown_recipient_domain,
 reject_unauth_destination,
 check_sender_access hash:/etc/postfix/sender_access,
 check_client_access hash:/etc/postfix/pop-before-smtp,
 permit_mynetworks
smtpd_relay_restrictions = permit_mynetworks,
 reject_unauth_pipelining,
 reject_non_fqdn_recipient,
 reject_unknown_recipient_domain,
 check_sender_access hash:/etc/postfix/sender_access,
 check_client_access hash:/etc/postfix/pop-before-smtp,
 permit_sasl_authenticated,
 defer_unauth_destination
smtpd_sasl_auth_enable = yes
smtpd_sasl_tls_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_restrictions = permit_mynetworks,
 reject_non_fqdn_sender,
 reject_unknown_sender_domain,
 permit_sasl_authenticated,
 reject_unlisted_sender,
 check_client_access hash:/etc/postfix/pop-before-smtp
smtpd_tls_security_level = may

Next up is Postfix's Master config. In culling through it I realized that the human-editable is easier to digest! Further, nearly all of it was the default, and I removed all of it except smtp, to set up a familiar pattern, so:

smtp        inet n - n - - smtpd
submission  inet n - n - - smtpd -o syslog_name=postfix/submission
   -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes
policyd-spf unix - n n - 0 spawn user=policyd-spf
   argv=/usr/libexec/postfix/policyd-spf

Then, Dovecot. It WAS my perception this is where "all the issues are", but unless there's something key not output by the above command, it looks pretty clean?! ... As with Postfix, I culled the parts that didn't seem pertinent to make this a lot easier to deal with. Here it is:

passdb {
  args = scheme=cram-md5 /somefile
  driver = passwd-file
}

passdb {
  args = /anEMPTYfile (not using it yet)
  driver = passwd-file
  master = yes
  pass = yes
}

service auth - matches postfix

service imap-login {
  inet_listener imaps {
    port = 993
    ssl = yes
  }
}
service pop3-login {
  inet_listener pop3s {
    port = 995
    ssl = yes
  }
}
service submission-login {
  inet_listener submission {
    port = 587
  }
}
ssl = required

certs, including our virtual domains, omitted

My perception now is that the most likely issue is the ORDER of the permits and rejects in Postfix. I'll be going through that (again!) a lot more carefully shortly.

Ongoing Testing and Results

Of course, the next step is to get users to connect while monitoring to see if there's any more spam getting through. Regarding spammer monitoring, I'm using:

watch -n 1 -p -b qshape

And:

tail -f /var/log/maillog | grep 'status=sent' | grep 'relay=' | grep -v 'relay=local'

If there are better ways, please share! Meanwhile:

Results Of Testing:

Result 1

We tested both internal and external valid user attempts with the new track-it-all configuration produces "a mountain" of materials. However, when a known wrong password is used, we get a clear message of that, and when a correct one is used, we get NO confirmation and the dovecot imap-login: Login: user= repeats several times per single connect attempt with NO explanation why that I've found?! Strange!

This appears to begin with said imap-login message, which includes the correct user, method, rip, lip and what I take is a TLS session ID is provided.

The next related message says, Added userdb setting: plugin/=yes Hmmm... IDk if that means there's some configuration detail I should have included but it knows to add or what.

The connections are not successful. Continuing on...

Result 2

I noted that there was no userdb declared, and at this point, having already tried going back to an entire older config tree, I figured it wasn't (yet) needed (because we aren't yet doing virtual users, but hope to this year), however, Dovecot wasn't visibly rejecting connections with valid passwords (as it is doing with known wrong ones), so I wondered if they were in fact getting through and then there was no user directory or some such. So, I tried it using an example syntax.

Dovecot refused to restart, so I removed it, however you can find the template I adapted here.

Result 3

As highlighted presently at the very top, for emphasis: With the server up relatively unchanged for over a day, and seeing no forwards, I decided to trust it overnight, and upon returning to the computer, I found TWO had gotten through. ...These entries were highlighted by my log-tracking tail / grep command noted above:

Feb 13 02:30:01 fs1 postfix/smtp[749779]: 1442CBE4C12: to=<[email protected]>, relay=mail.s1.alexdf.ru[82.202.199.173]:25, delay=3.4, delays=0.03/0.06/2.4/0.84, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A8D104D1D)
Feb 13 06:22:24 fs1 postfix/smtp[793411]: 8702BBE4C12: to=<[email protected]>, relay=mail.s2.alexdf.ru[188.68.206.204]:25, delay=3, delays=0.03/0.06/2.2/0.8, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as DAD424D20)

I instantly stopped both Postfix and Dovecot. I'm far from an expert at this kind analysis, but I know how to run grep! I presume that for the first one I want to focus in on 1442CBE4C12 and/or A8D104D1D, and on the second 8702BBE4C12 and/or DAD424D20. Analysis begins...

I only got output from the log and nothing from the queues - they're apparently empty. (And I had to turn mail back on to check the queues, which surprised me, so I turned off sending, first!

On the first:

Feb 13 02:29:58 fs1 postfix/cleanup[749650]: 1442CBE4C12: message-id=<[email protected]>
Feb 13 02:29:58 fs1 postfix/bounce[749778]: 6248FBE48BE: sender non-delivery notification: 1442CBE4C12
Feb 13 02:29:58 fs1 postfix/qmgr[644284]: 1442CBE4C12: from=<>, size=49321, nrcpt=1 (queue active)
Feb 13 02:30:01 fs1 postfix/smtp[749779]: 1442CBE4C12: to=<[email protected]>, relay=mail.s1.alexdf.ru[82.202.199.173]:25, delay=3.4, delays=0.03/0.06/2.4/0.84, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A8D104D1D)
Feb 13 02:30:01 fs1 postfix/qmgr[644284]: 1442CBE4C12: removed

The second:

Feb 13 06:22:21 fs1 postfix/cleanup[793292]: 8702BBE4C12: message-id=<[email protected]>
Feb 13 06:22:21 fs1 postfix/bounce[793410]: CFFC7BE41DA: sender non-delivery notification: 8702BBE4C12
Feb 13 06:22:21 fs1 postfix/qmgr[644284]: 8702BBE4C12: from=<>, size=49321, nrcpt=1 (queue active)
Feb 13 06:22:24 fs1 postfix/smtp[793411]: 8702BBE4C12: to=<[email protected]>, relay=mail.s2.alexdf.ru[188.68.206.204]:25, delay=3, delays=0.03/0.06/2.2/0.8, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as DAD424D20)
Feb 13 06:22:24 fs1 postfix/qmgr[644284]: 8702BBE4C12: removed

This gave NOTHING USEFUL for finding out HOW the spammers managed to do this!

1 Answer 1

0

Gawd this was painful! However I discovered:

Some Important Details

These were not easily realized...

First

I'm fairly sure this system was NOT an official open mail relay, despite the relays it did, though I'm going to try and confirm that. (More on this below.)

Second

I was wrong, and nobody corrected me, regarding the default_transport Postfix parameter; when set, it blocks ALL inbound and outbound from and to anything but the system it's running on ("localhost"). (That is ONLY a from and to on localhost works.)

In addition, default_transport, when set, also switches to a hard rejection, code 550, so no further attempts are supposed to be tried. And therefore it's kind of dangerous to use as I had previously suggested. (I didn't try mitigating it with soft_bounce, but that would likely work, however, it overrode my other error code setting, unknown_local_recipient_reject_code = 450, so test to be sure.)

Third

"The way" to block all sending but permit reception is to use both of the since Postfix v 2.1 bifurcated pair of smtpd_relay_restrictions and smtpd_recipient_restrictions, and also note that the documentation regarding "backward compatibility" is quite confusing - I wasn't the only person who's been at this for decades who gave that a pause and decided the only way to understand not only the documentation but also both these parameters was to probe the actual behavior. ... This is an important lesson for those who would pursue this kind of work in any professional form: the documentation is all too often wrong or confusing, people are all too certain about what just isn't the case, and the only genuine arbiter of this kind of truth is the actual behavior of the software.

Fourth

I was fooled by the substantial volume of inbound spam and so I incorrectly concluded email reception was still OK when Postfix was in fact rejecting ALL inbound mail that's following anything like the usual rules.

I'm still not sure how it was getting in when no ordinary email could, but it's not your ordinary spam, for example, there were no from or return-path addresses, we just got open and closing angle-brackets in the log with no content.

Nor am I sure just how to prevent reception, but a trusty SpamAssassin installation should do the trick.

Fifth

We continue to get a few of these now and then:

Feb 18 01:42:58 <our_server> postfix/smtp[2188531]: 2B647BE41F6: to=<[email protected]>, relay=mail.s1.alexdf.ru[5.53.127.107]:25, delay=4.2, delays=0.06/0.06/2.1/1.9, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 806D64DA7)
Feb 18 08:57:36 <our_server> postfix/smtp[2270044]: 4E1F2BE41F7: to=<[email protected]>, relay=mail.s1.alexdf.ru[188.68.219.83]:25, delay=3.4, delays=0.03/0.06/2.5/0.8, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as F01A54DDC)

It's pretty close to the same pattern every time with a few subtle variations so I think these are fishing for whether or not they can exploit the system again. These are never seen when default_transport is set to a non-empty value, so they should be able to be blocked somehow, as noted above. I'm seeking to explicitly reject these, but that's a point-in-time solution!

Conclusion

While I never found a true cause or solution, I'm still on the hunt for it and if I find one I'll be sure to update this answer. However, I think there's enough here to be useful for others, such as knowing that you can turn it all off, as with default_transport and STILL get spam. This is a bizarre thing to see, to be sure!

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .