CRITICAL UPDATE
While working on this, there'd been no spam relayed for over a single day so I left it up and overnight, TWO emails got relayed! ACK!
Here they are, from my tail / grep commands (specifics below):
Feb 13 02:30:01 fs1 postfix/smtp[749779]: 1442CBE4C12: to=<[email protected]>, relay=mail.s1.alexdf.ru[82.202.199.173]:25, delay=3.4, delays=0.03/0.06/2.4/0.84, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A8D104D1D)
Feb 13 06:22:24 fs1 postfix/smtp[793411]: 8702BBE4C12: to=<[email protected]>, relay=mail.s2.alexdf.ru[188.68.206.204]:25, delay=3, delays=0.03/0.06/2.2/0.8, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as DAD424D20)
So, I turned it off again. Further updates below.
Background
This question has a longer background with two other questions here as the troubled server was long established and I won't get into it AT ALL in this question, but how a working system got all screwed up and some of what's been done so far can be found here.
It's critical to note that at the time the spam crackers convinced the system to relay mail, we THOUGHT it was configured roughly as before, and nmap reported it wasn't an open relay! YET, as documented on that other page, some days later, the system was sending hundreds of thousands of relayed spam!
Regarding the other questions already asked,
the first was asking about how to stop it from being an open relay, which is, briefly put, to just configure Postfix thus:
default_transport = error: This server is presently not sending email
(Choose your own message, of course - this is not actually ours, but here, I'm being polite.)
The second asks How do I know for sure it's fixed, and, if not, fix it fast?, and it has an answer that may be correct but I don't feel I can trust. Here, it's not about testing the site; the question is regarding the interaction between the two packages as it relates to being used as an open relay, closing that, and returning service to valid users, even though the standard (albeit simple / simplistic) tests (such as from nmap) say it isn't an open relay - and never have that we've seen.
OF COURSE, the postfix and dovecot security related settings are all interrelated, but, well, we've been using these two packages on this system (updated as needed, of course) since the late 1990s, I personally set them up and have maintained them through all these years, updating, making tweaks, etc. But that doesn't mean I've kept up on all the changes over time and know what all the vulnerabilities are, and so forth. For example, I was unaware until recently of the bifurcation in v 2.1 of smtpd_recipient_restrictions which split relay issues to smtpd_relay_restrictions. But, I have surely read the vast majority of the official materials that are presently current. (I find some of it seems contradictory and / or unclear, but that comes with the territory.)
It may help to know the original installation had to be rebuilt from scratch in late January (less than a month ago), to the same OS version, and that shortly before that, it had gotten an update but hadn't yet been rebooted. So, at the time the saga began on Jan 19, 2023, we were running on our old settings and following a restoration, we accidentally got the default Dovecot configuration, and were puzzled why we had to hack on it enough to get our Dovecot-using users (mostly external) connecting in (though I'm not even sure that worked completely before the spammers got us).
This is a good spot to point out that the spammers seemed to crack us about a day after we hacked Dovecot's Auth section to get our remote users connecting, which was also about two days after the system came back up receiving email. We dropped the remote users ASAP in an effort to stop the spamming and valid remote connections have never worked since, though we thought that had been better configured later. A KEY GOAL now is getting our remote users connections working again WITHOUT making this an open relay again! (That's presuming, of course, it really isn't an open relay presently, but we thought we fixed this a few times already, only to have them return!)
The current versions are:
- Fedora Server v 37 with updates as of Jan 20 or so, 2023
- Kernel 6.0.7-301.fc37.x86_64
- postfix-3.7.3-1.fc37.x86_64
- dovecot-2.3.20-1.fc37.x86_64
Related But Ancillary Questions
While not strictly a part of the stated goals, I've been a little too distracted to do more than think about these: I would sure love to know how to restrict the outbound error messages to not send any reason for disconnection - the local logs are enough, thanks! They don't need to know there's "no such user", and so forth, but mailbox full is OK. And, I'd like to also know how I can permanently ban some IP addresses, just on my own observations of the IP addresses trying to crack my/our systems. For example, no IP address originating outside the USA has any validity connecting as a user, and in fact, the valid user list is short (though I don't know the exact IPs, and indeed CANNOT know the exact IPs of a number of valid users because of how most ISPs change IP addresses).
Present Configuration
In the already cited question page, someone said they thought it was "a stolen credential," and so ALL the credentials were replaced since we errantly hadn't been logging the auth (now we log ALL of it!), and had NO idea what credential may have been used. So, a brute force, trust none policy was implemented. The contents of said authorization materials are obviously beyond what should be shared here.
Before prattling off all the values, it's worth saying that at the time of this writing, NONE of the imap/imaps/pop3/pop3s connections are working. They're up, accepting connections but valid users can't get through and that has to be fixed ASAP.
Regarding what the two packages say about the config now, I used: postconf -f > file postconf -Ff > file and doveconf -n > file
The total contents are some 1404 lines! Too much to post here, of course. Just postfix's "main" entries comprise 1043 lines, however I found the heretofore unknown to me command LANG=C comm -23 <(postconf -n) <(postconf -d) > file which did an amazing job and returned only the postconf main entries that are not at the "compiled in default.", which are only 42 lines! So...
I went through it all and selected what seems appropriate here - if more is desired, just let me know. First Postfix's main config in order:
milter_default_action = accept
mydestination = $myhostname,
localhost.$mydomain,
localhost,
...60 more...
mynetworks = explicit list of IPs, no ranges
relay_domains = $mydestination,
...and most of the internal box IPs...
smtp_tls_security_level = may
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks,
check_helo_access hash:/etc/postfix/helo_access,
reject_invalid_helo_hostname,
reject_non_fqdn_helo_hostname
smtpd_recipient_restrictions = permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_pipelining,
reject_non_fqdn_recipient,
reject_unknown_recipient_domain,
reject_unauth_destination,
check_sender_access hash:/etc/postfix/sender_access,
check_client_access hash:/etc/postfix/pop-before-smtp,
permit_mynetworks
smtpd_relay_restrictions = permit_mynetworks,
reject_unauth_pipelining,
reject_non_fqdn_recipient,
reject_unknown_recipient_domain,
check_sender_access hash:/etc/postfix/sender_access,
check_client_access hash:/etc/postfix/pop-before-smtp,
permit_sasl_authenticated,
defer_unauth_destination
smtpd_sasl_auth_enable = yes
smtpd_sasl_tls_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_restrictions = permit_mynetworks,
reject_non_fqdn_sender,
reject_unknown_sender_domain,
permit_sasl_authenticated,
reject_unlisted_sender,
check_client_access hash:/etc/postfix/pop-before-smtp
smtpd_tls_security_level = may
Next up is Postfix's Master config. In culling through it I realized that the human-editable is easier to digest! Further, nearly all of it was the default, and I removed all of it except smtp, to set up a familiar pattern, so:
smtp inet n - n - - smtpd
submission inet n - n - - smtpd -o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes
policyd-spf unix - n n - 0 spawn user=policyd-spf
argv=/usr/libexec/postfix/policyd-spf
Then, Dovecot. It WAS my perception this is where "all the issues are", but unless there's something key not output by the above command, it looks pretty clean?! ... As with Postfix, I culled the parts that didn't seem pertinent to make this a lot easier to deal with. Here it is:
passdb {
args = scheme=cram-md5 /somefile
driver = passwd-file
}
passdb {
args = /anEMPTYfile (not using it yet)
driver = passwd-file
master = yes
pass = yes
}
service auth - matches postfix
service imap-login {
inet_listener imaps {
port = 993
ssl = yes
}
}
service pop3-login {
inet_listener pop3s {
port = 995
ssl = yes
}
}
service submission-login {
inet_listener submission {
port = 587
}
}
ssl = required
certs, including our virtual domains, omitted
My perception now is that the most likely issue is the ORDER of the permits and rejects in Postfix. I'll be going through that (again!) a lot more carefully shortly.
Ongoing Testing and Results
Of course, the next step is to get users to connect while monitoring to see if there's any more spam getting through. Regarding spammer monitoring, I'm using:
watch -n 1 -p -b qshape
And:
tail -f /var/log/maillog | grep 'status=sent' | grep 'relay=' | grep -v 'relay=local'
If there are better ways, please share! Meanwhile:
Results Of Testing:
Result 1
We tested both internal and external valid user attempts with the new track-it-all configuration produces "a mountain" of materials. However, when a known wrong password is used, we get a clear message of that, and when a correct one is used, we get NO confirmation and the dovecot imap-login: Login: user= repeats several times per single connect attempt with NO explanation why that I've found?! Strange!
This appears to begin with said imap-login message, which includes the correct user, method, rip, lip and what I take is a TLS session ID is provided.
The next related message says, Added userdb setting: plugin/=yes Hmmm... IDk if that means there's some configuration detail I should have included but it knows to add or what.
The connections are not successful. Continuing on...
Result 2
I noted that there was no userdb declared, and at this point, having already tried going back to an entire older config tree, I figured it wasn't (yet) needed (because we aren't yet doing virtual users, but hope to this year), however, Dovecot wasn't visibly rejecting connections with valid passwords (as it is doing with known wrong ones), so I wondered if they were in fact getting through and then there was no user directory or some such. So, I tried it using an example syntax.
Dovecot refused to restart, so I removed it, however you can find the template I adapted here.
Result 3
As highlighted presently at the very top, for emphasis: With the server up relatively unchanged for over a day, and seeing no forwards, I decided to trust it overnight, and upon returning to the computer, I found TWO had gotten through. ...These entries were highlighted by my log-tracking tail / grep command noted above:
Feb 13 02:30:01 fs1 postfix/smtp[749779]: 1442CBE4C12: to=<[email protected]>, relay=mail.s1.alexdf.ru[82.202.199.173]:25, delay=3.4, delays=0.03/0.06/2.4/0.84, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A8D104D1D)
Feb 13 06:22:24 fs1 postfix/smtp[793411]: 8702BBE4C12: to=<[email protected]>, relay=mail.s2.alexdf.ru[188.68.206.204]:25, delay=3, delays=0.03/0.06/2.2/0.8, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as DAD424D20)
I instantly stopped both Postfix and Dovecot. I'm far from an expert at this kind analysis, but I know how to run grep! I presume that for the first one I want to focus in on 1442CBE4C12 and/or A8D104D1D, and on the second 8702BBE4C12 and/or DAD424D20. Analysis begins...
I only got output from the log and nothing from the queues - they're apparently empty. (And I had to turn mail back on to check the queues, which surprised me, so I turned off sending, first!
On the first:
Feb 13 02:29:58 fs1 postfix/cleanup[749650]: 1442CBE4C12: message-id=<[email protected]>
Feb 13 02:29:58 fs1 postfix/bounce[749778]: 6248FBE48BE: sender non-delivery notification: 1442CBE4C12
Feb 13 02:29:58 fs1 postfix/qmgr[644284]: 1442CBE4C12: from=<>, size=49321, nrcpt=1 (queue active)
Feb 13 02:30:01 fs1 postfix/smtp[749779]: 1442CBE4C12: to=<[email protected]>, relay=mail.s1.alexdf.ru[82.202.199.173]:25, delay=3.4, delays=0.03/0.06/2.4/0.84, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A8D104D1D)
Feb 13 02:30:01 fs1 postfix/qmgr[644284]: 1442CBE4C12: removed
The second:
Feb 13 06:22:21 fs1 postfix/cleanup[793292]: 8702BBE4C12: message-id=<[email protected]>
Feb 13 06:22:21 fs1 postfix/bounce[793410]: CFFC7BE41DA: sender non-delivery notification: 8702BBE4C12
Feb 13 06:22:21 fs1 postfix/qmgr[644284]: 8702BBE4C12: from=<>, size=49321, nrcpt=1 (queue active)
Feb 13 06:22:24 fs1 postfix/smtp[793411]: 8702BBE4C12: to=<[email protected]>, relay=mail.s2.alexdf.ru[188.68.206.204]:25, delay=3, delays=0.03/0.06/2.2/0.8, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as DAD424D20)
Feb 13 06:22:24 fs1 postfix/qmgr[644284]: 8702BBE4C12: removed
This gave NOTHING USEFUL for finding out HOW the spammers managed to do this!
