Fail2Ban: RegEx to filter all 404 errors out of the Apache-log

Question

In my logs I regularly see loads of 404-errors where bots obviously do scan the server systematically for specific software installations. As this also causes loads of traffic, I want to ban them.

So based on some HowTos I tried to find a RegEx for fail2ban which filters these 404-requests out of the Apache-logs and based on this blocks the related IPs.

Unfortunately none of the RegEx patterns I found and tried do work. So as an example:

fail2ban-regex /var/log/apache2/other*access.log '^<HOST> - .* "(GET|POST|HEAD).*HTTP.*" 404 .*$'

(which looks good to me) does not find anything:

Date template hits:
|- [# of hits] date format
|  [1210] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 1210 lines, 0 ignored, 0 matched, 1210 missed

There defintely are several 404-Errors in that logfile. So what Am I doing wrong? How should a proper RegEx look lie to get all the 404s?

Thanks!

Answer 1

other_vhosts_access.log by default has a different LogFormat vhost_combined which is prefixed by a vhost:port when compared with the standard common LogFormat. You need to modify the regex accordingly.

Answer 2

Have you tried this one

failregex = ^<HOST> - .* "(GET|POST|HEAD).*HTTP.*" 404 .*$
ignoreregex =.*(robots.txt|favicon.ico|jpg|png)