Forum archive
What is the default state of Fail2Ban on a new server (e.g. WordPress server)?
Hi Jeremy,
I am attempting to configure some WordPress related Fail2Ban rules and not yet having much success (I have configured the rules as per spec / online examples but the rules have no effect). I will plug away at it but it would be helpful if I had a couple of answers to some context questions.
Q1: Is Fail2Ban / Linux Firewall (IpTables) enabled by default?
I have configured some Fail2Ban rules that trigger an IP address block using "action: iptables-allports" but don't see the rule in the iptables rule list in the Linux Firewall. If a Fail2Ban rule is added that creates an ip address block rule in iptables, shouldn't that iptable rule be visible in the Webmin Linux Firewall screen (maybe it's there and I am not seeing it)?
I also notice in the Webmin Linux Firewall screen the following:
Is iptables not enabled by default and thus, in turn, Fail2Ban does nothing?
Q2: Is Fail2Ban rule regex a subset of general regex semantics and, if so, is there a Fail2Ban regex doco site?
There seems to be some debate on the web about the structure of a Fail2Ban rule. Some claim that conditionals like "(POST | GET)" are not supported. In other examples it looks like the Fail2Ban rule can be structured as a set of parse strings seperated by line break (but unclear if that would be and AND or an OR).
Q3: What is the Fail2Ban polling interval and is it completed by CRON job?
What is the polling mechanism and duration and where is it configured or does it use inotify? It looks like it relies on systemd journals...in jail.local DEFAULT the backend=systemd is set.
If systemd is in place do we add "journalmatch = _SYSTEMD_UNIT=httpd.service" or "journalmatch = _SYSTEMD_UNIT=apache2.service"? Poking around it is not clear if apache is logging to systemd journal...it looks like start / stop type messages may be being logged but not http/https traffic...not sure...
==============================
UPDATE
==============================
I have sorted out a bunch of poorly documented jail.local quirks and believe I have a proper jail and filter configured as per fail2ban-regex test but still trying to get the jail action to trigger on the IP address of the matching records found by fail2ban-regex check. There is anecdotal net chatter that adding backend = polling to each jail in the jail.local file make the jail work as expected though inotify would be more efficient I believe. Will update with the successful config once it is working. Still interested in answers to Q1-Q3 above though :-)
I think I have everything configured correctly but still not banning any IP addresses...my action is to call iptables-allports action...wait a minute...the Ban Command for action iptables-allports is blank...there's a chunk of my life I won't get back. Why would it be blank! Now I have to learn iptables commands...ahhh crap.
Update2: I finally got IP addresses hitting xmlrpc.php to be banned. Had to tweak the regex, set usedns = no, use iptables[type=all, and set backend=auto to finally get filtered IP addresses blocked. Phew...what a tweaky marginally documented, brittle mess.
I am still interested in the answer to the afore-mentioned question which is why the default is systemd if apache is not logging http/https traffic to the systemd journal. Should we ne logging apache http/https traffic to systemd journal?
Update3: I spoke to soon...IP address is listed as being blocked in the Jails Status page...but I can still access the home page of the site using that "blocked" IP address. It is like Fail2ban thinks it is blocked but the iptables[type=all] action did nothing in iptables. I ran the sudo iptables -L -v command but it did not list the IP address as being banned. So...back to the Googling / ChatGPTing .
==============================
SUMMARY
==============================
Jeremy: You can consider this issue closed for now. After many hours of ChatGPT, Google A.I. and old school Google searchs and doc reading I got the basics sorted. It's a brittle construct...easy to break or miss something. I get now that systemd is the default and is useful for some f2b jails. For other Jails, like mine, you need to override with backend=auto to trigger it. Without pyinotify installed it defaults to poller(). You also need to set usedns=no or you can get some weird results and some extra overhead for a reverse DNS lookup. Stay away from Actions...I lost a day on the following action that was failing with a python error "action = iptables[name=wp-xmlrpc, type=all, blocktype=DROP, protocol=tcp, returntype=RETURN]". Also ignore banaction=, iptables-allports, iptables-multiport which have been deprecated for the action above...which fails with a python error.
/etc/fail2ban/filter.d/wordpress-xmlrpc.conf
/etc/fail2ban/jail.local
I am attempting to configure some WordPress related Fail2Ban rules and not yet having much success (I have configured the rules as per spec / online examples but the rules have no effect). I will plug away at it but it would be helpful if I had a couple of answers to some context questions.
Q1: Is Fail2Ban / Linux Firewall (IpTables) enabled by default?
I have configured some Fail2Ban rules that trigger an IP address block using "action: iptables-allports" but don't see the rule in the iptables rule list in the Linux Firewall. If a Fail2Ban rule is added that creates an ip address block rule in iptables, shouldn't that iptable rule be visible in the Webmin Linux Firewall screen (maybe it's there and I am not seeing it)?
I also notice in the Webmin Linux Firewall screen the following:
[ACTIVATE AT BOOT] () Yes (X) No Change this option to control whether your firewall is activated at boot time or not.Is iptables not enabled by default and thus, in turn, Fail2Ban does nothing?
Q2: Is Fail2Ban rule regex a subset of general regex semantics and, if so, is there a Fail2Ban regex doco site?
There seems to be some debate on the web about the structure of a Fail2Ban rule. Some claim that conditionals like "(POST | GET)" are not supported. In other examples it looks like the Fail2Ban rule can be structured as a set of parse strings seperated by line break (but unclear if that would be and AND or an OR).
Q3: What is the Fail2Ban polling interval and is it completed by CRON job?
What is the polling mechanism and duration and where is it configured or does it use inotify? It looks like it relies on systemd journals...in jail.local DEFAULT the backend=systemd is set.
If systemd is in place do we add "journalmatch = _SYSTEMD_UNIT=httpd.service" or "journalmatch = _SYSTEMD_UNIT=apache2.service"? Poking around it is not clear if apache is logging to systemd journal...it looks like start / stop type messages may be being logged but not http/https traffic...not sure...
==============================
UPDATE
==============================
I have sorted out a bunch of poorly documented jail.local quirks and believe I have a proper jail and filter configured as per fail2ban-regex test but still trying to get the jail action to trigger on the IP address of the matching records found by fail2ban-regex check. There is anecdotal net chatter that adding backend = polling to each jail in the jail.local file make the jail work as expected though inotify would be more efficient I believe. Will update with the successful config once it is working. Still interested in answers to Q1-Q3 above though :-)
I think I have everything configured correctly but still not banning any IP addresses...my action is to call iptables-allports action...wait a minute...the Ban Command for action iptables-allports is blank...there's a chunk of my life I won't get back. Why would it be blank! Now I have to learn iptables commands...ahhh crap.
Update2: I finally got IP addresses hitting xmlrpc.php to be banned. Had to tweak the regex, set usedns = no, use iptables[type=all, and set backend=auto to finally get filtered IP addresses blocked. Phew...what a tweaky marginally documented, brittle mess.
I am still interested in the answer to the afore-mentioned question which is why the default is systemd if apache is not logging http/https traffic to the systemd journal. Should we ne logging apache http/https traffic to systemd journal?
Update3: I spoke to soon...IP address is listed as being blocked in the Jails Status page...but I can still access the home page of the site using that "blocked" IP address. It is like Fail2ban thinks it is blocked but the iptables[type=all] action did nothing in iptables. I ran the sudo iptables -L -v command but it did not list the IP address as being banned. So...back to the Googling / ChatGPTing .
==============================
SUMMARY
==============================
Jeremy: You can consider this issue closed for now. After many hours of ChatGPT, Google A.I. and old school Google searchs and doc reading I got the basics sorted. It's a brittle construct...easy to break or miss something. I get now that systemd is the default and is useful for some f2b jails. For other Jails, like mine, you need to override with backend=auto to trigger it. Without pyinotify installed it defaults to poller(). You also need to set usedns=no or you can get some weird results and some extra overhead for a reverse DNS lookup. Stay away from Actions...I lost a day on the following action that was failing with a python error "action = iptables[name=wp-xmlrpc, type=all, blocktype=DROP, protocol=tcp, returntype=RETURN]". Also ignore banaction=, iptables-allports, iptables-multiport which have been deprecated for the action above...which fails with a python error.
/etc/fail2ban/filter.d/wordpress-xmlrpc.conf
[Definition]
failregex = ^.*.*"POST\s+/xmlrpc\.php.*$
^.*.*"GET\s+/xmlrpc\.php.*$
ignoreregex =/etc/fail2ban/jail.local
[wordpress-xmlrpc]
enabled = true
usedns = no
backend = auto
filter = wordpress-xmlrpc
logpath = /var/log/apache2/other_vhosts_access.log
port = http,https
maxretry = 0
maxdelay = 0
findtime = 1m
bantime = 1y
Replies (3)
Hi Tim, sorry I couldn't help sooner...
Reply 1Awesome thanks Tim.
Reply 2Your additional insight and info makes a ton of sense and adds to my base knowledge (which as I note is still mostly "theoretical".
As I said, I'd love to hear how your config goes. I'd love to provide a default fail2ban WordPress config for our appliance. I'd like to provide it for all appliances, but considering that our Wordpress appliance is one of the most popular, and one of - if not the- most popular CMS globally, it's a great candidate to be the first. I also anticipate that a pre-configured WordPress fail2ban conf would be something of a template for other apps.
Although TBH experience suggests that we need to be very careful with "tightening the security screws" too much. A couple of times now we've provided config that we believe to be a secure but usable default and have been proven wrong. Including Apache "mod_evasive" config is a perfect example. I thought I'd tested it pretty well, but it seemed there were so many usage scenarios that didn't match my experience that it lead to many people thinking that our appliances were "broken" - which I guess they were in their experience.
Anyway, I'm waffling... Thanks again mate and take care. Look forward to next time. :)
Cheers Jed. Appreciate your additional info.
Reply 3Regarding your question about findtime...this scopes the number of records in apache log that would be checked when the rule is invoked. Poller time defaults to 1 second but appears to dynamically change based on cumulative rule settings (haven't found an algorithm explanation so no one can say what the polling interval actually is). Findtime is a setting that influences the poll timer. Because we don't really know what the poll cycle will be findtime should not be set too short or it may keep missing periodic offenders. It can't be set too long because that impacts performance since it correlates to number of apache records parsed per rule. Attackers seem to attack once per minute continuously so that is the findtime setting target irrespective of the poller's actual polling duration.
When using systemd or pyinotify the rule is triggered immediately on message receipt so findtime scopes how far back to look as well but findtime is more applicable to maxretry>1 scenarios.
XMLrpc on a wordpress site is a canary-in-the-coalmine for attackers. Unless you use a few legacy WordPress plugins xmlrpc is unused and so for security we disable it in apache config. That does not stop attackers from continuously trying to invoke it and chewing up resources. So the rule, as configured above, blocks the IP address as soon as it tries. Since attackers routinely cycle through a bunch of attack vectors with xmlrpc as one of them...if you block an ip address for xmlrpc access...you don't have to test for all the other attack vectors. So a performant solution.
The exception is login brute force...some will brute force login but not xmlrpc...so a rule for that should also be applied. I am testing that dynamic rule right now.
Regarding bantime...Because hackers have a pool of IP addresses they cycle through you can't release them from jail too quickly. If you ban them forever your performance will degrade as the blocked iP address list gets huge. I set bantime to 1 year based on the following logic:
Periodic apache log reviews when a site is experiencing performance degradation due to attacks shows about 10 unique IP addresses per day attacking so this setting caps the ongoing IP Address block list to 3650 which I expect is a size that won't affect performance. Think of this setting as "auto catch-and-release". So the list size will grow continuously for a year then cap out at whatever accumulated list size that encompasses.
When using systemd or pyinotify the rule is triggered immediately on message receipt so findtime scopes how far back to look as well but findtime is more applicable to maxretry>1 scenarios.
XMLrpc on a wordpress site is a canary-in-the-coalmine for attackers. Unless you use a few legacy WordPress plugins xmlrpc is unused and so for security we disable it in apache config. That does not stop attackers from continuously trying to invoke it and chewing up resources. So the rule, as configured above, blocks the IP address as soon as it tries. Since attackers routinely cycle through a bunch of attack vectors with xmlrpc as one of them...if you block an ip address for xmlrpc access...you don't have to test for all the other attack vectors. So a performant solution.
The exception is login brute force...some will brute force login but not xmlrpc...so a rule for that should also be applied. I am testing that dynamic rule right now.
Regarding bantime...Because hackers have a pool of IP addresses they cycle through you can't release them from jail too quickly. If you ban them forever your performance will degrade as the blocked iP address list gets huge. I set bantime to 1 year based on the following logic:
Periodic apache log reviews when a site is experiencing performance degradation due to attacks shows about 10 unique IP addresses per day attacking so this setting caps the ongoing IP Address block list to 3650 which I expect is a size that won't affect performance. Think of this setting as "auto catch-and-release". So the list size will grow continuously for a year then cap out at whatever accumulated list size that encompasses.
Hi Tim, sorry I didn't get here sooner.
Thanks for posting back with your experience, tips and your final config. I've bookmarked this thread and hope to have a closer look at fail2ban myself sometime soon so I can improve the default implementation in the next TKL major version release.
TBH it looks like you now know more about fail2ban config than I do! My knowledge and understanding of fail2ban is pretty shallow and more theoretical than practical. I have also discovered that while on face value it seems pretty straight forward, that's not the reality. So I likely wouldn't have been much help if I'd arrived sooner.
Now you've found a solution you've likely already answered some of your own initial questions. Regardless, I'll provide some response to your initial questions and hopefully it's of value to others, if not you.
1. Our iptable rules are not enabled by default. However that should not matter for the purposes of fail2ban - or any other tool that has relevant permissions. The Linux firewall is implemented at the kernel network stack level. Firewall management tools (such as the Webmin module we pre-install) present it as a monolith thing, but that's just an abstraction. Webmin simply provides a mechanism to store pre-configured rules and apply/disable them on mass. It also provides a mechanism to reapply them on reboot (they are non-persistent by default).
2. As i noted above, I haven't played with fail2ban as much as I'd like so I can't speak much to what regex fail2ban uses/understands.
As you're likely aware, generally speaking regex has no standard - just conventions. Like you, I failed to find a clear, concise and explicit documentation of fail2ban's regex implementation. If you did find something in the end, please share a link. Despite that, my limited experience suggests that it follows general conventions and at least they provide 'fail2ban-regex' for regex testing.
3. fail2ban runs as a service. The interval between log scans is somewhat system resource dependent, but in general will be once per second or so. I'm sure that it's configurable, but TBH I've never investigated as the default seems pretty sane to me.
As you note, by default fail2ban only checks the systemd journal. I can't answer the rest of the detail of your question OTTOMH, but I would assume that fail2ban can filter the journal either by service name/s (i.e. 'apache2') or journal "tags" (a specific tag that a process provides when sending output to the journal).
As a bit of an aside, 'httpd.service' is not a thing on Debian. 'httpd' is the "proper" name of the Apache (organization) web server software - i.e. what we generally refer to as "Apache". Some distros (e.g. Red Hat) use the upstream 'httpd' name and config set up. Debian (and derivatives) have a more customized config layout (which IMO is far superior) and use the name 'apache2'.
Back on topic, as you also note the default TKL Apache journal output is fairly limited and the detail is only logged to the files in /var/log/apache2. FWIW what Apache logs to the the journal can be customized but that's a whole other conversation.
My WordPress experience is very limited but Drupal can be configured to send info directly to the journal - things that would be useful for fail2ban - things like URL, referrer, IP, HTTP code etc. It is still part of the 'apache2' journal but it is "tagged" as 'drupal'. I.e. internally each Drupal journal entry is '_SYSTEMD_UNIT=apache2.service' but also 'SYSLOG_IDENTIFIER=drupal'. Although that's all somewhat irrelevant now you have it configured to use the Apache text logs.
I'd be interested in the "real world" effect of your rule set but as I understand it (which may be wrong), 'findtime' is how far back in time it will read the log for regex matches. 'maxretries' is how many log entries that match your regex +1 (1 retry = 2 hits). I.e. if 'findtime = 1m' and 'maxretries = 2' means 2 matches in the last minute will trigger a ban. So if I understand correctly, your 'maxretry = 0' will ban all IPs that hit that endpoint which makes the 'findtime' essentially irrelevant. Is that your experience or am I missing something?
I hope that's of some value?!