Forum archive
Leantime vulnerability?
A while ago I installed Leantime on Proxmox using TKL's template. It worked fine.
I also run a mailserver (mailcow) at home and recently got listed again on Spamhaus' exploit and css blocklist.
The info I got from Spamhaus' website it the following:
Why was this IP listed?[my-public-ip] has been classified as part of a proxy network. There is a type of malware using this IP that installs a proxy that can be used for nearly anything, including sending spam or stealing customer data. This should be of more concern than a Spamhaus listing, which is a symptom and not the problem.The proxy is installed on a device - usually an Android mobile, firestick, smart doorbell, etc, but also iPads, and Windows computers - that is using your IP to send spam DIRECTLY to the internet via port 25: This is very often the result of third party "free" apps like VPNs, channel unlockers, streaming, etc being installed on someone's personal device, usually a phone.Technical informationImportant: If this IP operates as a mail server, it should look and behave like a mail server. The HELO currently used appears to be dynamic and that is behaviour commonly observed in malware/proxy networks.Recent connections:(IP, UTC timestamp, HELO value)[my-public-ip] 2024-04-12 05:50:00 leantimeImportant points:The HELOs are often dynamic-looking rDNS and usually claim to be from geographically very different networks OR spoofs of major brands.They can include impossible HELOs like "gmail.com", "outlook.com", "comcast.net" - Gmail, Outlook and Comcast do not use these. These are all fake.If the HELO does not make sense for the IP generating it, it should be looked at closely.There is often more than one compromised device.Guest networks should also be secured.This is a simple explanation of how it can work: https://www.spamhaus.com/resource-center/when-doorbells-go-rogue/Any devices with "free" VPNs, TV streaming, channel unlocking, or 3rd-party apps installed are the first things to check.What should be done about it?We very strongly recommend securing your firewall to not allow any packets outbound on port 25, except those coming from any email server(s) on your local network. Remote sending of email to servers on the Internet should still work if web-based, or configured properly to use port 587 using SMTP-AUTH. Guest networks should be secured too.After port 25 is outbound is secured, the proxy needs to be found and removed. We can only see what's coming from the NAT (public) IP; anything inside your network is visible only to you. You can start logging at the router or firewall to see what's trying to use port 25 and that should lead you right to the compromised device(s).
When I saw this report, I closed port 25 (which at the time pointed to mailcow) and killed the leantime container as the only device on my network with the HELO/hostname leantime, was the leantime container.
I checked my mailcow logs and they are clean as far as I could find with the help of the mailcow community.
I disconnected the network of the container and started examining it, but email sending was never set up either for TKL or leantime itself.
Does anyone have an idea what could be the problem/coulprit?
PS: If any more information required, ask for it and I'll try to post it.
Sorry for the bad formatting, I don't know HTML, but am used to markdown.
Replies (7)
Update
Reply 1As I said, thanks for posting.
Reply 2As I said in my welcome over on the other thread, thanks for posting.
Firstly, your timing is pretty good as there have been a couple of vulnerabilities posted regarding Leantime! So I'll get onto rebuilding that to include the latest fixed version and release an update this week. TBH, I thought I was subscribed to their alerts, so I'll need to explore why. Regardless, I don't think it's anything to do with those.
Regardless, I don't think that's related to your issue. I'll explain why I say that and what I think has happened and why.
Before I move on to what I think might have happened, if you haven't already, I do suggest updating Leantime itself. I.e. not the underlying TurnKey OS, but the Leantime app itself. The OS will auto install any security updates for itself. But updating Leantime realy should have interaction, so autoinstalling updates is not ideal - besides AFAIK there is no mechanism to do that anyway. So it's required to update Leantime itself manually. Since the release of our v18.0 appliance, updates should be possible via the Leantime web UI - rather than manual CLI updates in the older TurnKey release. If you need a hand with that, please open a new thread. Although it might be best to migrate your data to the new appliance, then update Leantime.
It may be possible and the message does specifically suggest that it's related to your TurnKey Leantime appliance. But unless there is confirmation that illegitimate emails are being sent, I don't think that the issue is related to some sort of malware infection in either Leantime or the underlying TurnKey server.
FWIW there is a (3 month old) post on Reddit which suggests that actually sending spam is not required to trigger that message and the corresponding Spamhause block. I have also read that Spamhause have tightened the way they decide whether traffic is spam or not, although I could not find any specific info from them (only third parties making the claim).
Whilst you say that email was not set up in your TurnKey Leantime appliance or Leantime itself, do you have some local firewall (e.g. a network firewall, or a firewall within your router) that stops anything other than your mailcow mailserver sending via port 25?
The reason why I ask, is TurnKey servers include Postfix (an MTA). And by default (i.e. if not configured to send emails via something else) Postfix will try to send mails directly via port 25. Leantime will use the local Postfix to send emails by default too. Even if you disable emails in Leantime, if you entered an email address on firstboot, then the OS will still attempt to send cron errors and automated OS security update notifications to you.
As the Spamhause message specifically notes a hostname of "leantime" (the TurnKey default for the Leantime appliance) and then goes on to say "If the HELO does not make sense for the IP generating it, it should be looked at closely." My assumption is that any mail related DNS records (e.g. DKIM, SPF, DMARC and perhaps others) note a domain - other than "leantime".
TLDR; I reckon that your TurnKey server has been trying to send emails as hostname "leantime" (rather than an appropriate FQDN - with required DNS records). And that has triggered Spamhause blacklisting.
If you want to check if your TurnKey server has been sending emails please check the logs:
journalctl -u postfix@-.service
To disable Postfix on your TurnKey server, this should do the trick:
systemctl disable postfix
Please post back after doing some investigations as it would be good to know if I'm right or if it's something else (and/or you're still stumped). If I am right, then perhaps we should reconsider our defaults?
Hmm, that sucks :(
Reply 3It sucks that that tweak didn't help. Although I wouldn't expect it to have caused an error? AFAIK it should have either fixed it or just caused it to have the same behavior as before.
Regardless, thanks for posting back and I'm glad to hear that you could work around it with a manual update.
Hopefully now you've updated, future CLI updates will work. If they don't please post back with any errors you get. If you don't get any error but it still doesn't work, still let me know and I can look into it further.
Changing all double quotes
Reply 4Changing all double quotes for single ones, didn't work (updater gave errors). Changing only the double quotes outside of the blocks also resulted in errors. Ended up manually updating. Will do that every now and then from now on. Thanks for you help.
AFAIK the issue with update is quotes in the config file
Reply 5From what I gather, the issue with the Leantime CLI updater is that we use double quotes in our config file updates/cutomisations - see this comment from a Leantime dev.
FWIW in PHP (similar to python) single and double quotes are interchangeable, so long as they are used in matching pairs. Apparently the old Leantime updater didn't anticipate that scenario though. It's since been updated to support either sort of quotes now, but we've updated our build to be consistent with the default use of single quotes in the config file regardless.
So AFAIK, changing the double quotes for single ones in the config file should make the CLI updater work.
Failing that (or instead if you prefer) a manual update should work fine.
Either way if you continue to have issues updating, please open a new thread and note the error message(s) and I'll do my best to help out.
I will setup the SMTP relay.
Reply 6I will setup the SMTP relay.
Is there any guide on how I can update Leantime? Running the leantime binary with the update command doesn't seem to do anything.
Do you want it to send emails?
Reply 7If you want it to send emails, then you could use your existing email server as a SMTP relay (assuming that it's capable of that - I'd be surprised if it wasn't). Confconsole supports setting up a custom SMTP relay:
Confconsole >> Advanced >> Mail relaying >> Custom
If you don't want it to send emails at all, then disable postfix (on the server). This should do it:
systemctl stop postfix systemctl disable postfix
To double check that worked:
systemctl status postfix*
It would indeed appear as if postfix has been trying to send emails to my email address, which is an iCloud address.
And indeed my SPF is set to only allow emails from my root domain and mail.mydomain.nl.
I don't know if I can provide you with any more logs to further investigate what caused is to go nuts and spam my IP into oblivion.