Paul Lindon's picture

Hi there!
It's me again from a few days ago regarding Concrete5.
I realize that whatever I do doesn't really work and I need to reconfigure out I do things. I'd love some tips on what I could do instead, or how I should be doing it. Apologies if this is out of scope.
I have 1 public facing dynamic IP. I have ports 80 and 443 forwarded to server 192.168.0.199 where Caddy is sitting. Caddy works similar to Nginx (albeit much simpler) in reverse proxying to various services with automated SSL via Let's Encrypt or ZeroSSL. 

Caddy then takes domains and redirects to the various servers hosting the various things. Config looks similar to:

media.domain.example {
        reverse_proxy 192.168.0.111:8000
        encode gzip zstd
}

radarr.domain.example {
        reverse_proxy 192.168.0.111:8001
        encode gzip zstd
}

sonarr.domain.example {
        reverse_proxy 192.168.0.111:8002
        encode gzip zstd
}

files.domain.example {
        reverse_proxy 192.168.0.111:8003
        encode gzip zstd
}

tasks.domain.example {
        reverse_proxy 192.168.0.122:8000
        encode gzip zstd
}

office.domain.example {
        reverse_proxy 192.168.0.133:8000
        encode gzip zstd
}

demo.domain.example {
        reverse_proxy 192.168.0.144
        encode gzip zstd
}

erp.domain.example {
        reverse_proxy 192.168.0.155:8000
        request_header Referer 192.168.0.155
        request_header Host 192.168.0.155

}

 

This is fantastic as I only need 1 public facing IP. The problem is Turnkey, is trying to be smart and do the same thing.

I am trying to setup the concrete5 container inside of Proxmox. I have set the title to just apache SSL since this is a problem that would affect just about every container using SSL.

For concrete5 where the apache rewrite logic only forces SSL to redirect on the login page, the normal webpage loads, but login will not work since both Caddy and Apache are fighting to give me SSL.

I believe I am simply missing some vital piece of information to understand how this should all work together, so I ask for some assistance if you're willing to offer it.

What would you recommend in terms of using reverse proxies with turnkey servers? How should this be setup?

 

Thank you again in advance for the help. Your knowledge is truly appreciated.

Forum: 
Tags: 
Paul Lindon's picture

Forgot to mention that on the DNS site, I have a single domain a.domain.example that is dynamically updated via python script to my current dynamic IP. The the various subdomains are just CNAME records that route to a.domain.example.

Timmy's picture

While I'm not reverse-proxying, I fetch certs on one machine then I have a shell script that passes them around to all the Apache servers.

Paul Lindon's picture

In that case I have a question, I was able to find .crt, .json and .key from caddy in: "/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/subdomain.domain.example"

I believe these are the certs Caddy create when requesting SSL from either let's encrypt or ZeroSSL.

I got the tip to add the .crt and .key to: etc/ssl/private/cert.pem & /etc/ssl/private/cert.key for apache. Unfortunately this does not work, and I believe it's because they're incompatible.

apache2 cert.key looks like:

-----BEGIN PRIVATE KEY-----
xxxxxxxxxxxxxxxxxxxxxxx
-----END PRIVATE KEY-----

Caddy cert.key looks like:

-----BEGIN EC PRIVATE KEY-----
xxxxxxxxxxxxxxxxx
-----END EC PRIVATE KEY-----

 

apache2 cert.pem looks like:

-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxx
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
xxxxxxxxxxxxxxxxxxx
-----END PRIVATE KEY-----
-----BEGIN DH PARAMETERS-----
xxxxxxxxxxxxxxxxxxxxx
-----END DH PARAMETERS-----

 

Caddy cert.crt looks like:

-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----


Is there something I'm doing wrong with this?

Jeremy Davis's picture

You should be able to just copy your cert & key over the top. But you'll need to do a few mods. Firstly, as you note, your cert and key are in a slightly different format to what we provide, but my googling suggests that Apache shouldn't care. So you should be able to use the key file as is. As for the cert, assuming they're all in /etc/ssl/private, just do this:

mv cert.pem cert.bak
cat caddy.crt > cert.pem
cat caddy.key >> cert.pem
cat dhaparams.pem >> cert.pem

Then restart Apache. If Apache fails to start or there is some other error, please share the relevant lines from the journal. E.g. to get Apache logs for the last ten minutes:

journalctl -u apache2 --since "10 minutes ago"

If it fails, my guess is that the cert is using a cipher that Apache isn't configured to use. As a hint, the Apache directive is "SSLCipherSuite" and it's counterpart in Caddy's config is "ciphers".

Good luck

Timmy's picture

There are a number of cert types and formats; you already figured out some require concatenation of two or more files together.

It looks like

EC, I would imagine, stands for Elliptical Curve key, where as the other, by lack of designation, I presume is an RSA type key.

I just had a workmate preach the wonders of Caddy to me yesterday (literally- kinda startling to see this thread after) so I'm not educated at all on it. But is there an ability to key key types? Conversely, I'm poorly educated on Apache but perhaps it can be set to use an EC key?

The DH stuff is Diffie-Hellman key exchange data. That lets two devices compute a shared secret for key exchanges. Again, I hadn't heard of Caddy until yesterday but I take it looking around revealed no DH data generated? Perhaps this is some configuration difference in how Apache or Caddy intend SSL/HTTPS to occur?

Hopefully someone with more experience hops in and lays it out.

Jeremy Davis's picture

Firstly, thanks for jumping in and trying to help out Timmy! :)

Assuming that your Caddy instance handles the public TLS termination (i.e. it serves the "proper" TLS certs, from Let's Encrypt), the only thing we need to worry about is the connection between Caddy and the Concrete CMS backend. Here are the configuration options as I see them:

  1. www  Caddy  vanilla http (port 80)  Concrete
  2. www  Caddy  self signed https (port 443)  Concrete
  3. www  Caddy  "proper" (e.g. LE) https (port 443)  Concrete (2 variations)
  4. www  Caddy  Caddy CA signed https (port 443)  Concrete

1. vanilla http (port 80)

The first is the simplest and as I understand it, what you were asking for initially. Essentially you just need to disable (i.e. remove from the Apache conf) the https redirect for the log in page. I.e. remove these lines from the Concrete Apache conf (/etc/apache2/sites-available/concrete.conf):

    RewriteEngine on
    RewriteRule ^(.*)/login$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Then restart Apache. Assuming the default file, here's a oneliner to do that (please note that this will remove all lines that include 'Rewrite'!):

sed -i "\|Rewrite|d" /etc/apache2/sites-available/concrete.conf && systemctl restart apache2

Then configure your reverse proxy (something) like this:

concrete.domain.example {
        reverse_proxy 192.168.0.xxx:80
        encode gzip zstd
}

Where 192.168.0.xxx is the private IP of your Concrete instance. Given your example config, I assume that without a port, it will default to port 80 (vanilla http), but personally I think it's best to be explicit.

As I noted previously, historically, this is how it was often done. However, the downside is that the traffic within the LAN is unencrypted, so snooping on passwords, etc is pretty easy because they're transmitted in plain text within the LAN. There is also no guarantee that the backend server actually is the Concrete instance, so MITM attacks are also trivial.

As any attack would require a malicious actor within your LAN, you may be comfortable with that? Obviously, the value/importance of the content is also a factor.

2. self signed https (port 443)

The next option is to just use the default self-signed TLS cert (or regenerate it) on Concrete (i.e. the one that TurnKey servers generate on first boot). TBH, I'm not actually 100% sure whether this is possible with Caddy (I've done it behind Nginx before), but my reading of the Caddy source code, suggests that it may be? Essentially, it requires that the validition of the TLS cert is skipped, because a self signed cert won't be trusted by Caddy by default.

TBH, I have no idea if/how that is defined in the config though? Random guess is something like this?:

concrete.domain.example {
        reverse_proxy 192.168.0.xxx:443
        insecure
        encode gzip zstd
}

Or perhaps one of these?:

        insecure true
        insecure yes

I am only guessing though, so you may need to consult the Caddy docs and/or Caddy community to get guidance on the exact config layout. (Or just exclude this option).

This pathway is still pretty simple and a clear improvement as it means that the traffic (between Caddy and Concrete) will be encrypted. So even if someone malicious is snooping on your network, they won't be able to snoop your password. However, it's still far from perfect as there is still no guarantee that the backend server actually is your Concrete server (so a MITM attack would again be trivial).

3. "proper" (e.g. LE) https (port 443)

3a. Concrete get's it's own cert

This option will require that your Concrete server has it's own legitimate CA signed cert (e.g. from Let's Encrypt). Not considering how you get the cert, this is a super simple option and once configured, should "just work" (similar to vanilla http). That will be as secure as you can get, as all traffic between Caddy and Concrete will be encrypted and if someone tries to do a MITM, the connection will fail (because there would be an invalid intermediate cert).

On the downside, your Concrete server will need to get it's own cert from LE. Whilst it doesn't make Caddy completely redundant (it's still providing the reverse proxy) it does somewhat minimise the value of centrally managed https termination (because you still need to manage the local LE cert on Concrete). The only ACME validation path supported by Confconsole in v17.x is HTTP-01 - which requires your server be publicly available via port 80. You could do that in Caddy I'm sure (i.e. reverse proxy ports 80 (for LE validation) & port 443 (for Concrete content). In the upcoming v18.x, DNS-01 ACME cert validation is an option. That is better IMO because you don't need to reverse proxy port 80, but it still defeats the idea of centrally managed TLS.

3b. cert copied from Caddy

A variation on the above would be to copy the valid Let's Encrypt certs from your Caddy instance to the Concrete instance. AFAIK that is essentially what Timmy is doing. Whilst this should work fine, the most common config is to get a combined cert on the host (i.e. Caddy) and use the same cert for all content being served. It's probably unlikely to ever be an issue, however having a unique certificate on each backend is the only way to ensure that the backend you are connecting to actually is the Concrete server. To ensure that, you'd need to get an individual cert for each backend server.

Either way...

Regardless, I imagine that the Caddy config for this option (i.e. LE cert for back - inc the variation) would be very similar to the first config example I gave (except via hostname on port 443):

concrete.domain.example {
        reverse_proxy concrete.domain.example:443
        encode gzip zstd
}

Note that you will need to create a hosts file entry for the domain (pointing to the private LAN IP). Otherwise Caddy will probably do a DNS lookup and then use it's own public IP, thus creating a never ending loop.

4. Caddy CA signed https (port 443)

TBH, this is me speculating on how flexible Caddy's CA (certificate authority) setup is. It appears to me that Caddy ships with the requirements for this pathway OOTB, but I'm not sure if they can easily be bent to do it this way? For reference, the "bending" I'm referring to is using the Caddy CA signed certs for the backend too. The bit I'm not 100% sure of is whether Caddy will check it's own CA when validatinig the cert of your backend servers?

FWIW I have a vague idea of how you would go about this with Nginx and OpenSSL, but I have zero experience with Caddy, so you'll definitely need to check the docs and/or ask on their forums if this doesn't "just work".

Assuming that what I think is possible, this last path has same advantages as #3, but without the need to get LE certs on your Concrete server (although I guess it hardly matters where they come from if you still need to copy them across). Another advantage is that you can set a long expiry time, so you only need to copy the cert once (not every 3 months - like LE certs require). You can also use issue certs for unique internal domain names and even IP addresses potentially.


Bottom line...

Please note that I haven't explicitly gone through and tested all this, but I feel fairly confident that what I've written is at least generally correct (there may be some details that require tweaks). Hopefully it's of some value to you?

As you can see, there are a range of options and none of them are "right"! It all comes down to what works for you and what your risk profile and risk appetite are.

Paul Lindon's picture

Seriously Jeremy, WOW!

Thank you, this is incredible, not only the fact that you took the time to write this all, but also for the fact that you came up with not 1, not 2, not 3, BUT 4 POSSIBLE SOLUTIONS! Thank you Jeremy :) This is very valuable information, not just for me but anyone else who comes across this page.

I decided for option 3a, that sounds the best, most secure and easiest to setup.

Option 1, as previously discovered caused issues as some of the traffic is HTTP, as you mentioned. I also got security warnings from Firefox on the login pages about password being sent as HTTP. No good.

Option 2, Honestly not a huge fan of self signed certs, mainly because it will give out error messages about being self signed. I like my websites to be appealing to the layman, and security warnings are always scary to them!

Option 3a..... Dead simple. Go to turnkey server, open confconsole, LE, get certs, enable autorenewal. Then go to my Caddyfile and set the setting:

subdomain.domain.example {
        reverse_proxy https://192.168.0.XXX {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}

This seems to work (Source for knowledge). Not entirely sure how or why this works, based on how I've interpreted it this transports both 80 and 443 traffic to 192.168.0.XXX:443 and skips verifying anything so the turnkey server does everything and Caddy simply proxies the traffic.

This is by far the easiest and fastest way to set it up and is easily reproducible across configs, in addition to just working :) Almost as simple as just the plaintext solution but just that much more secure.

I want to point out this is absolutely perfect for me in terms of security. Everything handled by the server, but only Caddy server is "open" to the internet.

Thank you again Jeremy!

Add new comment