Using Let's Encrypt with an Apache proxy to another webserver

[NotionCommotion]

I am having inconsistent results when having httpd act as a proxy to another webserver which uses a self-signed certificates and is running in a docker.  Sometimes it works but then I later attempt to do what I think is identical and get challenge failed errors.

Trying to understand why, I reviewed Let's Encrypts How It Works and below is my understanding:

Step 1 - Agent proves to LetsEncrypt that the web server controls a domain

1. Agent generates a new key pair (one time)
2. Agent asks LetsEncrypt required proof of control (one or all?):
   1. DNS record (What is this used for?)
   2. An HTTP resource under its URI (must create a file which will be accessed by LetsEncrypt) 
   3. Other?
3. LetsEncrypt provides random string to be signed with agent's private key to prove ownership.
4. LetsEncrypt then performs:
   1. Verify private key using random string and agent's public key (when is LetsEncrypt given public key?).
   2. Attempts to upload newly created HTTP resource.
   3. If valid, LetsEncrypt acknowledges agent identified by URI and public key (authorized key pair).
   4. Does LetsEncrypt provide a new key and is this the authorized key pair?

Step 2 - Agent requests (or renews or revokes) certificates for the domain.

1. Agent constructs CSR with public key and signs with private key and authorized key (this other key provided by LetsEncrypt?)
2. LetsEncrypt returns certificate with public agent's public key.

 

Below is my 443 to 8443 proxy and I have another virtual host which just redirects 80 to 443 (not shown).  The various SSLProxyVerify settings are my attempt to deal with the docker's webserver using self-signed certificates and while it sometimes worked, do not know if it is all correct.

Based on my new understanding of how Let's Encrypt works, I am trying to understand why I am having issues.  My thoughts are:

1. It will not be able to perform Step 1 and prove to Let's Encrypt that it controls the domain because Apache will pass the requests to the docker's webserver and the docker's webserver will not know anything about this scope.  Or maybe it will but doesn't have write permission to where to create the new HTTP resource?  Perhaps a workaround is to first create a traditional virtual host, allow Apache to prove to Let's Encrypt that it controls the domain, and then go back to the proxy?
2. Maybe I am having conflicts between using Apache's keys and the docker's webserver's keys?
3. Maybe my proxy configuration is incorrect and I just thought it was the same when it was working?
4. Maybe something else?

This has been totally frustrating and would appreciate any suggestions.  Thanks

 

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName my-site.net
    ProxyPreserveHost On
    ProxyRequests Off
#       <Proxy *>
#               Order deny,allow
#               Allow from all
#       </Proxy>
    SSLProxyEngine On
    SSLProxyVerify none
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerExpire Off
    SSLProxyCheckPeerName Off
    #SSLEngine On       #Not sure what this is
    ProxyPass / https://127.0.0.1:8443/
    ProxyPassReverse / https://127.0.0.1:8443/

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/my-site.net/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/my-site.net/privkey.pem
</VirtualHost>
</IfModule>

 

[NotionCommotion]

I don't understand why this is happening.  I prune out all existing dockers including volumes off a machine, remove certbot, waste out /etc/letsencrypt, reboot the server, turn off httpd, clone a git repository that doesn't have anything related to letsencrypt, execute docker-compose build, and then bring it up and...  Why does it still have reference to letsencrypt?

 

caddy_1     | {"level":"info","ts":1632674310.9699051,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1     | {"level":"info","ts":1632674310.9699159,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1     | {"level":"info","ts":1632674310.97,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00025b570"}
caddy_1     | {"level":"info","ts":1632674310.9813414,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy_1     | {"level":"info","ts":1632674310.9813673,"logger":"tls","msg":"finished cleaning storage units"}
caddy_1     | {"level":"info","ts":1632674310.9813404,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
caddy_1     | {"level":"info","ts":1632674310.9814408,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["mysite.net"]}
caddy_1     | {"level":"info","ts":1632674310.9816387,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1     | {"level":"info","ts":1632674310.9816458,"msg":"serving initial configuration"}
caddy_1     | {"level":"info","ts":1632674310.9818106,"logger":"tls.obtain","msg":"acquiring lock","identifier":"mysite.net"}
caddy_1     | {"level":"info","ts":1632674310.9828124,"logger":"tls.obtain","msg":"lock acquired","identifier":"mysite.net"}
caddy_1     | {"level":"info","ts":1632674310.9835498,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["mysite.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy_1     | {"level":"info","ts":1632674310.9835625,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["mysite.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy_1     | {"level":"info","ts":1632674311.5837862,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"mysite.net","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy_1     | {"level":"error","ts":1632674312.071876,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"mysite.net","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"Connection refused"}
caddy_1     | {"level":"error","ts":1632674312.0718951,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"mysite.net","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Connection refused","order":"https://acme-v02.api.letsencrypt.org/acme/order/215675790/27383624550","attempt":1,"max_attempts":3}
caddy_1     | {"level":"info","ts":1632674313.3718565,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"mysite.net","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy_1     | {"level":"error","ts":1632674314.2157142,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"mysite.net","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"Fetching http://mysite.net/.well-known/acme-challenge/dnzP3sIQ2PJ68Y3hdcNGEouiwpWIDU66_roQDMzp6v0: Connection refused"}
caddy_1     | {"level":"error","ts":1632674314.2157347,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"mysite.net","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Fetching http://mysite.net/.well-known/acme-challenge/dnzP3sIQ2PJ68Y3hdcNGEouiwpWIDU66_roQDMzp6v0: Connection refused","order":"https://acme-v02.api.letsencrypt.org/acme/order/215675790/27383633790","attempt":2,"max_attempts":3}
caddy_1     | {"level":"error","ts":1632674315.7577894,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mysite.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[mysite.net] solving challenges: mysite.net: no solvers available for remaining challenges (configured=[tls-alpn-01 http-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/215675790/27383649220) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy_1     | {"level":"warn","ts":1632674315.758133,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
caddy_1     | {"level":"info","ts":1632674316.6636553,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"UvTMxijwSYXU8puSPR-V1Q"}

 

[requinix]

If your public server is the one with the cert then it should be the one handling the cert. Think about it as SSL termination.

What this site does is use a custom authenticator script that writes certbot's generated key/token pair to a publicly-accessible /.well-known/acme-challenge file (plus a custom cleanup script to remove it). Your server could do that too - just don't proxy /.well-known/acme-challenge/* to the backend server.

[requinix]

7 hours ago, NotionCommotion said:

2. Agent asks LetsEncrypt required proof of control (one or all?):

1. DNS record (What is this used for?)
2. An HTTP resource under its URI (must create a file which will be accessed by LetsEncrypt) 
3. Other?

Just one. With certbot running on a public server, a file is much easier to work with than DNS. For larger organizations with multiple servers, the renewal process running somewhere internally and writing to DNS is easier than having to replicate files.

 

7 hours ago, NotionCommotion said:

4. LetsEncrypt then performs:

1. Verify private key using random string and agent's public key (when is LetsEncrypt given public key?).
2. Attempts to upload newly created HTTP resource.
3. If valid, LetsEncrypt acknowledges agent identified by URI and public key (authorized key pair).
4. Does LetsEncrypt provide a new key and is this the authorized key pair?

certbot handles all that magic for you. The important thing is that you are given a key and token to prove you have ownership, and those two need to be discoverable by Let's Encrypt.

 

7 hours ago, NotionCommotion said:

Step 2 - Agent requests (or renews or revokes) certificates for the domain.

1. Agent constructs CSR with public key and signs with private key and authorized key (this other key provided by LetsEncrypt?)
2. LetsEncrypt returns certificate with public agent's public key.

More certbot magic. It handles certificate files - you just point your webserver to the relevant file paths and specify a reload command for it to pick up any changes.

[NotionCommotion]

25 minutes ago, requinix said:

If your public server is the one with the cert then it should be the one handling the cert. Think about it as SSL termination.

What this site does is use a custom authenticator script that writes certbot's generated key/token pair to a publicly-accessible /.well-known/acme-challenge file (plus a custom cleanup script to remove it). Your server could do that too - just don't proxy /.well-known/acme-challenge/* to the backend server.

I agree that if my public server is the one with the cert then it should be the one handling the cert. I've since looked at files within the docker and think it is also trying to do so.  I think by random chance originally happened to execute things in some order that I didn't initially send /.well-known/acme-challenge/* to the backend server and then it was cached and then I thought all was good.  Sorry for asking but would you mind giving me dummy instructions on how to prevent proxying the challenge to the backend server?

PS.  Thanks for the clarification on our later post.  DNS part seems to make sense.  Lot of "magic" going on!  I like your reference to the "important" items.  Key and token given by Let's Encrypt to prove you have ownership.  Not sure about how I need to specify a reload command, however.

[requinix]

1 hour ago, NotionCommotion said:

I agree that if my public server is the one with the cert then it should be the one handling the cert. I've since looked at files within the docker and think it is also trying to do so.  I think by random chance originally happened to execute things in some order that I didn't initially send /.well-known/acme-challenge/* to the backend server and then it was cached and then I thought all was good.  Sorry for asking but would you mind giving me dummy instructions on how to prevent proxying the challenge to the backend server?

PS.  Thanks for the clarification on our later post.  DNS part seems to make sense.  Lot of "magic" going on!  I like your reference to the "important" items.  Key and token given by Let's Encrypt to prove you have ownership.  Not sure about how I need to specify a reload command, however.

The only place the Let's Encrypt stuff needs to happen is on the proxying server. Is it containerized too? Because I can't tell what "the docker" is supposed to mean.

The servers in the background handling regular requests don't even necessarily need SSL if it's inside your own network (and you trust it's secure against rogue servers or whatever), but otherwise you can use a self-signed cert just for the TLS aspect and make the proxying server ignore the insecurity - or even use a local cert authority, grant certs as needed, and of course install the CA's as a trusted root everywhere.

To tell Apache not to proxy a path, normally I would have the virtualhost configuration broken down into <Location>s, but if you don't need that then you can literally tell ProxyPass not to proxy the one path.

[NotionCommotion]

13 hours ago, requinix said:

The only place the Let's Encrypt stuff needs to happen is on the proxying server. Is it containerized too? Because I can't tell what "the docker" is supposed to mean.

No, the proxy server is not containerized and architecture is as follows:

• my-site.net points to my home IP.
• My home router forwards ports 80, 443, 8080, 8443, etc to a physical Linux server (AKA "server") on same ports (and server has firewalld opened for these same ports)
• Server has non-containerized httpd server which has a virtual host for my-site.net and proxy's to 80->8080 and 443->8443 (posted previously except since added directive to ignore forwarding "/.well-known/acme-challenge/")
• On the server, a docker is up with 8080->80 and 8443->443 and SERVER_NAME my-site.net.

13 hours ago, requinix said:

To tell Apache not to proxy a path, normally I would have the virtualhost configuration broken down into <Location>s, but if you don't need that then you can literally tell ProxyPass not to proxy the one path.

Thanks, added ProxyPass "/.well-known/acme-challenge/" "!" before other directives but still have issues as discussed next.

19 hours ago, NotionCommotion said:

I don't understand why this is happening.  I prune out all existing dockers including volumes off a machine, remove certbot, waste out /etc/letsencrypt, reboot the server, turn off httpd, clone a git repository that doesn't have anything related to letsencrypt, execute docker-compose build, and then bring it up and...  Why does it still have reference to letsencrypt?

As shown on my second post, attempting to troubleshoot I shutdown the httpd server and rebuilt the docker from scratch, but the caddy webserver running in a docker still tries to get a certificate from Let's Encrypt.   Could my home router or the Linux server somehow have cached my-site.net and causing this?  There isn't typically some sort of TLS/SSL daemon running on a Linux server which might cause this, is there?  I feel I need to deal with this first, no?  Or maybe not?

Maybe the log shown was even happening when I previously had things working as it appears that caddy will automatically try to get a certificate from Let's Encrypt?

Let me give it a try.  Stopping httpd and changing the caddy webserver running on the docker to use 80 and 443 instead of 8080 and 8443, and bring the docker back up.  All works!  Bringing it down, changing back to 8080 and 8443, starting httpd, and bringing the docker back up.  All works again!

Okay, immediate emergency solved, however, would still like to know why and what the future implications might be.  Seems that when running on 80/443, caddy got a certificate somehow from Let's Encrypt and saved it in the /data directory, and then when switched back to 8080/8443 didn't attempt to get another one.  Other that just dumb luck, how would one troubleshoot this?  Before doing this, the docker logs would not report anything when a request was made to it by the Apache proxy, but Apache would display a proxy error and var/log/httpd/error_log would report the following.  Is my current approach which seems to work suspect and how should one troubleshoot proxy errors?

[Sun Sep 26 08:15:15.517425 2021] [proxy_http:error] [pid 80478:tid 139837784958720] (103)Software caused connection abort: [client 12.34.56.78:51573] AH01102: error reading status line from remote server 127.0.0.1:8443
[Sun Sep 26 08:15:15.517463 2021] [proxy:error] [pid 80478:tid 139837784958720] [client 12.34.56.78:51573] AH00898: Error reading from remote server returned by /
[Sun Sep 26 08:15:15.562080 2021] [proxy_http:error] [pid 80478:tid 139837776566016] (103)Software caused connection abort: [client 12.34.56.78:51573] AH01102: error reading status line from remote server 127.0.0.1:8443, referer: https://mysite.net/
[Sun Sep 26 08:15:15.578823 2021] [proxy_http:error] [pid 80476:tid 139837709457152] (103)Software caused connection abort: [client 12.34.56.78:61193] AH01102: error reading status line from remote server 127.0.0.1:8443, referer: https://mysite.net/
[Sun Sep 26 08:15:15.578838 2021] [proxy:error] [pid 80476:tid 139837709457152] [client 12.34.56.78:61193] AH00898: Error reading from remote server returned by /favicon.ico, referer: https://mysite.net/
[Sun Sep 26 08:15:17.358509 2021] [proxy_http:error] [pid 80476:tid 139837155833600] (103)Software caused connection abort: [client 12.34.56.78:61193] AH01102: error reading status line from remote server 127.0.0.1:8443
[Sun Sep 26 08:15:17.372752 2021] [proxy_http:error] [pid 80478:tid 139837281658624] (103)Software caused connection abort: [client 12.34.56.78:61685] AH01102: error reading status line from remote server 127.0.0.1:8443
[Sun Sep 26 08:15:17.372778 2021] [proxy:error] [pid 80478:tid 139837281658624] [client 12.34.56.78:61685] AH00898: Error reading from remote server returned by /
[Sun Sep 26 08:15:17.395653 2021] [proxy_http:error] [pid 80478:tid 139837273265920] (103)Software caused connection abort: [client 12.34.56.78:61685] AH01102: error reading status line from remote server 127.0.0.1:8443, referer: https://mysite.net/
[Sun Sep 26 08:15:17.434038 2021] [proxy_http:error] [pid 80478:tid 139837256480512] (103)Software caused connection abort: [client 12.34.56.78:52594] AH01102: error reading status line from remote server 127.0.0.1:8443, referer: https://mysite.net/

 

[requinix]

Home network? Get rid of HTTPS on the intranet.

Have external HTTPS connections pass through the router to the non-containerized server, have that handle all the SSL as a terminator, and then proxy requests to the containerized server's non-SSL on 8080/80.

Far simpler.

[NotionCommotion]

5 hours ago, requinix said:

Home network? Get rid of HTTPS on the intranet.

Have external HTTPS connections pass through the router to the non-containerized server, have that handle all the SSL as a terminator, and then proxy requests to the containerized server's non-SSL on 8080/80.

Far simpler.

Home network is just in the interim and will not be used much longer.  Sounds like regardless of whether on a home network or not, you recommend having the non-containerized server handle all the SSL as a terminator, and then proxy requests to the containerized server's non-SSL on 8080/80.  Seems very reasonable.  I am pretty sure I previously tried such and got some sort of SSL client making a request to a non-SSL server error but expect I could configure the Apache proxy to allow.  Without having any more logs that "error reading status line from remote server" from the Apache log and nothing from the docker log, it was hard to know what was happening.  I had looked to see if there were any additional caddy logs other than those coming from the docker logs but didn't find any.  Guess I have this resolved enough but still would like to know how I should have better troubleshoot it.  Thanks

[requinix]

12 minutes ago, NotionCommotion said:

Sounds like regardless of whether on a home network or not, you recommend having the non-containerized server handle all the SSL as a terminator, and then proxy requests to the containerized server's non-SSL on 8080/80.

This external-server-terminates-SSL-and-proxies-internally-without-SSL ("offloading") is a common strategy, especially back in the days when SSL was (or at least people thought it was) computationally expensive. It's not as much now because everything's fast, but the convenience in not having to deploy more certificates across more machines is worthwhile.

The only requirement to doing that is securing the network against unwanted servers (ie, MITM attacks), but that really shouldn't be much of a concern. Because if it was a concern, it would really be a concern.

 

12 minutes ago, NotionCommotion said:

I am pretty sure I previously tried such and got some sort of SSL client making a request to a non-SSL server error but expect I could configure the Apache proxy to allow.

Apache shouldn't care...

 

12 minutes ago, NotionCommotion said:

Without having any more logs that "error reading status line from remote server" from the Apache log and nothing from the docker log, it was hard to know what was happening.  I had looked to see if there were any additional caddy logs other than those coming from the docker logs but didn't find any.  Guess I have this resolved enough but still would like to know how I should have better troubleshoot it.  Thanks

I tend to circumvent problem solving by declaring the problem is moot to begin with.

Is the error for every request or intermittent?

[NotionCommotion]

12 hours ago, requinix said:

I tend to circumvent problem solving by declaring the problem is moot to begin with.

Is the error for every request or intermittent?

Both every request and intermittent.  I randomly performed some actions and then didn't get the errors.  Then tried to re-build it and continuously got the errors.  The frustrating part was I got very little feedback indicating the cause.

It wasn't until I discovered that if I temporarily first disable Apache and have the webserver running in the container listen to 80/443, and then access it using the domain name (not localhost), the certificate gets saved and then I can revert back to 8080/8443 and access it via the proxy without error.  The problem should be moot if I disable SSL in the container's webserver, however, it took me way too long to discover it was SSL related.