I’m a bit frustrated with the SSL (Let’s Encrypt) auto-update handling. I posted already a year ago when an update broke the process (which was the solved by some code wizardry beyond my skill level) and now again, It seems the update didn’t happen as it was supposed to, and I end up with a myriad of messages from people having trouble accessing our SeaTable deployment (Developer Edition latest - self-hosted). Following the documentation, when the auto-update fails, the command /templates/renew_cert.sh should fix the issue, but this path doesn’t even exist!? I’m completely lost on how to fix this and why it doesn’t work in the first place. I’d be very grateful if someone could tell me why this is happening and mostly how to fix it as soon as possible. I went to through all the documentation files and nothing I could find would work as explained. Please let me know what information I can share to get this resolved. TIA
Edit: Was able to run the renew command (had to go into container first obviously) but always get an error message (invalid response).
Sorry to be a cumbersome here but I tried to whole day to resolve this on my own - with admittedly limited knowledge - but was not successful at all. I tried to follow the instructions from the last time renewal failed but I always get the same response error code:
mydomain.com:Verify error:xx.xx.xx.xx: Invalid response from http://mydomain.com/.well-known/acme-challenge/0OXncLFCTpCHpgb3MOyemKStG_jecBeiiphmLYbVNiY: 404
[private information redacted]
I went through every possible resource I could think of but I’m running against a wall. I’ll happily share any config or log file necessary to get this solved. Obviously (as it seems nobody else is having this problem), although I haven’t changed anything, there has to be some misconfiguration hidden somewhere. My limited Linux-Kung-Fu is just not sufficient to solve this by myself
This is not a SeaTable problem but a Let’s Encrypt/Certbot issue. SeaTable uses Let’s Encrypt for applying for and managing SSL certificates.
Re-run the renew command. Then check the error message. There is a link in the error message. Copy the link into your browser’s address bar and see what happens.
Hmm, is that really true? Because the system was running without problems and apart from upgrading SeaTable versions no changes were made at all. A year ago, you also mentioned that a fix from your side was necessary.
Either way, what I found out so far is that the renewal probably fails because the http://mydomain/.well-known/acme-challenge/ folder cannot be reached over port 80. I tested this with a text file I put there to see if it could be reached from the browser. This leads me to believe that there is maybe a misconfiguration in the seatable nginx.conf. I double checked against the file of a fresh install of another server and don’t see any anomalies there though. I also checked if a firewall rule like ufw is maybe interfering but ufw is disabled.
This seems like such a simple thing to fix and I just can’t find the culprit, it drives me crazy. Let’s Encrypt forums are not helpful either because they say as long as I cannot reach that folder over port 80 there’s nothing that can be done.
If anyone has any ideas or clues that I could follow…please save me?
Edit: Speaking of the challenge folder, when I delete it manually, it should be re-created during script execution as far as I understand In my case that is also not happening.Maybe a webroot path problem involved here?
[Fri Dec 2 08:33:35 CET 2022] _currentRoot='/var/www/'
[Fri Dec 2 08:33:35 CET 2022] wellknown_path='/var/www//.well-known/acme-challenge'
or is that double slash normal? Because later during execution it seems to be called correctly:
[Fri Dec 2 08:33:46 CET 2022] mydomain.com:Verify error:78.47.33.104: Invalid response from http://mydomain.com/.well-known/acme-challenge/v3oP1jN04jJ2LF5ykaZffyC-d3jO9PZR7jAOsvkVzV0: 404
Your attempt to debug is ok. Create a test.txt with some content at .well-known/acme-challenge/ folder and then try to reach this file with a curl command. Try this:
docker exec -it seatable bash # to enter the seatable docker container
echo "hello world" >> /var/www/.well-known/acme-challenge/test.txt
curl http://your-domain/.well-known/acme-challenge/test.txt
Now there could be two results:
if the result is hello world your should run the command /templates/renew_cert.sh
if there is another result like “301 redirect to https” then you have to fix your nginx.conf.
Please post your nginx.conf of the SeaTable container. I only need the part for port 80. My looks like
server {
listen 80;
server_name your-domain;
# for letsencrypt
location /.well-known/acme-challenge/ {
alias /var/www/challenges/;
try_files $uri =404;
}
location / {
if ($host = your-domain) {
return 301 https://$host$request_uri;
}
}
}
After you changed the nginx.conf, you have to reload nginx with the commands:
nginx -t
nginx -s reload
Then try again to reach the test.txt file with curl.
I followed your instructions and created the file. I can confirm that it was created by checking the alias folder /var/www/challenges/acme-challenge via terminal. When running the curl command from another terminal window outside the SeaTable network (I guess that’s the idea) or just a browser window, I get a 404 error:
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.22.0</center>
</body>
</html>
My nginx.conf looks like this (as far as I can see identical to yours):
upstream dtable_servers {
server 127.0.0.1:5000;
keepalive 15;
}
server {
listen 80;
server_name my-domain.com;
# for letsencrypt
location /.well-known/acme-challenge/ {
alias /var/www/challenges/;
try_files $uri =404;
}
location / {
if ($host = my-domain.com) {
return 301 https://$host$request_uri;
}
}
}
server {
server_name my-domain.com;
listen 443 ssl;
ssl_certificate /opt/ssl/my-domain.com.crt;
ssl_certificate_key /opt/ssl/my-domain.com.key;
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA;
ssl_session_cache shared:SSL:50m;
ssl_prefer_server_ciphers on;
# for letsencrypt
location /.well-known/acme-challenge/ {
alias /var/www/challenges/;
try_files $uri =404;
}
Probably stupid question: did you replace “your-domain” and “my-domain” with your specific url?
Because 404 means that nginx could not route your request. Even if the txt file is in the wrong place it should then show 301…
Therefore, I assume that there is a problem with your domain or that you are using ipv6 instead of ipv4.
But yes, ofc it is the specific URL. Also Seatable was running perfectly for more than a year already and is even now running without any problems over http as long as you accept the expired certificate.
It’s a mystery. What do you mean by using ipv6 instead of ipv4? How can I check? There are DNS records for A and AAAA if that is the question.
Please post the output of the following commands. Please replace with your domain:
curl http://your-domain/server-info/ # should return a 301
curl https://your-domain/server-info/ # should be something like {"version":"3.2.6","edition":"enterprise edition"}
curl http://your-domain/.well-known/acme-challenge
curl http://your-domain/.well-known/acme-challenge/
user@user ~ % curl http://my-domain.com/server-info/
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.22.0</center>
</body>
</html>
user@user ~ % curl https://my-domain.com/server-info/
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
user@user ~ % curl http://my-domain.com/.well-known/acme-challenge
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.22.0</center>
</body>
</html>
user@user ~ % curl https://my-domain.com/.well-known/acme-challenge
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Hey pueblo,
until here, everything looks fine. For more debugging, I need ssh-access to your system.
But what I can say is that this is for sure no SeaTable problem. As long as you cannot reach your test.txt file lets encrypt will not update your certificate. It has to be related with network, routing, firewall or nginx configuration error.
Christoph
In the meanwhile I can report the following findings (probably desperation attempts). I did compare the installation with a completely fresh install (although enterprise edition) on another server where the .well-known folder is accessible as supposed. Basically everything looks exactly the same apart from the contents of the accounts.conf file in the ssl folder within seatable-data. The fresh install looks like this:
Also in the ssl folder I can see two files that are two years old and are not present with the fresh install: mydomain.csr and mydomain.account.key. Are those maybe leftovers from older authentication attempts?
Also within container when I navigate to the var/www/challenges/acme-challenge folder I can see (next to the generated test.txt) the many challenge-files from my authorization attempts. Are those supposed to stay there when authorization is unsuccessful? I actually thought they were supposed to always get deleted when the process finishes?
And a final observation. The letsencrypt.log has no more entries after the day the certificate expired and the renewal failed. That’s quite strange with all the renewal attempts I did in the meanwhile
However all this probably still has nothing to do with the fact that the folder cannot be reached from the web!?
Only other difference I can think of is that the fresh install has Hetzner DNS nameservers while the problematic one has DNS managed by Cloudflare (A and AAAA records). They are not proxied though.
I did install it strictly following the SeaTable manual for Developer Edition (Docker). SSL was working fine for two years with one exception when an update broke something which was easily fixed with the kind support here. I do not have any special configurations at all, and apart from the initial setup - as per manual - no changes were made anywhere. Also, as mentioned before, apart from the expired certificate the deployment is running smoothly without any problems and perfectly reachable from the web.
So, I was able to make the test file accessible via curl by manually changing the ownership of the acme-challenge folder from root:root to www-data:www-data. I tried this because with the working fresh installation this is the ownership of the folder. Now when I run the renew script though it changes ownership back to root:root and the challenge fails again.
This is exactly the problem and as it seems it’s probably just a coincidence that I’m the only one encountering it so far and others might run into this as well.
Unfortunately the solution from the last post in the link above doesn’t work for me. Ownership is always again overwritten by the renew script and the challenge fails.
I had exactly this same problem this month, and like you narrowed it down to the file ownership problem.
My extremely inelegant solution was to have a second terminal window open, and repeatedly send the chown command to fix the permissions issue as the renewal script ran. (I probably issued the chown command a dozen times while the renewal script ran.) It was very kludgy but it worked. … Hopefully helpful.
In my case that is also not happening.Maybe a webroot path problem involved here?
