Bringing your entire infrastructure down using Caddy
If you've talked to me about server infrastructure and DevOps before, there's a reasonable chance that I have told you about our lord and saviour Caddy. If you're not aware about it, it's an HTTP/2 web server written in Go that has super simple configuration and automatic HTTPS certificates. Neat, right? Right.
Yesterday Caddy successfully crashed all websites running on my server thanks to automatic HTTPS certificates
Setting the stage
Yesterday I was moving my blog from blechi.at to pretzelhands.com - The reason being that the shared host that runs behind blechi.at (which is a domain I share with my dad), seems to block requests from various Asian countries. I don't know why it happens and I can't be assed to ask support about it.
Kill your infrastructure in 60 seconds
Caddy requests its certificates from Let's Encrypt, as everyone should be doing in
$CURRENT_YEAR. So while I was happily rearranging my config for pretzelhands.com, I restarted my server multiple times.
Apparently I had setup my service for Caddy wrong however, causing it to register another account with Let's Encrypt
every time I restarted the server. Usually this isn't a problem, because I restart my server maybe once or twice.
I was fiddling around a lot yesterday, however and kept on requesting more and more accounts.
Turns out that Let's Encrypt really don't like that.
Before I knew I hit a rate-limit and my entire server just came crashing down in a magical rainbow of red error messages. According to Let's Encrypt this rate limit for accounts applies to 10 accounts per 3 hours. I have been trying to wait the specified amount of time and request a fresh set of certificates. So far I haven't been successful. Maybe they're also pissy about me requesting multiple certificates for the same domain. I simply don't know.
So I did the next best thing and pulled up nginx, which was stilled installed on my server anyways. And now here we are.
How to prevent this
One thing I learned about very quickly yesterday is that Caddy offers a
-ca flag with which you can point it to the
staging environment of Let's Encrypt. That is what I should have done. I've already implented a little script and called
caddy-test. It will do just that.
I still love Caddy with all of my head, but this was a real shotgun to the foot moment. Woops. Now excuse me while I bring the rest of my infrastructure back up. Grmbl.