Hugo MinIO Transition Fun

Background

It’s not a dev blog if you don’t change everything about it every time you remember to publish a blog, and this post is no different. My blog has long been on Jekyll, but I also lose it (literally sometimes I can’t find the source) and forget about it.

Recently I’ve set up a MinIO cluster on my personal infrastructure for hosting some internal state and content (Forgejo assets, Matrix Synapse images, Outline assets, etc) and I’ve been looking for other ways to utilize this service I’m hosting. For the past couple of years my blog has been hosted in Azure in an Azure Blob fronted by Azure Front Door (for https). This is ok, but it’s a real pain to remember how to set it up and the terraform never fully encompassed all the options as some of them weren’t exposed to the API (at least when I first terraformed it).

I got to thinking that AWS lets you host a website from an s3 bucket, can we do the same with a MinIO bucket? It’s intended to be a closely compatible implementation of s3, so I decided to try it!

A new blogging framework has entered the chat

I first tried to write a new post in Jekyll, to give me something to publish and test, but I was tired and not able to debug clearly and I thought it was incompatible with the ruby version I was giving it, instead it was just that I didn’t read the error. But as I tend to do when I’m tired and not thinking clearly, I just burned it all down and reached for another blogging framework, Hugo.

That’s ok, Hugo has a Jekyll Importer and my content isn’t anything complicated, so that was easy. Ok, how do we deploy Hugo? That’s a good next step.

Hugo Deploy

Hugo Deploy lets me push straight to an s3 or s3-compatible bucket? That’s great! How lucky that I can stop using rclone and just let hugo be in charge of publishing. I love using every possible feature of a piece of software, so while rclone is awesome, why not use Hugo’s Deployment?!

Set up the MinIO Infra

  1. I create a MinIO Bucket (blog)
  2. Created a user blog with an s3 policy for read/write on the bucket:
{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Action": [
    "s3:*"
   ],
   "Resource": [
    "arn:aws:s3:::blog/*"
   ]
  }
 ]
}
  1. Set my AWS-compatible Environment Variables in my environment like so:
AWS_ACCESS_KEY_ID=<secret-key-id>
AWS_SECRET_ACCESS_KEY=<secret-key>
AWS_DEFAULT_REGION=ATL-USA
  1. Added a deploy block to my hugo.yaml following the instructions in the docs about s3-compatible endpoints

For S3-compatible endpoints, see https://gocloud.dev/howto/blob/#s3-compatible

deployment:
  targets:
    - name: production
      url: "s3://mybucket?endpoint=minio-us.infra.frodux.in:443&disableSSL=false&s3ForcePathStyle=true"
  1. Press the buttons, let’s go this is easy!
hugo deploy
Deploying to target "production" (s3://mybucket?endpoint=minio-us.infra.frodux.in:443&disableSSL=false&s3ForcePathStyle=true)
Error: open bucket s3://mybucket?endpoint=minio-us.infra.frodux.in:443&disableSSL=false&s3ForcePathStyle=true: unknown query parameter "disableSSL"

Debugging why deploy doesn’t work

Oh, well, that’s weird. That’s fine, maybe there’s a bit of drift in the docs, totally happens all the time. First we search the docs and GitHub Issues for Hugo. All I really found was this one issue, #6116, which spawned the docs that don’t work for me.

So I move towards trying harder, trying some alternative forms of the URL. I can read the docs and the go-code, this should be possible. Paring down the unsupported options, all the way down to `s3://blog?endpoint=minio-uss.infra.frodux.in" results in the following:

Deploying to target "production" (s3://blog?endpoint=minio-us.infra.frodux.in)
Error: blob (code=Unknown): operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "//blog./minio-us.infra.frodux.in?list-type=2&max-keys=1000": unsupported protocol scheme ""
Lots more attempts, if you'd like to see them

Ok, let’s then specify a protocol on the endpoint and use the secret awssdk=v2 option we found in the gocloud docs.

Deploying to target "production" (s3://blog?endpoint=https://minio-us.infra.frodux.in&awssdk=v2)
Error: blob (code=Unknown): operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://blog.minio-us.infra.frodux.in/?list-type=2&max-keys=1000": dial tcp: lookup blog.minio-us.infra.frodux.in: no such host

Wait why is it now making the endpoint blog.<endpoint>? Oh, let’s force path style instead of DNS style as documented in gocloud code

hugo deploy
Deploying to target "production" (s3://blog?endpoint=https://minio-us.infra.frodux.in&awssdk=v2&s3ForcePathStyle=true)
Error: open bucket s3://blog?endpoint=https://minio-us.infra.frodux.in&awssdk=v2&s3ForcePathStyle=true: unknown query parameter "s3ForcePathStyle"

or

hugo deploy
Deploying to target "production" (s3://blog?endpoint=https://minio-us.infra.frodux.in&awssdk=v2&use_path_style=true)
Error: open bucket s3://blog?endpoint=https://minio-us.infra.frodux.in&awssdk=v2&use_path_style=true: unknown query parameter "use_path_style"

Eventually I started doubting that this could work. The documentation said it did, maybe it did at one time, but not for me! So I started tracing the code to see where things were going wrong. Hugo seems to hand off all the s3 stuff to gocloud, so I made a go playground to understand some parsing and then started digging in deeper there. I found awssdk=v2 in the s3 section of the docs which then led me to how to support s3PathStyle (required for MinIO) which only seems to be documented in code.

Still, none of these flags worked, so I decided to see if maybe Hugo’s gocloud version was out of date. And it was, not significantly, just 1 release behind (0.39 to 0.40), but that’s something, so I checked the changelog and…

blob/s3blob: custom endpoints with s3 and aws sdk v2 by @caarlos0 in #3473

hmm this is at least related to my problem, doesn’t sound like exactly what I’m experiencing but maybe there are some implementation details I don’t understand.

Contributing to Hugo

I’m not quite sure how to test that this will fix my issue other than building hugo with this library. So it’s time to clone Hugo and attempt the upgrade. Thankfully the CONTRIBUTING.md is very newcomer friendly, so I was off and running building the version I wanted in just a few minutes. I first tried with the HEAD of their master branch, and it exhibited the same issue, so I bumped the gocloud version and tried again and suddenly it pushed my assets! Cool! Time to make a PR!

$ hugo version
hugo v0.141.0-DEV-d66f1aa08f087f99a631227fec4e2939f199cbc3+extended+withdeploy linux/amd64 BuildDate=2024-12-17T17:12:29Z

$ hugo deploy
Deploying to target "production" (s3://blog?endpoint=https://minio-us.infra.frodux.in&awssdk=v2&use_path_style=true&disable_https=false)
Identified 27 file(s) to upload, totaling 148 kB, and 0 file(s) to delete.
Success!
Success!

I added a new line with an example working s3 url (very close to my final URL) just to help the next person in the docs too: https://github.com/gohugoio/hugo/pull/13158, outlining why I was upgrading this and the relevant upgrade issue/links that indicated this was the right course of action.

It’s at this point I realize, I never asked for help from the Hugo community. Whoops! I was so busy just trying to get this to work, I didn’t pause and try to lean on the expertise of others. Well, to cover this in case someone comes along asking the question later, I created #13159. Maybe someone will say “oh X works you don’t need that PR to accomplish this”, but I think the PR is still good because we can be on the latest version of gocloud anyways.

Conclusion

Well, I can now publish to my MinIO bucket with my local built copy of hugo deploy, so I can now write this blog post, because <sarcasm>obviously without an entire new blog framework I couldn’t write a single new post</sarcasm>. I’ll post a follow-up PR on how to use NGINX to host the content inside of the MinIO Bucket, since my MinIO is only exposed to my Tailscale Tailnet and I want to have a nice cert, url, etc for the blog.

Code isn’t scary, it can be big and complicated and confusing, I learned again today that if you take the time to step through and see where things might be going wrong, you can learn something new and contribute something to the community while you’re at it!