So like many I’ve been playing with local LLMs, here is how I’ve setup secure access to my local instance, using tailscale and my golang light proxy

What you need

  • A machine with a suitable GPU (in my case a desktop with a rtx 4090 but could also be an apple silicon mac, an Nvidia DGX Spark, etc)
  • Docker (Docker desktop or something compatible)
  • Tailscale
  • Fortio proxy

That’s it!

Optionally install ollama directly (I did) if you prefer local ollama vs running inside Open Web UI’s image.

Enabling https/TLS certs for Tailscale

In the admin DNS page of the admin console login.tailscale.com/admin/dns, under HTTPS Certificates, select Enable HTTPS (and read the certificate transparency disclaimer).

If you haven’t done so already pick a “fun name” for your ts.net while you’re at it.

See more details or updates in Tailscale’s doc if needed.

Starting the LLM Web UI

From getting started docs:

docker run -d -p 3000:8080 -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

This gives you access on http://localhost:3000 but the next step will give https from anywhere on your tailscale network

Starting the https proxy

If you have go installed you can go install fortio.org/proxy@latest otherwise get the binary for your host at github.com/fortio/proxy/releases

Note: I use a binary directly on the host as accessing the tailscale daemon from inside docker isn’t easy, or rather I didn’t know how to map the unix domain socket it expects on linux to the host windows pipe for instance, if you know how, please let me know, I’d rather use docker run -d for the proxy too (the image is fortio/proxy:latest if you get it working)

proxy -tailscale -default-route localhost:3000 -timeout 0

Setting timeout to 0 allows slow download of new models (infinite max duration, use -timeout 20m for 20 minutes for instance)

(ps: on mac/linux you can run it through systemd, or plain nohup proxy & to keep it running; on windows start /b etc)

That’s it

Enjoy https://yourmachine.yourtsnet.ts.net/ access!

Secure access sshot