Building a Resilient Backup VPN for My Homelab with Headscale (Self‑Hosted Tailscale)

Why I Needed a “Backup VPN” for My Homelab

Like many homelab users, my primary remote-access solution has long been WireGuard. It’s fast, elegant, and simple.
However, WireGuard has one structural weakness:

It relies on UDP.

In practice, this becomes a real problem on:

• Public Wi‑Fi (cafés, airports, hotels)
• Corporate guest networks
• Cellular hotspots with aggressive NAT or filtering

I repeatedly ran into situations where:

• WireGuard would not connect at all
• Other VPN protocols also failed
• Only TCP 443 (HTTPS) traffic was allowed

At that point, speed no longer matters — connectivity does.

So my goal became very specific:

Design a fully self‑hosted, homelab‑native VPN that automatically falls back to TCP 443 when UDP is blocked.
No commercial VPNs. No subscriptions. No “black box” dependencies.

Why Not Just Use a Commercial VPN?

Commercial VPNs do solve the TCP 443 problem — but they introduce new tradeoffs:

• Recurring cost
• External trust dependency
• No direct access to my private LAN
• Extra hops and unpredictable routing

I wanted:

• Direct access to my home network
• Full control over routing and security
• Infrastructure I can reason about and document
• A dedicated solution that fits my homelab

This naturally led me to Tailscale — and then to Headscale.

Tailscale vs Headscale

• Tailscale: Managed control plane + WireGuard data plane
• Headscale: Open‑source, self‑hosted control plane compatible with Tailscale clients

Key insight:

You can self‑host the control plane (Headscale)
while still benefiting from Tailscale’s mature clients, NAT traversal, and DERP fallback.

This gives you:

• WireGuard performance on good networks
• Automatic TCP 443 fallback via DERP on hostile networks
• No vendor lock‑in

High‑Level Architecture

Key Design Choices

• Headscale as the single source of truth
• Dedicated subnet router (ts-router) instead of overloading existing hosts
• SNAT/MASQUERADE to avoid fragile return routing
• No exit node — this is a home access VPN, not a traffic tunnel

Setup Step 1: Headscale Control Plane (LXC)

Recommendation: Use a dedicated, single-purpose container. Do not mix this with Caddy or other business apps. Headscale is just a control plane HTTP service and does not need TUN device access.

Recommended Config:

• OS: Debian 12 or Ubuntu LTS
• CPU/RAM: 1 vCPU / 256MB+
• Network: Bridged to LAN (must be reachable by Caddy)

Configuration Tweaks

The config file is usually at /etc/headscale/config.yaml. The key requirements are: server_url must be your public HTTPS domain, and listen_addr should listen internally for Caddy to reverse proxy.

Example config:

1server_url: "https://headscale.example.com"
2listen_addr: "0.0.0.0:8080"
3
4# Disable headscale's built-in TLS handling
5tls_letsencrypt_hostname: ""
6tls_letsencrypt_cache_dir: ""

Caddy Reverse Proxy (TLS Termination)

On your Caddy machine/container:

1headscale.example.com {
2  reverse_proxy <HEADSCALE_LXC_IP>:8080
3}

Verify: curl -vk https://headscale.example.com/health should return 200 and {"status":"pass"}.

Create User & Pre-auth Key

1# Find the ID for your user (e.g., johnny)
2headscale users list
3
4# Create a pre-auth key for the router to join
5headscale preauthkeys create --user 1 --expiration 24h

Setup Step 2: Data Plane & Router (ts-router LXC)

This is the most critical part. The ts-router must have TUN access enabled, otherwise tailscaled will fail.

LXC Settings:

• TUN: ✅ Must be Enabled
• OS: Debian 12 (Recommended)
• Network: Static IP recommended

Install & Join Headscale

 1# Install Tailscale
 2curl -fsSL https://tailscale.com/install.sh | sh
 3systemctl enable --now tailscaled
 4
 5# Join and advertise routes
 6sudo tailscale up \
 7  --login-server=https://headscale.example.com \
 8  --authkey=tskey-xxxxxxxxxxxxxxxx \
 9  --advertise-routes=192.168.10.0/24 \
10  --accept-dns=false

After joining, you must enable the route on the Headscale side:

1headscale routes list
2# Find route ID
3headscale routes enable --route <ROUTE_ID>

Critical: Enable IP Forwarding & SNAT

This is where most setups fail.

1. Enable IPv4 Forwarding:

1echo 'net.ipv4.ip_forward=1' >/etc/sysctl.d/99-tailscale-router.conf
2sysctl -p /etc/sysctl.d/99-tailscale-router.conf

2. Configure SNAT/MASQUERADE:

Since your home gateway typically doesn't know how to route return traffic to 100.64.0.0/10, you must use SNAT to ensure response packets are routed back to the ts-router.

1# Temporary test
2iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -o eth0 -j MASQUERADE
3
4# Persistence (Recommended)
5apt install -y iptables-persistent
6# Choose YES to save rules during install

Once SNAT is in place, internal access should work immediately.

Setup Step 3: Client Configuration

macOS / Linux

Use the CLI to join. The key flag is --accept-routes so the system respects the advertised LAN routes.

1tailscale up \
2  --login-server=https://headscale.example.com \
3  --accept-routes

Android / iOS

1. Navigate to Settings (sometimes hidden under "Advanced" or requires tapping the version number) and change the Control URL to https://headscale.example.com.
2. Log in using an Auth Key or via the browser.
3. Enable "Accept routes" in the client settings.

Verification & Testing

1. Basic Connectivity

From an external network (e.g., via phone hotspot), try to ping an internal IP (like 192.168.10.1).

2. Simulate UDP Block (Automatic Fallback Test)

When UDP is blocked, Tailscale should automatically fall back to DERP relays (TCP 443).

Check status via CLI:

1tailscale status
2# Look for relay "sea" or similar, indicating relay usage

tailscale netcheck provides detailed status on UDP vs DERP connectivity.

Security & Trust Model

This setup intentionally avoids:

• Exposing the home router directly
• Opening inbound VPN ports on the LAN
• Granting full-tunnel exit-node access

Instead:

• Only authenticated devices join the tailnet
• Only approved subnet routes are accepted
• Traffic scope is limited to the home LAN

This aligns well with a least-privilege homelab philosophy.

Conclusion

This project wasn’t about chasing novelty — it was about eliminating a real reliability gap.

By leveraging Headscale and a dedicated router container, we built a system that:

• Uses WireGuard (UDP) whenever possible for performance
• Transparently falls back to TCP 443 (DERP) on restrictive networks
• Remains fully self‑hosted and auditable

Most importantly, it connects reliably anywhere — making it the perfect backup solution.