Liferay Tunnel (lfr-tunnel) End-to-End Server & DNS Setup Guide¶

This guide walks you through setting up a complete, production-grade lfr-tunnel gateway server from scratch. By following this guide, you will be able to replicate the exact infrastructure used for the official Sales Engineering gateway using your own public domains and VPS hosting.

1. Domain & DNS Configuration¶

Before setting up your VPS, you must configure your domain name to route visitor requests and developer tunnels to your gateway's public IP address.

1.1. Required DNS Records¶

Log into your DNS provider (e.g., Cloudflare, Route53, GoDaddy) and add the following records for your domain (e.g. yourdomain.com):

Type	Name	Value	TTL	Proxy Status (Cloudflare)	Description
A	`@`	`YOUR_VPS_PUBLIC_IP`	Auto	DNS Only (Grey Cloud)	Root domain pointing to VPS
A	`tunnel`	`YOUR_VPS_PUBLIC_IP`	Auto	DNS Only (Grey Cloud)	Control plane registration endpoint
A	`*`	`YOUR_VPS_PUBLIC_IP`	Auto	DNS Only (Grey Cloud)	Wildcard for active developer tunnels

Info

Disable Cloudflare Proxy (CDN)
You MUST set the Proxy Status to DNS Only (Grey Cloud). If Cloudflare proxies the traffic (Orange Cloud), it will interfere with Chisel's persistent WebSocket connections, block wildcard SSL verification via Nginx, and break direct TCP forwarding.

1.2. Email Security TXT Records¶

Because lfr-tunneld can send emails for user registration and administrative approvals, you must configure SPF, DKIM, and DMARC records to prevent mail servers from rejecting or flagging notifications as spam/forgery:

SPF (Sender Policy Framework): Add a TXT record for @:

v=spf1 ip4:YOUR_VPS_PUBLIC_IPV4 ip6:YOUR_VPS_IPV6 -all

DMARC (Domain-based Message Authentication): Add a TXT record for _dmarc:
```
v=DMARC1; p=reject; sp=reject; adkim=s; aspf=s;
```
DKIM (DomainKeys Identified Mail): Configure a TXT record *._domainkey:
```
v=DKIM1; p=
```

2. VPS Server Setup & Security Hardening¶

Provision a clean VPS running Ubuntu 22.04 LTS or Ubuntu 24.04 LTS (e.g., on DigitalOcean, Hetzner, AWS, or Linode) with at least 1 vCPU and 1GB RAM. If provisioning on AWS specifically, see the AWS EC2 Provisioning Guide first — it covers the AWS-specific steps (Elastic IP, security groups) before continuing with §2.1 below.

2.1. Basic OS & Package Updates¶

Once logged in via SSH as root, update all system packages:

apt update && apt upgrade -y

2.2. Create a Restricted Sudo User¶

Do not run services or manage the server as root directly. Create a new administrative user (e.g. adminuser):

# Add the new user
adduser adminuser

# Grant sudo permissions
usermod -aG sudo adminuser

2.3. Hardening SSH Configuration¶

Disable root login and password-based authentication to prevent brute-force attacks.

Authorize your SSH Key for the new user:

# Switch to the new user
su - adminuser
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# Paste your public SSH key into authorized_keys
nano ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
exit

Modify the SSH Daemon Configuration: Open /etc/ssh/sshd_config:
```
sudo nano /etc/ssh/sshd_config
```
Ensure the following directives are configured:
```
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
```
Restart the SSH service:
```
sudo systemctl restart sshd
```
Note: Keep your current terminal open and test logging in via a separate terminal to verify you can connect successfully before exiting.

2.4. Configure the Firewall (UFW)¶

Only expose necessary ports. Block all other traffic:

# Allow SSH
sudo ufw allow ssh

# Allow HTTP (port 80) and HTTPS (port 443)
sudo ufw allow http
sudo ufw allow https

# Enable the firewall
sudo ufw enable

# Check status
sudo ufw status

3. Nginx Reverse Proxy & Let's Encrypt Wildcard SSL¶

lfr-tunneld runs on localhost, while Nginx acts as the public-facing entrypoint, handling SSL termination and proxying traffic to the backend.

3.1. Install Nginx and Certbot¶

Install Nginx, Certbot, and the Certbot DNS plugin for your DNS provider (e.g., Cloudflare):

sudo apt install -y nginx certbot python3-certbot-nginx python3-certbot-dns-cloudflare

3.2. Obtain Wildcard SSL Certificates¶

Because dynamic developer subdomains (e.g., *.yourdomain.com) are routed on this server, you must obtain a wildcard certificate. Let's Encrypt only supports wildcard validation using the DNS-01 challenge.

Create a Cloudflare API Token with permissions to edit your domain's DNS zone files.

Save the token in a secure file on the VPS:

sudo mkdir -p /etc/letsencrypt
sudo nano /etc/letsencrypt/cloudflare.ini

Write the following:

dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN

Secure the credentials file:

sudo chmod 600 /etc/letsencrypt/cloudflare.ini

Run Certbot to fetch wildcard certificates for both root and wildcard subdomains:

sudo certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  --agree-tos \
  --no-eff-email \
  -m admin@yourdomain.com \
  -d yourdomain.com \
  -d *.yourdomain.com

3.3. Configure Nginx Configuration¶

Create an Nginx configuration file at /etc/nginx/sites-available/lfr-tunnel:

sudo nano /etc/nginx/sites-available/lfr-tunnel

Paste the following configuration (replace yourdomain.com with your actual domain):

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# 1. HTTP to HTTPS Force Redirect
server {
    listen 80;
    listen [::]:80;
    server_name yourdomain.com *.yourdomain.com;

    return 301 https://$host$request_uri;
}

# 2. Regional Edge Redirect (us.yourdomain.com -> yourdomain.com)
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name us.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    return 301 https://yourdomain.com$request_uri;
}

# 3. Main Landing Redirect (yourdomain.com -> portal.yourdomain.com)
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    return 301 https://portal.yourdomain.com$request_uri;
}

# 4. Control Plane & Portal Server
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name portal.yourdomain.com tunnel.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    # Root landing page (proxies to portal/dashboard of lfr-tunneld)
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
    }

    # Proxy CLI registration API to lfr-tunneld
    location /api/ {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Proxy Chisel WebSocket handshake endpoint
    location /tunnel {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# 3. Wildcard Subdomain Data Plane Routing
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name *.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto https;

        # Increase timeouts for uploading large files/assets
        client_max_body_size 500M;
        proxy_connect_timeout 120s;
        proxy_send_timeout 120s;
        proxy_read_timeout 120s;
    }
}

Enable the Nginx configuration and restart Nginx:

sudo ln -s /etc/nginx/sites-available/lfr-tunnel /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

3.4. Configure Nginx Maintenance Mode (Optional but Recommended)¶

To prevent 502 Bad Gateway errors from being displayed to users during database backups restoration or server updates, you can configure Nginx to automatically intercept traffic and serve a beautiful, static maintenance page when the trigger file is present.

Create the web root directory for the static maintenance assets:
```
sudo mkdir -p /var/www/lfr-tunnel
```
Update /etc/nginx/sites-available/lfr-tunnel by adding the following block inside both SSL server blocks (the Control Plane block yourdomain.com and the Wildcard Data Plane block *.yourdomain.com), just after the SSL certificate configurations:

    # Maintenance Check Block
    # Checks for the presence of the maintenance trigger file
    if (-f /var/lib/lfr-tunneld/maintenance.enable) {
        set $maintenance 1;
    }
    # Allow rendering the maintenance page itself without redirect loops
    if ($uri = "/maintenance.html") {
        set $maintenance 0;
    }
    if ($maintenance = 1) {
        return 503;
    }


    error_page 503 /maintenance.html;
    location = /maintenance.html {
        root /var/www/lfr-tunnel;
    }

Validate the configuration and reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

4. Install & Configure `lfr-tunneld`¶

4.1. Create a Restricted System User¶

To secure the host, create a dedicated system user lfr-tunnel with no shell or home directory to execute the daemon process:

sudo useradd -r -s /bin/false lfr-tunnel

4.2. Build the Server Binary¶

Build the Go binary locally on your computer and copy it to the VPS, or build it directly on the VPS if you have Go installed:

# Compile. `make build` rather than a bare `go build`: the Makefile pins GOTMPDIR, and the
# toolchain links inside GOTMPDIR before moving the result to -o, so a bare build leaves an
# unsigned binary in the system temp directory whatever -o says (#1337, #1859).
make build

# Install binary to system path
sudo cp bin/lfr-tunneld /usr/local/bin/
sudo chmod 755 /usr/local/bin/lfr-tunneld
sudo chown root:root /usr/local/bin/lfr-tunneld

4.3. Configuration Files Setup¶

Create a configuration directory and secure it so only the daemon can read the sensitive shared secret token:

sudo mkdir -p /etc/lfr-tunneld
sudo nano /etc/lfr-tunneld/server-config.yaml

Start from the committed reference, which lists every key the gateway accepts with a placeholder value:

sudo cp resources/server/server-config.example.yaml /etc/lfr-tunneld/server-config.yaml
sudo nano /etc/lfr-tunneld/server-config.yaml

It is deliberately the complete key list rather than a minimal sample. A test (TestExampleConfigCoversEveryField) fails the build if a setting is added to the code without appearing there, so it cannot quietly fall behind — an example that omits a setting is worse than no example, because it reads as authoritative.

The values in it are placeholders and describe no real deployment. Credentials — SMTP, webhook URLs, edge token_hash values — are left empty and supplied out of band; do not commit them.

At minimum you must set domains, owner.user_id, db_path and the listener addresses. The settings below are the ones most worth understanding before you start; the rest are safe at their defaults.

Before restarting the gateway on any config change, validate the file first:

sudo /usr/local/bin/lfr-tunneld -check-config -config /etc/lfr-tunneld/server-config.yaml

Exit 0 means it would start. Exit 1 prints the parse error — which is the difference between finding out now and finding out during a restart the control plane does not come back from.

Every flag `lfr-tunneld` accepts¶

Rendered from the parser rather than transcribed, and kept honest by a test that fails when a flag is added without documenting it (#2092):

Flag	Default	What it does
`-config <path>`	—	Path to `server-config.yaml`.
`-check-config`	off	Validate the configuration and exit, without starting the gateway or touching the database.
`-domains <list>`	—	Comma-separated wildcard domains, overriding `domains:` in the file.
`-bind <addr>`	—	HTTPS gateway bind address (e.g. `:443`), overriding `bind_addr:`.
`-http-bind <addr>`	—	HTTP gateway bind address (e.g. `:80`), overriding `http_bind_addr:`.
`-cert <path>`	—	Wildcard SSL certificate path, overriding `ssl_cert_file:`.
`-key <path>`	—	Wildcard SSL private key path, overriding `ssl_key_file:`.

All but -config and -check-config override a key the configuration file already sets. The systemd unit in this guide passes only -config, deliberately: a flag in the unit file is a setting that is not in server-config.yaml, so it is invisible to anyone reading the config and to -check-config.

Note

Slack & Microsoft Teams Notifications Configuration Liferay Tunnel supports two secure options for routing gateway notification alerts (user registration requests, rate limit blocks, abuse reports, manual IP bans) directly to your Slack or Teams channels:

Incoming Webhook URL (Preferred):
Create an Incoming Webhook App inside your corporate Slack or Teams workspace mapped to a target channel (public or private).
Add the generated secret webhook URL to the webhooks.slack_url or webhooks.teams_url parameter in server-config.yaml and set webhooks.enabled: true.
This uses rich markdown formatting (Slack Block Kit or Office 365 Message Cards) to deliver beautiful notifications asynchronously.
Slack Channel Email Address (Alternative):
Generate an email address for your target Slack channel inside Slack channel settings (under Integrations > Send emails to this channel).
Configure admin_notification_email to point directly to that address in server-config.yaml (e.g. admin_notification_email: '"lfr-tunnel-admin (Slack)" <lfr-tunnel-admin-xxx@liferay.slack.com>').
Our mail client fully parses name-formatted email recipients natively, ensuring standard SMTP notifications are safely delivered and displayed directly inside the channel without requiring Slack App authorization.

Note

Slack App "Add to Slack" Redirect URL (Distribution, Not Notifications) The two options above cover receiving alerts and don't require a Slack App at all. If you've instead created a real Slack App (e.g. to get an "Add to Slack" button so others can install it into their own workspace), Slack's app dashboard requires at least one Redirect URL before it will activate that button — this is a standard OAuth v2 installation flow, not a notification channel.

lfr-tunneld implements this redirect target at https://<your-domain>/api/integrations/slack/callback. To wire it up: 1. In your Slack app's dashboard, under OAuth & Permissions, add https://<your-domain>/api/integrations/slack/callback as a Redirect URL and pick the bot scopes your app needs. 2. Under Basic Information, note the App ID, Client ID, Client Secret, Signing Secret, and Verification Token. 3. Set the Client ID and Client Secret via secrets.env (§4.5) rather than server-config.yaml, since those two are the actual secrets:

# /etc/lfr-tunneld/secrets.env
LFT_SLACK_APP_ID=<App ID>
LFT_SLACK_CLIENT_ID=<Client ID>
LFT_SLACK_CLIENT_SECRET=<Client Secret>
LFT_SLACK_SIGNING_SECRET=<Signing Secret>
LFT_SLACK_VERIFICATION_TOKEN=<Verification Token>

(App ID and Client ID aren't secret on their own, but Slack's dashboard only hides the other three — placing all five together in secrets.env keeps the whole credential set in one root-only file instead of splitting it across two locations.) 4. Restart lfr-tunneld (sudo systemctl restart lfr-tunneld) so the new environment variables load, then click "Add to Slack" — the callback exchanges the one-time code for a workspace access token server-side and stores it in the gateway's database; the token is never displayed in a browser or committed to any file.

Tip

Native Multi-Factor Authentication (MFA / TOTP)
If enable_user_portal is set to true, users can activate 6-digit Time-Based One-Time Password (TOTP) MFA from their Account Settings tab. This secures passwordless portal sessions using two independent factors: possession of email (magic link) + possession of device (authenticator app). Gateway administrators can reset or disable a user's MFA status directly from the Admin Dashboard in case of lost devices.

# Access Control
allowed_email_domains:
  - "liferay.com"

# SMTP Relay Configuration (Required for registration & magic links)
# smtp_server is relay-agnostic: point it at a local Postfix daemon, a
# managed provider like Amazon SES, or any other SMTP relay -- see §4.4
# below for a walkthrough of both a self-hosted and a managed option.
# Note: Use your domain here instead of 127.0.0.1 to securely pass TLS verification
# with your Let's Encrypt certificates.
# If your relay needs a username/password (e.g. SES -- see §4.4.2), leave those two
# fields blank here and set LFT_SMTP_USERNAME/LFT_SMTP_PASSWORD instead, so the
# credentials never sit in this file at all. See §4.5's secrets.env for how.
smtp_server:
  host: "yourdomain.com"
  port: 25
  username: ""
  password: ""
  from_address: "Liferay Tunnel <noreply@yourdomain.com>"

Info

Owner Bootstrapping & Database Role Precedence
The owner.user_id and owner.name settings in the configuration file act solely as bootstrap properties. During the very first startup when the SQLite database is empty, the gateway automatically registers this email and name with the owner role. * Once the database is initialized, changes to owner.user_id and owner.name configuration keys will not modify or overwrite existing user records or names in the database. * The gateway code determines owner status by checking both the configured owner.user_id string and the user's role column in the database. Anyone assigned the owner role in the database gets full visibility privileges.

Apply restricted file permissions:

sudo chown -R lfr-tunnel:lfr-tunnel /etc/lfr-tunneld
sudo chmod 700 /etc/lfr-tunneld
sudo chmod 600 /etc/lfr-tunneld/server-config.yaml
# Note: if you later add /etc/lfr-tunneld/secrets.env (§4.5), don't re-run the
# recursive chown above afterward -- it would flip that file from root:root back to
# lfr-tunnel:lfr-tunnel, which still works but drops the "root-only" property.

4.4. Choosing an SMTP Relay¶

smtp_server in server-config.yaml is a plain host/port/username/password relay config — lfr-tunneld doesn't care what's on the other end, so pick whichever option fits your deployment. Two common choices are covered below: a self-hosted Postfix relay, or a managed provider like Amazon SES. Neither is required over the other — self-hosting keeps everything under your own control with no third-party account, while a managed relay avoids the ongoing IP-reputation and TLS/rDNS maintenance a self-hosted relay needs.

4.4.1. Option A: Local Postfix Email Relay (TLS Verification & rDNS Alignment)¶

If you are running a local Postfix daemon to send emails, you must securely configure Postfix to present your Let's Encrypt certificates and align its HELO SMTP banner (myhostname) with your public IP's PTR (reverse DNS) record to prevent major mail hosts (like Google or Microsoft) from flagging outbound notifications as spam/forgery.

To align the banner, bind the certificates to Postfix, and allow relaying from the domain's resolved IP, run the following:

# 1. Align SMTP HELO Banner with your public reverse DNS (PTR) record
sudo postconf -e "myhostname = tunnel.yourdomain.com"
sudo postconf -e "myorigin = yourdomain.com"

# 2. Configure Postfix to present your secure Let's Encrypt SSL certificates
sudo postconf -e "smtpd_tls_cert_file=/etc/letsencrypt/live/yourdomain.com/fullchain.pem"
sudo postconf -e "smtpd_tls_key_file=/etc/letsencrypt/live/yourdomain.com/privkey.pem"

# 3. Allow secure relaying from localhost and your VPS external IPs
sudo postconf -e "mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 YOUR_VPS_PUBLIC_IPV4 YOUR_VPS_PUBLIC_IPV6"

# 4. Restart Postfix to apply all updates
sudo systemctl restart postfix

Make sure that your smtp_server.host in the server-config.yaml points to your public domain (e.g., yourdomain.com) rather than 127.0.0.1 so that the hostname securely matches the Common Name (CN) of the certificate! Also ensure your VPS's external IP addresses are added to Postfix's mynetworks as shown above.

4.4.2. Option B: Amazon SES (Managed Relay)¶

Amazon SES is a managed alternative to a self-hosted Postfix relay — AWS handles IP reputation and delivery infrastructure, at the cost of requiring an AWS account.

Create a domain identity and get its DKIM tokens:

aws sesv2 create-email-identity --email-identity yourdomain.com --region us-east-1

Publish the 3 returned DKIM tokens as CNAME records: <token>._domainkey.yourdomain.com → <token>.dkim.amazonses.com. SES verifies domain ownership automatically once these propagate — no separate verification TXT record is needed:
```
aws sesv2 get-email-identity --email-identity yourdomain.com --region us-east-1 \
  --query '{Verification:VerificationStatus, Dkim:DkimAttributes.Status}'
```
SES accounts start in a sandbox (200 messages/day, verified recipients only) — request production access from the SES console's "Account dashboard" before relying on it for real user registrations/magic links.
Create an IAM user/role scoped to send-only (ses:SendEmail, ses:SendRawEmail), then generate SES SMTP credentials for it from the SES console's "SMTP Settings" page (it performs the IAM secret key → SMTP password conversion for you).

Point smtp_server at SES's regional SMTP endpoint, but don't put the SMTP username/password in server-config.yaml — set them via the secrets.env mechanism in §4.5 instead, so they never sit in the main config file (which gets read, backed up, and copied around far more than a dedicated secrets file):

smtp_server:
  host: "email-smtp.us-east-1.amazonaws.com"
  port: 587
  username: ""
  password: ""
  from_address: "Your App <noreply@yourdomain.com>"

# /etc/lfr-tunneld/secrets.env -- see §4.5 for permissions/systemd wiring
LFT_SMTP_USERNAME=<SES SMTP username>
LFT_SMTP_PASSWORD=<SES SMTP password>

Add include:amazonses.com to your domain's SPF TXT record so receiving mail servers see SES's sending IPs as authorized alongside DKIM.

4.5. systemd Service Setup¶

If your SMTP relay needs a username/password (e.g. SES — see §4.4.2's step 5), create a root-only secrets file instead of putting them in server-config.yaml:

sudo touch /etc/lfr-tunneld/secrets.env
sudo chown root:root /etc/lfr-tunneld/secrets.env
sudo chmod 600 /etc/lfr-tunneld/secrets.env
sudo nano /etc/lfr-tunneld/secrets.env

# /etc/lfr-tunneld/secrets.env
LFT_SMTP_USERNAME=<SES SMTP username>
LFT_SMTP_PASSWORD=<SES SMTP password>

systemd's EnvironmentFile= directive below is read by the systemd manager itself (running as root) before it execs and drops to User=lfr-tunnel, so 600 root:root is sufficient — the service process never needs its own read access to the file, and nothing in /etc/lfr-tunneld besides this one file needs to hold the credentials. This is the same "restricted secrets file" pattern used for client-side tokens in getting_started.md's Option C, applied at the systemd layer instead of a shell profile.

Create a systemd unit file at /etc/systemd/system/lfr-tunneld.service:

sudo nano /etc/systemd/system/lfr-tunneld.service

Paste the hardened service script:

[Unit]
Description=Liferay Tunnel Gateway Daemon
After=network.target

[Service]
Type=simple
User=lfr-tunnel
Group=lfr-tunnel
WorkingDirectory=/etc/lfr-tunneld
EnvironmentFile=-/etc/lfr-tunneld/secrets.env
ExecStart=/usr/local/bin/lfr-tunneld --config /etc/lfr-tunneld/server-config.yaml
Restart=on-failure
RestartSec=5s

# Security Hardening (systemd Sandboxing)
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
CapabilityBoundingSet=
ReadOnlyPaths=/usr/local/bin/lfr-tunneld
ReadWritePaths=/etc/lfr-tunneld

[Install]
WantedBy=multi-user.target

The leading - on EnvironmentFile=-/etc/lfr-tunneld/secrets.env makes it optional — deployments that don't need any LFT_* secret overrides (e.g. a local Postfix relay with no auth) can skip creating the file entirely rather than the service failing to start over a missing file.

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable lfr-tunneld
sudo systemctl start lfr-tunneld

Verify service is active and listening on localhost:

sudo systemctl status lfr-tunneld
sudo journalctl -u lfr-tunneld -n 50 -f

4.6. Service Self-Healing (Nginx & Watchdog)¶

To make your VPS gateway fully self-healing and immune to crashes, configuration dependency omissions, or system freezes, deploy passive and active watchdogs:

1. Nginx Systemd Auto-Restart Override¶

By default, Nginx does not automatically recover on failure in standard OS packages. To enable Nginx self-healing: 1. Create the systemd override directory:

sudo mkdir -p /etc/systemd/system/nginx.service.d/

2. Create /etc/systemd/system/nginx.service.d/override.conf:

[Service]
Restart=on-failure
RestartSec=5s

3. Reload systemd and restart Nginx:

sudo systemctl daemon-reload
sudo systemctl restart nginx

2. Active Watchdog Script & Timer¶

An active watchdog runs every minute to verify Nginx and lfr-tunneld are answering, heals missing Let's Encrypt configuration files on the fly, and restarts whatever is not responding.

The script is not reproduced here. It lives in this repo as scripts/common/gateway-watchdog.sh, with its unit and timer beside it, and scripts/common/setup-central-vps.sh and setup-edge-vps.sh install all three. This section used to inline a full copy, and that copy had already drifted -- it was missing the Environment=LFT_BACKEND_PORT= line the real unit carries, which is the line without which the script exits immediately. A stale copy of an operational script is worse than a pointer to it, because it looks authoritative (#1412).

If you are provisioning by hand rather than with the setup scripts:

sudo install -m 700 -o root -g root scripts/common/gateway-watchdog.sh /usr/local/bin/
sed "s/__BACKEND_PORT__/<your backend port>/" scripts/common/gateway-watchdog.service \
  | sudo tee /etc/systemd/system/gateway-watchdog.service > /dev/null
sudo cp scripts/common/gateway-watchdog.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now gateway-watchdog.timer

What the watchdog tells you¶

The watchdog heals the box; on its own it told nobody, so a gateway could go down, restart, and come back with no trace while portal users saw an unexplained outage (#1875).

It now appends each restart to a spool file -- /var/lib/lfr-tunnel/watchdog-events.jsonl by default, or LFT_WATCHDOG_SPOOL if the unit sets one -- and lfr-tunneld forwards anything new to the admin notification address at startup and hourly thereafter.

The notice arrives after recovery, not during the outage. That is inherent: the gateway being down is the event, so the notification cannot leave through the gateway while it is happening. For a service that does not come back, nothing here will tell you -- that needs a check that does not live on the box.
The alert reports how many restarts happened in the last hour. Three in an hour is a different problem from one: something is putting the service back into the state the watchdog keeps healing.
Set alert_notify_watchdog_restart to false in admin settings to turn it off. It defaults on, because the whole defect was that nobody was being told.
watchdog_spool_path in server-config.yaml overrides where the gateway looks. It must match whatever the watchdog writes -- if the two disagree the gateway reads a file nothing writes and reports no restarts at all, which looks exactly like a gateway that never needed one.

The spool is world-readable and holds timestamps and service names only. The gateway reads it and never writes it: the watchdog runs as root and lfr-tunneld does not, so the gateway keeps a high-water mark in its own settings instead, and the watchdog trims its own file to 200 lines.

4.7. Vanity Domain Hook (Custom Domains)¶

The Vanity Domain Hook lets users bring their own domain (e.g. demo.customer.com) and have lfr-tunneld automatically provision Nginx + a Let's Encrypt certificate for it on registration, and tear both down on release. It's an external script, configured and toggled from the Admin Dashboard's System Settings (owner-only), not this setup guide -- but the directories and services it needs have to exist on the box first, which this section covers. scripts/common/lfr-vanity-hook.sh in this repo is a ready-to-use reference implementation.

Why this needs its own directories. lfr-tunneld runs as the unprivileged lfr-tunnel user under ProtectSystem=strict + NoNewPrivileges=true (§4.5) -- by design, it and anything it execs (including this hook) cannot write to /etc/nginx/sites-enabled, the default /etc/letsencrypt, or reload Nginx. Rather than loosen that hardening, the hook gets two narrowly-scoped directories of its own, added to the service's ReadWritePaths=, plus a small root-owned watcher that reloads Nginx on its behalf:

Hand /etc/nginx/conf.d to the lfr-tunnel user. It's already glob-included by the distro's default nginx.conf (include /etc/nginx/conf.d/*.conf;), so this needs no nginx.conf edit -- just an ownership change:
```
sudo chown lfr-tunnel:lfr-tunnel /etc/nginx/conf.d
```
Create a dedicated ACME webroot, kept separate from /var/www/lfr-tunnel's static assets so file ownership stays simple:
```
sudo mkdir -p /var/www/lfr-tunnel-vanity
sudo chown -R lfr-tunnel:lfr-tunnel /var/www/lfr-tunnel-vanity
```
Add both paths to the lfr-tunneld.service unit's ReadWritePaths= from §4.5 (do not remove /etc/lfr-tunneld -- append to it):
```
ReadWritePaths=/etc/lfr-tunneld /etc/nginx/conf.d /var/www/lfr-tunnel-vanity
```
Certbot's own state (--config-dir/--work-dir/--logs-dir) points at /etc/lfr-tunneld/letsencrypt, which is already writable -- no third path needed there.

lfr-tunneld still can't reload Nginx itself, so a root-owned systemd path unit watches /etc/nginx/conf.d and reloads Nginx whenever the hook adds or removes a config file:

sudo tee /etc/systemd/system/nginx-vanity-reload.path > /dev/null << 'EOF'
[Unit]
Description=Watch for Vanity Domain Hook nginx config changes

[Path]
PathChanged=/etc/nginx/conf.d

[Install]
WantedBy=multi-user.target
EOF
sudo tee /etc/systemd/system/nginx-vanity-reload.service > /dev/null << 'EOF'
[Unit]
Description=Reload nginx after a Vanity Domain Hook config change

[Service]
Type=oneshot
ExecStart=/usr/sbin/nginx -s reload
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now nginx-vanity-reload.path

Configuring the hook itself, once the directories above exist: 1. Copy scripts/common/lfr-vanity-hook.sh to the box (e.g. /usr/local/bin/lfr-vanity-hook.sh, owned root:root, mode 755 -- it must live under a trusted, root-owned folder and be executable, which the Admin Dashboard's own path validation enforces when you set it). 2. Add its environment to /etc/lfr-tunneld/secrets.env (§4.5) -- these aren't secrets, but this is the file lfr-tunneld's systemd unit already loads via EnvironmentFile=:

# /etc/lfr-tunneld/secrets.env
NGINX_CONF_DIR=/etc/nginx/conf.d
WEBROOT_PATH=/var/www/lfr-tunnel-vanity
UPSTREAM_URL=http://127.0.0.1:8080   # match your -p / http_bind_addr port
CERTBOT_DIR=/etc/lfr-tunneld/letsencrypt

ACME_EMAIL is deliberately not set here -- lfr-tunneld always injects it itself from owner.user_id (§4.3), overriding anything set in this file, since the Owner is the operator actually responsible for the shared Certbot install. 3. sudo systemctl restart lfr-tunneld to pick up the new environment. 4. In the Admin Dashboard's System Settings, set Vanity Domain Hook Path to /usr/local/bin/lfr-vanity-hook.sh and enable it.

On failure, the hook alerts both the configured admin (webhook + email) and the requesting user directly (bypassing their notification preference, since it means their domain may currently have no valid certificate) -- see #945 for the design rationale.

5. Client CLI Setup & Connection¶

Now that your server is running, any user can connect to it using the lfr-tunnel CLI.

Configure Client Configuration File (e.g. ~/.lfr-tunnel/config.yaml):

server_url: "https://yourdomain.com"
auth_token: "YOUR_SHARED_SECRET_TOKEN_KEY"
subdomain: "my-dev-env"
ports:
  - 8080

Start the Tunnel:

lfr-tunnel -config ~/.lfr-tunnel/config.yaml

Your local Liferay instance running on port 8080 is now securely exposed to the web at https://my-dev-env.yourdomain.com!

6. Server Upgrades and Automated Deployment (`deploy.sh`)¶

When you need to rebuild and deploy a new version of lfr-tunneld to your public VPS, you can use the automated scripts/deploy.sh script included in the repository root.

This script: 1. Compiles the Go server gateway natively for Linux (GOOS=linux GOARCH=amd64) using secure path-trimming (-trimpath). 2. Uploads the newly compiled binary, error pages, and static web assets to your VPS via scp. 3. Connects via SSH to safely stop, copy, swap, and restart the lfr-tunneld daemon.

Standard Usage:¶

./scripts/deploy.sh

Specifying a Custom SSH Key (`-i`):¶

If your local SSH agent is empty, or you use a specific private key file/certificate to connect to your VPS, pass it explicitly using the -i flag:

./scripts/deploy.sh -i ~/.ssh/id_ed25519

7. Troubleshooting Deployment Failures¶

Issue: `Permission denied (publickey). scp: Connection closed`¶

When running ./scripts/deploy.sh, you may receive a Permission denied (publickey) error from your VPS server. This means SSH is not offering any private key credentials during the connection handshake.

Solution 1: Load your key into your active SSH Agent¶

Ensure your private key is actively loaded into your memory-backed SSH agent:

ssh-add ~/.ssh/id_ed25519

(Verify loaded keys using: ssh-add -l)

Solution 2: Explicitly pass your key file¶

Run the script while passing your identity file directly using the -i parameter:

./scripts/deploy.sh -i ~/.ssh/id_ed25519

Solution 3: Native macOS Keychain Integration (Recommended)¶

To prevent your Mac from "forgetting" your loaded SSH keys after reboots, configure your local SSH client to automatically load and retrieve key passphrases directly from your macOS Keychain on-demand.

Add the following block to your local ~/.ssh/config file (create it if it doesn't exist):

Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_ed25519

Then, save your passphrase once to the keychain:

ssh-add --apple-use-keychain ~/.ssh/id_ed25519

8. Enterprise Customization & Policy Hardening¶

Liferay Tunnel includes built-in compliance, deliverability, and administrative systems optimized for strict enterprise security parameters.

8.1. Configuring Custom Legal Policies (Privacy & Cookies)¶

By default, the gateway portal serves baseline legal disclosures at /privacy and /cookies describing generic SQLite database storage and session tracking. You can customize these disclosures using two distinct methods:

Method A: Configuration-Driven Redirects (Dynamic URLs)¶

Enterprise self-hosters can direct users to corporate legal disclosures hosted externally. Add the following fields to /etc/lfr-tunneld/server-config.yaml:

# Optional custom policy links (defaults to server fallbacks if empty)
privacy_policy_url: "https://yourcompany.com/privacy-policy"
cookie_policy_url: "https://yourcompany.com/cookie-disclosure"

Once updated and restarted, the portal footers, registration forms, and OOTB welcome pages will automatically point to those corporate links.

Method B: Intercepting at Nginx (Static Local Files)¶

If you prefer hosting custom policies natively on the VPS but separately from the Go binary, you can use Nginx's high-performance alias blocks.

Upload your custom HTML policies to /var/www/lfr-tunnel/policies/ (automatically done when running ./scripts/deploy.sh if policies are placed inside resources/server/policies/).

Add the following location blocks inside /etc/nginx/sites-available/lfr-tunnel (under the port 443 server block):

# Serve Custom Branded Legal Policies
location = /privacy {
    alias /var/www/lfr-tunnel/policies/privacy.html;
    default_type text/html;
}

location = /cookies {
    alias /var/www/lfr-tunnel/policies/cookies.html;
    default_type text/html;
}

Test and reload Nginx:

sudo nginx -t && sudo systemctl reload nginx

To comply with strict IT auditing frameworks (like SOC2 or ISO 27001), you can optionally force users to explicitly check a consent box agreeing to your legal policies before completing their profile setups.

Enable the enforcement flag inside server-config.yaml:
```
enforce_policy_consent: true
```
The User Experience: The Complete Profile Setup page (setup.html) will dynamically render a required checkbox. The user cannot submit setup without checking it.
The Audit Trail: The server will capture the exact timestamp of consent and save it in the SQLite database under policy_consent_at. This provides an airtight compliance record for the version that was current at registration; see 8.2.1 for keeping it accurate once the policy changes.

8.2.1. Re-asking When the Policy Changes¶

enforce_policy_consent above covers registration only, and it stamps a single policy_consent_at. That timestamp cannot say which text was agreed to, so before this existed a policy change reached nobody who had already registered.

Set policy_version to close that. Every user whose acceptance history does not contain that exact string is asked again, with a grace period to answer in:

policy_version: "2026-09"          # bump whenever the policy text changes
policy_consent_grace_days: 14      # how long they have, from their OWN first sight of it
policy_consent_warning_days: 5     # how long before the deadline the warnings escalate
policy_consent_stops_active_tunnels: false

Leave policy_version empty and nothing changes. That is the default, so an existing gateway keeps behaving exactly as it did on upgrade; re-consent starts only when an operator sets a version.

The version is an opaque label, compared for equality and never for order. A date, a content hash and "2" are all fine — but do not reuse a string somebody has already accepted, because that reads as "already agreed".

Phase	Portal	Clients and tunnels
Within grace	Usable. A gate appears at login offering Accept or Remind me later; a banner remains for the rest of the session	Working
Warning window	The banner escalates, the client warns at startup, and one email goes out	Working
After the deadline	Blocked until accepted	New tunnels refused. Tunnels already running are not interrupted

Three details worth knowing:

The clock starts when the user first sees the new version, recorded per user, not when you published it. Somebody on two weeks' leave gets the full window rather than a deadline that expired while they were away.
"Remind me later" lasts for one session. The gate is shown again at the next login, which is what stops the banner becoming wallpaper.
A reminder that fails to send is retried on the next sweep. The email is claimed before it is sent, so two gateway processes cannot both send it, and the claim is given back when the send fails -- an SMTP outage therefore delays the reminder rather than losing it while recording it as delivered. If mail and the database fail together the claim cannot be given back, and that one case is logged as [Consent] Could not release the policy warning claim, naming the user who is now recorded as warned without having been.
policy_consent_stops_active_tunnels is off by default and should usually stay off. Refusing the next tunnel enforces the policy completely — no tunnel survives a restart — whereas turning this on drops live connections the moment the sweep notices, which for a gateway used in customer demos is the outcome the warning window exists to prevent.

The audit trail is the user_acknowledgements table, which is append-only: every acceptance is kept with its version, timestamp, IP and user agent, so "what did this user agree to, and when" stays answerable across any number of policy revisions.

An administrator can ask a user's client for its diagnostic logs, and the gateway will only issue that request for a user who has explicitly turned diagnostic log sharing on. There is nothing to configure -- it is always available and always off until the user acts.

Where the user grants it: an unticked checkbox beneath the policy statement on the Complete Profile Setup page, and an on/off toggle in Account Settings in both portals. It is a separate control from the policy consent above, deliberately: policy acceptance is a condition of using the service, so a combined checkbox could not be refused without refusing the service, and a consent that cannot be refused is not consent.
Where it is stored: users.diagnostics_consent_at, a nullable timestamp. NULL means off. No migration backfills it, so every account that existed before this shipped is off and has never been asked.
Where it is enforced: on the gateway, at the moment the request would be issued -- not at login and not at client startup. A user who switches it off mid-session has the next request refused, with no message needing to reach their running client.
What an admin sees when it is refused: 403 and <email> has not enabled diagnostic log sharing. Ask them to turn it on; there is no override.
The audit trail: diagnostics.consent_granted, diagnostics.consent_withdrawn, diagnostics.collect_requested and diagnostics.collect_refused in admin_audit_log. Refused attempts are recorded too -- a run of them is a pattern worth being able to see.

If you deploy this onto a gateway with existing users, bump policy_version. The consent text describes a new flow of user data off the machine, so the privacy policy changes with it, and 8.2.1 is the mechanism that puts a changed policy in front of people who registered before it. Without a version bump no existing user is ever shown the change. Their diagnostics setting stays off either way -- this is about them being told, not about their consent state.

How the request reaches the client: on the /api/tunnel-status heartbeat the client already sends every five seconds (#1763). Only the gateway actually serving a session can hand it over, so when that gateway is an edge, central forwards the request down the edge control channel, the edge hands it to the client, and the acknowledgement and the collected bundle come back up the same way (#1991). An edge with no control connection is reported as unreachable rather than accepting a request nothing would carry.
What an admin sees if nobody picks it up: nothing is stored, and diagnostics.collect_expired is written after five minutes. A request that silently never happened must not look like one that did.

8.3. Customizing Client Binary Downloads & Commands (Self-Hosting & EDR Bypass)¶

By default, the Developer Portal Dashboard recommends client downloads and installation commands pointing directly to the project's official GitHub releases.

If your organization requires codesigned client executables (e.g., to bypass Endpoint Detection & Response (EDR) software like SentinelOne), or if you are running in a restricted intranet zone without public GitHub access, you can host the signed client binaries locally on the VPS (served via Nginx) and override the commands/links shown to developers.

Step 1: Configure Nginx to Serve Local Client Downloads¶

Upload the signed client binaries to /var/www/lfr-tunnel/static/downloads/ on the VPS. Add an Nginx alias location block under the port 443 server config block:

location /static/downloads/ {
    alias /var/www/lfr-tunnel/static/downloads/;
    autoindex off;
    add_header Content-Disposition 'attachment';
}

Step 2: Override Client Platform Configurations in `server-config.yaml`¶

Declare the client_platforms block to redirect developers to your self-hosted binaries and specify organization-sanctioned package manager formulae or installation scripts:

client_platforms:
  macos_arm64:
    url: "https://yourdomain.com/static/downloads/lfr-tunnel-darwin-arm64"
    cmd: "brew tap yourcompany/tap && brew install lfr-tunnel"
    cmd_fallback: "curl -sSfL https://yourdomain.com/static/downloads/install.sh | sh"
  macos_amd64:
    url: "https://yourdomain.com/static/downloads/lfr-tunnel-darwin-amd64"
    cmd: "brew tap yourcompany/tap && brew install lfr-tunnel"
    cmd_fallback: "curl -sSfL https://yourdomain.com/static/downloads/install.sh | sh"
  windows_amd64:
    url: "https://yourdomain.com/static/downloads/lfr-tunnel-windows-amd64.exe"
    cmd: "scoop bucket add yourcompany https://yourdomain.com/scoop && scoop install lfr-tunnel"
    cmd_fallback: "iwr https://yourdomain.com/static/downloads/install.ps1 | iex"
  linux_amd64:
    url: "https://yourdomain.com/static/downloads/lfr-tunnel-linux-amd64"
    cmd: "curl -sSfL https://yourdomain.com/static/downloads/install.sh | sh"

Step 3: Customizing the Docker Workaround Panel Visibility¶

If your users still encounter local EDR restrictions running CLI binaries natively, you can display a secondary, Docker-based client setup panel.

The Docker panel will automatically appear on the dashboard only if the docker_image parameter is declared in server-config.yaml. To hide this card entirely, simply leave docker_image empty or remove it:

# To enable the Docker card, declare the registry image:
docker_image: "peterjrichards/lfr-tunnel:latest"
docker_bypass_url: "https://github.com/peterrichards-lr/lfr-tunnel/blob/master/docs/liferay-se-guide.md#using-the-docker-wrapper-edr-bypass"

# To hide the Docker workaround card, leave it empty or comment it out:
# docker_image: ""

Step 4: Automating Multi-Platform Code Signing & Verification¶

To establish user trust and ensure EDR compatibility (SentinelOne/false-positive prevention) across Windows, macOS, and Linux, you can utilize lfr-tunnel-ops's build and sign subcommands (pkg/ops/build.go/pkg/ops/sign.go -- there is no separate shell script; scripts/sign-release.sh was replaced by this Go-based CLI a while back and no longer exists).

These subcommands let you build, sign, and update release checksums natively on your macOS MacBook.

A. Obtaining Signing Certificates¶

Windows (Authenticode):
Corporate IT Request: Ask your IT Department for a Code Signing Certificate from your organization's internal Active Directory Certificate Services (AD CS). Have them export it as a PKCS#12 (.p12 or .pfx) file.
Why it works: Active Directory domain-joined machines automatically trust certificates issued by the internal CA.
macOS (Developer ID):
Corporate IT Request: Ask IT to add your Apple ID to the company’s Apple Developer Team with Developer or Admin privileges, and export a Developer ID Application certificate.
Linux (GPG Signature):
Generate a GPG key pair locally on your machine:
```
gpg --full-generate-key
```

B. Temporary Self-Signed Certificates (Local EDR Testing)¶

For isolated testing or when corporate certificates are not yet available, you can sign binaries with a self-signed identity and configure a SentinelOne rule to trust it specifically for your machine:

Windows Self-Signed Key Generation (on macOS): Generate a temporary PKCS#12 bundle (temp_signing_key.p12):

# 1. Generate private key
openssl genrsa -out self-signed-key.key 2048

# 2. Create Certificate Signing Request
openssl req -new -key self-signed-key.key -out self-signed.csr -subj "/CN=Lfr-Tunnel Test Code Signing"

# 3. Create self-signed certificate with Code Signing extended key usage
openssl x509 -req -days 365 -in self-signed.csr -signkey self-signed-key.key -out self-signed-cert.crt -addtrust codeSigning

# 4. Package key and certificate
openssl pkcs12 -export -out temp_signing_key.p12 -inkey self-signed-key.key -in self-signed-cert.crt -name "Temp Code Sign"

SentinelOne Exception: Provide self-signed-cert.crt (public key only) to your SentinelOne administrator. They can create an exception under Exclusions -> Signatures (Authenticode) to whitelist binaries signed with this certificate.
macOS Self-Signed Key Generation:
Open the Keychain Access app.
Go to Keychain Access -> Certificate Assistant -> Create a Certificate...
Name: Temp Code Sign
Identity Type: Self Signed Root
Certificate Type: Code Signing
Click Create. Right-click the newly generated certificate to export it as a .cer file and send it to your SentinelOne administrator for macOS whitelisting.

C. Running `lfr-tunnel-ops build` and `sign`¶

build cross-compiles the client into dist/ first; sign then signs what it finds there. There is no interactive prompt mode -- both are entirely env-var-driven. Per platform, sign silently prints Skipping ... signing (no valid ... provided/found) and moves on if that platform's env vars aren't set, rather than asking you for anything.

build records what it produced in dist/build-manifest.json, and sign and deploy-clients both refuse to run if that manifest's source version no longer matches pkg/config/version.go (#1279). This is why: build's output is routinely piped, and a pipe that closes early -- | head -4 is the case that actually happened -- kills it with SIGPIPE before it compiles anything. Twice in one day the downloads page was populated with the previous version's binaries while the release, the tag, the gateway and latest_version all agreed on the new one. A user running --upgrade was told a new version was available, downloaded it, had the signature and integrity verified, installed it, and was still on the old version.

If you genuinely mean to ship artefacts that did not come from the current source, pass -allow-stale to either command. It has to be a deliberate act rather than the default.

Provide the variables below directly via shell or 1Password op run. <vault> is a placeholder -- substitute whichever 1Password vault actually holds these items for you (e.g. your personal Private vault, or an org-specific vault name like Employee):

lfr-tunnel-ops build

# macOS Codesigning
export LFT_MACOS_IDENTITY="71FC1F1B1AAF4504A6B098BA0BDC785979DF14F9"

# Windows Authenticode (private key + certificate attached as files on the same item)
export LFT_SIGN_KEY="op://<vault>/self-signed-windows-signing-key/<key-filename>.key"
export LFT_SIGN_CRT="op://<vault>/self-signed-windows-signing-key/<cert-filename>.crt"
export LFT_SIGN_PASS="op://<vault>/self-signed-windows-signing-key/password"

# self-signed-windows-signing-key must be a generic item type (e.g. Secure Note/Password)
# with the private key and certificate attached as separate file fields -- NOT an "SSH Key"
# category item. That category only accepts an SSH-format key and derives an SSH public
# key from it, which can't hold an arbitrary X.509 certificate (see #951). The `password`
# field is a plain custom field, not the SSH Key type's built-in passphrase handling.

# Alternatively: PKCS#12 bundle (.p12 / .pfx)
# export LFT_SIGN_P12="op://<vault>/self-signed-windows-signing-key/<bundle-filename>.p12"

# Linux GPG signing -- LFT_GPG_KEY is a real key ID/fingerprint (from `gpg --list-secret-keys`),
# not a 1Password item name. LFT_GPG_SECRET is that key's exported secret key (`gpg --armor
# --export-secret-keys <key-id>`), attached as a file on its own item. LFT_GPG_PASS is that
# key's passphrase, or the literal string "skip" if the key is unencrypted -- don't fall back
# to reusing LFT_SIGN_PASS here, they're unrelated keys.
export LFT_GPG_KEY="<gpg-key-id-or-fingerprint>"
export LFT_GPG_SECRET="op://<vault>/self-signed-linux-gpg-key/<key-filename>.asc"
export LFT_GPG_PASS="skip"

# minisign signature over the checksums manifest (self-upgrade integrity, see #949) --
# independent of the three platform signing steps above, always attempted.
export MINISIGN_SECRET_KEY="op://<vault>/self-signed-minisign-key/<key-filename>.key"
export MINISIGN_KEY_PASSWORD="op://<vault>/self-signed-minisign-key/password"

op run -- lfr-tunnel-ops sign

Output: Signed binaries land back in dist/, alongside dist/checksums.txt, dist/checksums.txt.minisig, dist/build-manifest.json, and a per-binary .asc for whichever platform's GPG step ran. Nothing lands in bin/. build-manifest.json is deliberately excluded from checksums.txt: that file is what --upgrade and install.sh verify a downloaded binary against, so it stays a list of downloadable artefacts.

8.4. Dual-Mode Gateway Maintenance (Bouncer Mode vs. Fire Curtain)¶

To accommodate different levels of maintenance urgency, the gateway supports two maintenance modalities configurable directly from the dashboard:

A. Soft Maintenance ("Bouncer Mode" - Admins & Owners)¶

Soft Maintenance is designed for routine tasks, upgrades, or database checks. It acts like a bouncer checking IDs at the door: 1. How to Trigger: * Navigate to the Users tab in the Admin Dashboard. * Under the Gateway Soft Maintenance Mode card, specify the Action Name, Reason, and Duration. * Select a countdown duration (e.g., Immediate, or 5 Minutes) and click "Enable Soft Maintenance". 2. Behavior & Experience: * All active dashboards show a prominent countdown warning banner: "⚠️ Scheduled Maintenance starting in X:XX minutes! All standard tunnels will be paused." * Once active, all standard client tunnels are forcefully dropped (KickLease), new CLI connections are blocked, and standard users are blocked from logging into the portal. * Admins and Owners remain fully unblocked—the control panel dashboard and API endpoints remain online so you can manage resources and disable maintenance directly from the UI.

B. Nginx Hard Maintenance ("Fire Curtain" - Owner Only)¶

For high-risk operations, database restores, or full server downtime, the Platform Owner can drop a Nginx "Fire Curtain" that completely seals the server: 1. How to Trigger: * Navigate to the Users tab (only visible if logged in as the owner). * Under the Nginx Iron Curtain Mode card, fill out the Action Name, Duration, and Reason. * Click "Enable Iron Curtain" and type LOCKOUT in all caps inside the safety prompt to confirm. 2. Behavior & Experience: * The Go backend immediately writes a maintenance.enable trigger file and copies the customized, localized maintenance.html fallback template to /var/lib/lfr-tunneld/ and /var/www/lfr-tunnel/. * Nginx immediately intercepts all incoming HTTP/HTTPS traffic at the reverse proxy layer, serving the static maintenance page. * Warning: This blocks everyone, including the Owner and the Admin Dashboard. You will be immediately disconnected. * Restoration: To lift the Fire Curtain, you must log in to the VPS via SSH and run sudo disable-maintenance.sh.

8.5. Automated Cloudflare Dynamic DNS (DDNS) Service Setup¶

If your VPS or gateway environment runs on a dynamic public IP address, you can configure our native background Cloudflare Dynamic DNS (DDNS) service.

This background service automatically polls your public IPv4 and IPv6 addresses every 5 minutes and dynamically syncs your Cloudflare DNS zone records for the root (@), wildcard (*), and your explicit SMTP mail host (tunnel) subdomains whenever an IP change is detected, keeping your tunnels and mail server HELO/rDNS alignment 100% self-healing!

Step 1: Place API token configuration¶

Create a secure configuration file at /etc/letsencrypt/cloudflare.ini (this matches the certbot API folder location):

# Cloudflare API Token (with Zone.DNS Edit permissions)
dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN

Apply restricted permissions to secure the token:

sudo chmod 600 /etc/letsencrypt/cloudflare.ini

Step 2: Install the DDNS Script¶

Move the script to your server's binary folder and make it executable:

sudo cp scripts/liferay/vm6/cloudflare-ddns.sh /usr/local/bin/cloudflare-ddns.sh
sudo chmod +x /usr/local/bin/cloudflare-ddns.sh

Step 3: Install the systemd service & timer¶

To automate running the script every 5 minutes natively using systemd:

Copy the systemd service file to /etc/systemd/system/cloudflare-ddns.service:

sudo cp scripts/liferay/vm6/cloudflare-ddns.service /etc/systemd/system/cloudflare-ddns.service

Copy the systemd timer file to /etc/systemd/system/cloudflare-ddns.timer:

sudo cp scripts/liferay/vm6/cloudflare-ddns.timer /etc/systemd/system/cloudflare-ddns.timer

Enable and start the timer service:

sudo systemctl daemon-reload
sudo systemctl enable cloudflare-ddns.timer
sudo systemctl start cloudflare-ddns.timer

Step 4: Verify the Service¶

Check that your systemd timer is active and scheduled:

systemctl list-timers | grep cloudflare

You can also trigger a manual DNS update check immediately to confirm it works:

sudo systemctl start cloudflare-ddns.service
sudo journalctl -u cloudflare-ddns.service -n 50

⚠️ Important RFC 4592 Wildcard Empty Non-Terminal (ENT) Warning¶

Under strict DNS specifications (RFC 4592), if you define an explicit record of any type (even just a TXT record, such as an SPF record on portal.yourdomain.com or tunnel.yourdomain.com), the wildcard record (*.yourdomain.com) is completely bypassed and deactivated for that specific subdomain prefix!

To prevent NXDOMAIN (or empty resolution) errors on subdomains that have associated SPF or DKIM TXT records, you must configure explicit, exact-match A and AAAA records for those subdomains (e.g., portal and tunnel).

Our dynamic cloudflare-ddns.sh script is natively hardened to automatically handle this. By default, it is configured with RECORD_NAMES=("@" "*" "tunnel" "portal") to keep all required exact-match IP mappings 100% synchronized in real-time alongside your wildcard records.

8.6. Customizing Translations & Email Templates at Runtime¶

To provide enterprise-grade flexibility and avoid needing a full software re-release just to edit email copy or add a new locale, lfr-tunneld supports dynamic, zero-recompilation filesystem overrides for both properties-based translations and HTML email templates.

On startup, the Go gateway daemon utilizes a Dual-Layer Loading Mechanism: 1. It scans /etc/lfr-tunneld/i18n/ and /etc/lfr-tunneld/templates/ first for local filesystem overrides. 2. It falls back to default Go-embedded assets bundled inside the compiled binary second.

1. Customizing Portal Vocabulary & Locales¶

To customize, edit, or add any portal translation key: 1. Create the localized properties configuration directory on your VPS:

sudo mkdir -p /etc/lfr-tunneld/i18n

2. Copy or create a standard Java-style .properties file matching the target locale (e.g., Language_ro.properties for Romanian, or Language.properties for the default English fallback):

sudo nano /etc/lfr-tunneld/i18n/Language_ro.properties

3. Populate with standard key=value lines:

portal.welcome=Bine ai venit pe Liferay Tunnel!
btn.login.with.email=Autentificare prin E-mail

4. Restart the service to apply:

sudo systemctl restart lfr-tunneld

2. Customizing Transactional Email HTML Templates¶

To customize the HTML layout or copy of transactional emails (e.g., magic_link.html, invitation.html, gdpr_delete.html): 1. Create the templates directory structured by language subfolders:

sudo mkdir -p /etc/lfr-tunneld/templates/en
sudo mkdir -p /etc/lfr-tunneld/templates/ro

2. Create or copy your custom HTML template file (e.g., /etc/lfr-tunneld/templates/ro/magic_link.html):

sudo nano /etc/lfr-tunneld/templates/ro/magic_link.html

3. Use standard Go html/template parameters (like {{.Name}}, {{.Link}}, {{.ReportLink}}) to dynamically interpolate values:

<p>Salut {{.Name}},</p>
<p>Folosește link-ul pentru a te conecta în siguranță:</p>
<p><a href="{{.Link}}" style="background:#0969da; color:#fff; padding:12px 24px;">Conectare</a></p>

4. Restart the service to apply instantly:

sudo systemctl restart lfr-tunneld

(Note: The server will automatically append the clean English fallback version at the bottom of all non-English emails, separated by a crisp visual divider!)

8.7. Gateway Maintenance & Backup Command-Line Utilities¶

To simplify remote management and updates, the deployment process automatically uploads and registers administrative command-line utilities in the system path (/usr/local/bin/) on the VPS. Run these directly as root or via sudo:

A. Putting the Gateway into Maintenance Mode¶

To temporarily put the gateway into maintenance mode (this immediately serves the themed Liferay-branded 503 Service Temporarily Unavailable fallback page to all standard clients, active tunnels, and user dashboards):

sudo enable-maintenance.sh

B. Restoring Gateway Back Online¶

To exit maintenance mode and resume normal routing:

sudo disable-maintenance.sh

C. Safe Database Restoration Sequence¶

To perform a safe database restore from a file backup without exposing users to broken pages or race conditions, run the coordinated restore script:

sudo restore-with-maintenance.sh [backup_file_path]

(This automatically enables maintenance mode, launches the backup restoration, and takes the gateway back online once the restoration completes successfully.)

8.8. Decoupled Client/Server Versioning Management¶

To prevent developers from seeing client CLI update warnings when you deploy cosmetic or backend changes to the server gateway, you can separate the latest client version from the server's running version.

Configuration keys in `server-config.yaml`:¶

min_client_version: Specifies the minimum client version required to connect to the gateway. If a client is older than this, it is hard-blocked and exits immediately.
latest_client_version: Specifies the latest recommended client CLI version. If set, the gateway will return this version as the latest_version to connecting client instances. If a client's version is older than this (but newer than min_client_version), it displays a soft update warning.

How it works:¶

Server-only fixes (e.g., v1.9.4):
Leave latest_client_version as "v1.9.3" in /etc/lfr-tunneld/server-config.yaml.
The server gateway will run on v1.9.4 (visible in the Admin Dashboard footer via the server_version API metadata), but it will report the latest client to be v1.9.3.
Existing developers running v1.9.3 will not see any upgrade warnings or console notices.
Client CLI upgrades (e.g., v1.10.0):
Deploy the new client to package managers.
Update /etc/lfr-tunneld/server-config.yaml to set latest_client_version: "v1.10.0".
Existing clients running v1.9.x will now be prompted with soft upgrade warnings to update to v1.10.0.

8.9. Automatic IP Bans (Rate Limiter)¶

The API rate limiter auto-bans an address after 50 refused requests. Since #1353 those bans expire, and escalate for repeat offenders, following the fail2ban model.

This matters because the blacklist is enforced ahead of all routing: a banned address gets a 403 on everything, including the admin portal, so it cannot reach /api/admin/blacklist/ to unban itself. A permanent automatic ban therefore had no in-band recovery -- and automatic bans act on a noisy signal, since shared NAT, CGNAT, mobile carriers and corporate egress all look like one very busy address.

Manual bans placed by an admin are unaffected: they never expire, because a person decided on them.

auto_ban:
  duration: 24h            # ban time for a first offence. 0 = never expires
  increment: true          # longer bans for repeat offenders
  factor: 2                # multiplier per prior ban: 24h, then 48h, 96h, ...
  max_duration: 168h       # cap, so escalation cannot become permanent by another route
  history_retention: 720h  # how long an expired ban is remembered, so escalation still sees it
  ignore:                  # never auto-banned
    - 127.0.0.1/32
    - ::1/128

The defaults above apply when the block is absent, so no change is needed to keep sensible behaviour. Set duration: 0 to restore the previous permanent-ban behaviour.

Keep loopback in ignore. Locking an operator out of their own gateway is a worse outcome than a missed ban, and the ban cannot be lifted from the banned address. Add your own office or VPN ranges here for the same reason.

8.10. Trusted Proxies and Client IP Resolution¶

The gateway binds loopback (http_bind_addr: "127.0.0.1:8080") and nginx terminates TLS in front of it. Every request therefore arrives from 127.0.0.1, and the real client address can only come from a forwarding header — which is exactly why those headers must not be believed unconditionally.

trusted_proxies names the hops whose X-Real-IP / X-Forwarded-For may be honoured:

trusted_proxies:
  - 127.0.0.1/32
  - ::1/128

Those are the defaults, and they match every documented deployment. A request arriving from anywhere else has its headers ignored and is attributed to its peer address instead.

Why it matters. The resolved address drives the per-tunnel IP whitelist, the API rate limiter and its auto-ban, and every audit log entry. Without the boundary, a deployment serving TLS directly (ssl_cert_file/ssl_key_file, no nginx) lets a visitor walk straight through an IP whitelist by naming an allowed address in a header.

Only widen this for a proxy you actually run. Naming a range you do not control hands everyone inside it the ability to choose their own client address.

Each entry must name exactly one host, and the gateway refuses to start otherwise (#1801). An entry is accepted when it is a bare address, a /32 or a /128 — or when the whole prefix lies inside address space no host on the internet can occupy: RFC1918 (10/8, 172.16/12, 192.168/16), CGNAT 100.64.0.0/10, link-local, loopback, or IPv6 ULA fc00::/7. That last allowance is what keeps a load-balancer subnet inside your own VPC working, so the rule refuses what is dangerous rather than what is unfamiliar.

The test is not prefix length — a /24 is no safer than an /8 in any way that matters. It is whether an attacker can obtain a source address inside the range, because that is the only thing trusting a forwarder grants on. So trusted_proxies: ["10.20.0.0/16"] starts and trusted_proxies: ["203.0.113.0/24"] does not, and neither does 10.0.0.0/7 — it begins in RFC1918 space but runs out of it into 11.0.0.0/8, and containment is checked over the whole prefix.

The gateway refuses rather than warning because there is no safe way to carry on. Starting with the entry dropped would attribute every visitor to the proxy instead: IP whitelists would deny everyone, the rate limiter's auto-ban would ban the proxy, and every audit entry would name the same host — the same outage, with nothing pointing at its cause. There is deliberately no opt-out key, since the legitimate wide case is already allowed above and an override could only ever re-enable the dangerous one. lfr-tunnel-ops check-config reports the same problem on a running gateway as an error-severity finding, so a fleet can be checked before any restart.

The same rule governs nginx's set_real_ip_from entries, which render-nginx-config refuses at render time — one rule, one implementation (pkg/nettrust), applied on both boundaries.

If you serve TLS directly and trust non-loopback ranges, the gateway logs a warning at startup: that combination means headers are honoured with nothing in front to sanitise them. That warning is about the missing sanitiser, not about width — a private range is within the width rule and still worth knowing about here.

The nginx side matters too. The generated config sets both headers to $remote_addr, rather than $proxy_add_x_forwarded_for which appends to whatever the client sent — leaving the leftmost entry caller-controlled. If you place a CDN or load balancer in front of nginx, revisit this: you would then want nginx's real_ip module with set_real_ip_from naming that upstream.

In a multi-node deployment nginx already does this, on both roles. $remote_addr is only the visitor when the visitor connected directly; when another gateway cross-proxies the request it is that gateway, and the lines above would then attribute the request to it. render-nginx-config emits a real_ip block naming the nodes that legitimately forward — the control plane, loopback and the peer edges on an edge (#1450, #1750, #1757); every edge plus loopback on central, which its own edges forward to whenever they hold no route for a served domain (#1767). It rewrites $remote_addr to the visitor before the proxy_set_header lines run, so the gateway's own resolution order is unaffected. It is not a way in: real_ip rewrites only when the immediate peer is one of the named addresses, so a visitor arriving directly is never rewritten however they forge the header. Every entry must be an exact address, never a range.

8.11. Anonymous Geographic Distribution (`country_db_path`)¶

The admin analytics page can show a geographic distribution panel: how many distinct users registered from each country during the current ISO week. It is off unless you supply a geo-IP database, and off is a fully supported state — the panel says so, registration is unaffected, and nothing else in the gateway changes.

No database file ships with the gateway, and none ever can. Every vendor below forbids redistribution, which is why obtaining the file is an operator step rather than a build step. That is also the only reason the setting exists: pkg/geo's package comment records the rest of the design, and the parts that shape this section are that the client IP is resolved to a country in memory and then discarded, that only per-country cardinalities are persisted, and that a country must reach at least 5 distinct users (geo.DefaultThreshold) before it is shown at all — below that it is folded into an OTHER bucket.

8.11.1. Which databases work¶

The file must be in MaxMind's .mmdb binary format, and the gateway reads the country out of it at country.iso_code. All three vendors below put it there, so all three free country databases work:

Vendor / edition	Record path	Account needed	Cost	Licence as published by the vendor
MaxMind GeoLite2 Country (and the paid GeoIP2 Country/City)	`country.iso_code`	Yes — an account and a generated licence key	GeoLite2 free; GeoIP2 is the paid product	GeoLite End User Licence Agreement
DB-IP IP to Country Lite	`country.iso_code`	No	Free	Creative Commons Attribution 4.0 International
IP2Location LITE DB1, MMDB edition	`country.iso_code`	Yes — a free account	Free	IP2Location LITE "Terms of Use"

Note

This table used to say IP2Location's MMDB edition uses a top-level country_code, and it does not (#1993). country_code is the column name in IP2Location's CSV and BIN editions; their MMDB edition is a deliberate drop-in MaxMind clone, down to database_type "GeoLite2-City", MaxMind's eight languages and MaxMind's nested record schema — measured on a real IP2LOCATION-LITE-DB11.MMDB, whose record for 8.8.8.8 carries the keys [city continent country location postal registered_country subdivisions] and no country_code anywhere. The gateway carried a second decode path for that schema from #1935 until #1993 removed it; it never matched a vendor file, and IP2Location resolved through country.iso_code the whole time.

If you deploy a file this list does not cover, the symptom is that it opens cleanly and then resolves every address to nothing, which the panel reports as "no country yet has enough distinct users" rather than as a problem with the file. To tell those two apart against a file you already have, point the vendor-compat tests at it — they log the record's own top-level keys and name every address that resolved to nothing:

LFT_GEO_TEST_DB=/path/to/your.mmdb make test PKG=./pkg/geo/ TEST_FLAGS="-test.v -test.run ARealDatabase"

DB-IP needs no MaxMind account and no code specific to it. It mirrors MaxMind's record schema, and this was measured rather than assumed: dbip-city-lite-2026-09.mmdb resolved 8.8.8.8 → US and 1.1.1.1 → AU against the unmodified resolver. If you want the panel working in the next ten minutes with no signup, that is the file to fetch.

8.11.2. It must be `.mmdb` — IP2Location's default download is not¶

IP2Location's default download for every LITE edition is a .BIN, which is their own proprietary format and not an mmdb. The gateway cannot read it, and no configuration makes it readable: you need the MMDB download of the same database, offered further down the same download page.

The remedy is a different download, not a different path or permission, so the gateway names it rather than reporting the opaque parse error the .BIN would otherwise produce:

geo: /etc/lfr-tunneld/geoip/IP2LOCATION-LITE-DB1.BIN is a .BIN file, which is IP2Location's
proprietary format and not MaxMind's .mmdb -- download the MMDB edition of the same database
instead: ...

That wording is bounded to files actually named .BIN; a genuinely corrupt MaxMind file still reports a plain open error, so it does not send you downloading a format you already have.

8.11.3. Prefer a country-level edition¶

IPinfo Lite is a fourth option, free and signup-light, and the only supported vendor whose record puts the code at a top-level country_code rather than nesting it under country (#2008). The gateway handles both shapes, so this matters only if you are reading the file yourself. Its country field holds the country NAME, which the gateway deliberately never decodes.

Pick the country edition — GeoLite2 Country, DB-IP IP to Country Lite, IP2Location LITE DB1. Not GeoLite2 City, not DB-IP City Lite, not DB11.

This is a privacy choice, not only a disk-space one. The gateway decodes the country and nothing else, and pkg/geo/resolver.go records why: decoding the whole record "would pull city, subdivision and lat/long into memory, which this feature has no use for and which are far more identifying than a country". A city edition puts that data on the gateway's disk where the country-level feature can never use it. Inspecting dbip-city-lite-2026-09.mmdb directly, it carries subdivisions, latitude and longitude keys alongside iso_code — present in the file, never read by this code path. A country edition cannot over-collect even by accident, and it is far smaller: that DB-IP city file is 121 MB uncompressed and IP2Location's DB11 .BIN 93 MB, against the ~15 MB IP2Location publish for DB1's MMDB.

8.11.4. Licensing — what you are agreeing to¶

Read the vendor's current terms before deploying. They have changed before, and the summary below is a starting point for that decision rather than a substitute for it. Where a licence file ships inside the download, that file is quoted directly and is the authoritative statement of what you received; where the claim comes from a vendor web page instead, it is labelled as such.

All three forbid or restrict redistribution. None of them can ship with this server, and in every case you download the file yourself.

MaxMind GeoLite2 — free of charge, but not an anonymous fetch: MaxMind's developer site states you must sign up for an account and then generate one or more licence keys to download the databases. Distribution is under the GeoLite End User Licence Agreement, not a permissive licence. Three clauses matter operationally. §3 requires attribution: "to the extent the Services contain any copyrightable elements those copyrightable elements are governed by the Creative Commons License. You must provide attribution of your use to MaxMind (an example of attribution: 'This product includes GeoLite Data created by MaxMind, available from https://www.maxmind.com.')". §6.1 requires MaxMind's prior written consent before disclosing the databases to a third party, and §6.3 requires you to cease use of and destroy old versions within thirty (30) days of a new GeoLite release, with written confirmation on request. That last clause turns "keep it current" from good practice into a licence obligation — see §8.11.9. These quotes are from MaxMind's published EULA, effective 2026-02-12 and read on 2026-09-16; check the current text, as this is exactly the kind of term an organisation may want its own sign-off on even though no money changes hands. GeoIP2 is the paid product; "a commercial licence is required" usually refers to GeoIP2, or to a use of GeoLite2 the EULA does not permit.

Note

An earlier version of this section said GeoLite2 required no credit line, on the reasoning that its EULA "restricts disclosure rather than requiring public acknowledgment". §6.1 does restrict disclosure — and §3 also requires attribution. Reading the EULA rather than reasoning from the other two vendors is what found it (#1921). All three supported vendors require a credit, and they require different ones, which is why the panel renders a per-provider line rather than one sentence.

DB-IP IP to Country Lite — the lightest obligations of the three. DB-IP's download page states the Lite databases are distributed under the Creative Commons Attribution 4.0 International License, that no account is needed, and that the file is refreshed monthly in both MMDB and CSV. The attribution requirement, as worded on that page: "In the case of a web application, you must include a link back to DB-IP.com on pages that display or use results from the database." This is from the vendor's page, not from a licence file inside the download — the .mmdb carries data and metadata only.

IP2Location LITE — free of charge and attribution-bearing, and the one where the shipped licence file is worth reading in full. LICENSE_LITE.TXT, inside the download itself, states:

You must agree to our "Terms of Use", which are published online at https://lite.ip2location.com/terms-of-use at all times.

You are not permitted to redistribute or resell this product.

and requires a specific acknowledgment wherever the database is used:

"[Your site name or product name] uses the IP2Location LITE database for <a href="https://lite.ip2location.com">IP geolocation</a>."

Clause 1 is the one to notice: it binds you to online terms that can change without the file you hold changing. README_LITE.TXT in the same archive also notes LITE accuracy is "up to Class C only", with the commercial editions sold as more accurate. The LITE download page additionally displays a CC BY-SA badge, which does not sit obviously alongside clause 2's flat prohibition on redistribution — if you need to redistribute, resolve that with IP2Location rather than relying on either statement.

Info

The portal renders the credit for you, and picks it from the file. Since #1921 the Geographic Distribution panel displays a per-provider attribution line in both portal arms, whenever a database is open — including the below-threshold state, where rows exist and are suppressed, because DB-IP's wording covers pages that display or use results. There is nothing for you to add.

The vendor is derived from the database's own database_type metadata, not configured, so there is no key to get wrong: DBIP-* renders DB-IP's CC BY 4.0 credit with a link back to db-ip.com, GeoLite2-*/GeoIP2-* renders MaxMind's §3 attribution, and IP2Location's MMDB editions render the acknowledgment LICENSE_LITE.TXT prescribes.

A file this build does not recognise renders an honest "vendor could not be identified" line instead of somebody else's credit. If you see that, the startup log's provider=unknown says the same thing, and you must add the credit your vendor requires by hand (§8.6 covers the translation and template override mechanisms). This is deliberate: printing a named vendor's acknowledgment over another vendor's data would be a false provenance claim and leave the real supplier's licence unmet.

8.11.5. File placement and permissions¶

Put the database inside the configuration directory the daemon already owns:

sudo mkdir -p /etc/lfr-tunneld/geoip
# Fetch the file on your workstation, verify it against the vendor's checksum, then copy it up.
sudo cp dbip-country-lite-2026-09.mmdb /etc/lfr-tunneld/geoip/country.mmdb
sudo chown -R lfr-tunnel:lfr-tunnel /etc/lfr-tunneld/geoip
sudo chmod 750 /etc/lfr-tunneld/geoip
sudo chmod 640 /etc/lfr-tunneld/geoip/country.mmdb

The daemon only ever reads the file, so read access for lfr-tunnel is all it needs.

The systemd sandbox in §4.5 rules out some otherwise obvious locations. ProtectHome=true makes /home and /root inaccessible to the service, and PrivateTmp=true gives it a private /tmp — so a database left in a login user's home directory or in /tmp is invisible to the daemon and reports as missing, however correct the path looks from your shell. /etc/lfr-tunneld is already ReadWritePaths=, and ProtectSystem=strict leaves the rest of the filesystem readable even though it is read-only, so /var/lib/lfr-tunneld/ or /usr/share/GeoIP/ work equally well if you prefer to keep a ~15 MB artefact out of /etc. If you do use a directory the unit has not been told about, confirm the service can read it before assuming the path is wrong.

Note the recursive chown above is scoped to geoip/, deliberately: re-running §4.3's recursive chown across the whole of /etc/lfr-tunneld would flip secrets.env back off root:root.

8.11.6. Setting the path, and what resolves it¶

# /etc/lfr-tunneld/server-config.yaml
country_db_path: "/etc/lfr-tunneld/geoip/country.mmdb"
country_db_provider: "dbip"   # maxmind | dbip | ip2location | ipinfo -- REQUIRED

Both keys are required. A path without a vendor leaves geographic distribution off, and the panel says so rather than guessing.

The vendor can also be chosen in the portal, under System Settings -> Geo-IP Vendor (#1995). It is a dropdown there rather than a text field, and it shows the exact credit the chosen vendor will publish before you publish it. A vendor set in the portal wins over this key, and the screen names whichever of the two is in force — so the file and the portal cannot disagree without the screen saying so. Changing it takes effect immediately: the vendor is a display value that never touches address decoding, so nothing has to be reopened or restarted.

country_db_path stays YAML-only, deliberately. It is a path on the gateway host, read at startup, and an admin session able to point the gateway at any readable file is a disclosure vector worth not opening. The portal shows it read-only, which is what you need to diagnose a wrong path.

That is not bureaucracy. Each supported vendor's licence obliges a different visible credit, and the file cannot be trusted to say which vendor published it. Measured against a real IP2LOCATION-LITE-DB11.MMDB:

database_type = "GeoLite2-City"          <- MaxMind's own string
description   = "GeoLite2City database"
languages     = [en de es fr ja pt-BR ru zh-CN]
country       = {geoname_id, iso_code, names{...}}   <- MaxMind's record schema

IP2Location ship a deliberate drop-in clone, so an IP2Location file is indistinguishable from a MaxMind one in every observable property. Deriving the vendor from the file credited MaxMind for IP2Location's data — a false statement about provenance, with IP2Location's own required acknowledgment left unshown (#1964).

Naming the vendor makes you the author of that claim. Get it wrong and the panel credits the wrong vendor, but that is then a misconfiguration to correct rather than something the gateway invented. The gateway logs a warning when your declared vendor disagrees with the file's own metadata, which catches a typo without making the common case fail — but it cannot tell which of you is right, and does not try.

Use an absolute path. The value is passed to the filesystem verbatim, so a relative path resolves against the daemon's working directory — which the unit file in §4.5 sets to /etc/lfr-tunneld, not to wherever you happened to be standing when you edited the config. No expansion of any kind happens: ~, $HOME and shell globs are not interpreted.

The key is also settable as the environment variable LFT_COUNTRY_DB_PATH, which overrides the YAML value. That is useful in a container image where the database is mounted at a path the baked-in config does not know, but there is no secret here, so secrets.env is not the natural home for it.

`geolite2_db_path` — the old spelling, still honoured¶

The setting was called geolite2_db_path until #1921, which named one vendor for what is really "where the country database lives". That spelling still works and is not going away: breaking a running deployment is not an improvement. Its environment variable, LFT_GEOLITE2_DB_PATH, still works too.

Config state	What the gateway opens
`country_db_path` only	that path
`geolite2_db_path` only	that path — nothing to change on an existing gateway
both, same path	that path, no warning
both, different paths	`country_db_path` wins, and the gateway logs a warning naming the file it opened

The neutral key winning is the deliberate direction. If the alias won, adding country_db_path to a config that still carried geolite2_db_path would do nothing at all — silently, with a startup log naming the old path and a panel that looks perfectly healthy. That is the one outcome you cannot diagnose from the outside. The reverse mistake announces itself in the same log line you are told to read in §8.11.7. Migrate the line rather than duplicating it.

8.11.7. Applying it, and confirming it took effect¶

Reload the daemon. A restart is not needed, and has not been since #1998.

sudo systemctl reload lfr-tunneld

SIGHUP re-reads country_db_path, its geolite2_db_path alias, and country_db_provider, and applies them to the running process — the old database is closed, the new one is opened, and the panel's attribution follows the file that is actually open. No tunnel is dropped. That is the whole point: a restart interrupts every tunnel central serves, which made verifying a vendor expensive enough not to do — which is how the IP2Location attribution stayed wrong until someone finally downloaded the file (#1964).

So switching vendor is: edit two lines, reload, look at the panel.

Four things worth knowing before you rely on it:

A failed reload keeps the database that is already open. A path with no file at it, an unreadable file, an undeclared or misspelled country_db_provider — each is logged and refused, and the panel keeps serving and keeps crediting the vendor of the file still open. A reload that turned a working panel off because of a typo would be worse than no reload at all. Check the journal rather than assuming (below).
Clearing country_db_path does turn the feature off. That is an instruction rather than a typo, so it is honoured — otherwise the feature could not be switched off without the restart this removes.
Only these keys reload. SIGHUP re-reads the whole file, but applies only edge_nodes (#1309) and the three geo keys above. Every other edit in the file is ignored until a restart, and the reload says so in its own log line — a key that appeared to reload and did not would leave you unable to tell which half of your edit is live (#1454).
lfr-tunnel-ops reconcile-server-config manages exactly three keys — session_duration, session_max_lifetime and policy_version — and leaves every other key on the live box alone. It will neither push nor report drift in country_db_path (or its geolite2_db_path alias).

A binary redeploy is not needed either. This is a configuration change; the gateway you are running already reads the keys.

Then read the journal. The log states the gateway's geo status unambiguously, and it emits exactly one of these (the admin panel now says the same thing — §8.11.8):

sudo journalctl -u lfr-tunneld -b | grep '\[Geo\]'

Log line	Meaning
`[Geo] Anonymous geographic distribution enabled` — with `path`, `provider` and `threshold`	Working. The file opened and the panel is live. `provider` is the vendor you declared in `country_db_provider` and decides which attribution the panel renders.
`[Geo] Geo-IP database not found; geographic distribution disabled` — with `path`	The path is set and there is no file there. Typo, wrong directory, or a location the sandbox hides (§8.11.5).
`[Geo] Failed to open geo-IP database; geographic distribution disabled` — with `path` and `error`	The file exists and could not be read: wrong format (a `.BIN` names itself here), truncated download, or permissions.
`[Geo] A geo-IP database is configured and readable but country_db_provider is not set` — with `path`	The file is fine; the vendor is not declared. Set it (§8.11.6).
`[Geo] Both country_db_path and geolite2_db_path are set` — with `using`	Both spellings name a different file. The neutral key won; remove the alias line.
nothing at all	Neither `country_db_path` nor `geolite2_db_path` is set. This is the default and is not an error.

After a systemctl reload there is one more line, and it is the one that tells you whether the reload landed:

Reload line	Meaning
`[Geo] Reload swapped the country database` — with `path` and `provider`	Applied. The panel now credits `provider`.
`[Geo] Reload refused the new country database; keeping the one already in force` — with `reason`	Not applied. The previous database is still serving and still credited. `reason` names which key to fix.
`[Geo] Reloaded country database config: no change; the open file was left alone.`	Neither geo key changed, so nothing was reopened. Expected when you reload to apply an `edge_nodes` edit.
`[Geo] Reload turned anonymous geographic distribution off`	`country_db_path` is now empty. Deliberate, and honoured.

Once enabled, the panel still needs data before it shows rows, and three entirely normal conditions delay that:

Countries are counted at registration, so nothing accumulates until users connect.
A country needs 5 distinct users in the current ISO week to be shown at all; below that it is folded into OTHER. On a small deployment the honest outcome is a panel that stays on "no country yet has enough distinct users" — that is the k-anonymity threshold doing its job, not a broken database.
Counts are written out by the hourly prune timer (prune_interval, default 1h), so allow up to an hour after the first registrations before expecting rows.

The country is derived from the resolved client IP, so §8.10 applies directly: a trusted_proxies list that does not match your real topology attributes visitors to the proxy, and the geographic panel will faithfully report the country your load balancer sits in.

8.11.8. The panel says which of the three off states you are in¶

A wrong path and an unset path used to render the same sentence — "No geo-IP database is configured" — so an operator who mistyped the database path was told they had never set it, and the startup log was the only place the difference existed (#1938). It is not any more. The panel shows one of:

What the panel says	State	What to do
No geo-IP database is configured…	Neither `country_db_path` nor its `geolite2_db_path` alias is set. The default, and not an error.	Nothing, unless you want the feature.
No file exists at the geo-IP database path configured in `country_db_path`… (and names the path it tried)	The path is set and there is no file there.	Compare the path it names against the file you installed — §8.11.5 if it looks right but the daemon cannot see it.
The geo-IP database configured in `country_db_path` could not be read… (names the path, and the open error)	The file exists and is unusable: wrong format, truncated, or unreadable by the daemon's user.	A `.BIN` says so in the quoted error — fetch the MMDB edition (§8.11.2). Otherwise re-download and check ownership.

The second and third name country_db_path whichever spelling you actually used: the two resolve to one value before anything opens a file (§8.11.6), and the path quoted back to you is the one the gateway tried.

The path is shown to admins only: /api/admin/analytics/locations is behind the same admin check as the rest of that page, and the response carries nothing extra when the feature is working.

The journal lines in §8.11.7 still say the same things and are still the fastest check over SSH; they are no longer the only place the distinction exists. Note that the panel describes the state the running process is in — it is decided once at startup, so a file you have just put in place shows as missing until you restart.

8.11.9. Keeping it current¶

There is no auto-update mechanism in this repo. No timer fetches a new database, nothing warns that the file is old, and a stale database degrades silently — addresses reassigned since the file was built resolve to the previous holder's country or to nothing at all, and the panel looks exactly as healthy either way.

Refreshing is therefore an operator job, and for MaxMind it is a licence obligation: the GeoLite EULA's §6.3 destruction-within-30-days clause (§8.11.4) is not satisfied by leaving last year's file in place. All three vendors publish roughly monthly.

The gateway memory-maps the database at startup and holds that mapping for the life of the process, which dictates how a refresh has to be done:

# 1. Download and verify the new file, then stage it alongside the live one.
sudo install -o lfr-tunnel -g lfr-tunnel -m 640 country-new.mmdb /etc/lfr-tunneld/geoip/country.mmdb.new

# 2. Atomic rename. The running daemon keeps its mapping of the OLD inode, which stays valid.
sudo mv /etc/lfr-tunneld/geoip/country.mmdb.new /etc/lfr-tunneld/geoip/country.mmdb

# 3. Restart to pick up the new file, and confirm it opened.
sudo systemctl restart lfr-tunneld
sudo journalctl -u lfr-tunneld -b | grep '\[Geo\]'

Never overwrite the file in place (cp onto the live path, or a downloader writing straight to it) while the daemon is running: the mapping is of the file's contents, so rewriting those bytes underneath a live process changes what lookups read mid-flight. Write-then-rename, as above.

If you automate this on a timer, keep the download, the verification and the restart in one script so a failed fetch cannot leave a half-written file in the live path. MaxMind publish geoipupdate for their own databases; DB-IP and IP2Location are plain HTTPS downloads. Either way, step 3 and its log check are what turns the automation from "it ran" into "it worked" — a refresh that silently left the daemon on the old mapping is the exact failure this section exists to make visible.

8.12. Tunnel Keepalive and the Client Reconnect Window¶

Two settings decide how a tunnel behaves when this gateway is restarted. Both exist because neither number was configurable when it mattered: a deploy used to take every attached client offline until somebody restarted it by hand (#1946).

tunnel_keepalive: "25s"        # how often the gateway pings each attached tunnel client
client_reconnect_window: "60s" # how long a client should keep trying to reattach to this gateway

tunnel_keepalive is the ping on the chisel control channel. Its counterpart is nginx's proxy_read_timeout on location /tunnel, which defaults to 60s: the ping is what stops an idle tunnel looking silent to nginx, so if you change that timeout, change this too. Leave it empty for the built-in 25s, which gives two pings per 60s window.
client_reconnect_window is advertised to clients on /api/version. It is how long a client keeps trying to reattach here before falling back to its own region failover, and it is the one knob in this pair you can move without a client release. Two caveats, both structural: it only steers clients new enough to read it, and a client clamps it to 20s-180s rather than trusting it -- too short undoes the fix, too long starves failover. Leave it empty and clients use their own default of 60s.

9. Asymmetric Outbound Routing Workaround (Dual-IP VPS)¶

If your VPS hosting provider allocates multiple public IPv4/IPv6 addresses to a single virtual instance (for example, a primary IP and a secondary IP), you may encounter outbound routing issues.

9.1. The Problem: Outbound Packet Drops¶

By default, the Linux kernel's route selection algorithm may dynamically select the secondary IP address as the source IP for outbound packets. If the provider's firewall blocks or drops traffic that initiates from the secondary IP (or if asymmetric routing is detected and dropped at the network edge), tasks requiring outbound connectivity from the VPS (such as Let's Encrypt ACME renewals, SMTP mail sending, or regional edge latency health checks) will fail.

9.2. The Solution: Pinned Route Source in Netplan¶

To guarantee that outbound connections originating from the VPS are consistently pinned to the primary IP, you must configure a persistent static default route specifying the primary IP as the source (from) in Netplan.

Open your Netplan configuration file (usually located at /etc/netplan/ e.g., /etc/netplan/50-cloud-init.yaml):
```
sudo nano /etc/netplan/50-cloud-init.yaml
```

Locate your network interface configuration and add the routes block under your interface (e.g., eth0). Specify your gateway IP under via and your primary IP (82.39.133.178) under from:

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: no
      addresses:
        - 82.39.133.178/24  # Primary IP
        - 82.39.133.179/24  # Secondary IP
      routes:
        - to: default
          via: 82.39.133.1   # Gateway IP (check via `ip route show`)
          from: 82.39.133.178 # Force primary IP as source for outbound traffic

Validate the Netplan configuration:
```
sudo netplan try
```
Apply the routing changes:
```
sudo netplan apply
```

Verify that outbound traffic is routing via the correct primary IP:

curl https://ifconfig.me
# Output should match your primary IP: 82.39.133.178

Last Updated: 2026-09-20 | Last Reviewed: 2026-09-20

Liferay Tunnel (lfr-tunnel) End-to-End Server & DNS Setup Guide¶

1. Domain & DNS Configuration¶

1.1. Required DNS Records¶

1.2. Email Security TXT Records¶

2. VPS Server Setup & Security Hardening¶

2.1. Basic OS & Package Updates¶

2.2. Create a Restricted Sudo User¶

2.3. Hardening SSH Configuration¶

2.4. Configure the Firewall (UFW)¶

3. Nginx Reverse Proxy & Let's Encrypt Wildcard SSL¶

3.1. Install Nginx and Certbot¶

3.2. Obtain Wildcard SSL Certificates¶

3.3. Configure Nginx Configuration¶

3.4. Configure Nginx Maintenance Mode (Optional but Recommended)¶

4. Install & Configure lfr-tunneld¶

4.1. Create a Restricted System User¶

4.2. Build the Server Binary¶

4.3. Configuration Files Setup¶

Every flag lfr-tunneld accepts¶

4.4. Choosing an SMTP Relay¶

4.4.1. Option A: Local Postfix Email Relay (TLS Verification & rDNS Alignment)¶

4.4.2. Option B: Amazon SES (Managed Relay)¶

4.5. systemd Service Setup¶

4.6. Service Self-Healing (Nginx & Watchdog)¶

1. Nginx Systemd Auto-Restart Override¶

2. Active Watchdog Script & Timer¶

What the watchdog tells you¶

4.7. Vanity Domain Hook (Custom Domains)¶

5. Client CLI Setup & Connection¶

6. Server Upgrades and Automated Deployment (deploy.sh)¶

Standard Usage:¶

Specifying a Custom SSH Key (-i):¶

7. Troubleshooting Deployment Failures¶

Issue: Permission denied (publickey). scp: Connection closed¶

Solution 1: Load your key into your active SSH Agent¶

Solution 2: Explicitly pass your key file¶

Solution 3: Native macOS Keychain Integration (Recommended)¶

8. Enterprise Customization & Policy Hardening¶

8.1. Configuring Custom Legal Policies (Privacy & Cookies)¶

Method A: Configuration-Driven Redirects (Dynamic URLs)¶

Method B: Intercepting at Nginx (Static Local Files)¶

8.2. Enforcing Policy Consent Audit Trails¶

8.2.1. Re-asking When the Policy Changes¶

8.2.2. Diagnostic Log Sharing Consent¶

8.3. Customizing Client Binary Downloads & Commands (Self-Hosting & EDR Bypass)¶

Step 1: Configure Nginx to Serve Local Client Downloads¶

Step 2: Override Client Platform Configurations in server-config.yaml¶

Step 3: Customizing the Docker Workaround Panel Visibility¶

Step 4: Automating Multi-Platform Code Signing & Verification¶

A. Obtaining Signing Certificates¶

B. Temporary Self-Signed Certificates (Local EDR Testing)¶

C. Running lfr-tunnel-ops build and sign¶

8.4. Dual-Mode Gateway Maintenance (Bouncer Mode vs. Fire Curtain)¶

A. Soft Maintenance ("Bouncer Mode" - Admins & Owners)¶

B. Nginx Hard Maintenance ("Fire Curtain" - Owner Only)¶

8.5. Automated Cloudflare Dynamic DNS (DDNS) Service Setup¶

Step 1: Place API token configuration¶

Step 2: Install the DDNS Script¶

Step 3: Install the systemd service & timer¶

Step 4: Verify the Service¶

⚠️ Important RFC 4592 Wildcard Empty Non-Terminal (ENT) Warning¶

8.6. Customizing Translations & Email Templates at Runtime¶

1. Customizing Portal Vocabulary & Locales¶

2. Customizing Transactional Email HTML Templates¶

8.7. Gateway Maintenance & Backup Command-Line Utilities¶

A. Putting the Gateway into Maintenance Mode¶

B. Restoring Gateway Back Online¶

C. Safe Database Restoration Sequence¶

8.8. Decoupled Client/Server Versioning Management¶

Configuration keys in server-config.yaml:¶

How it works:¶

8.9. Automatic IP Bans (Rate Limiter)¶

8.10. Trusted Proxies and Client IP Resolution¶

8.11. Anonymous Geographic Distribution (country_db_path)¶

8.11.1. Which databases work¶

8.11.2. It must be .mmdb — IP2Location's default download is not¶

8.11.3. Prefer a country-level edition¶

8.11.4. Licensing — what you are agreeing to¶

8.11.5. File placement and permissions¶

8.11.6. Setting the path, and what resolves it¶

4. Install & Configure `lfr-tunneld`¶

Every flag `lfr-tunneld` accepts¶

6. Server Upgrades and Automated Deployment (`deploy.sh`)¶

Specifying a Custom SSH Key (`-i`):¶

Issue: `Permission denied (publickey). scp: Connection closed`¶

Step 2: Override Client Platform Configurations in `server-config.yaml`¶

C. Running `lfr-tunnel-ops build` and `sign`¶

Configuration keys in `server-config.yaml`:¶

8.11. Anonymous Geographic Distribution (`country_db_path`)¶

8.11.2. It must be `.mmdb` — IP2Location's default download is not¶

`geolite2_db_path` — the old spelling, still honoured¶