Multi-Node Orchestration & Remote Node Setup Guide¶

This guide details how to prepare remote Linux compute nodes, configure multi-node target registries, run remote workloads, and perform live workload migrations with Liferay Docker Manager (LDM).

1. Preparing a Remote Compute Node¶

LDM can orchestrate workloads on any remote machine (cloud VM, local Linux server, or WSL instance) reachable over SSH with Docker installed.

Requirements on Remote Node¶

Operating System: Linux (Ubuntu, Debian, RHEL, AlmaLinux, Fedora) or macOS.
Utilities: openssh-server, rsync (or tar).
Docker: Docker Engine and Docker Compose V2.
User Permissions: SSH user must be a member of the docker group (non-sudo Docker access).

Universal Platform Compatibility¶

LDM multi-node orchestration can be used anywhere Docker and SSH are running. This includes:

Cloud Infrastructure: AWS EC2, GCP Compute Engine, Azure VMs, DigitalOcean Droplets.
Windows Compute Nodes: Windows 10/11 running WSL2 (Ubuntu / Debian).
Local Infrastructure: LAN Linux servers, macOS nodes, local VMs, bare-metal hardware.

Remote Node Setup Commands¶

Choose the package manager appropriate for your Linux distribution:

Option A: Amazon Linux 2023 / RHEL / Fedora (`dnf`)¶

# Update system package index and install Docker & rsync
sudo dnf install -y docker rsync

# Enable and start Docker service
sudo systemctl enable --now docker

# Add your SSH user (e.g. ec2-user) to the docker group
sudo usermod -aG docker $USER
newgrp docker

Option B: Ubuntu / Debian (`apt-get`)¶

# Update system package index and install prerequisites
sudo apt-get update && sudo apt-get install -y curl rsync ca-certificates gnupg

# Install Docker Engine and Docker Compose V2
curl -fsSL https://get.docker.com | sh

# Add your SSH user (e.g. ubuntu) to the docker group
sudo usermod -aG docker $USER
newgrp docker

Platform Example 1: AWS EC2 Compute Node Setup (Amazon Linux 2023 / Ubuntu)¶

Security Group Config: Inbound rules needed:
TCP Port 22 (SSH): Required for LDM orchestrator control plane (ldm target). Restrict to your IP or VPN for security.
TCP Port 80 (HTTP) & Port 443 (HTTPS): Optional. Required ONLY for direct browser access. Can be closed if using outbound tunnels (lfr-tunnel-docker).
Key Pair & Local SSH Config:
Download the .pem key (e.g. aws-key.pem) and restrict file permissions (chmod 400 ~/.ssh/aws-key.pem).
Add a host entry to ~/.ssh/config on your local machine so OpenSSH and Docker CLI automatically use the key pair for this host:
```
Host 203.0.113.20
    User ec2-user
    IdentityFile ~/.ssh/aws-key.pem
```
Register AWS Node:

# Amazon Linux 2023 uses ec2-user (Ubuntu uses ubuntu)
ldm target add node-a --host 203.0.113.20 --user ec2-user --key ~/.ssh/aws-key.pem
ldm target status node-a

Platform Example 2: Windows WSL2 Compute Node Setup¶

Enable OpenSSH inside WSL2:

sudo apt-get install -y openssh-server rsync
sudo service ssh start

Add Local Public Key to WSL2: Append your SSH public key (~/.ssh/id_rsa.pub) to ~/.ssh/authorized_keys in WSL2.
Register Windows WSL2 Node:

ldm target add win-wsl --host 192.168.1.50 --user developer --key ~/.ssh/id_rsa
ldm target status win-wsl

2. Remote Node Connectivity Verification¶

From your local machine, verify passwordless SSH and non-sudo Docker engine access:

# Test SSH access
ssh -i ~/.ssh/id_rsa user@remote-ip "echo 'SSH Connection OK'"

# Test non-sudo Docker access
ssh -i ~/.ssh/id_rsa user@remote-ip "docker info && docker compose version"

3. LDM Multi-Node Target Registry Management¶

Register and manage target compute nodes directly via the LDM CLI:

# 1. Register a new compute target node
ldm target add prod-node --host 34.200.10.5 --user ubuntu --key ~/.ssh/id_rsa

# 2. List target nodes and verify active target
ldm target ls

# 3. Test target connectivity and Docker engine health probe
ldm target status prod-node

# 4. Set default active target compute node
ldm target use prod-node

When a Node's IP Address Changes¶

A node's address is held in two places: the host in ~/.ldmrc, and the endpoint of the Docker CLI context of the same name. Every remote command dials the context, so both must agree.

ldm target add writes both, which makes re-registering the node the reliable way to move it. Since LDM-#1797 it merges onto the stored node, so only the setting that actually moved needs restating -- the SSH key, the MAC pin and the default flag are kept:

ldm target add prod-node --host <new-ip>

ldm target status compares the two and warns when they disagree, because the probe result belongs to whichever host the context names -- not the one the table shows:

⚠️  Target 'node-a' is registered as 203.0.113.10, but its Docker context points
    at 203.0.113.20. The status above was probed against 203.0.113.20.
💡  Next step: Re-register the node to rebuild the context: ldm target add node-a …

Tip

On AWS, an instance without an Elastic IP is assigned a new public address every time it is stopped and started. Allocating an Elastic IP per node removes this class of problem entirely; note that AWS bills for public IPv4 addresses either way.

If you use ldm node power wake (or scripts/manage_target_nodes.py in the repository) to power nodes on, it resolves the new address and updates ~/.ldmrc, the node-power config and the Docker context together. ldm node power sync-dns does the same for a node that is already running. Starting an instance from the AWS console -- or via the node-power-manager workflow, which runs on a CI runner and updates that runner's files -- leaves your own machine unaware, and re-registering is the fix.

When a Node's SSH User Changes¶

The user is held in the same two places -- user in ~/.ldmrc, and the ssh://<user>@<host> endpoint of the Docker context -- and they drift for the same reasons. The failure is harder to read than a changed IP, because ldm target ls keeps showing a correct configuration while every command fails as somebody else:

$ docker --context node-a version
ldm-automation@203.0.113.10: Permission denied (publickey,...).

ldm doctor compares the two and reports the disagreement (LDM-#1797):

[node-a] SSH user     DRIFTED (ldm-automation)

Re-register to rebuild the context. Nothing else about the node is disturbed:

ldm target add node-a --user ec2-user

SSH Key Selection and `IdentitiesOnly`¶

Docker's context transport invokes ssh without -i, so it relies on ~/.ssh/config and the SSH agent. A catch-all block like this one prevents the node's key from ever being offered, even when the agent holds it:

Host *
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes

Give the node its own block above any Host * block, since SSH takes the first value it finds for each keyword:

Host 203.0.113.10
  User ec2-user
  IdentityFile ~/.ssh/aws-key.pem
  IdentitiesOnly yes

The symptom otherwise is Permission denied (publickey) from docker --context <node> while ssh -i <key> <user>@<host> succeeds.

4. Remote Workload Deployment & Execution¶

Deploy and manage workloads on remote nodes seamlessly:

# Deploy and run project workload on remote node
ldm run my-project --target prod-node

# Stream remote container logs
ldm logs my-project --target prod-node

# Open interactive shell inside remote Liferay container
ldm shell my-project --target prod-node

# Run safe SELECT query against remote database
ldm db query "SELECT userId, emailAddress FROM User_" --target prod-node

# Lifecycle management on remote node
ldm stop my-project --target prod-node
ldm start my-project --target prod-node
ldm restart my-project --target prod-node

5. Live Target Workload Migration¶

Migrate a running workload live between target compute nodes with zero manual data loss:

# Migrate workload from local machine to remote compute node
ldm target migrate local prod-node

# Migrate workload from one remote node to another
ldm target migrate prod-node aws-west

Migration Pipeline Overview¶

Pre-flight Target Probe: Verifies SSH and Docker connectivity on source and destination targets.
Snapshot Creation: Takes a database and named volume snapshot on the source node.
Graceful Stack Stop: Stops containers on the source node.
Workspace Directory Sync: Auto-syncs workspace files via rsync or tar stream.
Target Metadata Reassignment: Updates .liferay-docker.meta to point to the destination target.
Workload Launch: Spins up containers and verifies health on the destination compute node.

If the sync in step 4 fails, the migration stops there and exits 3. The project keeps its original target, its files on the source node are untouched, and the snapshot from step 2 is still available -- so you can fix the transfer and run the migration again. Nothing is reassigned or started on the destination, because a project recorded as living on a node that never received its files is far harder to diagnose than a migration that refused.

A failed sync outside migration -- an ldm run against a remote target, say -- warns rather than refusing, and the node may then be missing recent changes or holding files the project no longer has. The transfer's own error output is in ~/.ldm/last-command.log. Note that rsync abandons --delete for the whole transfer after an I/O error, so a node that hits this repeatedly accumulates stale files until the cause is fixed.

6. Compute Node Sizing & Hardware Recommendations¶

Liferay DXP/Portal stacks consist of three main memory-intensive server engines: the Liferay Tomcat JVM, PostgreSQL database, and Elasticsearch search engine. Choosing the correct compute node hardware prevents Out-Of-Memory (OOM) kernel crashes during deployment and snapshot restoration.

Component Resource Allocation Breakdown¶

Service Component	Recommended Memory	Recommended CPU	Details & Tuning Notes
Liferay Tomcat JVM	4 GB – 8 GB	2 vCPUs	Default heap `-Xms2g -Xmx4g` + Metaspace (`768m`) + OS thread stacks.
PostgreSQL 16 DB	1 GB – 2 GB	1 vCPU	Shared buffers, `work_mem`, and connection pools.
Elasticsearch 8 / OpenSearch	2 GB – 4 GB	1 vCPU	JVM Heap (`-Xms1g -Xmx2g`) + Lucene `mmap` page cache.
Traefik & LDM Helpers	256 MB – 512 MB	0.5 vCPUs	Reverse proxy routing, zero-config SSL cert generation.
Operating System & Docker	1 GB	0.5 vCPUs	Linux kernel buffers, Docker daemon, systemd services.
Total Stack Requirement	8.5 GB – 15.5 GB	4 vCPUs	Ideal target envelope for 1 active LDM stack.

AWS EC2 Instance Selection Guide¶

EC2 Instance Type	vCPUs	Memory	Evaluation & Usage Verdict
`t3.medium`	2	4 GB	❌ Under-Spec: Will crash immediately with Out-Of-Memory (OOM) kills when launching Liferay + Elasticsearch together.
`t3.large`	2	8 GB	⚠️ Minimum Spec: Bare minimum for a single Liferay instance. Risky during heavy database migrations or search reindexing.
`t3.xlarge`	4	16 GB	✅ THE GOLDEN SWEET SPOT: Perfect balance for 1–2 active LDM stacks. Ample RAM for Tomcat JVM, Elasticsearch, and DB snapshot imports without swap thrashing.
`t3.2xlarge`	8	32 GB	🚀 Multi-Stack / Heavy Production: Recommended for multi-tenant compute nodes running 3+ concurrent LDM project stacks.

7. Remote Node Security & Outbound Tunneling¶

When hosting LDM stacks on a remote AWS EC2 compute node, you can choose between Direct Inbound Routing and Outbound Tunnel Routing.

A. Direct Inbound Routing (Public IP + Traefik)¶

Security Group Inbound Rules: Port 22 (SSH for ldm target), Port 80 (HTTP), and Port 443 (HTTPS).
Primary LDM Domain Resolution: LDM automatically manages /etc/hosts entries on your local machine (ldm doctor --fix-hosts), allowing you to use any custom domain structure you want (my-project.local, dev.company.test, etc.) pointing to the remote node IP.
Optional Wildcard DNS Alternative (nip.io / sslip.io): For developers who prefer not to modify their local /etc/hosts file or lack local elevated privileges, wildcard DNS services like nip.io or sslip.io (e.g. https://my-project.34.200.10.5.nip.io) automatically route traffic to the target IP without any system file edits.

B. Outbound Tunnel Routing (`lfr-tunnel-docker` / `ngrok`)¶

Security Group Inbound Rules: Port 22 (SSH ONLY). You can CLOSE ports 80 & 443 entirely!
How It Works: The LDM tunnel agent (lfr-tunnel-docker sidecar) runs inside the remote Docker container network on EC2. It creates an outbound encrypted WebSocket connection to a public tunnel gateway (*.lfr.live / *.lfr-demo.online).
Benefits:
Zero Inbound HTTP/HTTPS Firewall Exposure: Your EC2 instance has no public web ports exposed to scanners or bot attacks.
Instant Public Subdomains: Access your remote stack via a HTTPS URL (e.g. https://my-subdomain.lfr-demo.online) without configuring DNS records or AWS Elastic IPs.

Running a Tunnel on a Remote Compute Node¶

# Boot project on remote node with outbound tunneling enabled
ldm run my-project --target prod-node --share --share-provider lfr-tunnel-docker --share-subdomain my-aws-demo

Last Updated: 2026-10-23 | Last Reviewed: 2026-10-23