From 94e0b682141325bbac556ac8264fa2d94b25093c Mon Sep 17 00:00:00 2001 From: Taylor Downs Date: Mon, 19 Oct 2026 10:46:59 +0200 Subject: [PATCH 1/2] explain clustering --- docs/deploy/requirements.md | 35 ++++++++++++++----- .../current/deploy/requirements.md | 30 +++++++++++++--- 2 files changed, 52 insertions(+), 13 deletions(-) diff --git a/docs/deploy/requirements.md b/docs/deploy/requirements.md index de80a04288f..32e41fc789e 100644 --- a/docs/deploy/requirements.md +++ b/docs/deploy/requirements.md @@ -30,11 +30,13 @@ your deployment partner can better estimate your total cost of ownership. 4. Will the deployment require high-availability? (i.e., if OpenFn will receive requests in real-time from other applications rather than run cron-based jobs, then at least two instances of OpenFn should be run simultaneously - behind a load-balancer, making use of “distributed Erlang” to ensure graceful - application redundancy; if OpenFn will not be responsible for receiving - requests and will only be responsible for making relatively time-independent - outbound requests on a cron schedule, the importance of maintaining a - zero-downtime system is slightly reduced.) + behind a load-balancer, all active at once and joined into one Erlang cluster + (see + [High-Availability on Docker Engine](#c-high-availability-on-docker-engine)); + if OpenFn will not be responsible for receiving requests and will only be + responsible for making relatively time-independent outbound requests on a + cron schedule, the importance of maintaining a zero-downtime system is + slightly reduced.) 5. Does OpenFn need to plug into tooling you already run, such as your own monitoring and alerting, secrets manager, CI/CD pipelines, or change-control process? Integrating with customer-specific tooling is scoped separately from @@ -184,11 +186,27 @@ balancer, which also handles HTTPS. If one host fails, the others keep serving requests. Use a replicated or managed Postgres service so the database doesn't become the weak point. +:::tip Run every node, with no spares + +Keep all your Lightning nodes running behind the load balancer. Don't set up a +hot or cold spare: every node serves traffic, and if one fails the others take +on its load. The nodes join one Erlang cluster on their own, so you don't need +to build any messaging between them. + +Outside Kubernetes, Lightning finds other nodes with gossip by default. Gossip +uses UDP multicast, which most cloud networks block between VMs. On Docker +Engine across several hosts, set `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_ENABLED` to +`true` so nodes find each other through the Postgres database they already +share. + +::: + ```mermaid flowchart TB users(Users & webhooks) --> lb1 lb1 --> ex1 & ex2 ex1 & ex2 -- SQL --> db1 + ex1 <-. Erlang cluster .-> ex2 subgraph lb["Load balancer"] lb1(HTTPS termination) end @@ -206,9 +224,10 @@ flowchart TB ### (d) High-Availability on Kubernetes Run the web app and the worker in separate node pools that scale on their own, -each sized for its workload. This is how OpenFn's own cloud runs. For the -database, use a high-availability managed Postgres service or run Postgres in -the cluster. +each sized for its workload. This is how OpenFn's own cloud runs. As on Docker +Engine, every web app pod is active, and on GKE the pods join one Erlang cluster +on their own. For the database, use a high-availability managed Postgres service +or run Postgres in the cluster. ```mermaid flowchart TB diff --git a/i18n/es/docusaurus-plugin-content-docs/current/deploy/requirements.md b/i18n/es/docusaurus-plugin-content-docs/current/deploy/requirements.md index a39fdaeb153..da8516ca75a 100644 --- a/i18n/es/docusaurus-plugin-content-docs/current/deploy/requirements.md +++ b/i18n/es/docusaurus-plugin-content-docs/current/deploy/requirements.md @@ -1,6 +1,6 @@ --- title: Requisitos -translation_source_hash: de80a04288f28b1b038411af676bcb22d3961597 +translation_source_hash: 32e41fc789ea45ec11ccab3e4e6c01de1135f452 translation_review_status: machine --- @@ -33,8 +33,9 @@ para que tu socio de despliegue pueda estimar mejor el costo total de propiedad. 4. ¿El despliegue requerirá alta disponibilidad? (Es decir, si OpenFn va a recibir solicitudes en tiempo real desde otras aplicaciones en lugar de ejecutar jobs basados en cron, se deberían ejecutar al menos dos instancias - de OpenFn a la vez detrás de un balanceador de carga, usando "Erlang - distribuido" para lograr una redundancia de la aplicación sin interrupciones. + de OpenFn a la vez detrás de un balanceador de carga, todas activas y unidas + en un solo clúster de Erlang (consulta + [Alta disponibilidad en Docker Engine](#c-high-availability-on-docker-engine)). Si OpenFn no va a recibir solicitudes y solo va a hacer solicitudes salientes con un horario cron, donde el momento exacto importa poco, mantener un sistema sin tiempo de inactividad es algo menos importante). @@ -194,11 +195,28 @@ balanceador de carga, que también se encarga de HTTPS. Si un host falla, los demás siguen atendiendo las solicitudes. Usa un servicio de Postgres replicado o administrado para que la base de datos no se convierta en el punto débil. +:::tip Ejecuta todos los nodos, sin nodos de reserva + +Mantén todos tus nodos de Lightning en funcionamiento detrás del balanceador de +carga. No configures un nodo de reserva, ni en caliente ni en frío: todos los +nodos atienden tráfico y, si uno falla, los demás asumen su carga. Los nodos se +unen por su cuenta en un solo clúster de Erlang, así que no tienes que construir +ningún mecanismo de comunicación entre ellos. + +Fuera de Kubernetes, Lightning encuentra los demás nodos mediante gossip de +forma predeterminada. Gossip usa multidifusión UDP, que la mayoría de las redes +en la nube bloquean entre máquinas virtuales. En Docker Engine con varios hosts, +establece `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_ENABLED` en `true` para que los +nodos se encuentren a través de la base de datos Postgres que ya comparten. + +::: + ```mermaid flowchart TB users(Users & webhooks) --> lb1 lb1 --> ex1 & ex2 ex1 & ex2 -- SQL --> db1 + ex1 <-. Erlang cluster .-> ex2 subgraph lb["Load balancer"] lb1(HTTPS termination) end @@ -217,8 +235,10 @@ flowchart TB Ejecuta la aplicación web y el worker en grupos de nodos separados que escalan por su cuenta, cada uno dimensionado para su carga de trabajo. Así funciona la -propia nube de OpenFn. Para la base de datos, usa un servicio administrado de -Postgres con alta disponibilidad o ejecuta Postgres en el clúster. +propia nube de OpenFn. Igual que en Docker Engine, todos los pods de la +aplicación web están activos y, en GKE, se unen por su cuenta en un solo clúster +de Erlang. Para la base de datos, usa un servicio administrado de Postgres con +alta disponibilidad o ejecuta Postgres en el clúster. ```mermaid flowchart TB From 4257405b06b70f91233d388cb65093173ad24770 Mon Sep 17 00:00:00 2001 From: Taylor Downs Date: Tue, 20 Oct 2026 10:40:36 +0200 Subject: [PATCH 2/2] remove misleading text --- docs/deploy/requirements.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/deploy/requirements.md b/docs/deploy/requirements.md index 32e41fc789e..975344dc89b 100644 --- a/docs/deploy/requirements.md +++ b/docs/deploy/requirements.md @@ -225,9 +225,9 @@ flowchart TB Run the web app and the worker in separate node pools that scale on their own, each sized for its workload. This is how OpenFn's own cloud runs. As on Docker -Engine, every web app pod is active, and on GKE the pods join one Erlang cluster -on their own. For the database, use a high-availability managed Postgres service -or run Postgres in the cluster. +Engine, every web app pod is active, and on GKE the pods join one Erlang +cluster. For the database, use a high-availability managed Postgres service or +run Postgres in the cluster. ```mermaid flowchart TB