ERSAP Java

Micro-services framework for distributed scientific data-stream processing. Engines written in Java, C++, or Python are wired into pipelines via a YAML services file and driven by an orchestrator over ZeroMQ.

---

Build

Requires Java 14+.

git clone https://github.com/JeffersonLab/ersap-java.git
cd ersap-java
./gradlew deploy          # installs into $ERSAP_HOME

---

Run a pipeline

export ERSAP_HOME=/path/to/ersap
export ERSAP_USER_DATA=/path/to/user/data
export ERSAP_MONITOR_FE="<monitor-ip>%9000_java"   # enables metric forwarding

ersap-shell

Inside the shell:

set session      myrun
set servicesFile $ERSAP_USER_DATA/config/pipeline.yaml
set inputDir     $ERSAP_USER_DATA/data/input
set outputDir    $ERSAP_USER_DATA/data/output
run local

---

Observability

1. Start the Monitor Front-End (dedicated node or separate terminal):

j_dpe --host <monitor-ip> --port 9000 --session myrun

2. Start the Prometheus exporter (same node as Monitor FE):

java -cp "$ERSAP_HOME/lib/*" \
     org.jlab.epsci.ersap.util.prometheus.PrometheusExporter \
     --monitor-host <monitor-ip> --monitor-port 9000 \
     --session '*' --prometheus-port 9095

3. Start Prometheus + Grafana via Docker Compose:

# Edit docker/observability/prometheus/prometheus.yml — set target to <monitor-ip>:9095
cd docker/observability
docker compose up -d
open http://localhost:3000    # admin / changeme

---

Perlmutter (NERSC) — multi-node Slurm deployment

One-time setup on the login node ($HOME is the same global filesystem on all nodes):

# Prometheus
PROM_VER=2.53.0
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VER}/prometheus-${PROM_VER}.linux-amd64.tar.gz
tar xzf prometheus-${PROM_VER}.linux-amd64.tar.gz && mv prometheus-${PROM_VER}.linux-amd64 $HOME/prometheus
mkdir -p $HOME/prometheus/data

# Grafana
GRAF_VER=11.1.0
wget https://dl.grafana.com/oss/release/grafana-${GRAF_VER}.linux-amd64.tar.gz
tar xzf grafana-${GRAF_VER}.linux-amd64.tar.gz && mv grafana-v${GRAF_VER} $HOME/grafana
mkdir -p $HOME/grafana/{data,logs,plugins}

Deploy config files (once, from the login node):

bash ~/ersap-java/perlmutter-setup/deploy.sh

This copies the Prometheus scrape config, Grafana custom.ini, datasource, dashboard provisioning, and the ERSAP Overview dashboard into the correct $HOME locations.

Submit a job. Three single-node scripts are provided (see HOWTO-perlmutter.md for a quick cheat sheet):

Script	Purpose
perlmutter-ersap-monitor.slurm	monitor stack only
perlmutter-ersap-processor.slurm	one pipeline container reporting to a remote monitor
perlmutter-ersap-allinone.slurm	monitor and one pipeline on the same node

Use port 19000 instead of 9000 (9000 is busy on Perlmutter); j_dpe --port and --monitor-port must match.

Monitor-only allocation (one compute node)

Use perlmutter-ersap-monitor.slurm when you want the monitor stack (j_dpe + PrometheusExporter + Prometheus + Grafana) to live in its own long-running SLURM job, independent of any processing-node allocation. Pipeline nodes launched elsewhere point at it via ERSAP_MONITOR_FE.

Submit (from the repo root; logs/ must exist before SLURM opens the job's stdout file):

mkdir -p logs
sbatch perlmutter-ersap-monitor.slurm

Edit the #SBATCH --account= line first if your NERSC repo is not amsc016. All other settings (ERSAP_HOME, ports, session, timeouts) can be overridden by exporting them before sbatch.

Discover the allocated monitor node — hostname discovery is a compute-node operation, not a submit-host one:

squeue -j <job-id> -o "%.18i %.9P %.30j %.8u %.2t %.10M %.6D %R"
scontrol show job <job-id>
scontrol show hostnames "$(squeue -h -j <job-id> -o '%N')"
cat  logs/monitor-<job-id>/monitor-info.txt
source logs/monitor-<job-id>/monitor.env   # exposes MONITOR_HOST / ERSAP_MONITOR_FE / ports

monitor-info.txt records the SLURM job id, short hostname, FQDN, expanded node list, endpoints, PIDs, launch commands, and log paths. monitor.env is the machine-readable subset that pipeline nodes can source.

Verify readiness. The job only prints the "ERSAP monitor successfully started" banner after every service is listening, curl http://.../metrics, /-/ready, and /api/health all respond, ersap_prometheus_exporter_up == 1 (i.e. the exporter is attached to j_dpe), and no child has exited. A failure at any step aborts the job with a non-zero exit code.

Follow the logs (on the login node — $HOME is shared with the compute node):

tail -f logs/monitor-<job-id>.out
tail -f logs/monitor-<job-id>/j_dpe.log
tail -f logs/monitor-<job-id>/exporter.log
tail -f logs/monitor-<job-id>/prometheus.log
tail -f logs/monitor-<job-id>/grafana.log

Inspect the running node directly:

NODE=$(squeue -h -j <job-id> -o '%N')
ssh "$NODE" 'ss -tlnp | grep -E ":(19000|9095|9090|3000)"'
ssh "$NODE" 'ps -o pid,cmd -p $(pgrep -d, -u $USER -f "j_dpe|PrometheusExporter|prometheus|grafana-server")'

Connect from your laptop (the info file also prints this line):

ssh -N \
  -L 3000:<monitor-node>:3000 \
  -L 9090:<monitor-node>:9090 \
  <user>@perlmutter.nersc.gov
# Grafana:    http://localhost:3000  (admin / changeme)
# Prometheus: http://localhost:9090

Shut down — clean cancellation triggers the batch script's SIGTERM trap, which sends SIGTERM to every captured child PID, waits up to 20 s, then SIGKILLs stragglers, and writes STATE=stopped to the status file:

scancel <job-id>

---

Docker

# Build
docker build -t ersap-java -f docker/Dockerfile .

# Run a DPE
docker run --rm -it --network=host \
  -v "$PWD/data:/usr/local/ersap/data" \
  ersap-java j_dpe --host 0.0.0.0 --port 7771 --session myrun

---

Contact: sro@jlab.org