feat: made simplens k8s native

.devcontainer/devcontainer.json

@@ -19,7 +19,7 @@
         }
     },
     "postCreateCommand": "npm install && cd dashboard && npm install",
-    "postStartCommand": "docker compose -f docker-compose.infra.yaml up -d",
+    "postStartCommand": "docker compose -f ./docker-compose/docker-compose.infra.yaml up -d",
     "remoteEnv": {
         "NODE_ENV": "development",
         "INFRA_HOST": "localhost"

.env.example

@@ -1,122 +1,46 @@
-# ============================================
-# INFRASTRUCTURE HOST CONFIGURATION
-# ============================================
-# INFRA_HOST: Used by docker-compose.infra.yaml for Kafka & MongoDB configuration
-#   - Same system:      Leave unset (defaults to host.docker.internal)
-#   - Separate systems: Set to the infrastructure machine's IP/hostname
-# ============================================
-# INFRA_HOST=192.168.1.100         # Uncomment and set when running infra on a separate machine
+# ==============================================================================
+# SimpleNS Essential Environment Configuration
+# ==============================================================================
+# NOTE: All operational settings (worker batch sizes, poll intervals, retries,
+# deduplication TTLs, recovery thresholds, and log levels) are now managed
+# dynamically via MongoDB and the Admin Dashboard with live zero-downtime hot-reloading!
+#
+# Administrator credentials are configured interactively on first launch via
+# the Dashboard Setup UI (/setup) and stored securely in MongoDB.
+# ==============================================================================
 
-# ============================================
-# CONNECTION URLS (for application services in docker-compose.yaml)
-# ============================================
-# Same system:      Use host.docker.internal as the host
-# Separate systems: Use the infrastructure machine's IP/hostname
-# ============================================
-
-NODE_ENV=                          # Set to either production or development
 # ============================================
 # API SERVER
 # ============================================
-NS_API_KEY=                        # Generate via: openssl rand -base64 32
+NODE_ENV=production                # Set to 'production', 'development', or 'test'
 PORT=3000                          # API server port
-MAX_BATCH_REQ_LIMIT=1000           # Max batch request size
+JWT_SECRET=                        # REQUIRED: Secret key for JWT admin tokens (min 32 chars, generate with: openssl rand -base64 32)
 
 # ============================================
 # DATABASE
 # ============================================
-MONGO_URI=                         # MongoDB connection string (If you are using self hosted instance, it should be configured with replicaSet: mongodb://host:port/simplens?replicaSet=rs0)
-
+# MongoDB connection URI (configured with replicaSet for change streams/transactions)
+MONGO_URI=mongodb://mongo:27017/simplens?replicaSet=rs0
 # ============================================
 # KAFKA
 # ============================================
-BROKERS=                           # Kafka brokers list (host:port if self-hosted)
-
-# Kafka Topic Partitions (Core Topics)
-DELAYED_PARTITION=1                # Partitions for delayed_notification topic
-NOTIFICATION_STATUS_PARTITION=1    # Partitions for notification_status topic
-
+BROKERS=kafka:9093             # Comma-separated Kafka broker addresses
 
 # ============================================
 # REDIS
 # ============================================
-REDIS_URL=                        # Redis connection URL (redis://host:port if self-hosted)
-
-# ============================================
-# PLUGIN SYSTEM
-# ============================================
-SIMPLENS_CONFIG_PATH=./simplens.config.yaml   # Path to plugin configuration file
-PROCESSOR_CHANNEL=all                          # 'all' for all channels, or specific: 'email', 'sms'
-
-
-# ============================================
-# BACKGROUND WORKER
-# ============================================
-OUTBOX_POLL_INTERVAL_MS=5000       # Poll every 5 seconds
-OUTBOX_CLEANUP_INTERVAL_MS=60000   # Cleanup every 60 seconds
-OUTBOX_BATCH_SIZE=100              # Process 100 entries per poll
-OUTBOX_RETENTION_MS=300000         # Keep published entries for 5 minutes
-OUTBOX_CLAIM_TIMEOUT_MS=30000      # Claim timeout for crashed workers
-
-# ============================================
-# RETRY & IDEMPOTENCY
-# ============================================
-IDEMPOTENCY_TTL_SECONDS=86400      # Keep idempotency keys for 24 hours
-MAX_RETRY_COUNT=5                  # Max retries before marking as failed
-PROCESSING_TTL_SECONDS=120         # Processing lock held for 2 minutes
-
-# ============================================
-# DELAYED NOTIFICATIONS
-# ============================================
-DELAYED_POLL_INTERVAL_MS=1000      # Poll Redis every 1 second
-DELAYED_BATCH_SIZE=10              # Fetch 10 due events per poll
-MAX_POLLER_RETRIES=3               # Max retries before DLQ
-
-# ============================================
-# RECOVERY SERVICE
-# ============================================
-RECOVERY_POLL_INTERVAL_MS=60000            # Run recovery every 1 minute
-PROCESSING_STUCK_THRESHOLD_MS=300000       # Mark as stuck after 5 minutes
-PENDING_STUCK_THRESHOLD_MS=300000          # Mark as orphaned after 5 minutes
-RECOVERY_BATCH_SIZE=50                     # Process 50 stuck notifications per poll
-RECOVERY_CLAIM_TIMEOUT_MS=60000            # Claim timeout for recovery cron worker
-
-# ============================================
-# CLEANUP (runs as part of recovery)
-# ============================================
-CLEANUP_RESOLVED_ALERTS_RETENTION_MS=86400000             # Keep resolved alerts for 24 hours
-CLEANUP_PROCESSED_STATUS_OUTBOX_RETENTION_MS=86400000     # Keep processed status outbox for 24 hours
+REDIS_URL=redis://redis:6379  # Redis connection URL (used for pub/sub sync & locks)
 
 # ============================================
 # LOGGING (Grafana/Loki)
 # ============================================
-LOKI_URL=                          # Loki URL for log aggregation (http://host:port if self-hosted)
-LOG_LEVEL=info                     # Log level: debug, info, warn, error
-LOG_TO_FILE=true                   # Enable file logging
+LOKI_URL=http://loki:3100     # Optional Loki endpoint for centralized logging
+LOG_LEVEL=info                     # Initial bootstrap log level (debug, info, warn, error)
+LOG_TO_FILE=true                   # Enable file transport logging to /logs directory
 
 # ============================================
 # ADMIN DASHBOARD
 # ============================================
-AUTH_SECRET=                       # Generate via: openssl rand -base64 32
-ADMIN_USERNAME=admin               # Dashboard admin username
-ADMIN_PASSWORD=                    # Dashboard admin password
-AUTH_TRUST_HOST=true               # Trust host for auth
-API_BASE_URL=                      # Notification service API URL
-WEBHOOK_HOST=                      # Webhook host for callbacks
-WEBHOOK_PORT=3002                  # Dashboard webhook port
-DASHBOARD_PORT=3002                # Dashboard port
-BASE_PATH=                         # Configurable BASE_PATH to render the admin dashboard
-HTTPS_COOKIE=                      # Controls the storage of session cookies in http and https mode. (If set false, then the cookies can be stored in http connection)
-
-
-# ============================================
-# PLUGIN SPECIFIC ENV VARIABLES (Partition variables + Plugin credentials)
-# ============================================
-
-# Channel-Specific Partitions (Dynamic - based on simplens.config.yaml channels)
-# Topic: {CHANNEL}_notification, Env: {CHANNEL}_PARTITION
-MOCK_PARTITION=1                  # Partitions for mock_notification topic
-
-# PLUGIN CREDENTIALS (@simplens/nodemailer-gmail) (You need to install this plugin separately. Not included with simplens-core)
-EMAIL_USER=                        # Gmail address
-EMAIL_PASS=                        # Gmail app password (not regular password!)
+API_BASE_URL=http://api:3000/sns  # Core Notification service API URL
+DASHBOARD_PORT=3002                     # Next.js Dashboard port
+BASE_PATH=/sns                          # Optional subpath (e.g. /dashboard) or empty for root

.env.test

@@ -3,6 +3,7 @@
 
 NODE_ENV=test
 PORT=3099
+JWT_SECRET=test-jwt-secret-key-for-testing-purposes-only
 
 # Test Database - uses in-memory MongoDB via mongodb-memory-server
 # The MONGO_URI below is a placeholder; tests use MongoMemoryServer
@@ -14,9 +15,6 @@ REDIS_URL=redis://localhost:6380
 # Test Kafka - fully mocked in tests
 BROKERS=localhost:9093
 
-# API Authentication
-NS_API_KEY=test-api-key-for-testing
-
 # Limits
 MAX_BATCH_REQ_LIMIT=1000
 DELAYED_PARTITION=1

AGENTS.md

@@ -21,6 +21,9 @@ Use TypeScript with strict typing and existing `@src/*` path aliases. Match curr
 
 Plan changes before writing code. Follow low-level design principles: single responsibility, clear interfaces, useful dependency inversion, and explicit error handling. Do not duplicate logic; extract shared behavior into focused utilities, services, or test helpers. Always use `axios` instead of the plain `fetch` API in this project.
 
+## Type & Schema Centralisation
+All TypeScript interfaces and types that live inside `src/` **must** be declared in `src/types/types.ts`. All Zod (or other validation library) schemas must be declared in `src/types/schemas.ts`. Individual source files must **not** define their own local interfaces, types, or schemas — import everything from `@src/types/types` or `@src/types/schemas` instead. This rule applies to every layer: API controllers, services, plugins, workers, processors, utilities, and database modules.
+
 ## Testing Guidelines
 Vitest is the test runner. Integration tests also use `supertest`, `mongodb-memory-server`, and Redis mocks. Name test files with `.test.ts`, for example `tests/unit/plugins/loader.test.ts`.
 
@@ -31,5 +34,11 @@ Use short Conventional Commit style messages such as `feat: add email retry poli
 
 PRs should include a summary, linked issue when applicable, test evidence, and screenshots or GIFs for `dashboard/` UI changes. Husky `pre-push` runs both lint commands.
 
+## Dynamic Configuration & Dashboard Settings
+Whenever any configuration field is added, modified, or removed in the dynamic operational configuration (`src/types/schemas.ts` in `operationalSettingsSchema`, `src/config/env.config.ts`, or `src/config/dynamic-config.service.ts`), you **must** also update the dashboard:
+1. Update the `OperationalSettings` interface in `dashboard/lib/types.ts`.
+2. Update the corresponding card and input fields in `dashboard/app/settings/page.tsx` so administrators can view and tune the parameter live with appropriate labels, inputs, and validation hints.
+
 ## Security & Configuration Tips
-Do not commit secrets. Use `.env.example` as the template for `.env`, and update it when adding configuration. For local full-stack work, prefer `docker-compose.dev.yaml`; for core-only work, start infrastructure with `docker compose -f docker-compose.infra.yaml up -d`.
+Do not commit secrets. Use `.env.example` as the template for `.env`, and update it when adding configuration. For local full-stack work, use `./docker-compose/scripts/switch-env.ps1 -Env local -Rebuild` (or `docker compose -f docker-compose/docker-compose.local.yaml up -d`); for shared infrastructure, use `docker compose -f docker-compose/docker-compose.infra.yaml up -d` (or `./docker-compose/scripts/switch-env.ps1 -Env infra-only`).
+

CONTRIBUTING.md

@@ -23,7 +23,7 @@ This repository contains:
 Recommended development paths:
 
 - Use `.devcontainer` (preferred for easy dev setup without any clutter).
-- Use `docker-compose.dev.yaml` for the full stack.
+- Use `docker-compose/` for full-stack Docker development with the environment switcher.
 - Run app code locally only if MongoDB, Redis, and Kafka are available.
 - Create the `simplens.config.yaml` file for any local development option below.
    - Refer the [Plugin Configuration Guide](https://www.simplens.in/docs/core/self-hosting#plugin-configuration) for manually creating `simplens.config.yaml` configuration file.
@@ -40,18 +40,21 @@ Recommended development paths:
 2. Let post-create scripts install dependencies automatically.
 3. Start developing with local source mounted in the container.
 
-### Option B: Docker Compose (`docker-compose.dev.yaml`)
+### Option B: Docker Compose (`docker-compose/`)
 
-Use when you want the full stack via Docker:
+Use when you want the full stack via Docker with the environment switcher:
 
 ```bash
-docker compose -f docker-compose.dev.yaml build
-docker compose -f docker-compose.dev.yaml up -d
+# Windows
+./docker-compose/scripts/switch-env.ps1 -Env local -Rebuild
+
+# Linux/macOS
+./docker-compose/scripts/switch-env.sh local --rebuild
 ```
 
 Note:
 
-- Code changes are not always reflected automatically in containerized runtime images.
+- Code changes are rebuilt into the local image when passing `-Rebuild` or `--rebuild`.
 - Rebuild/restart relevant services after source changes when needed.
 
 ### Option C: Run Core Locally + Infra Services
@@ -75,12 +78,12 @@ cp .env.example .env
 Copy-Item .env.example .env
 ```
 
-3. Start required infrastructure (MongoDB, Kafka, Redis) via Docker:
+3. Start required infrastructure (MongoDB, Kafka, Redis, Loki, Grafana) via Docker:
 
-- Set the env variable `INFRA_HOST=localhost` before starting the services in `docker-compose.infra.yaml`
+- Set the env variable `INFRA_HOST=localhost` before starting the services in `docker-compose/docker-compose.infra.yaml`
 
 ```bash
-docker compose -f docker-compose.infra.yaml up -d
+docker compose -f docker-compose/docker-compose.infra.yaml up -d
 ```
 
 ## Development Commands