PoC: typed REST contract with DRF + drf-spectacular, piloted on /rest/rooms

.github/workflows/django-tests.yml

@@ -0,0 +1,72 @@
+name: "Django tests"
+
+on:
+  pull_request:
+    paths:
+      - "docker/web/**"
+      - ".github/workflows/django-tests.yml"
+  push:
+    branches:
+      - "main"
+      - "devel"
+    paths:
+      - "docker/web/**"
+      - ".github/workflows/django-tests.yml"
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12" # Matches the python:3.12 base image in docker/Dockerfile
+          cache: pip
+          cache-dependency-path: docker/web/requirements.txt
+
+      - name: Install dependencies
+        run: pip install -r docker/web/requirements.txt unittest-xml-reporting
+
+      - name: Run tests
+        run: docker/web/run_tests.sh -v 2
+        env:
+          NSPM_TEST_JUNIT_DIR: ${{ github.workspace }}/test-results
+
+      # annotate_only writes the job summary and failure annotations without creating a
+      # check run, which needs checks: write and so fails on pull requests from forks.
+      - name: Report test results
+        if: ${{ !cancelled() }}
+        uses: mikepenz/action-junit-report@v6
+        with:
+          report_paths: test-results/*.xml
+          annotate_only: true
+          detailed_summary: true
+          include_passed: true
+          require_tests: true
+
+  # test_openapi.py checks openapi.yaml against the DRF views; this checks the UI's generated
+  # TypeScript types against openapi.yaml. Together they catch a contract change committed
+  # without running docker/web/generate_api.sh.
+  api-types:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: docker/web/nspanelmanager/web/react-components
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20" # Matches the node:20 image in docker/Dockerfile
+
+      - name: Regenerate TypeScript types from openapi.yaml
+        run: npm run generate:api-types
+
+      - name: Check the committed types are up to date
+        run: |
+          if [ -n "$(git status --porcelain -- src/api/schema.d.ts)" ]; then
+            git diff -- src/api/schema.d.ts
+            echo "::error file=docker/web/nspanelmanager/web/react-components/src/api/schema.d.ts::src/api/schema.d.ts is out of date with docker/web/openapi.yaml. Run docker/web/generate_api.sh and commit the result."
+            exit 1
+          fi

CONTRIBUTING.md

@@ -0,0 +1,41 @@
+# Contributing
+
+## Changing the REST API
+
+The React UI and Django share a typed contract for the `/rest/*` endpoints that have been ported to Django REST Framework (currently `/rest/rooms` and `/rest/rooms/<id>`; the rest are still plain Django views and aren't covered yet). The chain is:
+
+```
+web/serializers.py + DRF views in web/rest.py
+        │  drf-spectacular
+        ▼
+docker/web/openapi.yaml                      (committed)
+        │  openapi-typescript
+        ▼
+react-components/src/api/schema.d.ts         (committed)
+        │  openapi-fetch
+        ▼
+api.GET / api.PUT / … in react-components/src/api/client.ts
+```
+
+Rename or retype a serializer field and, once the files are regenerated, every UI caller that uses the old name stops compiling.
+
+### After changing a serializer or a DRF view
+
+1. Run `docker/web/generate_api.sh`. It needs the Django requirements (see `docker/web/run_tests.sh`) and Node/npm. It rewrites `openapi.yaml` and `schema.d.ts`.
+2. Fix whatever `npx tsc -b` in `react-components/` now reports.
+3. Commit both generated files with your change.
+
+CI fails if you skip step 1:
+
+- `web/tests/test_openapi.py` (Django tests) fails if `openapi.yaml` doesn't match the views, or if drf-spectacular warns about a view it can't describe, which would otherwise come out as a loose type.
+- The `api-types` job in `.github/workflows/django-tests.yml` fails if `schema.d.ts` doesn't match `openapi.yaml`.
+
+### Conventions for DRF views
+
+- Field names on the wire are snake_case. If a model field is named differently, map it in the serializer (`display_order = serializers.IntegerField(source="displayOrder")`), not in the view.
+- Request serializers include `WarnOnUnknownFieldsMixin`. In production it logs a warning for fields the serializer doesn't know and otherwise ignores them, so a UI and server from different versions still work together. Under the test settings (`REST_REJECT_UNKNOWN_FIELDS = True`) it rejects them with a 400, so a test whose request body has drifted from the serializer fails.
+- Responses keep the `{"status": "ok", ...}` envelope. Errors are always `ErrorResponseSerializer` (`{"status": "error", "message": ..., "errors"?: {field: [msg]}}`), produced by `web.rest.exception_handler`. List each error status a view can return in `@extend_schema(responses=...)` so it shows up in the UI's types.
+
+### Calling the API from the UI
+
+Use `api` from `src/api/client.ts` rather than `fetch()`. Pass request bodies as typed values (e.g. form data from `useForm<RoomUpdateRequest>()`): openapi-fetch infers `body` generically, so an inline object literal isn't checked for extra keys. In production the server only logs a warning for unknown fields, so a typo there fails quietly.

docker/web/generate_api.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+# Regenerate the REST API contract after changing a serializer or a DRF view:
+#   1. docker/web/openapi.yaml, from the DRF views (drf-spectacular). web/tests/test_openapi.py fails if this is stale.
+#   2. The UI's TypeScript types, src/api/schema.d.ts, from openapi.yaml (openapi-typescript).
+# Needs the Django requirements (see run_tests.sh) and Node/npm. Commit both files.
+# See "Changing the REST API" in CONTRIBUTING.md. CI fails if either file is stale.
+set -e
+cd "$(dirname "$0")"
+
+NSPM_DATA_DIR="$(mktemp -d)" python nspanelmanager/manage.py spectacular --validate --fail-on-warn --file openapi.yaml
+
+# generate:api-types runs openapi-typescript with npx rather than as a devDependency: it declares a
+# typescript ^5 peer dependency and the UI is on TypeScript 6. CI runs the same script.
+cd nspanelmanager/web/react-components
+npm run generate:api-types

docker/web/nspanelmanager/nspanelmanager/settings.py

@@ -28,9 +28,12 @@
 # Quick-start development settings - unsuitable for production
 # See https://docs.djangoproject.com/en/4.1/howto/deployment/checklist/
 
+# Persistent data (secret key, database). Overridable so that tests can run outside the container.
+DATA_DIR = environment("NSPM_DATA_DIR", default="/data")
+
 # SECURITY WARNING: keep the secret key used in production secret!
 SECRET_KEY = ""
-SECRET_KEY_PATH = "/data/secret.key"
+SECRET_KEY_PATH = os.path.join(DATA_DIR, "secret.key")
 if os.path.exists(SECRET_KEY_PATH):
     with open(SECRET_KEY_PATH) as f:
         SECRET_KEY = f.read()
@@ -59,6 +62,8 @@
     #'django.contrib.staticfiles', Disabled and make use of django_components.safer_staticfiles instead. It will not serve out .py or .html files
     "django_components.safer_staticfiles",
     "django_components",
+    "rest_framework",
+    "drf_spectacular",
 ]
 
 STATICFILES_DIRS = [os.path.join(BASE_DIR, "web/components"), os.path.join(BASE_DIR, "web/react-components/dist")]
@@ -107,7 +112,7 @@
 # Database
 # https://docs.djangoproject.com/en/4.1/ref/settings/#databases
 
-DATABASES = {"default": {"ENGINE": "django.db.backends.sqlite3", "NAME": "/data/nspanelmanager_db.sqlite3", "timeout": 20}}
+DATABASES = {"default": {"ENGINE": "django.db.backends.sqlite3", "NAME": os.path.join(DATA_DIR, "nspanelmanager_db.sqlite3"), "timeout": 20}}
 
 LOGGING = {
     "version": 1,
@@ -166,3 +171,29 @@
 DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
 
 X_FRAME_OPTIONS = "SAMEORIGIN"
+
+# Django REST Framework, used by the /rest/* views ported from plain Django views.
+REST_FRAMEWORK = {
+    # Like the plain views it replaces, the API has no login and no CSRF check (DRF only checks CSRF in SessionAuthentication).
+    "DEFAULT_AUTHENTICATION_CLASSES": [],
+    "DEFAULT_PERMISSION_CLASSES": ["rest_framework.permissions.AllowAny"],
+    "UNAUTHENTICATED_USER": None,
+    "DEFAULT_PARSER_CLASSES": ["rest_framework.parsers.JSONParser"],
+    "DEFAULT_RENDERER_CLASSES": ["rest_framework.renderers.JSONRenderer"],
+    "EXCEPTION_HANDLER": "web.rest.exception_handler",
+    "DEFAULT_SCHEMA_CLASS": "drf_spectacular.openapi.AutoSchema",
+}
+
+# Fields a DRF request serializer doesn't know are logged and ignored, so UIs and servers from different
+# versions stay compatible. test_settings.py turns this on to reject them with a 400 (see web/serializers.py).
+REST_REJECT_UNKNOWN_FIELDS = False
+
+SPECTACULAR_SETTINGS = {
+    "TITLE": "NSPanelManager REST API",
+    "DESCRIPTION": "The /rest/* API used by the web UI. Generated by drf-spectacular; see docker/web/generate_api.sh.",
+    "VERSION": "1",
+    "SERVE_INCLUDE_SCHEMA": False,
+    # Separate request/response schemas, so read-only fields like `id` aren't required in request bodies.
+    "COMPONENT_SPLIT_REQUEST": True,
+    "ENUM_NAME_OVERRIDES": {"OkStatusEnum": ["ok"], "ErrorStatusEnum": ["error"]},
+}

docker/web/nspanelmanager/nspanelmanager/test_settings.py

@@ -0,0 +1,34 @@
+"""
+Settings used when running the Django test suite (see docker/web/run_tests.sh).
+
+Identical to the production settings except that persistent data (the secret key)
+is written to a throw-away directory instead of /data. The Django test runner
+automatically uses an in-memory SQLite database, so the real database is never touched.
+"""
+
+import os
+import tempfile
+
+os.environ.setdefault("NSPM_DATA_DIR", tempfile.mkdtemp(prefix="nspm-test-"))
+
+from .settings import *
+
+MEDIA_ROOT = os.path.join(DATA_DIR, "media")
+
+# A test request body with a field the serializer doesn't know has drifted from the API: fail it with a 400
+# instead of only logging a warning as production does.
+REST_REJECT_UNKNOWN_FIELDS = True
+
+# Hashing speed is irrelevant for tests.
+PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"]
+
+# Many views log full tracebacks when rejecting bad input, which the tests do on purpose.
+# Set NSPM_TEST_LOG_LEVEL=INFO (or DEBUG) to see them when debugging a failing test.
+LOGGING["root"]["level"] = os.environ.get("NSPM_TEST_LOG_LEVEL", "CRITICAL")
+LOGGING["loggers"] = {"django.request": {"level": os.environ.get("NSPM_TEST_LOG_LEVEL", "CRITICAL")}}
+
+# CI sets NSPM_TEST_JUNIT_DIR to also write JUnit XML results (needs unittest-xml-reporting).
+if os.environ.get("NSPM_TEST_JUNIT_DIR"):
+    TEST_RUNNER = "xmlrunner.extra.djangotestrunner.XMLTestRunner"
+    TEST_OUTPUT_DIR = os.environ["NSPM_TEST_JUNIT_DIR"]
+    TEST_OUTPUT_FILE_NAME = "django-tests.xml"

docker/web/nspanelmanager/web/api.py

@@ -231,13 +231,13 @@ def get_nspanel_config(request):
         base["button2_mqtt_topic"] = get_nspanel_setting_with_default(nspanel.id, "button2_mqtt_topic", "")
         base["button2_mqtt_payload"] = get_nspanel_setting_with_default(nspanel.id, "button2_mqtt_payload", "")
 
-        if nspanel.button1_detached_mode_light:
-            base["button1_detached_light"] = nspanel.button1_detached_mode_light.id
+        if nspanel.button1_detached_mode_entity:
+            base["button1_detached_light"] = nspanel.button1_detached_mode_entity.id
         else:
-            base["button1_detached_mode_light"] = -1
+            base["button1_detached_light"] = -1
 
-        if nspanel.button2_detached_mode_light:
-            base["button2_detached_light"] = nspanel.button2_detached_mode_light.id
+        if nspanel.button2_detached_mode_entity:
+            base["button2_detached_light"] = nspanel.button2_detached_mode_entity.id
         else:
             base["button2_detached_light"] = -1
         base["rooms"] = []

docker/web/nspanelmanager/web/apps.py

@@ -74,28 +74,34 @@ def send_mqttmanager_reload_command():
 
 
 def create_entity_pages_for_all_rooms():
-    from .models import Room, RoomEntitiesPage
+    from .models import Room
 
     for room in Room.objects.all():
-        # Check if an existing scenes page exists, if it does not, create one
-        existing_pages = RoomEntitiesPage.objects.filter(room=room, is_scenes_page=True)
-        if not existing_pages.exists():
-            page = RoomEntitiesPage()
-            page.room = room
-            page.display_order = 0
-            page.page_type = 4
-            page.is_scenes_page = True
-            page.save()
-
-        # Check if an existing entities page exists, if it doesn not, create one.
-        existing_pages = RoomEntitiesPage.objects.filter(room=room, is_scenes_page=False)
-        if not existing_pages.exists():
-            page = RoomEntitiesPage()
-            page.room = room
-            page.display_order = 0
-            page.page_type = 4
-            page.is_scenes_page = False
-            page.save()
+        create_entity_pages_for_room(room)
+
+
+def create_entity_pages_for_room(room):
+    from .models import RoomEntitiesPage
+
+    # Check if an existing scenes page exists, if it does not, create one
+    existing_pages = RoomEntitiesPage.objects.filter(room=room, is_scenes_page=True)
+    if not existing_pages.exists():
+        page = RoomEntitiesPage()
+        page.room = room
+        page.display_order = 0
+        page.page_type = 4
+        page.is_scenes_page = True
+        page.save()
+
+    # Check if an existing entities page exists, if it doesn not, create one.
+    existing_pages = RoomEntitiesPage.objects.filter(room=room, is_scenes_page=False)
+    if not existing_pages.exists():
+        page = RoomEntitiesPage()
+        page.room = room
+        page.display_order = 0
+        page.page_type = 4
+        page.is_scenes_page = False
+        page.save()
 
 
 def create_global_scenes_page():

docker/web/nspanelmanager/web/htmx.py

@@ -105,6 +105,7 @@ def nspanel_delete(request, nspanel_id):
         if request.method == "DELETE":
             nspanel = NSPanel.objects.get(id=nspanel_id)
             nspanel.delete()
+            send_mqttmanager_reload_command()
             response = HttpResponse("", status=200)
             response["HX-Redirect"] = "/"
             return response
@@ -267,6 +268,7 @@ def relay_group_save(request):
             rg = RelayGroup()
         rg.friendly_name = request.POST["name"]
         rg.save()
+        send_mqttmanager_reload_command()
 
         response = HttpResponse()
         response["HX-Refresh"] = "true"
@@ -278,6 +280,7 @@ def relay_group_delete(request, relay_group_id):
     if request.method == "DELETE":
         rg = RelayGroup.objects.get(id=relay_group_id)
         rg.delete()
+        send_mqttmanager_reload_command()
 
         response = HttpResponse()
         response["HX-Refresh"] = "true"
@@ -538,32 +541,6 @@ def partial_remove_entity_from_page_slot(request, page_id, slot_id):
     )
 
 
-def partial_edit_entities_page(request, page_id):
-    data = {
-        "page_id": page_id,
-    }
-    return render(request, "partial/edit_entities_page.html", data)
-
-
-def partial_save_edit_entities_page(request, page_id, page_type):
-    page = RoomEntitiesPage.objects.get(id=page_id)
-    page.page_type = page_type
-    page.save()
-
-    room_id = 0
-    if page.room:
-        room_id = page.room.id
-    send_mqttmanager_reload_command()
-    entities_pages = NSPanelRoomEntitiesPages()
-    return entities_pages.get(
-        request=request,
-        view="edit_room",
-        room_id=room_id,
-        is_scenes_pages=page.is_scenes_page,
-        is_global_scenes_page=(page.room is None),
-    )
-
-
 def get_entity_in_page_slot(page_id, slot_id):
     page = RoomEntitiesPage.objects.get(id=page_id)
     entities = Entity.objects.filter(entities_page=page, room_view_position=slot_id)