Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
d196356
Initial commit
giurgiur99 Nov 11, 2026
93495d1
remove ref
giurgiur99 Nov 11, 2026
9971999
fix: reject cross-index public queries and exit when the database is …
giurgiur99 Nov 11, 2026
05c2ac3
remove ocean.js ref
giurgiur99 Nov 11, 2026
fa6e608
TODO: Remove later: do not review delete files
giurgiur99 Nov 11, 2026
f99b15e
fix: address review comments and keep public read access to the DDO …
giurgiur99 Nov 12, 2026
29b2d8f
fix: paginate internal scans and retry the indexer version read
giurgiur99 Nov 12, 2026
0c2b57a
fixes
giurgiur99 Nov 12, 2026
50c4efe
await in pull image
giurgiur99 Nov 12, 2026
e878c41
fix: always configure the database in setupNodeEnv.sh
giurgiur99 Nov 12, 2026
d9cd572
fix: refresh job writes at once in the Elasticsearch job store
giurgiur99 Nov 12, 2026
4300a44
fix small comments
giurgiur99 Nov 12, 2026
48da82e
default db pass
giurgiur99 Nov 13, 2026
dcf76b4
Merge branch 'main' into feat/elasticsearch-only-db-1551
giurgiur99 Nov 13, 2026
e7a10de
Merge branch 'main' into feat/elasticsearch-only-db-1551
giurgiur99 Nov 13, 2026
79b698e
Merge branch 'main' into feat/elasticsearch-only-db-1551
giurgiur99 Nov 13, 2026
041af91
test: drop removed DB_TYPE from subsidy allowed-images config test
giurgiur99 Nov 13, 2026
23931d2
Sqlite default, add support for postgres
giurgiur99 Nov 17, 2026
b00b616
use lock
giurgiur99 Nov 17, 2026
015611b
exclude files for review
giurgiur99 Nov 17, 2026
3b02344
fixes
giurgiur99 Nov 17, 2026
cd2cbbe
coderabbit fixes
giurgiur99 Nov 17, 2026
60b996a
write dbtype
giurgiur99 Nov 17, 2026
64063ca
fix bad action run
giurgiur99 Nov 17, 2026
1f55faf
debug and fixes
giurgiur99 Nov 17, 2026
de8ce07
remove debug
giurgiur99 Nov 18, 2026
44d78de
fix(database): keep Elasticsearch stores working after a reconnect, s…
giurgiur99 Nov 18, 2026
3155f65
fix(node-metrics): keep the Elasticsearch roll-up correct after a par…
giurgiur99 Nov 18, 2026
489f5f2
simplify
giurgiur99 Nov 18, 2026
ed892be
fix: stop a starting indexer, align Elasticsearch DDO store with SQL,…
giurgiur99 Nov 18, 2026
31f67a0
fix: sweep stale service locks, make the Elasticsearch metrics roll-u…
giurgiur99 Nov 18, 2026
039dae1
feat(database): require DB_TYPE, no guessing from DB_URL, no leftover…
giurgiur99 Nov 18, 2026
356a60a
refactor(node-metrics): first Elasticsearch roll-up of an hour wins, …
giurgiur99 Nov 18, 2026
a2ca568
drop the stale-lock sweep, takeover is enough
giurgiur99 Nov 18, 2026
6d2b894
fix flacky tests (#1617)
alexcos20 Nov 18, 2026
ce11a47
database-neutral DDO search, delete the Elasticsearch query translator
giurgiur99 Nov 19, 2026
8e5fbd4
whitelist groups, unlisted flag and DID/address text search
giurgiur99 Nov 19, 2026
e2254d4
remove the Aquarius class, DDO search and indexing on ProviderInstance
giurgiur99 Nov 19, 2026
7bed070
system tests
giurgiur99 Nov 19, 2026
8e3b6e1
docs, comments and types cleanup for the database and query layer
giurgiur99 Nov 19, 2026
11501d1
under the 100-file limit;
giurgiur99 Nov 19, 2026
5adc742
Merge branch 'main' into feat/elasticsearch-only-db-1551
giurgiur99 Nov 19, 2026
9f2987d
coderabbit fixes
giurgiur99 Nov 19, 2026
57d9c22
fix coderabbit 2
giurgiur99 Nov 19, 2026
861e4e2
add test
giurgiur99 Nov 19, 2026
c4726af
fix cutoff review
giurgiur99 Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .coderabbit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
# TODO: delete this file after PR #1601 (pluggable database) is merged.
reviews:
# Files removed with SQLite, Typesense, the Elasticsearch query translator and the Aquarius
# routes. They no longer exist, so they need no review.
path_filters:
- '!src/@types/DDO/IDdoStateQuery.ts'
- '!src/@types/DDO/IMetadataQuery.ts'
- '!src/@types/DDO/SearchQuery.ts'
- '!src/components/database/ElasticSearchDdoStateQuery.ts'
- '!src/components/database/ElasticSearchMetadataQuery.ts'
- '!src/components/httpRoutes/aquarius.ts'
- '!schemas/**'
- '!scripts/inspect-job-metrics.mjs'
- '!typesense-compose.yml'
- '!src/@types/Typesense.ts'
- '!src/components/database/SQLLite*.ts'
- '!src/components/database/sqlite*.ts'
- '!src/components/database/Typesense*.ts'
- '!src/components/database/typesense*.ts'
- '!src/test/data/ddoSchema.ts'
- '!src/test/data/nonceSchema.ts'
- '!src/test/integration/nonce.test.ts'
- '!src/test/integration/typesense.test.ts'
- '!src/test/unit/sqliteClient.test.ts'
# More than 100 files: CodeRabbit skips the review. Docs, the lock file, agent files and
# tests that change only to fit the new database (new test files are still reviewed).
- '!docs/**'
- '!package-lock.json'
- '!AGENTS.md'
- '!CLAUDE.md'
- '!README.md'
- '!.github/copilot-instructions.md'
- '!src/test/.env.test'
- '!src/test/config.json'
- '!src/test/integration/compute.test.ts'
- '!src/test/integration/configDatabase.test.ts'
- '!src/test/integration/database.test.ts'
- '!src/test/integration/elasticsearch.test.ts'
- '!src/test/integration/imageCleanup.test.ts'
- '!src/test/integration/indexer.test.ts'
- '!src/test/integration/logs.test.ts'
- '!src/test/integration/maintenanceMode.test.ts'
- '!src/test/integration/persistentStorage.test.ts'
- '!src/test/unit/auth/token.test.ts'
- '!src/test/unit/compute.test.ts'
- '!src/test/unit/config.test.ts'
- '!src/test/unit/database/getJobsFilters.test.ts'
- '!src/test/unit/escrowFilters.test.ts'
- '!src/test/unit/fileInfoHandler.test.ts'
- '!src/test/unit/indexer/validation.test.ts'
- '!src/test/unit/maintenanceCoverage.test.ts'
- '!src/test/unit/maintenanceMode.test.ts'
- '!src/test/unit/nodeMetricsHandlers.test.ts'
- '!src/test/unit/oceanP2P.test.ts'
- '!src/test/unit/p2p/provideLimiter.test.ts'
- '!src/test/unit/p2p/republishStoredDdos.test.ts'
- '!src/test/unit/persistentStorageSharing.test.ts'
- '!src/test/unit/safeUrl.test.ts'
- '!src/test/unit/service/serviceDiskUsage.test.ts'
- '!src/test/unit/service/serviceJobsDatabase.test.ts'
- '!src/test/unit/service/serviceRestartRace.test.ts'
- '!src/test/utils/hooks.ts'
- '!src/test/utils/utils.ts'
- '!src/test/performance/util.js'
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,3 @@ src/test
coverage
docker-compose.yml
elasticsearch-compose.yml
typesense-compose.yml
7 changes: 5 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,18 @@
#check env.md file for examples and descriptions on each variable

#----------------- REQUIRED --------------------------
#This is the only required/mandatory variable
#Node will simply not run without this variable
#The node does not run without these two variables
#DB_TYPE: sqlite, postgresql or elasticsearch (the last two also need DB_URL, DB_USERNAME and DB_PASSWORD)
#All the other variables can remain blank (because they have defaults) or simply commented
export PRIVATE_KEY=REPLACE_ME
export DB_TYPE=sqlite
#-----------------------------------------------------

## core
export RPCS=
export DB_URL=
export DB_USERNAME=
export DB_PASSWORD=
export IPFS_GATEWAY=https://ipfs.io/
export ARWEAVE_GATEWAY=https://arweave.net/
export LOAD_INITIAL_DDOS=
Expand Down
12 changes: 4 additions & 8 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,7 @@ export class FileInfoHandler extends CommandHandler {

### Database Abstraction Layer

Multiple database backends via `DatabaseFactory` pattern:

- **Elasticsearch**: DDO metadata, indexer state, orders (async)
- **Typesense**: Search-optimized metadata queries
- **SQLite**: Nonce tracking, config storage, c2d Database (local)
Stores are built by `DatabaseFactory` from `DB_TYPE`: required: `sqlite` (no other setting), `postgresql` or `elasticsearch` (both need `DB_URL`, `DB_USERNAME`, `DB_PASSWORD`). SQLite and PostgreSQL share the stores in `src/components/database/sql/`; Elasticsearch uses the `Elastic*` files. The stores hold DDO metadata, indexer state, orders and the node state (nonces, config, auth tokens, buckets, node metrics, C2D jobs, service jobs, image records, service locks).

Access via singleton `Database` class: `db.ddo`, `db.indexer`, `db.order`, `db.c2d`, `db.nonce`, etc.

Expand All @@ -64,15 +60,15 @@ Central coordinator managing all components. Initialize with configuration and o
### Building & Testing

- **Build**: `npm run build` → compiles TypeScript to `./dist/`
- **Tests**: `npm run test` runs unit + integration (requires Docker Compose for Typesense)
- **Tests**: `npm run test` runs unit + integration (integration tests need Barge; PostgreSQL or Elasticsearch only when `DB_TYPE` selects one)
- **Dev**: Use `npm run build:tsc` in watch mode, or Node's `--experimental-specifier-resolution=node` for ESM imports

### Configuration

Environment-driven via `.env` or `CONFIG_PATH=/path/to/config.json`:

- **PRIVATE_KEY** (required): Node identity for P2P + crypto operations
- **DB_URL**: Elasticsearch/Typesense connection
- **DB_TYPE** / **DB_URL**: database selection (required, no default) and server URL
- **RPCS**: JSON mapping network IDs to RPC endpoints
- **OPERATOR_SERVICE_URL**: C2D cluster URLs (array of strings)
- See [docs/env.md](docs/env.md) for 40+ environment variables
Expand Down Expand Up @@ -108,7 +104,7 @@ validateCommandParameters(command, ['required', 'fields']) // throws ValidatePar

### DDO/Asset Schema Validation

DDOs validated against SHACL schemas in [schemas/](schemas/) (v1, v3, v5, v7). Indexer enforces validation; check `ValidateDDOHandler` for signature verification.
DDOs validated against SHACL schemas from `@oceanprotocol/ddo-js` (`validateDDO` in `src/utils/asset.ts`). Indexer enforces validation; check `ValidateDDOHandler` for signature verification.

### Error Response Format

Expand Down
114 changes: 95 additions & 19 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,49 @@ jobs:

test_unit:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- db: sqlite
- db: postgresql
db_url: 'postgresql://localhost:5432/ocean'
db_username: 'ocean'
db_password: 'changeme'
- db: elasticsearch
db_url: 'http://localhost:9200'
db_username: 'elastic'
db_password: 'changeme'
# a service with an empty image does not start: each leg starts its database only
services:
postgresql:
image: ${{ matrix.db == 'postgresql' && 'postgres:17' || '' }}
env:
POSTGRES_USER: ocean
POSTGRES_PASSWORD: changeme
POSTGRES_DB: ocean
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U ocean"
--health-interval 5s
--health-timeout 5s
--health-retries 20
elasticsearch:
image: ${{ matrix.db == 'elasticsearch' && 'elasticsearch:8.5.1' || '' }}
env:
discovery.type: single-node
ELASTIC_PASSWORD: changeme
xpack.security.enabled: 'true'
xpack.security.http.ssl.enabled: 'false'
ES_JAVA_OPTS: -Xms512m -Xmx512m
ports:
- 9200:9200
options: >-
--health-cmd "curl -sf -u elastic:changeme 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s'"
--health-interval 10s
--health-timeout 5s
--health-retries 20
steps:
- uses: actions/checkout@v7
with:
Expand All @@ -102,19 +145,51 @@ jobs:
IPFS_GATEWAY: http://172.15.0.16:8080/
ARWEAVE_GATEWAY: https://arweave.net/
RPCS: '{ "1": {"rpc": "https://rpc.eth.gateway.fm", "chainId": 1, "network": "mainet", "chunkSize": 100}, "137": {"rpc": "https://polygon.meowrpc.com", "chainId": 137, "network": "polygon", "chunkSize": 100 }, "80001": {"rpc": "https://rpc-mumbai.maticvigil.com","chainId": 80001, "network": "polygon-mumbai", "chunkSize": 100 } }'
DB_URL: 'http://localhost:8108/?apiKey=xyz'
DB_TYPE: 'typesense'
DB_USERNAME: 'elastic'
DB_PASSWORD: 'changeme'
DB_TYPE: ${{ matrix.db }}
DB_URL: ${{ matrix.db_url }}
DB_USERNAME: ${{ matrix.db_username }}
DB_PASSWORD: ${{ matrix.db_password }}
# sniffing returns the container-internal address, unreachable from the runner
ELASTICSEARCH_SNIFF_ON_START: 'false'
ELASTICSEARCH_SNIFF_INTERVAL: 'false'
ELASTICSEARCH_SNIFF_ON_CONNECTION_FAULT: 'false'
FEE_TOKENS: '{ "1": "0x967da4048cD07aB37855c090aAF366e4ce1b9F48", "137": "0x282d8efCe846A88B159800bd4130ad77443Fa1A1", "80001": "0xd8992Ed72C445c35Cb4A2be468568Ed1079357c8", "56": "0xDCe07662CA8EbC241316a15B611c89711414Dd1a" }'
FEE_AMOUNT: '{ "amount": 1, "unit": "MB" }'
- uses: actions/upload-artifact@v7
with:
name: coverage
name: coverage-unit-${{ matrix.db }}
path: coverage/

test_integration:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- db: sqlite
- db: postgresql
db_url: 'postgresql://localhost:5432/ocean'
db_username: 'ocean'
db_password: 'changeme'
# the Elasticsearch of Barge
- db: elasticsearch
db_url: 'http://localhost:9200'
db_username: 'elastic'
db_password: 'changeme'
services:
postgresql:
image: ${{ matrix.db == 'postgresql' && 'postgres:17' || '' }}
env:
POSTGRES_USER: ocean
POSTGRES_PASSWORD: changeme
POSTGRES_DB: ocean
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U ocean"
--health-interval 5s
--health-timeout 5s
--health-retries 20
steps:
- uses: actions/checkout@v7
with:
Expand Down Expand Up @@ -148,7 +223,7 @@ jobs:
- name: Run Barge
working-directory: ${{ github.workspace }}/barge
run: |
bash -x start_ocean.sh --no-node --with-typesense 2>&1 > start_ocean.log &
bash -x start_ocean.sh --no-node 2>&1 > start_ocean.log &
- run: npm ci
- run: npm run build
- run: docker image ls
Expand All @@ -163,7 +238,7 @@ jobs:
[ -f "$HOME/.ocean/ocean-contracts/artifacts/ready" ] && break
done
- name: docker logs
run: docker logs ocean-ocean-contracts-1 && docker logs ocean-typesense-1
run: docker logs ocean-ocean-contracts-1
if: ${{ failure() }}
- name: Build Docker registry auths
id: docker_auths
Expand Down Expand Up @@ -191,19 +266,21 @@ jobs:
IPFS_GATEWAY: http://172.15.0.16:8080/
ARWEAVE_GATEWAY: https://arweave.net/
RPCS: '{ "8996": {"rpc": "http://127.0.0.1:8545", "chainId": 8996, "network": "development", "chunkSize": 100}}'
DB_URL: 'http://localhost:9200'
DB_TYPE: 'elasticsearch'
DB_TYPE: ${{ matrix.db }}
DB_URL: ${{ matrix.db_url }}
DB_USERNAME: ${{ matrix.db_username }}
DB_PASSWORD: ${{ matrix.db_password }}
FEE_TOKENS: '{ "1": "0x967da4048cD07aB37855c090aAF366e4ce1b9F48", "137": "0x282d8efCe846A88B159800bd4130ad77443Fa1A1", "80001": "0xd8992Ed72C445c35Cb4A2be468568Ed1079357c8", "56": "0xDCe07662CA8EbC241316a15B611c89711414Dd1a" }'
FEE_AMOUNT: '{ "amount": 1, "unit": "MB" }'
ASSET_PURGATORY_URL: 'https://raw.githubusercontent.com/oceanprotocol/list-purgatory/main/list-assets.json'
ACCOUNT_PURGATORY_URL: 'https://raw.githubusercontent.com/oceanprotocol/list-purgatory/main/list-accounts.json'
DOCKER_REGISTRY_AUTHS: ${{ steps.docker_auths.outputs.value }}
- name: docker logs
run: docker logs ocean-ocean-contracts-1 && docker logs ocean-typesense-1
run: docker logs ocean-ocean-contracts-1
if: ${{ failure() }}
- uses: actions/upload-artifact@v7
with:
name: coverage
name: coverage-integration-${{ matrix.db }}
path: coverage/

test_system:
Expand Down Expand Up @@ -250,7 +327,7 @@ jobs:
- name: Run Barge
working-directory: ${{ github.workspace }}/barge
run: |
bash -x start_ocean.sh --no-node --with-typesense 2>&1 > start_ocean.log &
bash -x start_ocean.sh --no-node 2>&1 > start_ocean.log &
- run: docker image ls
- name: Delete default runner images
run: |
Expand All @@ -260,13 +337,16 @@ jobs:
with:
persist-credentials: false
repository: 'oceanprotocol/ocean-cli'
ref: 'feature/subsidy_provider'
# the CLI and ocean.js without Aquarius (/api/query, /api/ddo); back to the default
# branches when both are merged
ref: 'feat/node-neutral-query'
path: 'ocean-cli'
- name: Checkout Ocean-js
uses: actions/checkout@v7
with:
persist-credentials: false
repository: 'oceanprotocol/ocean.js'
ref: 'feat/node-neutral-query'
path: 'ocean.js'
- name: Build ocean-js
working-directory: ${{ github.workspace }}/ocean.js
Expand Down Expand Up @@ -302,7 +382,7 @@ jobs:
done

- name: docker logs
run: docker logs ocean-contracts-1 && docker logs ocean-typesense-1
run: docker logs ocean-contracts-1
if: ${{ failure() }}

- name: Build Docker registry auths
Expand All @@ -325,23 +405,20 @@ jobs:
run: |
npm run start > ocean-node.log 2>&1 &
env:
DB_TYPE: sqlite
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
IPFS_GATEWAY: http://172.15.0.16:8080/
ARWEAVE_GATEWAY: https://arweave.net/
P2P_ipV4BindTcpPort: 8000
HTTP_API_PORT: 8001
RPCS: '{ "8996": {"rpc": "http://127.0.0.1:8545", "chainId": 8996, "network": "development", "chunkSize": 100} }'
INDEXER_NETWORKS: '[8996]'
DB_URL: 'http://localhost:9200'
FEE_TOKENS: '{ "1": "0x967da4048cD07aB37855c090aAF366e4ce1b9F48", "137": "0x282d8efCe846A88B159800bd4130ad77443Fa1A1", "80001": "0xd8992Ed72C445c35Cb4A2be468568Ed1079357c8", "56": "0xDCe07662CA8EbC241316a15B611c89711414Dd1a" }'
FEE_AMOUNT: '{ "amount": 1, "unit": "MB" }'
AUTHORIZED_DECRYPTERS: '["0xe2DD09d719Da89e5a3D0F2549c7E24566e947260"]'
P2P_ENABLE_UPNP: 'false'
P2P_ENABLE_AUTONAT: 'false'
ALLOWED_ADMINS: '["0xe2DD09d719Da89e5a3D0F2549c7E24566e947260"]'
DB_TYPE: 'elasticsearch'
DB_USERNAME: 'elastic'
DB_PASSWORD: 'changeme'
MAX_REQ_PER_MINUTE: 320
MAX_CONNECTIONS_PER_MINUTE: 320
DOCKER_COMPUTE_ENVIRONMENTS: '[{"socketPath":"/var/run/docker.sock","environments":[{"storageExpiry":604800,"maxJobDuration":3600,"minJobDuration":60,"resources":[{"id":"cpu","total":4,"max":4,"min":1,"type":"cpu"},{"id":"ram","total":10,"max":10,"min":1,"type":"ram"},{"id":"disk","total":10,"max":10,"min":0,"type":"disk"}],"fees":{"8996":[{"prices":[{"id":"cpu","price":1}]}]},"free":{"maxJobDuration":60,"maxJobs":3,"resources":[{"id":"cpu","max":1},{"id":"ram","max":1},{"id":"disk","max":1}]}}]}]'
Expand Down Expand Up @@ -377,6 +454,5 @@ jobs:
fi
echo "=== Docker Logs ==="
docker logs ocean-contracts-1 || echo "No ocean-contracts-1 container"
docker logs ocean-typesense-1 || echo "No ocean-typesense-1 container"
echo "=== System Processes ==="
ps aux | grep -E "(node|ocean)" || echo "No node/ocean processes found"
3 changes: 0 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,6 @@ out
.yarn/install-state.gz
.pnp.*

# Typesense
typesense-data

# IDEA
.idea

Expand Down
Loading
Loading