FOUR-33159 POC Performance issue when opening the column configuration in a saved search

[devmiguelangel]

Issue & Reproduction Steps

FOUR-33159 POC Performance issue when opening the column configuration in a saved search

Problem

Opening Columns on a Request Saved Search with fulltext PMQL (e.g. (fulltext LIKE "%test%")) took ~76s and could 500 with MySQL error 1038 (Out of sort memory).

Two independent bottlenecks were hit on every load:

1. HasDataColumns::getData() re-executed the full Saved Search PMQL (including LOWER(data) LIKE '%test%') just to sample recent data JSON for column names.
2. For unscoped (global fulltext) searches, SavedSearch::getProcessesAttribute() fell back to Process::get() (all processes) and then ran a recursive ExportManager screen walk per process, adding ~50s of PHP.

Solution

Column discovery no longer depends on the user's PMQL or on sort_buffer_size.

• Bounded, PMQL-free sampling — HasDataColumns::getData() now uses column_discovery_query (type base query, optionally scoped by process_id), orders by indexed id DESC, and limits to 10 rows. The fulltext clause is never applied. MySQL 1038 is caught and returns an empty sample instead of a 500.
• Variable Finder instead of ExportManager — new ProcessVariableDiscoveryService resolves data.* fields from var_finder_variables (indexed) for both scoped and unscoped searches. Screen fallback uses ScreensInProcess (direct BPMN screenRef scan) capped to the 25 most recently updated processes, not the recursive export graph.
• Separate pagination for available columns — GET /api/1.1/processes/variables?onlyAvailable= now paginates Saved Search available + data columns independently. Process variables load in a second phase. Empty processIds returns the unscoped catalog instead of an empty page. Active columns are excluded in SQL. Fixed a duplicate paginate() call in the controller.
• Caching — Cache::flexible + lock + last-known-good for both the discovery service and data_columns, keyed on saved search / process / Variable Finder state.
• Frontend — VariablesList.vue pages available columns until meta.last_page, then switches to process variables. The edit screen no longer background-fetches every page.

Scope

• Saved Search listing results are unchanged; scopeFilter() / valueAliasFullText() are untouched.
• Task and Collection Saved Searches still resolve columns.
• No MySQL configuration change is required.

Related Tickets & Packages

https://processmaker.atlassian.net/browse/FOUR-33159
https://github.com/ProcessMaker/package-savedsearch/pull/695

Code Review Checklist

• I have pulled this code locally and tested it on my instance, along with any associated packages.
• This code adheres to ProcessMaker Coding Guidelines.
• This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
• This solution fixes the bug reported in the original ticket.
• This solution does not alter the expected output of a component in a way that would break existing Processes.
• This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
• This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
• This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
• This ticket conforms to the PRD associated with this part of ProcessMaker.

ci:package-savedsearch:FOUR-33159

.

[cursor]

PR Summary

Medium Risk
Changes pagination and discovery paths for a core Saved Search API; caching and fallbacks reduce outage risk but behavior shifts (separate onlyAvailable paging, empty processIds unscoped catalog).

Overview
Fixes slow/failing Saved Search column configuration by refactoring how process variables and available columns are loaded from GET /api/1.1/processes/variables.

ProcessVariableDiscoveryService centralizes resolving data.* columns: Variable Finder when available, otherwise screen fields via ScreensInProcess (replacing per-process ExportManager walks). Unscoped requests with empty processIds use a bounded catalog (25 recent processes or full finder). Results are cached with Cache::flexible, locks, and last-known-good payloads; MySQL 1038 sort-memory errors fall back to screens instead of 500s.

ProcessVariableController routes onlyAvailable through its own paginator (saved-search available + data columns, no merged process-variable page). Process-variable pagination excludes active saved-search columns in SQL, adds orderBy on finder queries, and delegates screen/fallback paths to the service. Removes duplicate paginate() and inline ExportManager screen scanning.

Tests cover fulltext + empty processIds, independent onlyAvailable pagination, active-column exclusion, no PMQL/data scan on onlyAvailable, and finder sort-memory fallback.

^{Reviewed by Cursor Bugbot for commit 2d4f89b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit eff5d27. Configure here.}

[decisions-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube