feat(ensemble_test_runner): add ensemble test runner workflow and enh… - #2371
Merged
Merged
Conversation
…ance documentation - Introduced a new GitHub Actions workflow for the Ensemble Test Runner, enabling automated testing for standalone and mixed Flutter/Ensemble applications. - Updated README and documentation to clarify usage, including the new `target` syntax for element matching and the requirement for `startScreen` in standalone tests. - Added example host application structure to demonstrate integration with Ensemble. - Enhanced schema definitions to support new testing capabilities and element targeting.
…umentation - Added support for capturing step screenshots and logging app console, API, and storage logs in the HTML report for host tests. - Introduced `skip` functionality for password fields to prevent test failures when secure content is visible. - Updated example host application to demonstrate new API calls and logging features. - Enhanced configuration options in `config.yaml` for enabling per-step screenshots and API mocks. - Improved test cases to validate new features, ensuring robust testing of host interactions.
…and documentation - Updated device.dart to handle uncaught plugin errors gracefully during device info retrieval. - Enhanced README and documentation to clarify the integration testing mode and its usage. - Modified application_yaml_tests.dart to support conditional execution based on the test mode. - Improved test harness to ensure native plugins are registered during integration tests. - Added new tests to validate device info mock behavior and integration mode functionality.
…e clarity - Removed unnecessary imports from various files to streamline the codebase. - Updated a method call to use `await` for better asynchronous handling in `mock_composition.dart`. - Enhanced variable scoping in `local_action_executor.dart` for improved readability.
…ub Actions - Introduced a new GitHub Actions workflow for the Ensemble Test Runner, enabling automated analysis, testing, and example suite execution for Flutter applications. - The workflow includes steps for code checkout, Flutter SDK setup, Melos installation, package analysis, schema drift checks, and running tests for both the main application and example widgets.
…g in test execution - Enhanced the `mapExecutorFailure` method to provide more accurate error classifications for tap/wait-for-id timeouts, distinguishing between `elementNotFound` and `elementNotInteractable`. - Updated the `TestStepExecutor` to throw specific errors when widgets are off-screen or not hit-testable. - Added unit tests to validate the new error mapping logic and ensure correct behavior for various timeout scenarios. - Cleaned up imports and improved code clarity in the local execution session.
…ing capabilities - Introduced timeout support for tap, double tap, and long press actions in the test framework, allowing for more flexible interaction handling. - Improved the resolution of element targets by adding support for actionable controls and refining the logic for handling text locators. - Enhanced error handling in the local execution session to better manage visibility assertions and element existence checks. - Updated the host application test driver to ensure proper runtime initialization based on execution mode. - Added new tests to validate the behavior of structured locators and timeout functionality in various scenarios.
…dling - Added support for secondary failure tracking during host test execution, allowing for better diagnostics of application errors. - Improved the `ensureHostFixturesSupported` function to validate API mocks and storage fixtures before test execution. - Enhanced error handling to classify pending application errors as crashes after screenshots are taken. - Introduced new tests to validate the behavior of API mocks and error handling in various scenarios, ensuring robust test coverage.
- Removed specific push path triggers from the Ensemble Test Runner workflow, streamlining the conditions under which the workflow is executed.
…egration handling - Updated the README to clarify the usage of `runApplicationYamlTests` and its integration with the `ensembleTestExecutionMode` dart-define. - Simplified the main entry point for application YAML tests to directly call `runApplicationYamlTests`. - Enhanced the CLI to adapt host entries for integration testing, ensuring proper binding selection based on the execution mode. - Improved the handling of integration and widget test modes, allowing for more flexible test execution. - Added documentation to emphasize the preferred usage of `runApplicationYamlTests` for better clarity.
- Updated the `toString` method in `TestExecutionError` to return only the message, improving clarity in error reporting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Extends
ensemble_test_runnerso standalone Ensemble, pure Flutter, and mixed Flutter+Ensemble apps share one YAML engine, CLI, reports, and locator/action model—without framework selectors or a second runner.Also addresses PR review P1s: negative structured assertions, integration viewport/service ownership, host-driver mock branching, element-identity Finders, text-locator collapse, fixtures-before-launch, screenshot error handling, plus partial teardown, structured
timeoutMs, and retry duration summing.Related Issue
N/A (extends #2367 / #2368 test-runner work)
Type of Change
What Has Changed
LocalTestExecutionSession+YamlStepDispatcher)example_hostmixed Flutter + Ensemble demotarget/within/occurrence/ text) with element-identity resolution and actionable text collapseensembleTestHostOwnsServices)How to Test
cd tools/ensemble_test_runner/example flutter pub get dart run ensemble_test_runner:ensemble_test --tests-dir=tests --test-entry=test/ensemble_yaml_tests.dart --mode=widgetcd tools/ensemble_test_runner/example_host flutter pub get dart run ensemble_test_runner:ensemble_test --tests-dir=tests --test-entry=test/application_yaml_tests.dart --mode=widgetScreenshots / Videos
HTML reports from widget suites under:
tools/ensemble_test_runner/example/build/ensemble_test_runner/report/tools/ensemble_test_runner/example_host/build/ensemble_test_runner/report/Checklist
flutter analyzeand addressed any new warningsflutter testand all tests pass