Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
447c4d3
SDK-7414: open the CLI flow for WebdriverIO-cucumber
AdityaHirapara Sep 2, 2026
78a90f2
feat(cli): add WdioCucumberTestFramework and drive it from cucumber's…
AdityaHirapara Sep 2, 2026
7299385
feat(cli): widen cucumber's shared dispatch gates and hook payload
AdityaHirapara Sep 2, 2026
71fa9bb
fix(cli): honour ignoreHooksStatus when marking cucumber sessions
AdityaHirapara Sep 2, 2026
e6596eb
fix(cli): route turboscale marking, hook-failure verdicts, a11y gate …
AdityaHirapara Sep 3, 2026
bdf9cb1
fix(cli): mark the cucumber session failed when a build hook fails wi…
AdityaHirapara Sep 3, 2026
650ca23
fix(cli): fail the cucumber session when any scenario fails, whatever…
AdityaHirapara Sep 4, 2026
28aa160
fix(a11y): decode goog:chromeOptions when it arrives over gRPC
AdityaHirapara Sep 6, 2026
e7f74f5
fix(cli): honour preferScenarioName on the CLI flow, and keep it out …
AdityaHirapara Sep 6, 2026
f1444f6
refactor(a11y): simplify the mocha-only guard on the hook scan gate
AdityaHirapara Sep 8, 2026
7573393
docs(cli): trim the cucumber hook comments in service.ts
AdityaHirapara Sep 8, 2026
ed012cf
refactor(cli): let automateModule own the preferScenarioName rename
AdityaHirapara Sep 8, 2026
a16ef31
docs(cli): drop parity-table references from cucumber comments
AdityaHirapara Sep 9, 2026
f8f5f6a
docs(cli): condense the BEFORE_ALL cascade comment
AdityaHirapara Sep 9, 2026
1382903
fix(cli): send the raw uri in bdd_meta_info.feature.path
AdityaHirapara Sep 9, 2026
e35a3d1
test(cli): name the cucumber framework suite as the main test file
AdityaHirapara Sep 9, 2026
8c550bd
test(cli): drop migration-plan references from test names and labels
AdityaHirapara Sep 9, 2026
17ed99f
fix(cli): relativise bdd_meta_info.feature.path, which 1382903 did not
AdityaHirapara Sep 9, 2026
f45497d
docs(cli): condense the build-level hook comment
AdityaHirapara Sep 9, 2026
f84fc39
fix(cli): status-mark a session even when setSessionName is false
AdityaHirapara Sep 9, 2026
59a03de
fix(cli): register the driver when every product is turned off
AdityaHirapara Sep 9, 2026
31c5cc7
test(cli): merge the driver-registration tests into service.test.ts
AdityaHirapara Sep 9, 2026
8d63e5f
test(cli): fold service.preferScenarioName.cli.test.ts into service.t…
AdityaHirapara Sep 9, 2026
5fd444d
test(cli): fold the automateModule satellites into its main suite
AdityaHirapara Sep 9, 2026
e9f9aaa
Merge branch 'main' into SDK-7414/wdio-cucumber-platformisation
AdityaHirapara Sep 9, 2026
6a73c32
style(test): drop an unnecessary semicolon failing lint
AdityaHirapara Sep 9, 2026
bb7eacf
Merge branch 'main' into SDK-7414/wdio-cucumber-platformisation
AdityaHirapara Sep 11, 2026
343cc78
fix(cli): decouple scenario naming from the status opt-out, and key r…
AdityaHirapara Sep 17, 2026
bc28ca6
test(cli): pin the tag-scan contract, and make the chrome-options dro…
AdityaHirapara Sep 17, 2026
72fbc49
fix(a11y): report only real chrome-options drops, and pin the untouch…
AdityaHirapara Sep 17, 2026
615130a
Merge branch 'main' into SDK-7414/wdio-cucumber-platformisation
AdityaHirapara Sep 17, 2026
6844406
chore(changeset): auto-generate from PR template (minor)
github-actions[bot] Sep 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/pr-191.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@wdio/browserstack-service": minor
---

- WebdriverIO + CucumberJS now runs on the BrowserStack CLI flow, the same path Mocha already uses. Reporting, accessibility, Percy and session naming behave as before — no config change is needed.
- Fixed: sessions were left unmarked pass/fail when setSessionName: false was set. Naming and status are independent options again.
- Fixed: the accessibility extension was not applied on non-BrowserStack infrastructure, leaving scans empty on otherwise green runs.
2 changes: 1 addition & 1 deletion packages/browserstack-service/src/cli/cliUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ const CLI_DOWNLOAD_TMP_SUFFIX = '.zip'
export class CLIUtils {
static automationFrameworkDetail = {}
static testFrameworkDetail = {}
static CLISupportedFrameworks = ['mocha']
static CLISupportedFrameworks = ['mocha', 'cucumber']

static isDevelopmentEnv() {
return process.env.BROWSERSTACK_CLI_ENV === 'development'
Expand Down

Large diffs are not rendered by default.

11 changes: 10 additions & 1 deletion packages/browserstack-service/src/cli/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { BROWSERSTACK_ACCESSIBILITY, BROWSERSTACK_OBSERVABILITY, BROWSERSTACK_TE
import type { Options } from '@wdio/types'
import TestOpsConfig from '../testOps/testOpsConfig.js'
import WdioMochaTestFramework from './frameworks/wdioMochaTestFramework.js'
import WdioCucumberTestFramework from './frameworks/wdioCucumberTestFramework.js'
import WdioAutomationFramework from './frameworks/wdioAutomationFramework.js'
import WebdriverIOModule from './modules/webdriverIOModule.js'
import AccessibilityModule from './modules/accessibilityModule.js'
Expand Down Expand Up @@ -47,7 +48,7 @@ export class BrowserstackCLI {
modulesLoaded = false
binSessionId: string | null = null
modules: Record<string, BaseModule> = {}
testFramework: WdioMochaTestFramework|null = null
testFramework: WdioMochaTestFramework|WdioCucumberTestFramework|null = null
cliParams: Record<string, string> | null = null
automationFramework: WdioAutomationFramework|null = null
SDK_CLI_BIN_PATH: string | null = null
Expand Down Expand Up @@ -548,7 +549,15 @@ export class BrowserstackCLI {
const testFrameworkDetail = CLIUtils.getTestFrameworkDetail()
if (testFrameworkDetail.name.toLowerCase() === 'webdriverio-mocha') {
this.testFramework = new WdioMochaTestFramework([testFrameworkDetail.name], testFrameworkDetail.version, this.binSessionId as string)
return
}
if (testFrameworkDetail.name.toLowerCase() === 'webdriverio-cucumber') {
this.testFramework = new WdioCucumberTestFramework([testFrameworkDetail.name], testFrameworkDetail.version, this.binSessionId as string)
return
}
// An unmatched name leaves testFramework null, and every CLI event then no-ops with no
// error of any kind. Name it so the silence is diagnosable.
this.logger.error(`setupTestFramework: no CLI test framework registered for name=${testFrameworkDetail.name}; test events will not be tracked`)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Warning — [GRACEFUL DEGRADATION] Unconditional error log fires on every WDIO Jasmine run, a supported-by-design configuration

Problem

The new fall-through log is unconditional: any testFrameworkDetail.name that is not webdriverio-mocha or webdriverio-cucumber reaches it and emits at error level.

launcher.ts:345 sets that name mechanically as WDIO_NAMING_PREFIX + config.framework, so the three WDIO runner values produce webdriverio-mocha, webdriverio-cucumber and webdriverio-jasmine. Jasmine is a first-class WDIO framework this service supports, and it has no CLI test-framework class by design — so after this change every Jasmine run on the CLI/binary flow logs setupTestFramework: no CLI test framework registered for name=webdriverio-jasmine; test events will not be tracked at error level, on a path where nothing is actually broken relative to the shipped behaviour.

Two consequences:

  1. Customers on a supported configuration now see a BrowserStack error in their test output every run. That is the shape that generates support tickets, and it contradicts SH-12, which asks a feature that cannot run to disable itself at debug level rather than surface as a failure.
  2. The diagnostic value the comment is after ("name it so the silence is diagnosable") is diluted: the genuinely interesting case — a name nobody expected — is now indistinguishable from the routine Jasmine case, because both print the same error line.

The intent of the added log is right; only its severity and its indiscriminate reach are wrong.

Suggested Fix

Separate the known, expected unregistered frameworks from the unexpected ones, and drop the expected case to debug:

const name = testFrameworkDetail.name.toLowerCase()
if (name === 'webdriverio-mocha') { ... return }
if (name === 'webdriverio-cucumber') { ... return }

// Jasmine has no CLI test framework by design — expected, not an error.
if (name === 'webdriverio-jasmine') {
    this.logger.debug(`setupTestFramework: no CLI test framework for name=${testFrameworkDetail.name}; test events are not tracked for this framework`)
    return
}
this.logger.warn(`setupTestFramework: unrecognised test framework name=${testFrameworkDetail.name}; test events will not be tracked`)

If keeping a single branch is preferred, at minimum demote the level to debug/warn — an unsupported-by-design framework is not an error condition in the customer's run.

Confidence: 🟢 cards/_shared.md SH-12: a feature that cannot run must disable itself loudly at debug level, leaving the test unaffected; packs/default.md DEF-02 requires the signal be surfaced, not shouted on an expected path

— SDK PR Review Agent

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not valid — jasmine never reaches this line.

setupTestFramework() is called from loadModules(), which runs inside bootstrap(), and both entry points to bootstrap() are gated on the CLI allow-list:

  • launcher.ts:343if (CLIUtils.checkCLISupportedFrameworks(config.framework) && !isMultiremote)
  • service.ts:176 — the same gate in the worker

with CLISupportedFrameworks = ['mocha', 'cucumber'] (cliUtils.ts:50). Jasmine is not in that list, so the CLI never starts for a jasmine run and this method is never entered.

The comment's own phrasing points at it: "every Jasmine run on the CLI/binary flow" — jasmine has no CLI/binary flow. The log therefore fires only for the genuinely unexpected name, which is the case it was written for, and error level is appropriate there.

}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,14 @@ export default class AccessibilityModule extends BaseModule {
if (!this.accessibility) {
return
}
// Open the scan gate for the hook window so DOM-changing commands issued inside
// before/beforeEach/afterEach/after hooks trigger scans (web per-command path). The
// following onBeforeTest re-computes the per-test gate, so this only affects the hook.
if (this.autoScanning && sessionId !== undefined && sessionId !== null) {
// Open the scan gate for the hook window, so commands issued inside hooks are scanned.
// Mocha-only, as legacy gates the identical write (`this._framework === 'mocha'`): it
// relies on the following onBeforeTest re-computing the gate, which holds only where
// beforeEach precedes beforeTest. Cucumber inverts that ordering, so there the write
// would land last and force the gate permanently open.
const frameworkName = String(TestFramework.getState(testInstance, TestFrameworkConstants.KEY_TEST_FRAMEWORK_NAME) || '')
const isMocha = frameworkName.toLowerCase().includes('mocha')
if (this.autoScanning && isMocha && sessionId) {
Comment on lines +98 to +99

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Warning — [CORRECTNESS] Narrowing the hook-window gate to mocha silently drops jasmine hook scans

Problem

The hook-window scan gate is now written only when the framework name contains mocha. Before this change the same write happened for every WDIO runner, so jasmine silently loses it: DOM-changing commands issued inside a jasmine beforeEach/afterEach are no longer scanned on the web per-command path. Nothing errors and nothing logs — the customer just sees fewer accessibility scans on the dashboard.

The rationale in the comment only justifies excluding cucumber: cucumber's scenario boundary precedes its Before hooks, so the write would land last and force the gate permanently open. That reasoning does not apply to jasmine — like mocha, jasmine runs beforeEach before beforeTest, so the following onBeforeTest re-computes the per-test gate and the hook-window write is harmless. The discriminator the code actually needs is the hook/test ordering, not the identity mocha.

Verified: the legacy handler does gate this identically (accessibility-handler.ts:477if (this._framework === 'mocha' && this._sessionId)), so this is legacy parity and may well be deliberate. But the sibling hunk in the same PR explicitly reasons about jasmine ("mocha and jasmine keep the exact 3-arg behaviour") while this one does not mention it at all, which reads as an oversight rather than a decision. packs/observability.md OB-03 (second shape) names exactly this pattern: a path gated on a framework name silently excludes the variants that string does not match, and the change must cover every variant in the runner matrix.

Suggested Fix

Either widen the gate to the frameworks whose ordering makes the write safe, i.e. exclude only cucumber:

const frameworkName = String(TestFramework.getState(testInstance, TestFrameworkConstants.KEY_TEST_FRAMEWORK_NAME) || '').toLowerCase()
// cucumber's scenario boundary precedes its Before hooks, so this write would land last and
// force the gate permanently open; mocha and jasmine both run beforeEach before beforeTest.
const reopensSafely = !frameworkName.includes('cucumber')
if (this.autoScanning && reopensSafely && sessionId) {
    this.accessibilityMap.set(sessionId, true)
}

…or, if dropping jasmine hook-window scans is intended (strict parity with accessibility-handler.ts:477), say so in the comment — "jasmine is excluded for parity with legacy, which also gates on _framework === 'mocha'" — and add a jasmine case to the new test block so the choice is pinned rather than incidental.

Confidence: 🟢 packs/observability.md OB-03 second shape (framework-name gating silently excludes the variants the string does not match); verified against legacy gate accessibility-handler.ts:477 and the runner hook ordering the comment relies on

— SDK PR Review Agent

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is legacy parity, and deliberate — but you're right that the comment doesn't say so.

Legacy gates identically at accessibility-handler.ts:515:

if (this._framework === 'mocha' && this._sessionId) {

So jasmine never had hook-window scans on either flow; nothing is lost by this change. Parity is the bar for this migration, so widening the gate to jasmine would be new behaviour rather than a fix.

Taking the second half of your suggestion: the comment should record that as a decision rather than leaving it to read as an oversight, since the sibling hunk does reason about jasmine explicitly. No behaviour change.

this.accessibilityMap.set(sessionId, true)
}
} catch (error) {
Expand Down Expand Up @@ -343,7 +347,13 @@ export default class AccessibilityModule extends BaseModule {

const sessionId = AutomationFramework.getState(autoInstance, AutomationFrameworkConstants.KEY_FRAMEWORK_SESSION_ID)
const accessibilityOptions = this.config.accessibilityOptions
const shouldScanTest = this.autoScanning && shouldScanTestForAccessibility(suiteTitle, test.title || '', accessibilityOptions as Record<string, string> | undefined) && this.accessibility
// Cucumber filters scans by gherkin tag, which needs the world object and the 6-arg
// form of shouldScanTestForAccessibility; the 3-arg form matches include/exclude tags
// against the test title instead and so silently scans every scenario. `args.world` is
// only ever populated on the cucumber path, so mocha and jasmine keep the exact 3-arg
// behaviour — both extra args arrive undefined/false and the tag branch is not taken.
const world = args.world as { [key: string]: unknown } | undefined
const shouldScanTest = this.autoScanning && shouldScanTestForAccessibility(suiteTitle, test.title || '', accessibilityOptions as Record<string, string> | undefined, world, Boolean(world)) && this.accessibility

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Warning — [TEST COVERAGE] Cucumber tag-filtering path ships with no test; the blanket mock hides the positional contract

Problem

The behavioural half of this unit that a customer can actually see — cucumber scans now filtered by gherkin tag instead of by substring match on the scenario title — lands with no test. The four tests added in this PR all exercise the onHookStart mocha/cucumber gate; none asserts that onBeforeTest forwards world and the isCucumber flag.

That gap is load-bearing here because the module test suite mocks the collaborator unconditionally (shouldScanTestForAccessibility: vi.fn().mockReturnValue(true) at tests/cli/modules/accessibilityModule.test.ts:33). With a blanket true, no existing test can fail if the two new arguments are dropped, reordered, or land in the wrong positional slots — and the real signature is positional (util.ts:528(suiteTitle, testTitle, accessibilityOptions?, world?, isCucumber?)). I verified the call in the diff matches that order, so this is a missing-regression-guard finding, not a live bug.

packs/default.md DEF-12 applies: a behavioural change with no test where the repo has an established test location for it — this module is densely tested and the same PR already edits that exact file. (Note also that the comment says "the 6-arg form"; the function takes five parameters.)

Suggested Fix

Add one assertion-on-the-call test next to the new onHookStart block, e.g.:

it('passes the cucumber world through so scans are filtered by gherkin tag', async () => {
    const world = { pickle: { tags: [{ name: '@a11y' }] } }
    await accessibilityModule.onBeforeTest({ instance: mockTestInstance, world, test: { title: 't' } } as any)
    expect(vi.mocked(shouldScanTestForAccessibility)).toHaveBeenCalledWith(
        expect.anything(), 't', expect.anything(), world, true)
})

it('keeps the 3-arg behaviour for mocha — no world, isCucumber false', async () => {
    await accessibilityModule.onBeforeTest({ instance: mockTestInstance, test: { title: 't' } } as any)
    expect(vi.mocked(shouldScanTestForAccessibility)).toHaveBeenCalledWith(
        expect.anything(), 't', expect.anything(), undefined, false)
})

This pins the positional contract with util.ts that the blanket mock currently hides.

Confidence: 🟢 packs/default.md DEF-12 + packs/tests.md TS-04: the collaborator is mocked to a constant true, so no existing test can catch a positional-argument regression on the new 5-arg call

— SDK PR Review Agent

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bc28ca6 — this was a real gap, and the most substantive of the test-coverage findings.

The 3-arg → 6-arg switch is new product behaviour (cucumber filtering scans by gherkin tag) and shipped with nothing pinning it. Its correctness rests entirely on args.world being populated on the cucumber path only, and that invariant was asserted by a comment rather than a test — so a future change populating world elsewhere would silently engage the tag branch for mocha, with no failure to catch it.

Both halves are now pinned:

it('passes the world through so cucumber can filter scans by tag', ...)
    expect(shouldScanTestForAccessibility).toHaveBeenCalledWith('Feature', 'Scenario', expect.anything(), world, true)

it('leaves the tag branch untaken when no world is supplied', ...)
    expect(shouldScanTestForAccessibility).toHaveBeenCalledWith('Suite', 'Test', expect.anything(), undefined, false)

Falsified: reverting the call to the 3-arg form fails both.


this.accessibilityMap.set(sessionId, shouldScanTest)

Expand Down
Loading
Loading