Skip to content

Re-enable the video export tests - #26086

Open
jkmassel wants to merge 1 commit into
trunkfrom
jkmassel/unskip-video-export-tests
Open

jkmassel wants to merge 1 commit into
trunkfrom
jkmassel/unskip-video-export-tests

Conversation

@jkmassel

@jkmassel jkmassel commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Re-enables the three skipped video export tests. They weren't broken — their 2–3s timeouts left too little headroom for an AVFoundation transcode on a busy or freshly reset simulator.

Summary

  • MediaURLExporterTests.testThatURLExportingVideoWorks / testThatURLExportingVideoWithoutGPSWorks — skipped in Sidebar: Enhance split view behavior #23627 as "too flaky in iOS 18"
  • ItemProviderMediaExporterTests.testThatVideoIsExported — skipped in 2023 as "takes too long"
  • Timeouts 2–3s → 60s. A timeout only bounds how long a real failure takes to report; a passing test isn't slowed down.
  • validateVideoExport compares the exported duration with accuracy: 0.01 instead of exact == — the export re-encodes (AVAssetExportPresetHighestQuality), so the output duration isn't guaranteed to match the source bit-for-bit. testThatVideoIsExported already uses the same tolerance.

Root Cause

Neither skip commit records a failure, but the timings make the cause clear. Each test transcodes a 3.47s 640×360 fixture; on an idle machine that takes ~0.5s, but the tail is long:

Condition p50 Max
Warm simulator, idle machine 0.51–0.62s 6.07s
Freshly erased simulator (what CI's reset_simulator: true does) 0.69–1.02s 9.83s
CPU saturated (32 busy processes on 16 cores) 4.93–5.83s 20.11s

Even the idle case exceeded the old 3s timeout. 60s gives ~3× headroom over the worst case we could produce.

Test Plan

Local stress on the CI configuration (Xcode 27.0, iOS 27.0 simulator), via test-without-building on one build. Per-test counts are first-attempt executions (-test-iterations runs fixed iterations, no retries):

  • 100 warm iterations — 300/300 passed
  • 50 iterations under CPU saturation — 150/150 passed
  • 20 runs, simulator erased before each — 54/54 passed across the 18 runs that executed. The other 2 never started: the test host failed to launch on the just-erased simulator (Simulator device failed to launch org.wordpress), before any test ran — a harness issue, not these tests.
  • 2 full WordPressUnitTests runs on an erased simulator — green, 6/6 video executions passed
  • Earlier probe on iOS 26.4 — 40/40 passed
  • Repeated CI runs — 20 extra builds of this commit (#34682–#34701) plus the PR build: all 21 green, and 0 failed first-attempt executions of the three video tests in Test Engine (63/63 passed; 0.4–0.7s each on CI). The first-attempt failures that did show up were all pre-existing flakes unrelated to this change: PinghubWebSocketTests.testReceiveManyMessage (13/20 runs), BlogJetpackTests.testSyncBlogsAndSignOut (7/20), ReaderTopicSwiftTest.testReaderSiteTopicUpdated (2/20), CommentDetailViewModelTests.actionsWaitForReplyCountAfterDetailRenders (1/20).

The three video export tests were skipped as flaky or slow. Locally they
pass every time, but their 2-3s timeouts have little headroom: a warm run
took up to 6s, and runs under CPU contention took up to 20s.

Raise the timeouts to 60s, which only bounds how long a real failure
takes to report, and compare the exported duration with a tolerance
rather than exact equality, since re-encoding can shift it slightly.
@jkmassel jkmassel added this to the 27.4 milestone Sep 25, 2026
@jkmassel jkmassel added the Testing Unit and UI Tests and Tooling label Sep 25, 2026
@jkmassel jkmassel self-assigned this Sep 25, 2026
@wpmobilebot

wpmobilebot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.
App NameJetpack
ConfigurationRelease-Alpha
Build Number34703
VersionPR #26086
Bundle IDcom.jetpack.alpha
Commitf5a9e85
Installation URL6gfa8gik6hui8
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot

wpmobilebot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.
App NameWordPress
ConfigurationRelease-Alpha
Build Number34703
VersionPR #26086
Bundle IDorg.wordpress.alpha
Commitf5a9e85
Installation URL7cj93t60r3gg8
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@crazytonyli

Copy link
Copy Markdown
Contributor

Maybe we should try using a smaller video, to see if the unit tests can run faster?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Testing Unit and UI Tests and Tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants