Repository navigation
chore: elide 1.6.x compat - #12
Merged
Merged
Conversation
Signed-off-by: Sam Gammon <sam@elide.dev>
- jtreg: pipe javac stderr through the filter instead of a process substitution, so golden-file comparisons never see truncated output - cpython: move the suite to 3.13 (v3.13.14) and stub `_opcode`, which 3.13's test.support imports but GraalPy lacks - wpt: give wptserve's TLS listeners no ports (a failed TLS child shut down every listener), fetch `common/` fixtures, prefer shimmed META scripts over upstream `.sub.js` templates, add HTTPS remote origins
Node's runner keeps .tmp.N under the checkout's test/ directory. The harness put it in a sibling workspace dir, outside the process cwd, where Elide's path-require confinement rejects modules that tests write into their tmpdir and then require.
Two summary/report fixes: 1. The final summary showed REGRESSED or RED even when tests newly passed. The classifier now judges progress on net terms: new passes (vs the expectations) and fixed tests (vs the previous run) are weighed against drift regressions. A run whose floor advanced is IMPROVED when otherwise clean, or ADVANCED (blue) when regressions/unratcheted failures remain, with a headline prompting a ratchet. Drift regressions that match or exceed the gains are still REGRESSED. Markdown indexes mark the same case with a blue dot and the harness log prints ADVANCED instead of RED. 2. Percentages and SVGs excluded skipped/suppressed tests from the denominator, which let muting flatter the numbers. Every view now shows two rates over the whole selection: the overall pass rate (skips count against it) and the rate vs expectations (only regressions count against it, so 100% means at or above the baseline). The per-suite chart shows the full result mix plus a baseline bar; the top chart draws both bars per suite with a legend. Regenerates the top-level index/README/charts and the latest per-run charts, adds the 1.6.0+dd7116a51 run reports, and updates briefings that still described skips as leaving the denominator.
Add a repeatable --filter flag to the launcher (alias --test-filter) so agents can run tests by name across suites, e.g. `--filter 'cpython-core:*time*'` or `--filter 'test262:*Temporal*' --filter 'node-api:*buffer*'`. - Patterns are globs matched case-insensitively against any part of the test's path/id; anchored path globs still act as prefixes. A suite prefix scopes the pattern and selects that suite when --suite is absent; an unprefixed pattern applies to every selected suite. A prefix naming an unselected suite is a usage error. - A filter only narrows the registry/manifest/--include selection. Each adapter logs `filter ...: selected N of M` before running. - harness/src/filter.ts holds parsing, matching and suite selection; the harness CLI passes patterns into AdapterContext.filter. - test262 resolves the file list itself when a filter is active and hands test262-harness explicit paths in chunks, since it only takes anchored, case-sensitive minimatch globs. - cpython-core filters at two levels: module names (or the pattern's leading dotted segment) select modules, then the same patterns go to the regrtest driver as --match-re to prune individual cases. - The Bali launcher accepts --filter and forwards it to jdk-jtreg. - README gains a "Targeted runs" section; each BRIEFING shows an example.
- Raise the per-test/case timeout default from 60s to 65s across the cpython, node-api, test262 and wpt adapters. - Drop manifest entries for stdlib modules PEP 594 removed in 3.13, and follow test_unicode's rename to test_str.
Skips still count against the pass rate; summarize the skip reasons in node-api, wpt-wintertc, and cpython-core expectations so it's clear they are structurally unrunnable tests, not hidden Elide failures.
…ain ratchet wording - "411 new passs": the plural helper appended "s" blindly; words ending in "s" now take "es". - ADVANCED runs (regressions alongside new passes) are marked ⬆️ in the launcher summary and the rendered reports; GAINED keeps 🔵. - Replace "ratchet to lock in the gain" / "lock it in" with "run with --ratchet to update the baseline" everywhere it appeared.
registry.toml pinned wpt, cpython (per case) and node-api to 60s, which overrode the adapters' new 65s defaults.
Latest run for cpython-core, node-api and wpt-wintertc; refreshes the top-level report index to point at it.
Charts now use a single-hue purple ramp where lightness carries the meaning (pass darkest, then fail, error, skip lightest) and every bar keeps a text label, so nothing depends on red/green hue. The top chart legend gains an "overall, with regressions" swatch. The README shrinks to the intro, the generated compatibility block, a short how-it-works, a quick start and links. Operational detail moves to docs/running.md (flags, --filter, concurrency, Bali), docs/reports.md (report layout, why tests are skipped, ratchet, analysis CLI) and docs/contributing.md (development, layout). Briefings now point at the running doc for targeted runs.
Ratcheting should run against the reports on the base branch; the 1.6.0+dd7116a51 runs recorded here were exploratory. Working-tree copies are left in place (untracked) for the run in progress.
A case timeout or an interpreter crash killed the shard's driver and silently dropped every module after it in the shard: whole manifest groups reported no results. The driver now ends with a completion sentinel (it exits 1 on any failing test, so the exit code cannot tell a crash apart), the case a crash takes down is reported, and the shard relaunches on the modules after the one it died in.
CPython's suite starts threads and child interpreters throughout; a sandbox
that refuses them crashes the interpreter mid-module. Network stays on
loopback because test.support.requires('network') lets everything through
outside regrtest.
…er launch - --timeout-scale N (TIMEOUT_SCALE) multiplies registry timeoutMs and caseTimeoutMs; dev builds run far slower than the release builds the limits are sized for. - Each cpython driver launch gets an empty cwd: CPython's TESTFN embeds the pid, container pids repeat, and a killed or resumed shard left files that broke the next test with the same name.
- The cpython driver leaves via os._exit after its completion sentinel, so a non-daemon thread a test left behind cannot hold the shard open. - The adapter always reads the driver's phase lines and treats silence between cases (module setup/teardown, a hung import) like a stuck case, instead of waiting out the whole shard budget. - test262's per-test limit is the registry caseTimeoutMs, so it scales. - Adapter test fakes print the completion sentinel the driver now emits.
Signed-off-by: Sam Gammon <sam@elide.dev>
bin/run.ts strips a suite prefix before invoking the harness; cli.ts now applies the same parsing so host runs with --filter 'node-api:*x*' select tests.
Each run rebuilds its overlay under the work dir, so two runs sharing .harness/work clobber each other's tests.
The final summary table could run to ~200 columns, mostly from the Changes column. It now fits the terminal: the full table when it fits, the Changes column wrapped when that suffices, and a stacked block per suite on narrow screens. Width comes from the TTY, then $COLUMNS; non-TTY output (CI logs, pipes) keeps the full table. Layout lives in bin/summary-table.ts so it is unit-testable. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Elide will not expose Node's internal modules.
Each case past the 65 s driver limit ended its whole module (test_socket, test_shutil, test_zipfile64, ...); skip them per case so the rest of the module runs. The stray SIGVTALRM/SIGPROF itimer tests are gated because their signal kills a later module's driver.
New expectation classes: `[exclude]` (globs) and `[exclude-skipped]`
(patterns over a runtime skip message). Matching tests are counted as
`excluded` and leave both the numerator and the denominator; a failure is
never excluded by message.
cpython-core excludes skips CPython itself takes on linux-x64 (implementation
details, C accelerators, C-API test modules, other platforms; 2,062 in the
last run). node-api skips tests that need Node internals (`// Flags:
--expose-internals` or `require('internal/…')`) with a marker message and
excludes them.
The runner now runs each `// META: variant=` as its own process with that
`location.search`, as wptrunner does. Before, all variants ran in one
process and the XHR variant's timeout left the portable TextDecoder
subtests of encoding/single-byte-decoder NOTRUN (reported as skips).
Browser-only files (document, iframes, navigation, media elements,
fetchLater) move from [skip] to [exclude], and `[exclude-ids]` excludes the
DOM-only subtests of mixed files (url/toascii's <a>/<area> cases,
single-byte-decoder's XHR/document variants), so their portable subtests
run and count. fetch/content-length/{parsing,too-long}.window.js use only
fetch() and are no longer skipped.
…verlay keeps real umask/cwd test_dis, test_peepholer, test_capi, test_dict_version and test__opcode test CPython's bytecode, C API and internals, not the language or stdlib. The node-api overlay stubbed process.umask/cwd unconditionally, hiding Elide's real implementations from test-process-umask*/chdir; the stubs now install only when the runtime lacks them.
The suite checkout is mounted read-only, and CPython's tests write beside their sources: import_helper.forget unlinks a .pyc in every sys.path entry, which a read-only mount refuses with EROFS even when none exists (test_bdb, test_runpy, test_traceback, test_importlib, ...). Lib and the driver are copied into the workspace; other top-level entries are linked.
…odule The shim reported isMainThread: true in every thread, so workers took main-thread paths (common's umask setup threw 'not supported in workers') and ~30 worker tests failed. It now re-exports node:worker_threads when the runtime provides it.
… shard multiprocessing children and resource trackers could outlive a killed or exited driver while holding its stdout, so the shard waited forever for EOF (a full run stalled for 25 minutes). The driver now runs in its own process group, killed on timeout and as soon as the driver exits.
A cpython-core run exhausted the WSL VM's memory (global OOM killed the user session): multiprocessing tests now start a full runtime (~0.9 GiB) per pool worker, one per CPU, across up to 64 concurrent drivers. The container gets a memory ceiling (70% of host memory, or HARNESS_MEMORY) so an OOM stays inside it, and cpython-core runs at most 12 drivers.
…evious one test_unittest's TestBreak (and test_signal) send the process SIGINT; the runtime can run the handler late, so the KeyboardInterrupt landed while the next modules imported and dropped them whole (test_ipaddress, test_posixpath, test_strftime: ~300 passes). The driver restores the default SIGINT handler per module and retries once when the interrupt arrives before the module's tests start.
cpython-core: unskip file-I/O and subprocess cases that now run; exclude more C-API test classes, refcount-finalization tests, and skips for packages the harness does not install. node-api: unskip node:vfs tests, which now pass.
A tracing GC leaves dropped files open until it collects; test_bz2's testOpenDel drops 10k, pushing later subprocess pipes past select()'s FD_SETSIZE. Collect after a test when more than 256 fds are open.
Serve the document origin as localhost so get-host-info's remote host is 127.0.0.1 on the same server (www1.127.0.0.1 is not a valid URL host, so every cross-origin test silently ran same-origin). Map www*.localhost to loopback in the container, and run at most 8 server-backed files at once: a saturated wptserve dropped connections (the integrity 'HTTP body write failed' flake) and pushed cors files past the bridge timeout.
Server runtimes (Node, Deno) do not enforce CORS; with real cross-origin hosts those tests fail instead of passing by accident. Exclude preflight, server-forbidden, origin-KO and CORP-same tests as 'unsupported:'. The suite report now lists exclusions by reason, and changes.md no longer counts an out-of-scope test as regressed or fixed.
Only what never applies to Elide is 'unsupported' (Node internals, DOM, CORS, CPython bytecode/internals, refcount GC timing); CPython C API and accelerator surfaces are 'out of scope', platform and harness gaps are 'not applicable'. Move five CPython implementation-detail skips to exclude.
Every suite at 100% vs expectations with no regressions: cpython-core 90.0%, node-api 81.8%, wpt-wintertc 95.1%, javac-jtreg 93.0%, test262 93.0%.
Signed-off-by: Sam Gammon <sam@elide.dev>
Signed-off-by: Sam Gammon <sam@elide.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upcoming