diff --git a/CHANGELOG.md b/CHANGELOG.md index e38188c721..0387fee1c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,14 @@ file is the human-readable summary of what changed and why. ### Core +- **ripgrep downloads work without GitHub access** — when `rg` is not + installed, CoDev Code fetches it on first use from GitHub releases, which + corporate networks commonly block. The download now falls back to the CoDev + site's own mirror of the official binaries, and each attempt is bounded to + 30 seconds so a proxy that swallows traffic can't stall first-time file + search for minutes. GitHub remains the primary source; nothing changes on + networks where it is reachable. + - **`@` file mentions work on Windows** — on Windows the `@` popup is powered by a ripgrep-based fallback (the native fuzzy finder is disabled there), and that fallback could end up permanently empty with no error shown: if diff --git a/DIVERGENCES.md b/DIVERGENCES.md index deab0678de..3866b37cfc 100644 --- a/DIVERGENCES.md +++ b/DIVERGENCES.md @@ -167,7 +167,7 @@ from the squash message first, or use a plain merge. | `packages/tui/src/component/prompt/autocomplete.tsx`, `packages/app/src/components/prompt-input.tsx`, `packages/opencode/src/cli/cmd/run/footer.prompt.tsx`, `packages/core/src/filesystem/search.ts`, `packages/tui/test/fixture/tui-sdk.ts`, `packages/opencode/test/cli/run/footer.view.test.tsx`, `packages/core/test/filesystem/search.test.ts` | `@` mention autocomplete lists file/directory results before references/agents/MCP resources; TUI and run-CLI menus cap the leading files so up to 3 non-file matches stay in the first visible page; web app group rank reordered to recent → file → reference → agent → resource (extracted into fork-only `prompt-input/at-options.ts`). TUI and run-CLI file resources track menu visibility so opening on an empty query fetches (previously nothing refetched until the query changed); core's ripgrep-fallback `find` returns its first entries for an empty query to mirror fff. Tests: tui-sdk fixture's `FetchHandler` gains an optional `method` arg, `renderFooter` accepts `findFiles`/`agents` overrides + a mention-menu ordering test, core search tests cover the fallback's empty query (fork-only test files: `packages/tui/test/prompt-autocomplete.test.tsx`, `packages/app/test-browser/prompt-at-options.test.ts`, `packages/app/src/components/prompt-input/at-options{,.test}.ts`) | Files are the dominant `@` use case; they should rank first and appear on a bare `@`. | Keep our ordering/reserve logic, resource source tracking, empty-query branch, fixture/test additions; take upstream's surrounding changes. | | `packages/slack/.gitignore`, `sdks/vscode/.gitignore` | Added the missing trailing newline (content unchanged) | Upstream's copies end without a newline; POSIX text files should end with one, and tools that concatenate or append to these files misbehave without it. | Whitespace-only: keep the trailing newline; take upstream's entries. | | `packages/tui/src/config/keybind.ts`, `packages/tui/src/component/prompt/index.tsx`, `packages/tui/src/app.tsx` | Permission-mode indicator + shift+tab cycle. keybind.ts: added `permission_mode_cycle: keybind("shift+tab", ...)` + its `permission.mode` CommandMap entry; `agent_cycle_reverse` default `"shift+tab"` → `"none"` (tab still cycles agents forward; previous-agent stays rebindable). prompt/index.tsx: the footer badge next to the agent name always renders in prompt mode — gray `manual` (`theme.textMuted`) in normal mode, red `auto` (`theme.error`) in auto mode (upstream only showed gray `auto`); the idle footer's shortcut hints gain a `shift+tab modes` text (via a `modeShortcut` `useCommandShortcut("permission.mode")` memo) left of `tab agents`, in the same no-usage `Match` arm. app.tsx: `"permission.mode"` added to `appBindingCommands` so the binding is gathered. | The permission mode is always visible (manual/auto) with auto in red as a danger signal, and shift+tab toggles it without opening the palette. | Keep the `permission_mode_cycle` definition + mapping, the `agent_cycle_reverse` `"none"` default, the badge `Show` block, the `modeShortcut` memo + hint text, and the `appBindingCommands` entry; take upstream's surrounding changes. If upstream reworks the auto badge or rebinds shift+tab, re-apply on top and re-audit the shift+tab conflict. | -| `packages/core/src/filesystem/search.ts`, `packages/core/src/ripgrep/binary.ts`, `packages/core/src/filesystem.ts`, `packages/core/src/tool/read-filesystem.ts`, `packages/core/test/filesystem/search.test.ts`, `packages/core/test/location-filesystem.test.ts` | Windows file-search fallback robustness. search.ts: the background ripgrep index walk no longer dies silently (`Effect.orDie` in a forked fiber) — each attempt resets the accumulated file/directory state, failures log a warning ("file index scan failed, file search may be empty"), and the walk retries forever on a 1s exponential backoff capped at 30s spacing, so installing rg mid-session self-heals; directory entries are suffixed and detected with `"/"` instead of `path.sep` (both ripgrep and fff layers), and fallback `find` queries normalize `\` to `/` and collapse doubled slashes (directory tab-expansion appends `/` to an already-`/`-suffixed entry). binary.ts: `Effect.cached` on ripgrep binary resolution (which memoizes failure exits for the process lifetime) replaced with a success-only cache behind a semaphore, so a failed rg download (offline, blocked proxy) retries on the next use instead of bricking search until restart. filesystem.ts `list` + read-filesystem.ts `list`: directory suffix `path.sep` → `"/"`; `fs.list` also `/`-normalizes `path.relative` output. Tests: search.test.ts pins `"/"`-suffixed directories, adds query-normalization coverage, and a stub-Ripgrep scan-retry test asserting the partial first walk is discarded; location-filesystem.test.ts pins `"src/"`. | On Windows fff is disabled by default (upstream `e4300e9b74`), so the `@` mention popup and file search depend entirely on the ripgrep fallback; a missing `rg.exe` plus one failed download left the index permanently empty with no user-visible signal (the CI-workflows row's choco preinstall papered over the same swallowed failure). `\`-suffixed directory entries also broke `endsWith("/")` directory-mime detection in the run CLI and poisoned drill-down queries with backslashes that can never match the `/`-normalized index. | Keep our retry/reset scan, success-only binary cache, `"/"` suffixes, and query normalization; take upstream's surrounding changes. If upstream reworks the fallback or re-enables fff on win32, preserve the invariants: entry paths are `/`-normalized with `/`-suffixed directories on every platform, index-scan failures are logged and retried, and a failed binary download is never cached. | +| `packages/core/src/filesystem/search.ts`, `packages/core/src/ripgrep/binary.ts`, `packages/core/src/filesystem.ts`, `packages/core/src/tool/read-filesystem.ts`, `packages/core/test/filesystem/search.test.ts`, `packages/core/test/location-filesystem.test.ts`, `packages/core/test/ripgrep-binary.test.ts`, `packages/core/test/preload.ts` | Windows file-search fallback robustness. search.ts: the background ripgrep index walk no longer dies silently (`Effect.orDie` in a forked fiber) — each attempt resets the accumulated file/directory state, failures log a warning ("file index scan failed, file search may be empty"), and the walk retries forever on a 1s exponential backoff capped at 30s spacing, so installing rg mid-session self-heals; directory entries are suffixed and detected with `"/"` instead of `path.sep` (both ripgrep and fff layers), and fallback `find` queries normalize `\` to `/` and collapse doubled slashes (directory tab-expansion appends `/` to an already-`/`-suffixed entry). binary.ts: `Effect.cached` on ripgrep binary resolution (which memoizes failure exits for the process lifetime) replaced with a success-only cache behind a semaphore, so a failed rg download (offline, blocked proxy) retries on the next use instead of bricking search until restart; the download tail tries GitHub first (each attempt bounded to 30s) and falls back to the CoDev landing-page mirror (`https://netmind.viettel.vn/codev/docs/code/downloads/rg---[.exe]`, a raw executable — no archive/extract — staged in a scoped temp dir then copied into place), via `Effect.catchCause` so a proxy block page (200 + HTML) that dies in `extract` as a defect still reaches the mirror. filesystem.ts `list` + read-filesystem.ts `list`: directory suffix `path.sep` → `"/"`; `fs.list` also `/`-normalizes `path.relative` output. Tests: search.test.ts pins `"/"`-suffixed directories, adds query-normalization coverage, and a stub-Ripgrep scan-retry test asserting the partial first walk is discarded; location-filesystem.test.ts pins `"src/"`; ripgrep-binary.test.ts adds mocked-HTTP coverage of the fallback order, success caching, and failure non-caching; preload.ts redirects `XDG_CACHE_HOME` to a fresh temp dir so no test can read or clobber the developer's real `~/.cache/codev`. | On Windows fff is disabled by default (upstream `e4300e9b74`), so the `@` mention popup and file search depend entirely on the ripgrep fallback; a missing `rg.exe` plus one failed download left the index permanently empty with no user-visible signal (the CI-workflows row's choco preinstall papered over the same swallowed failure). `\`-suffixed directory entries also broke `endsWith("/")` directory-mime detection in the run CLI and poisoned drill-down queries with backslashes that can never match the `/`-normalized index. | Keep our retry/reset scan, success-only binary cache, `"/"` suffixes, and query normalization; take upstream's surrounding changes. If upstream reworks the fallback or re-enables fff on win32, preserve the invariants: entry paths are `/`-normalized with `/`-suffixed directories on every platform, index-scan failures are logged and retried, a failed binary download is never cached, and the GitHub download keeps its CoDev-mirror fallback — the mirror filename scheme must match what codev-landing-page hosts under `public/docs/code/downloads` (and codevhub's `RG_VERSION` pin in its `src/lib/ripgrep.ts`); bump all three together when changing `VERSION`. | | `packages/tui/src/context/permission.tsx`, `packages/tui/src/app.tsx`, `packages/tui/src/ui/dialog-confirm.tsx`, `packages/tui/src/ui/toast.tsx` | Auto-approve consent gate. permission.tsx: mode always starts `"normal"` (upstream initialized it from `args.auto`); `toggle()` replaced by `set(mode)` plus a per-run `consented` flag + `consent()`. app.tsx: shared `enableAutoApprove()` (title in the `AUTO_APPROVE_TITLE` module const, body from an `autoApproveMessage` accessor, and a shorter one-line `autoApproveToast` accessor for the toast, which only has a few seconds on screen; both redden the word "dangerous" with `theme.error` and are passed unevaluated because renderables can only be created during render) shows a `DialogConfirm` opened on cancel and, on confirm, a warning toast under the same title, then sets the mode; both entry points funnel through it — an `onMount` when `args.auto` is set, and the `permission.mode` command (shift+tab / palette), which enables via the gate and disables directly. Consent is asked once per run; later toggles are silent. dialog-confirm.tsx: optional `defaultChoice` prop + 5th `DialogConfirm.show` arg selecting the initially-active button (omitted keeps upstream's confirm-first default); `message` widened to `JSX.Element \| (() => JSX.Element)` and called at the render site. toast.tsx: same `message` widening, resolved by a `message()` helper in `Toast()` (fork-only test file: `packages/tui/test/auto-approve-consent.view.test.tsx`). | Auto-approve is dangerous, so `--auto` (and its hidden aliases) and the first shift+tab into auto require explicit confirmation; declining or dismissing the dialog stays in manual mode, opening on cancel keeps a stray return from granting it, and the risk word is red like the `auto` badge. | Keep the consent flag + gate flow, the `toggle()` removal, the `defaultChoice` prop, and the accessor-aware `message` on both dialog and toast; take upstream's surrounding changes. If upstream adds new writers of the permission mode, route their enable path through `enableAutoApprove()`. | ## Known rebrand gaps diff --git a/packages/core/src/ripgrep/binary.ts b/packages/core/src/ripgrep/binary.ts index b81c24113f..1b60c9bdaf 100644 --- a/packages/core/src/ripgrep/binary.ts +++ b/packages/core/src/ripgrep/binary.ts @@ -12,6 +12,13 @@ import { which } from "../util/which" export namespace RipgrepBinary { const VERSION = "15.1.0" + // Fallback for networks where GitHub is blocked: the CoDev landing page + // mirrors the official executables for this VERSION (raw, not archived) at + // rg---[.exe]. Bumping VERSION requires hosting the + // new binaries there first; codevhub pins the same version in lib/ripgrep.ts. + // Base64 (like codev-gateway's const.ts) to keep the host out of casual + // greps, not as a secret. + const MIRROR = atob("aHR0cHM6Ly9uZXRtaW5kLnZpZXR0ZWwudm4vY29kZXYvZG9jcy9jb2RlL2Rvd25sb2Fkcw==") const PLATFORM = { "arm64-darwin": { platform: "aarch64-apple-darwin", extension: "tar.gz" }, "arm64-linux": { platform: "aarch64-unknown-linux-gnu", extension: "tar.gz" }, @@ -93,6 +100,21 @@ export namespace RipgrepBinary { // broken until restart. Cache successful resolution only. const resolved = { filepath: undefined as string | undefined } const semaphore = Semaphore.makeUnsafe(1) + const download = Effect.fnUntraced(function* (url: string) { + yield* Effect.logInfo("downloading ripgrep", { url }) + const bytes = yield* HttpClientRequest.get(url).pipe( + http.execute, + Effect.flatMap((response) => response.arrayBuffer), + // Bounded per attempt: a proxy that blackholes instead of refusing + // would otherwise sit on OS-level TCP timeouts for minutes before + // the fallback source is even tried. + Effect.timeout("30 seconds"), + Effect.mapError((cause) => (cause instanceof Error ? cause : new Error(String(cause)))), + ) + if (bytes.byteLength === 0) return yield* Effect.fail(new Error(`failed to download ripgrep from ${url}`)) + return new Uint8Array(bytes) + }) + const resolve = Effect.gen(function* () { const system = yield* Effect.sync(() => which(process.platform === "win32" ? "rg.exe" : "rg")) if (system && (yield* fs.isFile(system).pipe(Effect.orDie))) return system @@ -104,22 +126,43 @@ export namespace RipgrepBinary { const config = PLATFORM[platformKey] if (!config) throw new Error(`unsupported platform for ripgrep: ${platformKey}`) - const filename = `ripgrep-${VERSION}-${config.platform}.${config.extension}` - const url = `https://github.com/BurntSushi/ripgrep/releases/download/${VERSION}/${filename}` - const archive = path.join(Global.Path.bin, filename) - - yield* Effect.logInfo("downloading ripgrep", { url }) yield* fs.ensureDir(Global.Path.bin).pipe(Effect.orDie) - const bytes = yield* HttpClientRequest.get(url).pipe( - http.execute, - Effect.flatMap((response) => response.arrayBuffer), - Effect.mapError((cause) => (cause instanceof Error ? cause : new Error(String(cause)))), - ) - if (bytes.byteLength === 0) throw new Error(`failed to download ripgrep from ${url}`) - yield* fs.writeWithDirs(archive, new Uint8Array(bytes)) - yield* extract(archive, config, target) - yield* fs.remove(archive, { force: true }).pipe(Effect.ignore) + const fromGithub = Effect.gen(function* () { + const filename = `ripgrep-${VERSION}-${config.platform}.${config.extension}` + const archive = path.join(Global.Path.bin, filename) + const bytes = yield* download(`https://github.com/BurntSushi/ripgrep/releases/download/${VERSION}/${filename}`) + yield* fs.writeWithDirs(archive, bytes) + yield* extract(archive, config, target) + yield* fs.remove(archive, { force: true }).pipe(Effect.ignore) + }) + + // The mirror hosts the raw executable (no archive), named with node's + // platform/arch vocabulary — see codev-landing-page + // public/docs/code/downloads. Staged into a scoped temp dir first: the + // cache check above is bare existence, so partial bytes written to + // `target` directly would permanently satisfy it. + const fromMirror = Effect.gen(function* () { + const bytes = yield* download( + `${MIRROR}/rg-${VERSION}-${process.platform}-${process.arch}${process.platform === "win32" ? ".exe" : ""}`, + ) + const dir = yield* fs.makeTempDirectoryScoped({ directory: Global.Path.bin, prefix: "ripgrep-" }) + const staged = path.join(dir, path.basename(target)) + yield* fs.writeWithDirs(staged, bytes) + yield* fs.copyFile(staged, target) + if (process.platform !== "win32") yield* fs.chmod(target, 0o755) + }).pipe(Effect.scoped) + + // catchCause, not catch: a proxy serving a block page returns 200 with + // HTML, which passes filterStatusOk and dies in extract as a defect — + // that case must reach the mirror too. + yield* fromGithub.pipe( + Effect.catchCause((cause) => + Effect.logWarning("github ripgrep download failed, trying the CoDev mirror", { cause }).pipe( + Effect.flatMap(() => fromMirror), + ), + ), + ) return target }) return Service.of({ diff --git a/packages/core/test/preload.ts b/packages/core/test/preload.ts index 9ff0a6e455..977289fbb7 100644 --- a/packages/core/test/preload.ts +++ b/packages/core/test/preload.ts @@ -1,5 +1,13 @@ import path from "path" +import fs from "fs" +import os from "os" process.env.CODEV_DB = ":memory:" process.env.CODEV_MODELS_PATH = path.join(import.meta.dir, "plugin", "fixtures", "models-dev.json") process.env.CODEV_DISABLE_MODELS_FETCH = "true" +// Redirect the XDG cache (Global.Path.cache / Global.Path.bin) into a fresh +// temp dir so tests never read or clobber the developer's real ~/.cache/codev +// — ripgrep-binary.test.ts writes fake bytes at the rg target path. Must +// happen here: xdg-basedir snapshots env when first imported, and bun runs +// the whole suite in one process. +process.env.XDG_CACHE_HOME = fs.mkdtempSync(path.join(os.tmpdir(), "codev-test-cache-")) diff --git a/packages/core/test/ripgrep-binary.test.ts b/packages/core/test/ripgrep-binary.test.ts new file mode 100644 index 0000000000..699ed795e3 --- /dev/null +++ b/packages/core/test/ripgrep-binary.test.ts @@ -0,0 +1,144 @@ +import { afterEach, beforeEach, describe, expect } from "bun:test" +import { Effect, Exit, Layer, Ref } from "effect" +import { HttpClient, HttpClientResponse } from "effect/unstable/http" +import { AppNodeBuilder } from "@opencode-ai/core/effect/app-node-builder" +import { LayerNodePlatform } from "@opencode-ai/core/effect/app-node-platform" +import { Global } from "@opencode-ai/core/global" +import { RipgrepBinary } from "@opencode-ai/core/ripgrep/binary" +import { it } from "./lib/effect" +import { readdir, readFile, rm, stat } from "fs/promises" +import path from "path" + +// Pins the mirror filename contract: these names must match what +// codev-landing-page hosts under public/docs/code/downloads. +const MIRROR_URL = `https://netmind.viettel.vn/codev/docs/code/downloads/rg-15.1.0-${process.platform}-${process.arch}${process.platform === "win32" ? ".exe" : ""}` +const GITHUB_PREFIX = "https://github.com/BurntSushi/ripgrep/releases/download/15.1.0/" + +const target = path.join(Global.Path.bin, process.platform === "win32" ? "rg.exe" : "rg") +const binaryBytes = new TextEncoder().encode("fake-rg-binary") + +interface MockState { + github: { status: number; body?: Uint8Array } + mirror: { status: number; body?: Uint8Array } + calls: string[] +} + +const makeMockClient = (state: Ref.Ref) => + HttpClient.make((request) => + Effect.gen(function* () { + yield* Ref.update(state, (s) => ({ ...s, calls: [...s.calls, request.url] })) + const s = yield* Ref.get(state) + const route = request.url.startsWith(GITHUB_PREFIX) ? s.github : s.mirror + return HttpClientResponse.fromWeb( + request, + new Response(route.body ?? (route.status === 200 ? new Uint8Array() : "error"), { status: route.status }), + ) + }), + ) + +// Layer.fresh: RipgrepBinary.layer is a module-level constant holding the +// success-only resolution cache and its semaphore — without fresh, every test +// would see the first test's cached filepath through the global layer MemoMap. +const buildLayer = (state: Ref.Ref) => + Layer.fresh( + AppNodeBuilder.build(RipgrepBinary.node, [ + [LayerNodePlatform.httpClient, Layer.succeed(HttpClient.HttpClient, makeMockClient(state))], + ]), + ) + +const targetExists = Effect.promise(() => + stat(target).then( + () => true, + () => false, + ), +) + +// Resolution prefers a PATH rg; blank it so the download path always runs. +// which() appends Global.Path.bin (redirected to a temp dir by preload.ts) to +// its search path, so target cleanup in beforeEach keeps that empty too. +const env = { PATH: process.env.PATH, Path: process.env.Path } +beforeEach(async () => { + process.env.PATH = "" + delete process.env.Path + await rm(target, { force: true }) +}) +afterEach(async () => { + if (env.PATH === undefined) delete process.env.PATH + else process.env.PATH = env.PATH + if (env.Path === undefined) delete process.env.Path + else process.env.Path = env.Path + // The fake binary must not outlive this file: later suites (ripgrep.test.ts) + // resolve the real rg through the same Global.Path.bin. + await rm(target, { force: true }) +}) + +describe("RipgrepBinary mirror fallback", () => { + it.live("falls back to the CoDev mirror when the GitHub download fails", () => + Effect.gen(function* () { + const state = yield* Ref.make({ + github: { status: 500 }, + mirror: { status: 200, body: binaryBytes }, + calls: [], + }) + const resolved = yield* Effect.gen(function* () { + const binary = yield* RipgrepBinary.Service + return yield* binary.filepath + }).pipe(Effect.provide(buildLayer(state))) + expect(resolved).toBe(target) + const written = yield* Effect.promise(() => readFile(target)) + expect(new Uint8Array(written)).toEqual(binaryBytes) + const final = yield* Ref.get(state) + expect(final.calls).toHaveLength(2) + expect(final.calls[0]?.startsWith(GITHUB_PREFIX)).toBe(true) + expect(final.calls[1]).toBe(MIRROR_URL) + // The scoped staging temp dir must be gone; only the executable remains. + expect(yield* Effect.promise(() => readdir(Global.Path.bin))).toEqual([path.basename(target)]) + if (process.platform !== "win32") { + const mode = yield* Effect.promise(() => stat(target).then((s) => s.mode)) + expect(mode & 0o111).not.toBe(0) + } + }), + ) + + it.live("caches success: a second resolution makes no further requests", () => + Effect.gen(function* () { + const state = yield* Ref.make({ + github: { status: 500 }, + mirror: { status: 200, body: binaryBytes }, + calls: [], + }) + yield* Effect.gen(function* () { + const binary = yield* RipgrepBinary.Service + expect(yield* binary.filepath).toBe(target) + expect(yield* binary.filepath).toBe(target) + }).pipe(Effect.provide(buildLayer(state))) + const final = yield* Ref.get(state) + expect(final.calls).toHaveLength(2) + }), + ) + + it.live("does not cache failure, and an empty mirror body never creates the target", () => + Effect.gen(function* () { + // Empty 200 from the mirror: the degenerate corporate-proxy response. + // It must fail resolution rather than write a zero-byte rg that the + // bare-existence cache check would trust forever. + const state = yield* Ref.make({ + github: { status: 500 }, + mirror: { status: 200, body: new Uint8Array() }, + calls: [], + }) + yield* Effect.gen(function* () { + const binary = yield* RipgrepBinary.Service + const first = yield* binary.filepath.pipe(Effect.exit) + expect(Exit.isFailure(first)).toBe(true) + expect(yield* targetExists).toBe(false) + // The failure must not be memoized: with the mirror healthy again the + // same service instance recovers without a restart. + yield* Ref.update(state, (s) => ({ ...s, mirror: { status: 200, body: binaryBytes } })) + expect(yield* binary.filepath).toBe(target) + }).pipe(Effect.provide(buildLayer(state))) + const final = yield* Ref.get(state) + expect(final.calls).toHaveLength(4) + }), + ) +})