diff --git a/docs/avatar-contract.md b/docs/avatar-contract.md index 33212244..ab3d7c5c 100644 --- a/docs/avatar-contract.md +++ b/docs/avatar-contract.md @@ -318,22 +318,52 @@ in the cost column for a borrowed mascot. ### Captions -`#captions-bar` lives inside `#jim-stage`, under the avatar. It sizes to the -turn rather than clamping: a fixed `max-height` cut sentences in half behind a -scrollbar nobody reaches for mid-interview. `CAPTION_MAX_CHARS` already bounds -the text, so the remaining `min(40vh, 20rem)` cap is a backstop, not the normal -case. - -It is a live subtitle, not a log. The transcript tab keeps every turn, so the -bar hides after `CAPTION_IDLE_HIDE_MS` (12 s) with nobody speaking and returns -on the next word. `init` arms the timer for the "Listening..." placeholder too, -or it would sit over the editor for the whole interview having said nothing. +`#captions-bar` lives inside `#jim-stage`, under the avatar. It grows to fit +its text, with `max-height: min(40vh, 20rem)` and scrolling as a backstop. + +A completed Jim turn is displayed in full, without `CAPTION_MAX_CHARS` +truncation or an idle timeout. It stays until the candidate's next transcribed +speech or an edit that changes the code text. Speech replaces it with the +candidate's caption; a code text change hides the bar. Clicking or focusing +the editor and switching languages preserve Jim's caption. A new Jim turn +replaces the previous one and starts at the top of the caption bar. + +Keeping the last turn lets the candidate reread a hint or follow-up question +while thinking, without switching away from the problem. The Transcript tab +still keeps every turn. + +Jim captions reveal gradually, even after final text arrives, because +generation can finish while audio is still queued. An edit during the reveal +is remembered for that Jim segment: its final text finishes revealing, then +the bar hides without another edit. While revealing, Jim captions keep the +`CAPTION_MAX_CHARS` window and `CAPTION_IDLE_HIDE_MS` timeout, refreshed by +pacing updates. Candidate captions keep their 160-character window and 12 s +timeout. The "Listening..." placeholder also keeps its 12 s timeout. + +A stream with `lk.transcription_final` set to `"true"` must be read +successfully to its end before its turn can be retained. Once the reveal +reaches the end, the full turn stays without an idle timeout. A broken text +stream leaves partial captions temporary. The reveal uses a fixed rate; +audio playout timing is independent. Updates for already dismissed or +replaced Jim segment IDs are ignored. Final copies of earlier candidate turns +do not dismiss Jim's current turn. + +A longer interim update to the same candidate segment can pause Jim's +interim reveal without retiring it. Jim's next update can resume the reveal. +A new candidate segment or a candidate response after Jim's final retires +Jim's segment. ## Checks -`node --test tests/browser/avatar.test.js` covers the behavior with a fake -clock, a fake mount, and a fake model. It cannot see WebGL, a canvas, or a -layout, and no assertion in it should claim to. +`node --test tests/browser/avatar.test.js tests/browser/captions.test.js` +covers avatar behavior with a fake clock, a fake mount, and a fake model. +Caption tests use a fake clock and actual page functions to cover retention, +dismissal, final-stream handling, and editor input. These checks cannot see +WebGL, a canvas, or a layout, and no assertion in them should claim to. + +`node --test tests/browser/editor-pipeline.test.js` runs Chromium against the +interview page. It checks caption retention during editor focus, caret +movement, and language switches, plus scroll reset between Jim turns. `BROWSER_CHECK_FLOW=avatar ./scripts/browser-check.sh` is the only check that runs the renderer. It asserts the mount leaves `loading` and then asserts diff --git a/tests/browser/avatar.test.js b/tests/browser/avatar.test.js index 1f54fc90..ab95dacb 100644 --- a/tests/browser/avatar.test.js +++ b/tests/browser/avatar.test.js @@ -720,59 +720,23 @@ test("avatar blink and breath stay inside their envelope", () => { assert.ok(Math.abs(breathOffset(1234)) <= BREATH_AMPLITUDE); }); -// What the executable tests cannot see: which track the analyser is pointed at. -// Getting this wrong is not a rendering bug, it is the avatar watching the -// candidate, so it is worth a text assertion. -// Captions are a live subtitle under Jim's face, not a log; the transcript tab -// keeps every turn. Two properties, both invisible to a node test at runtime, -// so they are pinned where they are decided. -test("captions size to the turn and retire when nobody is speaking", () => { +// Caption timing is covered by executable tests in captions.test.js. +// These source assertions pin CSS rules; Node does not render the layout. +test("caption CSS keeps a viewport cap and scrolling", () => { const css = read("web/styles.css"); - const bar = css.slice(css.indexOf(".jim-stage .captions-bar")); + const start = css.indexOf(".jim-stage .captions-bar"); + assert.ok(start >= 0, "the caption sizing rule must exist"); + const bar = css.slice(start); const rule = bar.slice(0, bar.indexOf("}")); - // A fixed max-height cut sentences in half behind a scrollbar. The cap that - // remains is viewport-relative, a backstop rather than the normal case. + assert.doesNotMatch( rule, /max-height:\s*\d+(\.\d+)?rem;/, - "a fixed max-height truncates the turn instead of fitting it", + "captions must retain a viewport-relative height cap", ); assert.match(rule, /max-height:\s*min\(40vh/); - - // Shown on every new caption, hidden after an idle stretch, and the - // placeholder is armed too or it would sit there for the whole interview. - assert.match(script, /const CAPTION_IDLE_HIDE_MS = \d+;/); - const show = functionBody(script, "showCaptions"); - assert.match(show, /nodes\.captionsBar\.hidden = false;/); - assert.match( - show, - /clearTimeout\(captionIdleTimer\)/, - "a new turn must reset the countdown, not stack timers", - ); - assert.match( - show, - /setTimeout\([\s\S]*?nodes\.captionsBar\.hidden = true;[\s\S]*?CAPTION_IDLE_HIDE_MS\)/, - ); - assert.match( - script, - /nodes\.captionsText\.textContent = `\[\$\{speaker[^`]*`;\n\s*showCaptions\(\);/, - "every caption update must un-hide the bar", - ); - - // The reveal has to keep asking for the next tick while there is more of the - // turn to show. Dropping the re-arm leaves Jim's line frozen part-way through - // a sentence, which reads as a stall rather than as a bug. - const pace = functionBody(script, "paceInterviewerCaption"); - assert.match( - pace, - /caption\.timer = setTimeout\(paceInterviewerCaption, CAPTION_TICK_MS\);/, - "an unfinished turn must schedule its next tick", - ); - assert.match( - pace, - /caption\.shown < caption\.text\.length/, - "and stop scheduling once the whole turn is on screen", - ); + assert.match(rule, /overflow-y:\s*auto;/); + assert.match(rule, /pointer-events:\s*auto;/); }); // A node test has no layout engine, so it cannot see whether the stage diff --git a/tests/browser/captions.test.js b/tests/browser/captions.test.js new file mode 100644 index 00000000..09a9877d --- /dev/null +++ b/tests/browser/captions.test.js @@ -0,0 +1,672 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { functionBody, read } from "./source.js"; + +let captionCopies = 0; + +async function setupCaptions(t) { + // Caption state is module-level, so each case imports a fresh copy. + const captions = await import( + `../../web/captions.js?copy=${++captionCopies}` + ); + const nodes = { + captionsText: { textContent: "" }, + captionsBar: { hidden: true }, + }; + captions.initCaptions({ nodes }); + t.mock.timers.enable({ apis: ["setTimeout"] }); + return { captions, nodes }; +} + +function finishJim(t, nodes, text) { + const expected = `[Jim]: ${text}`; + for (let ticks = 0; ticks < text.length; ticks++) { + if (nodes.captionsText.textContent === expected) break; + t.mock.timers.tick(300); + } + assert.equal(nodes.captionsText.textContent, expected); +} + +function finishDismissal(t, nodes, text) { + for ( + let ticks = 0; + ticks < text.length && !nodes.captionsBar.hidden; + ticks++ + ) { + t.mock.timers.tick(300); + } + assert.equal( + nodes.captionsBar.hidden, + true, + "a pending edit must dismiss Jim after the final reveal", + ); +} + +// Execute page logic against test dependencies without its browser startup. +function loadInterviewFunction(name, scope) { + const body = `${functionBody(read("web/interview.js"), name)}\n}`; + return new Function("scope", `with (scope) { ${body}\nreturn ${name}; }`)( + scope, + ); +} + +test("the listening placeholder hides after twelve seconds", async (t) => { + const { captions, nodes } = await setupCaptions(t); + nodes.captionsText.textContent = "Listening..."; + captions.showCaptions(); + + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(11999); + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(1); + assert.equal(nodes.captionsBar.hidden, true); +}); + +test("candidate captions keep their window and renew the idle timeout", async (t) => { + const { captions, nodes } = await setupCaptions(t); + const text = + `${"I am checking the input constraints ".repeat(6)}` + + "before choosing a data structure. I will use a hash map."; + + captions.updateCaptions("you", text, "you-1"); + assert.equal(nodes.captionsText.textContent, "[You]: I will use a hash map."); + assert.equal(nodes.captionsBar.hidden, false); + + t.mock.timers.tick(11000); + assert.equal(nodes.captionsBar.hidden, false); + + captions.updateCaptions("you", "I will scan the array once.", "you-1"); + assert.equal( + nodes.captionsText.textContent, + "[You]: I will scan the array once.", + ); + + // The previous update's deadline must not hide the renewed caption. + t.mock.timers.tick(1000); + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(10999); + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(1); + assert.equal(nodes.captionsBar.hidden, true); +}); + +test("interim interviewer captions pace cumulative updates", async (t) => { + const { captions, nodes } = await setupCaptions(t); + + captions.updateCaptions("interviewer", "Use a hash map.", "jim-1"); + assert.equal(nodes.captionsText.textContent, "[Jim]: Use a ha"); + assert.equal(nodes.captionsBar.hidden, false); + + t.mock.timers.tick(300); + assert.equal(nodes.captionsText.textContent, "[Jim]: Use a hash map."); + + captions.updateCaptions("interviewer", "Use a", "jim-1"); + assert.equal(nodes.captionsText.textContent, "[Jim]: Use a hash map."); + + captions.updateCaptions("interviewer", "Try sorting.", "jim-2"); + assert.equal(nodes.captionsText.textContent, "[Jim]: Try sort"); + t.mock.timers.tick(300); + assert.equal(nodes.captionsText.textContent, "[Jim]: Try sorting."); +}); + +test("a finished interviewer turn is shown in full", async (t) => { + const { captions, nodes } = await setupCaptions(t); + const text = + "Start by checking how often each value appears in the input. " + + "Store counts in a hash map so repeated values are handled consistently. " + + "Then scan the array once and compare each value with its required complement."; + + assert.ok(text.length > 160); + + captions.updateCaptions("interviewer", text, "jim-1"); + const beforeFinal = nodes.captionsText.textContent; + captions.updateCaptions("interviewer", text, "jim-1", { final: true }); + + assert.equal(nodes.captionsText.textContent, beforeFinal); + + t.mock.timers.tick(300); + assert.equal(nodes.captionsText.textContent, `[Jim]: ${text.slice(0, 16)}`); + + finishJim(t, nodes, text); + + // Later ticks must neither trim nor hide the retained turn. + t.mock.timers.tick(300); + assert.equal(nodes.captionsText.textContent, `[Jim]: ${text}`); + t.mock.timers.tick(60000); + assert.equal(nodes.captionsBar.hidden, false); +}); + +test("a finished interviewer turn outlives idle timers", async (t) => { + const { captions, nodes } = await setupCaptions(t); + + captions.showCaptions(); + t.mock.timers.tick(11000); + + captions.updateCaptions("interviewer", "Think.", "jim-1", { + final: true, + }); + + // The placeholder's old deadline must not hide the completed turn. + t.mock.timers.tick(1000); + assert.equal(nodes.captionsBar.hidden, false); + + t.mock.timers.tick(60000); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(nodes.captionsText.textContent, "[Jim]: Think."); +}); + +test("candidate speech replaces Jim without late revival", async (t) => { + const { captions, nodes } = await setupCaptions(t); + + captions.updateCaptions("interviewer", "Use a hash map.", "jim-1", { + final: true, + }); + finishJim(t, nodes, "Use a hash map."); + captions.updateCaptions("you", "I will count the values.", "you-1"); + + assert.equal( + nodes.captionsText.textContent, + "[You]: I will count the values.", + ); + + captions.dismissCaptions(); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal( + nodes.captionsText.textContent, + "[You]: I will count the values.", + ); + + t.mock.timers.tick(1000); + + for (const final of [false, true]) { + captions.updateCaptions( + "interviewer", + "Use a hash map and count each value.", + "jim-1", + { final }, + ); + assert.equal( + nodes.captionsText.textContent, + "[You]: I will count the values.", + ); + assert.equal(nodes.captionsBar.hidden, false); + } + + t.mock.timers.tick(10999); + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(1); + assert.equal(nodes.captionsBar.hidden, true); +}); + +for (const final of [false, true]) { + test(`a code edit during ${final ? "a final" : "an interim"} Jim reveal is remembered`, async (t) => { + const { captions, nodes } = await setupCaptions(t); + const text = "Use a hash map and count each value."; + + captions.updateCaptions("interviewer", text, "jim-1", { final }); + const beforeEdit = nodes.captionsText.textContent; + + captions.dismissCaptions(); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(nodes.captionsText.textContent, beforeEdit); + + t.mock.timers.tick(300); + assert.equal(nodes.captionsBar.hidden, false); + assert.notEqual(nodes.captionsText.textContent, beforeEdit); + + captions.updateCaptions("interviewer", text, "jim-1", { + final: true, + }); + assert.equal(nodes.captionsBar.hidden, false); + + finishDismissal(t, nodes, text); + + for (const lateFinal of [false, true]) { + captions.updateCaptions( + "interviewer", + `${text} Then scan the array.`, + "jim-1", + { final: lateFinal }, + ); + assert.equal(nodes.captionsBar.hidden, true); + } + + t.mock.timers.tick(60000); + assert.equal(nodes.captionsBar.hidden, true); + }); +} + +test("a dismissed completed Jim turn stays hidden", async (t) => { + const { captions, nodes } = await setupCaptions(t); + + captions.updateCaptions("interviewer", "Use a hash map.", "jim-1", { + final: true, + }); + finishJim(t, nodes, "Use a hash map."); + captions.dismissCaptions(); + assert.equal(nodes.captionsBar.hidden, true); + + t.mock.timers.tick(300); + assert.equal(nodes.captionsBar.hidden, true); + + for (const lateFinal of [false, true]) { + captions.updateCaptions( + "interviewer", + "Use a hash map and count each value.", + "jim-1", + { final: lateFinal }, + ); + assert.equal(nodes.captionsBar.hidden, true); + } + + captions.updateCaptions("interviewer", "Try sorting.", "jim-2", { + final: true, + }); + finishJim(t, nodes, "Try sorting."); + assert.equal(nodes.captionsText.textContent, "[Jim]: Try sorting."); + assert.equal(nodes.captionsBar.hidden, false); + + t.mock.timers.tick(60000); + assert.equal(nodes.captionsBar.hidden, false); +}); + +test("a new Jim turn replaces the previous turn despite late updates", async (t) => { + const { captions, nodes } = await setupCaptions(t); + + captions.updateCaptions("interviewer", "Use a hash map.", "jim-1", { + final: true, + }); + captions.updateCaptions("interviewer", "Try sorting.", "jim-2"); + assert.equal(nodes.captionsText.textContent, "[Jim]: Try sort"); + + t.mock.timers.tick(300); + captions.updateCaptions("interviewer", "Try sorting.", "jim-2", { + final: true, + }); + + captions.updateCaptions( + "interviewer", + "Use a hash map and count each value.", + "jim-1", + { final: true }, + ); + assert.equal(nodes.captionsText.textContent, "[Jim]: Try sorting."); + + t.mock.timers.tick(60000); + assert.equal(nodes.captionsBar.hidden, false); +}); + +for (const { name, final, broken } of [ + { + name: "a complete final stream retains the caption", + final: true, + broken: false, + }, + { + name: "an interrupted final stream keeps partial captions temporary", + final: true, + broken: true, + }, + { + name: "an interim stream keeps captions temporary", + final: false, + broken: false, + }, +]) { + test(name, async (t) => { + const { captions, nodes } = await setupCaptions(t); + const updates = []; + const consume = loadInterviewFunction("consumeTranscript", { + isCurrentAgent: () => true, + responseWindowIndex: () => 0, + updateTranscriptSegment: () => {}, + recordReplay: () => {}, + updateCaptions(speaker, text, id, options = {}) { + updates.push(options.final === true); + captions.updateCaptions(speaker, text, id, options); + }, + }); + + async function* chunks() { + yield "Think"; + if (broken) throw new Error("stream interrupted"); + yield "."; + } + + const reader = chunks(); + reader.info = { + attributes: { + "lk.segment_id": "jim-1", + "lk.transcription_final": String(final), + }, + }; + + await consume({ localParticipant: { identity: "candidate" } }, reader, { + identity: "jim", + }); + + assert.deepEqual(updates, broken ? [false, false] : [false, false, final]); + assert.equal( + nodes.captionsText.textContent, + broken ? "[Jim]: Think" : "[Jim]: Think.", + ); + + t.mock.timers.tick(12000); + assert.equal(nodes.captionsBar.hidden, broken || !final); + }); +} + +test("only changed editor text dismisses Jim", async (t) => { + const { captions, nodes } = await setupCaptions(t); + const events = []; + let input; + nodes.editor = { + value: "", + addEventListener(type, handler) { + assert.equal(type, "input"); + input = handler; + }, + }; + const state = { + language: "javascript", + codeByLanguage: { javascript: "" }, + }; + + const binding = functionBody(read("web/interview.js"), "bindEvents"); + const start = binding.indexOf('nodes.editor.addEventListener("input",'); + const end = binding.indexOf( + '\n nodes.report.addEventListener("click"', + start, + ); + assert.ok(start >= 0, "the editor input listener must exist"); + assert.ok(end > start, "the listener boundary must exist"); + + new Function("scope", `with (scope) { ${binding.slice(start, end)} }`)({ + nodes, + state, + dismissCaptions: captions.dismissCaptions, + flushPendingLanguagePublish: () => events.push("language"), + scheduleEditorPaint: () => events.push("paint"), + flushPendingEditorPublish: () => events.push("publish"), + codePublishTimer: null, + CODE_PUBLISH_DEBOUNCE_MS: 300, + setTimeout, + clearTimeout, + }); + + captions.updateCaptions("interviewer", "Think.", "jim-1", { + final: true, + }); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(typeof input, "function"); + + input(); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(state.codeByLanguage.javascript, ""); + assert.deepEqual(events, ["language", "paint"]); + t.mock.timers.tick(300); + assert.deepEqual(events, ["language", "paint", "publish"]); + events.length = 0; + + nodes.editor.value = "const answer = 42;"; + input(); + assert.equal(nodes.captionsBar.hidden, true); + assert.equal(state.codeByLanguage.javascript, "const answer = 42;"); + assert.deepEqual(events, ["language", "paint"]); + + t.mock.timers.tick(299); + assert.deepEqual(events, ["language", "paint"]); + t.mock.timers.tick(1); + assert.deepEqual(events, ["language", "paint", "publish"]); + + captions.updateCaptions("interviewer", "Think about the input.", "jim-1", { + final: true, + }); + assert.equal(nodes.captionsBar.hidden, true); + captions.updateCaptions("interviewer", "Try another approach.", "jim-2", { + final: true, + }); + finishJim(t, nodes, "Try another approach."); + input(); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(nodes.captionsText.textContent, "[Jim]: Try another approach."); + t.mock.timers.tick(12000); + assert.equal(nodes.captionsBar.hidden, false); +}); + +test("a candidate final replay does not dismiss Jim", async (t) => { + const { captions, nodes } = await setupCaptions(t); + const jimText = "Use a hash map and count each value."; + + captions.updateCaptions("you", "I will", "you-1"); + captions.updateCaptions("interviewer", jimText, "jim-1"); + const interimJimText = nodes.captionsText.textContent; + + const consume = loadInterviewFunction("consumeTranscript", { + isCurrentAgent: () => true, + responseWindowIndex: () => 0, + updateTranscriptSegment: () => {}, + recordReplay: () => {}, + updateCaptions: captions.updateCaptions, + }); + + async function* chunks() { + yield "I will count"; + yield " the values."; + } + + const reader = chunks(); + reader.info = { + attributes: { + "lk.segment_id": "you-1", + "lk.transcription_final": "true", + }, + }; + + await consume({ localParticipant: { identity: "candidate" } }, reader, { + identity: "candidate", + }); + + assert.equal(nodes.captionsText.textContent, interimJimText); + + captions.updateCaptions("interviewer", jimText, "jim-1", { + final: true, + }); + finishJim(t, nodes, jimText); + assert.equal(nodes.captionsText.textContent, `[Jim]: ${jimText}`); + + for (const final of [false, true]) { + captions.updateCaptions( + "you", + "I will count the values and check their complements.", + "you-1", + { final }, + ); + assert.equal(nodes.captionsText.textContent, `[Jim]: ${jimText}`); + } + + t.mock.timers.tick(12000); + assert.equal(nodes.captionsBar.hidden, false); + + captions.updateCaptions("you", "I will compare complements.", "you-2"); + assert.equal( + nodes.captionsText.textContent, + "[You]: I will compare complements.", + ); + t.mock.timers.tick(11999); + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(1); + assert.equal(nodes.captionsBar.hidden, true); +}); + +test("candidate speech pauses a live Jim turn without retiring it", async (t) => { + const { captions, nodes } = await setupCaptions(t); + + captions.updateCaptions("you", "I will", "you-1"); + captions.updateCaptions("interviewer", "Use a hash map.", "jim-1"); + captions.updateCaptions("you", "I will count the values.", "you-1"); + + assert.equal( + nodes.captionsText.textContent, + "[You]: I will count the values.", + ); + + t.mock.timers.tick(300); + assert.equal( + nodes.captionsText.textContent, + "[You]: I will count the values.", + ); + + const text = "Use a hash map and count each value."; + captions.updateCaptions("interviewer", text, "jim-1", { + final: true, + }); + finishJim(t, nodes, text); + + t.mock.timers.tick(12000); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(nodes.captionsText.textContent, `[Jim]: ${text}`); +}); + +test("a final-only Jim turn still reveals gradually", async (t) => { + const { captions, nodes } = await setupCaptions(t); + + captions.updateCaptions("interviewer", "Use a hash map.", "jim-1", { + final: true, + }); + + assert.equal(nodes.captionsText.textContent, "[Jim]: Use a ha"); + assert.equal(nodes.captionsBar.hidden, false); + + t.mock.timers.tick(300); + assert.equal(nodes.captionsText.textContent, "[Jim]: Use a hash map."); + + t.mock.timers.tick(12000); + assert.equal(nodes.captionsBar.hidden, false); +}); + +for (const candidateId of ["you-1", null]) { + test(`candidate speech retires a revealing final Jim turn (${candidateId ?? "no ID"})`, async (t) => { + const { captions, nodes } = await setupCaptions(t); + const text = "Use a hash map and count each value."; + const candidateText = "I will count the values."; + + captions.updateCaptions("interviewer", text, "jim-1", { + final: true, + }); + assert.notEqual(nodes.captionsText.textContent, `[Jim]: ${text}`); + + captions.updateCaptions("you", candidateText, candidateId); + assert.equal(nodes.captionsText.textContent, `[You]: ${candidateText}`); + + t.mock.timers.tick(300); + assert.equal(nodes.captionsText.textContent, `[You]: ${candidateText}`); + + for (const final of [false, true]) { + captions.updateCaptions( + "interviewer", + `${text} Then scan the array.`, + "jim-1", + { final }, + ); + assert.equal(nodes.captionsText.textContent, `[You]: ${candidateText}`); + assert.equal(nodes.captionsBar.hidden, false); + } + + t.mock.timers.tick(11699); + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(1); + assert.equal(nodes.captionsBar.hidden, true); + }); +} + +test("a new candidate segment retires a live Jim turn", async (t) => { + const { captions, nodes } = await setupCaptions(t); + const jimText = "Use a hash map and count each value."; + const candidateText = "Now I will compare complements."; + + captions.updateCaptions("you", "I will count the values.", "you-1"); + captions.updateCaptions("interviewer", jimText, "jim-1"); + captions.updateCaptions("you", candidateText, "you-2"); + + assert.equal(nodes.captionsText.textContent, `[You]: ${candidateText}`); + + t.mock.timers.tick(300); + assert.equal(nodes.captionsText.textContent, `[You]: ${candidateText}`); + + for (const final of [false, true]) { + captions.updateCaptions( + "interviewer", + `${jimText} Then scan the array.`, + "jim-1", + { final }, + ); + assert.equal(nodes.captionsText.textContent, `[You]: ${candidateText}`); + assert.equal(nodes.captionsBar.hidden, false); + } + + t.mock.timers.tick(11699); + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(1); + assert.equal(nodes.captionsBar.hidden, true); +}); + +test("a pending code edit does not dismiss the next Jim turn", async (t) => { + const { captions, nodes } = await setupCaptions(t); + + captions.updateCaptions( + "interviewer", + "Use a hash map and count each value.", + "jim-1", + ); + captions.dismissCaptions(); + + captions.updateCaptions("interviewer", "Try sorting.", "jim-2", { + final: true, + }); + finishJim(t, nodes, "Try sorting."); + + t.mock.timers.tick(13000); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(nodes.captionsText.textContent, "[Jim]: Try sorting."); + + for (const final of [false, true]) { + captions.updateCaptions( + "interviewer", + "Use a hash map and count each value. Then scan the array.", + "jim-1", + { final }, + ); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(nodes.captionsText.textContent, "[Jim]: Try sorting."); + } +}); + +test("a pending code edit preserves the candidate idle deadline", async (t) => { + const { captions, nodes } = await setupCaptions(t); + const jimText = "Use a hash map and count each value."; + const candidateText = "I will count the values."; + + captions.updateCaptions("you", "I will", "you-1"); + captions.updateCaptions("interviewer", jimText, "jim-1"); + captions.updateCaptions("you", candidateText, "you-1"); + + t.mock.timers.tick(11000); + captions.dismissCaptions(); + assert.equal(nodes.captionsBar.hidden, false); + assert.equal(nodes.captionsText.textContent, `[You]: ${candidateText}`); + + t.mock.timers.tick(999); + assert.equal(nodes.captionsBar.hidden, false); + t.mock.timers.tick(1); + assert.equal(nodes.captionsBar.hidden, true); + + captions.updateCaptions("interviewer", jimText, "jim-1", { + final: true, + }); + assert.equal(nodes.captionsBar.hidden, false); + + finishDismissal(t, nodes, jimText); + t.mock.timers.tick(60000); + assert.equal(nodes.captionsBar.hidden, true); +}); diff --git a/tests/browser/editor-pipeline.test.js b/tests/browser/editor-pipeline.test.js index a1032797..7fd34a82 100644 --- a/tests/browser/editor-pipeline.test.js +++ b/tests/browser/editor-pipeline.test.js @@ -657,6 +657,122 @@ test("disabled auto indentation retains parsed bracket highlighting", async (t) } }); +test("Jim captions survive editor navigation and language switches", async (t) => { + const ctx = await editorPage(t, { + beforeLoad: (page) => page.clock.install(), + }); + if (!ctx) return; + const { page, errors } = ctx; + const editor = page.locator("#editor"); + const bar = page.locator("#captions-bar"); + const code = "const answer = 42;"; + const text = "Use a hash map and count each value."; + + const assertJim = async (message) => { + await page.evaluate( + () => new Promise((resolve) => requestAnimationFrame(resolve)), + ); + assert.equal(await bar.isVisible(), true, message); + assert.equal( + await page.locator("#captions-text").textContent(), + `[Jim]: ${text}`, + message, + ); + }; + + try { + await page.evaluate(async () => { + await import("/interview.js"); + document.querySelector("#audio-check").hidden = true; + (await import("/avatar/stage.js")).setStageCovered(false); + }); + + await page.locator('[data-language="javascript"]').click(); + await editor.fill(code); + await page.locator("#problem-tab").click(); + assert.equal( + await editor.evaluate((node) => document.activeElement === node), + false, + "setup must move focus away from the editor", + ); + + await page.evaluate(async (captionText) => { + const captions = await import("/captions.js"); + captions.updateCaptions( + "interviewer", + captionText, + "jim-editor-navigation", + { final: true }, + ); + }, text); + await page.clock.runFor(300 * Math.ceil(text.length / 8)); + await page.clock.runFor(13000); + await assertJim("a completed Jim caption must outlive the idle timeout"); + + await page.evaluate(() => { + document + .querySelector("#editor") + .dispatchEvent(new Event("input", { bubbles: true })); + }); + assert.equal(await editor.inputValue(), code); + await assertJim("an input event without changed text must preserve Jim"); + + await editor.focus(); + assert.equal( + await editor.evaluate((node) => document.activeElement === node), + true, + ); + await assertJim("focusing the editor must preserve Jim"); + + await editor.click({ position: { x: 8, y: 8 } }); + await assertJim("clicking the editor must preserve Jim"); + + await page.evaluate(() => { + document.querySelector("#editor").setSelectionRange(0, 0); + }); + await page.keyboard.press("ArrowRight"); + assert.deepEqual(await caretState(page), { + value: code, + start: 1, + end: 1, + }); + await assertJim("moving the caret must preserve Jim"); + + const python = page.locator('[data-language="python"]'); + await python.click(); + assert.equal( + await python.evaluate((button) => button.classList.contains("selected")), + true, + "the language switch must run", + ); + assert.notEqual( + await editor.inputValue(), + code, + "the language switch must restore a different buffer", + ); + await assertJim("switching languages must preserve Jim"); + + await page.locator('[data-language="javascript"]').click(); + assert.equal(await editor.inputValue(), code); + await assertJim("returning to the saved buffer must preserve Jim"); + + await editor.focus(); + await assertJim("refocusing after a switch must preserve Jim"); + + await editor.fill(`${code} `); + assert.equal(await editor.inputValue(), `${code} `); + assert.equal( + await bar.isVisible(), + false, + "changing the code text must dismiss Jim", + ); + + assert.deepEqual(errors, []); + } finally { + await page.close(); + } +}); + test("disabled bracket auto-close leaves insertion and deletion to the browser", async (t) => { const ctx = await editorPage(t, { params: { editorAutoClose: "0" } }); if (!ctx) return; @@ -809,6 +925,82 @@ test("plain editor text follows the highlight layer's color setting", async (t) } }); +test("a new Jim turn starts at the top of the caption bar", async (t) => { + const ctx = await editorPage(t, { + beforeLoad: (page) => page.clock.install(), + }); + if (!ctx) return; + const { page, errors } = ctx; + const bar = page.locator("#captions-bar"); + const captionText = page.locator("#captions-text"); + const firstText = + "Start by checking how often each value appears in the input. ".repeat(40); + const secondText = + "Then scan the array and check the boundary cases. ".repeat(40); + + const showJim = async (text, id) => { + await page.evaluate( + async ({ text, id }) => { + const captions = await import("/captions.js"); + captions.updateCaptions("interviewer", text, id, { final: true }); + await new Promise((resolve) => requestAnimationFrame(resolve)); + }, + { text, id }, + ); + await page.clock.runFor(300 * Math.ceil(text.length / 8)); + }; + + const scrollDown = async () => { + const position = await bar.evaluate((node) => { + node.scrollTop = node.scrollHeight; + return node.scrollTop; + }); + assert.ok(position > 0, "the test must actually scroll the caption bar"); + return position; + }; + + try { + await page.evaluate(async () => { + await import("/interview.js"); + document.querySelector("#audio-check").hidden = true; + (await import("/avatar/stage.js")).setStageCovered(false); + }); + + await showJim(firstText, "jim-scroll-1"); + assert.equal(await bar.isVisible(), true); + assert.equal(await captionText.textContent(), `[Jim]: ${firstText}`); + + const firstPosition = await scrollDown(); + await showJim(firstText, "jim-scroll-1"); + assert.equal( + await bar.evaluate((node) => node.scrollTop), + firstPosition, + "an update to the same turn must preserve the reading position", + ); + + await showJim(secondText, "jim-scroll-2"); + assert.equal(await captionText.textContent(), `[Jim]: ${secondText}`); + assert.equal( + await bar.evaluate((node) => node.scrollTop), + 0, + "a new Jim turn must start at the top", + ); + + const secondPosition = await scrollDown(); + await showJim(firstText, "jim-scroll-1"); + assert.equal(await captionText.textContent(), `[Jim]: ${secondText}`); + assert.equal( + await bar.evaluate((node) => node.scrollTop), + secondPosition, + "a retired turn must not move the current reading position", + ); + + assert.deepEqual(errors, []); + } finally { + await page.close(); + } +}); + test("a proportional font keeps the textarea and highlight metrics aligned", async (t) => { for (const editorHighlight of ["0", "1"]) { const ctx = await editorPage(t, { diff --git a/web/captions.js b/web/captions.js index 82ce7316..39d1fd4c 100644 --- a/web/captions.js +++ b/web/captions.js @@ -1,10 +1,8 @@ -/// The live subtitle over the editor. +/// Captions over the editor. /// -/// A subtitle, not a log: the transcript tab already keeps every turn, so this -/// shows what is being said now and retires when nobody is speaking. Split out -/// of `interview.js` because the pacing state is three variables and one timer -/// that nothing else may touch, and the reveal is the only thing in the page -/// that repaints on a 300ms tick. +/// Keep completed interviewer turns readable while the candidate thinks. +/// Interviewer captions reveal gradually, including final copies. Candidate +/// captions retain their bounded window and idle timeout. import { captionWindow } from "./lib.js"; @@ -17,22 +15,46 @@ export function initCaptions(deps) { const CAPTION_TICK_MS = 300; const CAPTION_CHARS_PER_TICK = 8; -// One uninterrupted spoken turn is unbounded, and the caption line sits in the -// header: rendered whole, a long answer pushes the timer and the controls off -// the screen. The tail is the part being spoken. +// Bound live captions until Jim's final text is fully revealed. const CAPTION_MAX_CHARS = 160; -// Long enough to finish reading a sentence that just landed, short enough that -// a silent pause clears the editor's corner. +// Candidate captions and the listening placeholder should clear when idle. const CAPTION_IDLE_HIDE_MS = 12000; let captionIdleTimer = null; -let interviewerCaption = { id: null, text: "", shown: 0, timer: null }; +let interviewerCaption = { + id: null, + text: "", + shown: 0, + timer: null, + final: false, + pendingDismissal: false, +}; -export function updateCaptions(speaker, text, id = null) { +// Each publish opens a separate stream, so replaced turns can arrive late. +const retiredInterviewerIds = new Set(); +const retiredCandidateIds = new Set(); +let candidateCaption = { id: null, text: "", replaced: false }; + +export function updateCaptions( + speaker, + text, + id = null, + { final = false, finalStream = final } = {}, +) { if (!nodes.captionsText) return; if (speaker === "interviewer") { + if (retiredInterviewerIds.has(id)) return; + if (candidateCaption.id != null) candidateCaption.replaced = true; if (interviewerCaption.id !== id) { - clearTimeout(interviewerCaption.timer); - interviewerCaption = { id, text: "", shown: 0, timer: null }; + retireInterviewerCaption(); + if (nodes.captionsBar) nodes.captionsBar.scrollTop = 0; + interviewerCaption = { + id, + text: "", + shown: 0, + timer: null, + final: false, + pendingDismissal: false, + }; } // The longest, not the latest. Every interim publish carries the whole turn // so far on a stream of its own, so one that lands out of order is a @@ -40,14 +62,40 @@ export function updateCaptions(speaker, text, id = null) { // reveal to a few characters and replay the line. if (text.length > interviewerCaption.text.length) interviewerCaption.text = text; + if (final) interviewerCaption.final = true; + + // Final text can arrive while audio is still queued. + if (interviewerCaption.final) retireCandidateCaption(); if (!interviewerCaption.timer) paceInterviewerCaption(); return; } - // Stop pacing Jim: the next tick would otherwise repaint his line over the - // candidate's. A later chunk of the same segment restarts it through the - // `!interviewerCaption.timer` check below. - clearTimeout(interviewerCaption.timer); - interviewerCaption.timer = null; + + if (retiredCandidateIds.has(id)) return; + const newCandidateSegment = candidateCaption.id !== id; + if (newCandidateSegment) { + retireCandidateCaption(); + candidateCaption = { id, text: "", replaced: false }; + } + + // A closing copy is not a new response to the interviewer. + if ( + candidateCaption.replaced && + (finalStream || text.length <= candidateCaption.text.length) + ) { + return; + } + + if (text.length > candidateCaption.text.length) candidateCaption.text = text; + candidateCaption.replaced = false; + + // A new candidate segment supersedes Jim. Otherwise, only a final Jim + // turn is retired; a live turn can resume after late candidate speech. + if (newCandidateSegment || interviewerCaption.final) { + retireInterviewerCaption(); + } else { + clearTimeout(interviewerCaption.timer); + interviewerCaption.timer = null; + } const clipped = captionWindow(text, CAPTION_MAX_CHARS); nodes.captionsText.textContent = `[${speaker === "you" ? "You" : "Jim"}]: ${clipped}`; showCaptions(); @@ -59,8 +107,16 @@ export function paceInterviewerCaption() { caption.text.length, caption.shown + CAPTION_CHARS_PER_TICK, ); - nodes.captionsText.textContent = `[Jim]: ${captionWindow(caption.text.slice(0, caption.shown), CAPTION_MAX_CHARS)}`; - showCaptions(); + const retained = caption.final && caption.shown === caption.text.length; + const text = retained + ? caption.text + : captionWindow(caption.text.slice(0, caption.shown), CAPTION_MAX_CHARS); + nodes.captionsText.textContent = `[Jim]: ${text}`; + if (retained && caption.pendingDismissal) { + dismissCaptions(); + return; + } + showCaptions({ persistent: retained }); if (caption.shown < caption.text.length) { caption.timer = setTimeout(paceInterviewerCaption, CAPTION_TICK_MS); } else { @@ -68,15 +124,56 @@ export function paceInterviewerCaption() { } } -/// Captions are a live subtitle, not a log: the transcript tab already keeps -/// every turn. Leaving the last thing anyone said pinned over the editor for -/// the rest of the interview is just an obstruction, so the bar retires once -/// nobody has spoken for a while and comes back on the next word. -export function showCaptions() { +/// Completed interviewer turns remain available while the candidate thinks. +/// Other captions expire so an idle subtitle does not cover the editor. +export function showCaptions({ persistent = false } = {}) { if (!nodes.captionsBar) return; nodes.captionsBar.hidden = false; clearTimeout(captionIdleTimer); + captionIdleTimer = null; + + if (persistent) return; + captionIdleTimer = setTimeout(() => { nodes.captionsBar.hidden = true; }, CAPTION_IDLE_HIDE_MS); } + +/// Remember a code response until Jim's final text finishes revealing. +export function dismissCaptions() { + if ( + !interviewerCaption.text || + retiredInterviewerIds.has(interviewerCaption.id) + ) { + return; + } + + interviewerCaption.pendingDismissal = true; + if ( + !interviewerCaption.final || + interviewerCaption.shown < interviewerCaption.text.length + ) { + return; + } + + retireInterviewerCaption(); + clearTimeout(captionIdleTimer); + captionIdleTimer = null; + if (nodes.captionsBar) nodes.captionsBar.hidden = true; +} + +function retireInterviewerCaption() { + clearTimeout(interviewerCaption.timer); + interviewerCaption.timer = null; + interviewerCaption.final = false; + interviewerCaption.pendingDismissal = false; + if (interviewerCaption.id != null) { + retiredInterviewerIds.add(interviewerCaption.id); + } +} + +function retireCandidateCaption() { + if (candidateCaption.id != null) { + retiredCandidateIds.add(candidateCaption.id); + } +} diff --git a/web/interview.js b/web/interview.js index bfab536b..8d5ee7cc 100644 --- a/web/interview.js +++ b/web/interview.js @@ -126,7 +126,12 @@ import { startIntegrityWorker, stopIntegrityWorker, } from "./integrity.js"; -import { initCaptions, showCaptions, updateCaptions } from "./captions.js"; +import { + dismissCaptions, + initCaptions, + showCaptions, + updateCaptions, +} from "./captions.js"; import { consentGiven, initRecording, @@ -973,6 +978,10 @@ function bindEvents() { } }); nodes.editor.addEventListener("input", () => { + // Only a changed code buffer counts as the candidate's response. + if (nodes.editor.value !== state.codeByLanguage[state.language]) { + dismissCaptions(); + } // Before the buffer below is taken, because this sends the tab as it // arrived. Typing inside the switch's own debounce window cancels it // otherwise, and the single packet that survives carries the switch and the @@ -1980,6 +1989,7 @@ async function consumeTranscript(room, reader, participant) { const responseWindow = responseWindowIndex(); const final = attrs["lk.transcription_final"] === "true"; let text = ""; + let completed = false; try { for await (const chunk of reader) { text += chunk; @@ -1987,14 +1997,19 @@ async function consumeTranscript(room, reader, participant) { // Not `text.trim()`: this runs per chunk and would copy the whole turn so // far just to ask whether it is empty. Non-empty is enough here, and the // post-loop write below does the trim once. - if (text) updateCaptions(speaker, text, id); + if (text) updateCaptions(speaker, text, id, { finalStream: final }); } + // A final stream that breaks must not pin its partial text. + completed = true; } catch { // Keep whatever arrived before the stream broke. } if (text.trim()) { updateTranscriptSegment(id, speaker, text, final); - updateCaptions(speaker, text, id); + updateCaptions(speaker, text, id, { + final: final && completed, + finalStream: final, + }); // Once per turn, at the end of the stream, not once per chunk: a chunk is a // few words and a turn is a sentence, and the replay is read as sentences. recordReplay("transcript", { speaker, text: text.trim(), responseWindow }); diff --git a/web/styles.css b/web/styles.css index 69cc7e84..0db474ad 100644 --- a/web/styles.css +++ b/web/styles.css @@ -845,10 +845,8 @@ p { overflow: hidden; } -/* Under the avatar, and only as wide as it. It grows to fit the turn instead of - clamping: a fixed max-height cut sentences in half behind a scrollbar nobody - in an interview is going to reach for. `CAPTION_MAX_CHARS` already bounds the - text, and the viewport cap is a backstop, not the normal case. */ +/* Completed Jim turns keep their full text. The viewport cap bounds the bar's + height while scrolling keeps longer turns readable. */ .jim-stage .captions-bar { margin: 0.35rem 0 0; max-height: min(40vh, 20rem);