diff --git a/.claude/skills/check-vacuum/SKILL.md b/.claude/skills/check-vacuum/SKILL.md new file mode 100644 index 0000000..dcb5685 --- /dev/null +++ b/.claude/skills/check-vacuum/SKILL.md @@ -0,0 +1,15 @@ +--- +name: check-vacuum +description: Check the Lab 5 notebook and use focused specialists to review the environment, requirements, and code features. +disable-model-invocation: true +--- + +# Check Vacuum + +1. Work from the CS440-Lab5 repository root. Run `python scripts/check_project.py`. +2. Report errors and unfinished TODOs. A passing check does not prove that Q-learning works. +3. Invoke `environment-specialist` with `context/environment.md` and only the environment cell from `q_learning.ipynb`. Ask it to check movement, rewards, reset, and terminal handling. +4. Invoke `requirements-specialist` with `context/requirements.md` and the check summary. Ask it to identify missing Lab 5 deliverables. +5. Invoke `configuration-reviewer` with `.claude/settings.json`, this skill, and the three specialist definitions. Supply the checker output and only relevant learning-function excerpts if reviewing Q-learning. +6. Combine the specialists' short findings. Use actual subagent calls and report any invocation failure. +7. Return: checks run, hyperparameters, findings, and remaining work. Do not change files unless requested or invent training results. diff --git a/.claude/skills/compare-hyperparameters/SKILL.md b/.claude/skills/compare-hyperparameters/SKILL.md new file mode 100644 index 0000000..7a40fbe --- /dev/null +++ b/.claude/skills/compare-hyperparameters/SKILL.md @@ -0,0 +1,18 @@ +--- +name: compare-hyperparameters +description: Sweep alpha, epsilon, and gamma over several seeds and summarize the trade-offs, to support the hyperparameter explanation the lab asks for. +disable-model-invocation: true +argument-hint: "[--alpha ...] [--epsilon ...] [--gamma ...] [--seeds N]" +--- + +# Compare Hyperparameters + +1. Work from the CS440-Lab5 repository root. Run `python scripts/sweep_hyperparameters.py $ARGUMENTS`. With no arguments it sweeps alpha and epsilon around the notebook's current values over 5 seeds. Training seeds and evaluation seeds are disjoint from the notebook's own seeds. +2. Add `--out sweep_results.json` if the user wants the table saved. Otherwise do not write files. The script never changes the notebook or `results.json`. +3. Summarize the table: success rate (mean and range across seeds), average evaluation reward and steps, and late-training traps for each setting. +4. Explain the trade-offs only as far as the numbers show them: + - alpha: how fast values move versus how noisy they are. + - epsilon: exploration of rarely visited cells versus wasted or risky steps while training. + - gamma: how strongly the +10 goal outweighs the per-step cost. +5. Say when differences fall within the seed-to-seed range and cannot be told apart. Do not call any setting "best" or "optimal" on that basis alone. The sweep covers a small grid and a small number of seeds. +6. Return: the settings tried, the table, what the evidence supports, and a suggested wording for the README's hyperparameter explanation. Do not edit the README or the notebook parameters unless asked, and do not invent results. diff --git a/.claude/skills/verify-results/SKILL.md b/.claude/skills/verify-results/SKILL.md new file mode 100644 index 0000000..27e7b47 --- /dev/null +++ b/.claude/skills/verify-results/SKILL.md @@ -0,0 +1,14 @@ +--- +name: verify-results +description: Sanity-check results.json for consistency and compare the learned policy with a model-based optimum, then review the claims made about it. +disable-model-invocation: true +--- + +# Verify Results + +1. Work from the CS440-Lab5 repository root. If `results.json` is missing, say so and stop; do not create or invent results. Tell the user to run the notebook's training and recording cells. +2. Run `python scripts/verify_results.py`. It checks that outcome counts add up, that reward matches steps, and that evaluation is not better than the 6-step best case. It also compares each learned action with a value-iteration solution. That solution is used only for verification and is never given to the learner. +3. Report any `INCONSISTENT` lines first. Policy differences from optimal are findings, not failures. Note which cells differ and by how much, and that rarely visited cells are expected to be less accurate. +4. Read the Results section of `README.md` and check each number and claim against `results.json` and the script output. Flag any claim the output does not support, such as "optimal", "best", or an explanation of why a cell's action was chosen. +5. Note that one seed shows only that a configuration worked in that run, and that the policy is evaluated from the start cell only. +6. Return: the recorded timestamp, consistency result, cells that differ from optimal, README claims that need correcting, and what remains unverified. Do not edit files unless asked, and do not rerun training. diff --git a/intent/intent.md b/intent/intent.md new file mode 100644 index 0000000..553e007 --- /dev/null +++ b/intent/intent.md @@ -0,0 +1,32 @@ +# Vacuum Robot Q-Learning Project + +## Objective +Implement Q-learning for a vacuum robot in a 4x4 grid environment to navigate from the start position (0,0) to the reward location (3,3). + +## Environment Specifications +- **Grid Size**: 4x4 cells +- **Start Position**: (0,0) +- **Reward Location**: (3,3) - provides positive reward upon reaching +- **Obstacle**: Located at (1,1) - robot cannot move into this cell +- **Trap**: Located at (1,3) - incurs penalty if entered + +## Robot Actions +Available operations: +1. **Left** - Move one cell left +2. **Right** - Move one cell right +3. **Up** - Move one cell up +4. **Down** - Move one cell down +5. **Suck** - Clean the current cell (not used initially as environment is clean) +6. **NoOp** - Stay in place (incurs cost, not beneficial) + +## Constraints & Assumptions +- Environment starts clean, so Suck action is not needed initially +- NoOp action incurs a cost without progressing toward the goal, so it should be avoided +- Robot cannot move out of the environment or into obstacle cells +- Goal: Minimize cumulative cost to reach (3,3) using Q-learning + +## Implementation Requirements +- Implement Q-table initialization and updates +- Handle state transitions respecting obstacles and traps +- Apply appropriate action selection (e.g., epsilon-greedy) +- Track visited states for learning progress \ No newline at end of file diff --git a/mini_coding_agent.py b/mini_coding_agent.py index 1bf2403..63e0ca8 100644 --- a/mini_coding_agent.py +++ b/mini_coding_agent.py @@ -12,7 +12,7 @@ DOC_NAMES = ("AGENTS.md", "README.md", "pyproject.toml", "package.json") -HELP_TEXT = "/help, /memory, /session, /reset, /exit" +HELP_TEXT = "/help, /memory, /session, /skills, /reset, /exit" WELCOME_ART = ( "/\\ /\\\\", "{ `---' }", @@ -27,6 +27,8 @@ "/help Show this help message.", "/memory Show the agent's distilled working memory.", "/session Show the path to the saved session file.", + "/skills List discovered skills and the skill log path.", + "/ Run a task with that skill explicitly.", "/reset Clear the current session history and memory.", "/exit Exit the agent.", ] @@ -34,6 +36,13 @@ MAX_TOOL_OUTPUT = 4000 MAX_HISTORY = 12000 IGNORED_PATH_NAMES = {".git", ".mini-coding-agent", "__pycache__", ".pytest_cache", ".ruff_cache", ".venv", "venv"} +SKILL_DIRS = ("skills", "skill", ".claude/skills") +SKILL_THRESHOLD = 0.25 +STOPWORDS = { + "a", "an", "and", "are", "as", "at", "be", "by", "can", "do", "for", "from", "how", "i", "in", + "is", "it", "me", "my", "of", "on", "or", "please", "so", "that", "the", "this", "to", "use", + "using", "we", "what", "with", "you", "your", +} ############################## #### Six Agent Components #### @@ -44,6 +53,7 @@ # 4) Context Reduction And Output Management -> clip, history_text # 5) Transcripts, Memory, And Resumption -> SessionStore, record, note_tool, ask, reset # 6) Delegation And Bounded Subagents -> tool_delegate +# 7) Agent Skills -> Skill, SkillRegistry, SkillLog, select_skill, activate_skill, tool_use_skill def now(): @@ -222,6 +232,129 @@ def complete(self, prompt, max_new_tokens): return data.get("response", "") +############################## +#### 7) Agent Skills ######### +############################## +class Skill: + def __init__(self, name, description, keywords, body, path): + self.name = name + self.description = description + self.keywords = keywords + self.body = body + self.path = path + + def match_text(self): + return " ".join([self.name.replace("-", " "), self.description, " ".join(self.keywords)]) + + +class SkillRegistry: + """Finds skill folders (skills//SKILL.md) and picks the one that fits a prompt.""" + + def __init__(self, skills, threshold=SKILL_THRESHOLD): + self.skills = {skill.name: skill for skill in skills} + self.threshold = threshold + + @classmethod + def discover(cls, root, threshold=SKILL_THRESHOLD): + root = Path(root) + skills = [] + for folder in SKILL_DIRS: + base = root / folder + if not base.is_dir(): + continue + for skill_dir in sorted(item for item in base.iterdir() if item.is_dir()): + for file_name in ("SKILL.md", "skill.md"): + path = skill_dir / file_name + if path.is_file(): + skills.append(cls.load(path)) + break + return cls(skills, threshold) + + @staticmethod + def load(path): + text = path.read_text(encoding="utf-8", errors="replace") + meta, body = {}, text + if text.startswith("---"): + parts = text.split("---", 2) + if len(parts) == 3: + for line in parts[1].splitlines(): + if ":" in line: + key, value = line.split(":", 1) + meta[key.strip().lower()] = value.strip().strip("\"'") + body = parts[2] + keywords = [word.strip() for word in meta.get("keywords", "").split(",") if word.strip()] + return Skill( + name=meta.get("name") or path.parent.name, + description=meta.get("description", ""), + keywords=keywords, + body=body.strip(), + path=path, + ) + + @staticmethod + def tokens(text): + words = re.findall(r"[a-z0-9]+", str(text).lower()) + result = [] + for word in words: + if word in STOPWORDS: + continue + for suffix in ("ing", "ed", "es", "s"): + if len(word) > len(suffix) + 2 and word.endswith(suffix): + word = word[: -len(suffix)] + break + result.append(word) + return result + + @classmethod + def similarity(cls, prompt, skill): + """Cosine similarity between word counts of the prompt and the skill's name/description/keywords.""" + left = cls.counts(cls.tokens(prompt)) + right = cls.counts(cls.tokens(skill.match_text())) + dot = sum(left[word] * right.get(word, 0) for word in left) + norm = (sum(v * v for v in left.values()) ** 0.5) * (sum(v * v for v in right.values()) ** 0.5) + return round(dot / norm, 3) if norm else 0.0 + + @staticmethod + def counts(words): + counts = {} + for word in words: + counts[word] = counts.get(word, 0) + 1 + return counts + + def match(self, prompt): + """Return (best_skill_or_None, scores, trigger). An explicit /skill-name always wins.""" + lowered = str(prompt).lower() + for name, skill in self.skills.items(): + if f"/{name.lower()}" in lowered: + return skill, {name: 1.0}, "explicit" + scores = {name: self.similarity(prompt, skill) for name, skill in self.skills.items()} + if not scores: + return None, scores, "none" + best = max(scores, key=scores.get) + if scores[best] >= self.threshold: + return self.skills[best], scores, "similarity" + return None, scores, "none" + + def text(self): + if not self.skills: + return "- none" + return "\n".join(f"- {skill.name}: {skill.description}" for skill in self.skills.values()) + + +class SkillLog: + """Appends one JSON line per event: prompt, similarity check, executed skill.""" + + def __init__(self, path): + self.path = Path(path) + self.path.parent.mkdir(parents=True, exist_ok=True) + + def write(self, event, **fields): + entry = {"time": now(), "event": event, **fields} + with self.path.open("a", encoding="utf-8") as handle: + handle.write(json.dumps(entry) + "\n") + return entry + + class MiniAgent: def __init__( self, @@ -235,6 +368,7 @@ def __init__( depth=0, max_depth=1, read_only=False, + skills=None, ): self.model_client = model_client self.workspace = workspace @@ -253,6 +387,10 @@ def __init__( "history": [], "memory": {"task": "", "files": [], "notes": []}, } + # 7) Agent Skills: find skill folders and open the skill log. + self.skills = skills if skills is not None else SkillRegistry.discover(self.root) + self.skill_log = SkillLog(self.root / ".mini-coding-agent" / "skills.log") + self.active_skill = None self.tools = self.build_tools() self.prefix = self.build_prefix() self.session_path = self.session_store.save(self.session) @@ -325,6 +463,13 @@ def build_tools(self): "description": "Ask a bounded read-only child agent to investigate.", "run": self.tool_delegate, } + if self.skills.skills: + tools["use_skill"] = { + "schema": {"name": "str"}, + "risky": False, + "description": "Load a listed skill's instructions when the task needs that skill.", + "run": self.tool_use_skill, + } return tools ############################################ @@ -364,11 +509,14 @@ def build_prefix(self): "- New files should be complete and runnable, including obvious imports.", "- Do not repeat the same tool call with the same arguments if it did not help. Choose a different tool or return a final answer.", "- Required tool arguments must not be empty. Do not call read_file, write_file, patch_file, run_shell, or delegate with args={}.", + "- If an Active skill section is present, follow its instructions step by step.", + "- If the task needs one of the listed skills and none is active, call use_skill with its name.", ]) return "\n\n".join([ "You are Mini-Coding-Agent, a small local coding agent running through Ollama.", "Rules:\n" + rules, "Tools:\n" + tool_text, + "Skills:\n" + self.skills.text(), "Valid response examples:\n" + examples, self.workspace.text(), ]) @@ -420,13 +568,68 @@ def history_text(self): #### 2) Prompt Shape And Cache Reuse (Continued) ####### ######################################################## def prompt(self, user_message): - return "\n\n".join([ - self.prefix, - self.memory_text(), + sections = [self.prefix, self.memory_text()] + if self.active_skill: + sections.append(self.skill_text()) + sections += [ "Transcript:\n" + self.history_text(), "Current user request:\n" + user_message, + ] + return "\n\n".join(sections) + + ############################## + #### 7) Agent Skills ######### + ############################## + def skill_text(self): + skill = self.active_skill + return "\n".join([ + f"Active skill: {skill.name}", + f"- source: {skill.path.relative_to(self.root)}", + "- instructions:", + clip(skill.body, 3000), ]) + def select_skill(self, user_message): + """Log the prompt, run the similarity check, and activate the best skill if it passes.""" + self.skill_log.write("prompt", session=self.session["id"], prompt=user_message) + skill, scores, trigger = self.skills.match(user_message) + self.skill_log.write( + "similarity_check", + session=self.session["id"], + scores=scores, + threshold=self.skills.threshold, + best=max(scores, key=scores.get) if scores else None, + selected=skill.name if skill else None, + ) + score_text = ", ".join(f"{name}={score}" for name, score in sorted(scores.items(), key=lambda kv: -kv[1])) + print(f"[skills] similarity: {score_text or 'no skills found'} (threshold {self.skills.threshold})", file=sys.stderr) + if skill: + self.activate_skill(skill, trigger, scores.get(skill.name)) + else: + print("[skills] no skill selected", file=sys.stderr) + return skill + + def activate_skill(self, skill, trigger, score=None): + self.active_skill = skill + self.remember(self.session["memory"]["notes"], f"active skill: {skill.name}", 5) + self.skill_log.write( + "skill_executed", + session=self.session["id"], + skill=skill.name, + trigger=trigger, + score=score, + path=str(skill.path.relative_to(self.root)), + ) + print(f"[skills] executing skill '{skill.name}' (trigger: {trigger})", file=sys.stderr) + + def tool_use_skill(self, args): + name = str(args.get("name", "")).strip() + skill = self.skills.skills.get(name) + if skill is None: + raise ValueError(f"unknown skill '{name}'") + self.activate_skill(skill, "tool") + return f"skill '{skill.name}' loaded. Follow these instructions:\n{skill.body}" + ############################################### #### 5) Session Memory (Continued) ########### ############################################### @@ -447,6 +650,7 @@ def ask(self, user_message): if not memory["task"]: memory["task"] = clip(user_message.strip(), 300) self.record({"role": "user", "content": user_message, "created_at": now()}) + self.select_skill(user_message) tool_steps = 0 attempts = 0 @@ -530,6 +734,7 @@ def tool_example(self, name): "write_file": 'def binary_search(nums, target):\n return -1\n', "patch_file": 'return -1return mid', "delegate": '{"name":"delegate","args":{"task":"inspect README.md","max_steps":3}}', + "use_skill": '{"name":"use_skill","args":{"name":"pr-review"}}', } return examples.get(name, "") @@ -599,6 +804,12 @@ def validate_tool(self, name, args): raise ValueError("task must not be empty") return + if name == "use_skill": + skill_name = str(args.get("name", "")).strip() + if skill_name not in self.skills.skills: + raise ValueError(f"unknown skill '{skill_name}'; choose one of: {', '.join(self.skills.skills)}") + return + def approve(self, name, args): if self.read_only: return False @@ -717,6 +928,7 @@ def extract_raw(text, tag): def reset(self): self.session["history"] = [] self.session["memory"] = {"task": "", "files": [], "notes": []} + self.active_skill = None self.session_store.save(self.session) def path_is_within_root(self, resolved): @@ -860,6 +1072,7 @@ def tool_delegate(self, args): depth=self.depth + 1, max_depth=self.max_depth, read_only=True, + skills=self.skills, ) child.session["memory"]["task"] = task child.session["memory"]["notes"] = [clip(self.history_text(), 300)] @@ -1003,6 +1216,10 @@ def main(argv=None): if user_input == "/session": print(agent.session_path) continue + if user_input == "/skills": + print("Skills:\n" + agent.skills.text()) + print(f"Log: {agent.skill_log.path}") + continue if user_input == "/reset": agent.reset() print("session reset") @@ -1016,4 +1233,4 @@ def main(argv=None): if __name__ == "__main__": - raise SystemExit(main()) + raise SystemExit(main()) \ No newline at end of file