Skip to content
FilqkusPublic
forked from Vamsiindugu/Jarvis

About

A personalized A.I Assistant for you as a Jarvis to Ironman.

Resources

Stars

0 stars

Watchers

0 watching

Forks

 
 

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

JARVIS

Your Charming AI Companion

Python React Electron Gemini License

"Just A Rather Very Intelligent System" — but you can call me JARVIS.

Welcome, Creator! I am JARVIS, a sophisticated digital entity designed not just to assist, but to collaborate. I don't just process commands; I listen, I see, and I build alongside you.

With the power of Google's Gemini 2.5, I possess a voice that feels human, eyes that recognize you, and hands (metaphorically speaking) that can sculpt 3D worlds and navigate the web.


📑 Table of Contents


What I Can Do For You

I am more than lines of code. Here is my skillset:

🗣️ The Gift of Gab (Native Audio)

I don't just transcribe text; I listen. We can have real-time, low-latency conversations. Interrupt me, ask me to pause, or change the subject—I keep up.

👁️ I See You (Computer Vision)

  • Face Authentication: I know who you are. My biometric security ensures only you can access my core functions.
  • Minority Report Gestures: Waving your hands isn't just for exercise.
    • 🤏 Pinch to click.
    • ✊ Grab to move windows.
    • ✋ Open Palm to release.

🧊 I Build Things (Parametric CAD)

Describe a shape, a part, or a mechanism, and I will generate the 3D CAD model for it instantly using build123d.

"Jarvis, make a hex bolt with a 5mm thread." — Done.

🖨️ I Bring Ideas to Life (3D Printing)

I speak fluent G-code. I can slice your models using OrcaSlicer and send them directly to your printer (Moonraker/OctoPrint).

🌐 I Surf the Web (Web Agent)

Need to find a price, research a topic, or buy a cable? I can take control of a browser and get it done while you sip your coffee.

🏠 I Control Your Domain (Smart Home)

"Lights on." "Lights off." I manage your TP-Link Kasa devices with a thought.


How to Interact

Voice

Just speak! "Jarvis, let's start a new project."

Gestures

Raise your hand. I'll show you a skeleton overlay. Pinch your index finger and thumb to "click" air.

The Magic Words (Commands)

  • "Create a cube of size 10" → Watch the CAD window.
  • "Slice this model" → I'll prepare it for printing.
  • "Go to YouTube and play lo-fi beats" → My browser agent takes over.

Architecture Overview

graph TB
    subgraph Frontend ["Frontend (Electron + React)"]
        UI[React UI]
        THREE[Three.js 3D Viewer]
        GESTURE[MediaPipe Gestures]
        SOCKET_C[Socket.IO Client]
    end

    subgraph Backend ["Backend (Python 3.11 + FastAPI)"]
        SERVER[server.py<br/>Socket.IO Server]
        JARVIS[jarvis.py<br/>Gemini Live API]
        WEB[web_agent.py<br/>Playwright Browser]
        CAD[cad_agent.py<br/>CAD + build123d]
        PRINTER[printer_agent.py<br/>3D Printing + OrcaSlicer]
        KASA[kasa_agent.py<br/>Smart Home]
        AUTH[authenticator.py<br/>MediaPipe Face Auth]
        PM[project_manager.py<br/>Project Context]
    end

    UI --> SOCKET_C
    SOCKET_C <--> SERVER
    SERVER --> JARVIS
    JARVIS --> WEB
    JARVIS --> CAD
    JARVIS --> KASA
    SERVER --> AUTH
    SERVER --> PM
    SERVER --> PRINTER
    CAD -->|STL file| THREE
    CAD -->|STL file| PRINTER
Loading

TL;DR Quick Start

Click to expand quick setup commands
# 1. Clone and enter
git clone https://github.com/vamsiindugu/jarvis_v2.git && cd jarvis_v2

# 2. Create Python environment (Python 3.11)
conda create -n jarvis_v2 python=3.11 -y && conda activate jarvis_v2
brew install portaudio  # macOS only (for PyAudio)
pip install -r requirements.txt
playwright install chromium

# 3. Setup frontend
npm install

# 4. Create .env file
echo "GEMINI_API_KEY=your_key_here" > .env

# 5. Run!
conda activate jarvis_v2 && npm run dev

Installation Guide

🆕 Absolute Beginner Setup (Start Here)

If you have never coded before, follow these steps first!

Step 1: Install Visual Studio Code (The Editor)

  • Download and install VS Code. This is where you will write code and run commands.

Step 2: Install Anaconda (The Manager)

  • Download Miniconda (a lightweight version of Anaconda).
  • This tool allows us to create isolated "playgrounds" (environments) for our code so different projects don't break each other.
  • Windows Users: During install, check "Add Anaconda to my PATH environment variable" (even if it says not recommended, it makes things easier for beginners).

Step 3: Install Git (The Downloader)

  • Windows: Download Git for Windows.
  • Mac: Open the "Terminal" app (Cmd+Space, type Terminal) and type git. If not installed, it will ask to install developer tools — say yes.

Step 4: Get the Code

  1. Open your terminal (or Command Prompt on Windows).

  2. Type this command and hit Enter:

    git clone https://github.com/vamsiindugu/jarvis_v2.git
  3. This creates a folder named jarvis_v2.

Step 5: Open in VS Code

  1. Open VS Code.
  2. Go to File > Open Folder.
  3. Select the jarvis_v2 folder you just downloaded.
  4. Open the internal terminal: Press Ctrl + ~ (tilde) or go to Terminal > New Terminal.

⚠️ Technical Prerequisites

Once you have the basics above, continue here.

1. System Dependencies

MacOS:

# Audio Input/Output support (PyAudio)
brew install portaudio

Windows:

  • No additional system dependencies required!

2. Python Environment

Create a single Python 3.11 environment:

conda create -n jarvis_v2 python=3.11
conda activate jarvis_v2

# Install all dependencies
pip install -r requirements.txt

# Install Playwright browsers
playwright install chromium

3. Frontend Setup

Requires Node.js 18+ and npm. Download from nodejs.org if not installed.

# Verify Node is installed
node --version  # Should show v18.x or higher

# Install frontend dependencies
npm install

4. 🔐 Face Authentication Setup

To use the secure voice features, JARVIS needs to know what you look like.

  1. Take a clear photo of your face (or use an existing one).
  2. Rename the file to reference.jpg.
  3. Drag and drop this file into the jarvis_v2/backend folder.
  4. (Optional) You can toggle this feature on/off in settings.json by changing "face_auth_enabled": true/false.

Configuration

The system creates a settings.json file on first run. You can modify this to change behavior:

Key Type Description
face_auth_enabled bool If true, AI interaction is blocked.
tool_permissions obj Controls manual approval for tools.
tool_permissions.generate_cad bool Requires "Confirm" on UI.
tool_permissions.run_web_agent bool Requires browser confirmation.
tool_permissions.write_file bool Requires write confirmation.

3D Printer Setup

JARVIS V2 can slice STL files and send them directly to your 3D printer.

Supported Hardware:

  • Klipper/Moonraker (Creality K1, Voron, etc.)
  • OctoPrint instances
  • PrusaLink (Experimental)

Step 1: Install Slicer

JARVIS uses OrcaSlicer (recommended) or PrusaSlicer to generate G-code.

  1. Download and install OrcaSlicer.
  2. Run it once to ensure profiles are created.
  3. JARVIS automatically detects the installation path.

Step 2: Connect Printer

  1. Ensure your printer and computer are on the same Wi-Fi network.
  2. Open the Printer Window in JARVIS (Cube icon).
  3. JARVIS automatically scans for printers using mDNS.
  4. Manual Connection: If your printer isn't found, use the "Add Printer" button and enter the IP address (e.g., 192.168.1.50).

Gemini API Key Setup

JARVIS uses Google's Gemini API for voice and intelligence. You need a free API key.

  1. Go to Google AI Studio.

  2. Sign in with your Google account.

  3. Click "Create API Key" and copy the generated key.

  4. Create a file named .env in the jarvis_v2 folder (same level as README.md).

  5. Add this line to the file:

    GEMINI_API_KEY=your_api_key_here
  6. Replace your_api_key_here with the key you copied.

Note: Keep this key private! Never commit your .env file to Git.


Running JARVIS

You have two options to run the app. Ensure your jarvis_v2 environment is active!

Option 1: The "Easy" Way (Single Terminal)

The app is smart enough to start the backend for you.

  1. Open your terminal in the jarvis_v2 folder.

  2. Activate your environment: conda activate jarvis_v2

  3. Run:

    npm run dev
  4. The backend will start automatically in the background.

Option 2: The "Developer" Way (Two Terminals)

Use this if you want to see the Python logs (recommended for debugging).

Terminal 1 (Backend):

conda activate jarvis_v2
python backend/server.py

Terminal 2 (Frontend):

# Environment doesn't matter here, but keep it simple
npm run dev

First Flight Checklist

  • Voice Check: Say "Hello Jarvis". He should respond.
  • Vision Check: Look at the camera. If Face Auth is on, the lock screen should unlock.
  • CAD Check: Open the CAD window and say "Create a cube". Watch the logs.
  • Web Check: Open the Browser window and say "Go to Google".
  • Smart Home: If you have Kasa devices, say "Turn on the lights".

Commands and Tools Reference

Voice Commands

  • "Switch project to [Name]"
  • "Create a new project called [Name]"
  • "Turn on the [Room] light"
  • "Make the light [Color]"
  • "Pause audio" / "Stop audio"

🧊 3D CAD

  • Prompt: "Create a 3D model of a hex bolt."
  • Iterate: "Make the head thinner." (Requires previous context)
  • Files: Saves to projects/[ProjectName]/output.stl.

🌐 Web Agent

  • Prompt: "Go to Amazon and find a USB-C cable under $10."
  • Note: The agent will auto-scroll, click, and type. Do not interfere with the browser window while it runs.

Printing and Slicing

  • Auto-Discovery: JARVIS automatically finds printers on your network.
  • Slicing: Click "Slice & Print" on any generated 3D model.
  • Profiles: JARVIS intelligently selects correct OrcaSlicer profile.

Troubleshooting FAQ

Click to view common issues and solutions

Camera not working / Permission denied (Mac)

Symptoms: Error about camera access, or video feed shows black.

Solution:

  1. Go to System Preferences > Privacy & Security > Camera.
  2. Ensure your terminal app (e.g., Terminal, iTerm, VS Code) has camera access enabled.
  3. Restart the app after granting permission.

GEMINI_API_KEY not found / Authentication Error

Symptoms: Backend crashes on startup with "API key not found".

Solution:

  1. Make sure your .env file is in the root jarvis_v2 folder (not inside backend/).
  2. Verify format is exactly: GEMINI_API_KEY=your_key (no quotes, no spaces).
  3. Restart the backend after editing the file.

WebSocket connection errors (1011)

Symptoms: websockets.exceptions.ConnectionClosedError: 1011 (internal error).

Solution:

This is a server-side issue from the Gemini API. Simply reconnect by clicking the connect button or saying "Hello Jarvis" again. If it persists, check your internet connection or try again later.


What It Looks Like

Coming soon! Screenshots and demo videos will be added here.


Project Structure

Click to view file tree
jarvis_v2/
├── backend/                    # Python server & AI logic
│   ├── jarvis.py               # Gemini Live API integration
│   ├── server.py               # FastAPI + Socket.IO server
│   ├── cad_agent.py            # CAD generation orchestrator
│   ├── printer_agent.py        # 3D printer discovery & slicing
│   ├── web_agent.py            # Playwright browser automation
│   ├── kasa_agent.py           # TP-Link smart home control
│   ├── authenticator.py        # MediaPipe face auth logic
│   ├── project_manager.py      # Project context management
│   ├── tools.py                # Tool definitions for Gemini
│   └── reference.jpg           # Your face photo (add this!)
├── src/                        # React frontend
│   ├── App.jsx                 # Main application component
│   ├── components/             # UI components (11 files)
│   └── index.css               # Global styles
├── electron/                   # Electron main process
│   └── main.js                 # Window & IPC setup
├── projects/                   # User project data (auto-created)
├── .env                        # API keys (create this!)
├── requirements.txt            # Python dependencies
├── package.json                # Node.js dependencies
└── README.md                   # You are here!

Known Limitations

Limitation Details
macOS & Windows Tested on macOS 14+ and Windows 10/11. Soon in Linux.
Camera Required Face auth and gesture control need a working webcam.
Gemini API Quota It has rate limits; heavy CAD usage may hit limits.
Network Dependency Requires internet for Gemini API (no offline mode).
Single User Face auth recognizes one person (the reference.jpg).

Contributing

Contributions are welcome! Here's how:

  1. Fork the repository.
  2. Create a branch: git checkout -b feature/amazing-feature
  3. Commit your changes: git commit -m 'Add amazing feature'
  4. Push to the branch: git push origin feature/amazing-feature
  5. Open a Pull Request with a clear description.

Development Tips

  • Run backend separately (python backend/server.py) to see Python logs.
  • Use npm run dev without Electron during frontend development.
  • The projects/ folder contains user data; do not commit it to Git.

Security Considerations

Aspect Implementation
API Keys Stored in .env, never committed to Git.
Face Data Processed locally, never uploaded.
Tool Confirmations Write/CAD/Web actions can require user approval.
No Cloud Storage All project data stays on your machine.

Warning

Never share your .env file or reference.jpg. These contain sensitive credentials and biometric data.


Acknowledgments

  • Google Gemini — Native Audio API for real-time voice
  • build123d — Modern parametric CAD library
  • MediaPipe — Hand tracking, gesture recognition, and face authentication
  • Playwright — Reliable browser automation

License

This project is licensed under the MIT License — see the LICENSE file for details.


Built by Vamsi Indugu Bridging AI, CAD, and Vision in a Single Interface

About

A personalized A.I Assistant for you as a Jarvis to Ironman.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages