"Just A Rather Very Intelligent System" — but you can call me JARVIS.
Welcome, Creator! I am JARVIS, a sophisticated digital entity designed not just to assist, but to collaborate. I don't just process commands; I listen, I see, and I build alongside you.
With the power of Google's Gemini 2.5, I possess a voice that feels human, eyes that recognize you, and hands (metaphorically speaking) that can sculpt 3D worlds and navigate the web.
- ✨ What I Can Do For You
- 🎮 How to Interact
- 🏗️ Architecture Overview
- ⚡ TL;DR Quick Start
- 🛠️ Installation Guide
- ⚙️ Configuration
- 🖨️ 3D Printer Setup
- 🔑 Gemini API Key Setup
- 🚀 Running JARVIS
- ✅ First Flight Checklist
▶️ Commands and Tools Reference- ❓ Troubleshooting FAQ
I am more than lines of code. Here is my skillset:
I don't just transcribe text; I listen. We can have real-time, low-latency conversations. Interrupt me, ask me to pause, or change the subject—I keep up.
- Face Authentication: I know who you are. My biometric security ensures only you can access my core functions.
- Minority Report Gestures: Waving your hands isn't just for exercise.
- 🤏 Pinch to click.
- ✊ Grab to move windows.
- ✋ Open Palm to release.
Describe a shape, a part, or a mechanism, and I will generate the 3D CAD model for it instantly using build123d.
"Jarvis, make a hex bolt with a 5mm thread." — Done.
I speak fluent G-code. I can slice your models using OrcaSlicer and send them directly to your printer (Moonraker/OctoPrint).
Need to find a price, research a topic, or buy a cable? I can take control of a browser and get it done while you sip your coffee.
"Lights on." "Lights off." I manage your TP-Link Kasa devices with a thought.
Just speak! "Jarvis, let's start a new project."
Raise your hand. I'll show you a skeleton overlay. Pinch your index finger and thumb to "click" air.
- "Create a cube of size 10" → Watch the CAD window.
- "Slice this model" → I'll prepare it for printing.
- "Go to YouTube and play lo-fi beats" → My browser agent takes over.
graph TB
subgraph Frontend ["Frontend (Electron + React)"]
UI[React UI]
THREE[Three.js 3D Viewer]
GESTURE[MediaPipe Gestures]
SOCKET_C[Socket.IO Client]
end
subgraph Backend ["Backend (Python 3.11 + FastAPI)"]
SERVER[server.py<br/>Socket.IO Server]
JARVIS[jarvis.py<br/>Gemini Live API]
WEB[web_agent.py<br/>Playwright Browser]
CAD[cad_agent.py<br/>CAD + build123d]
PRINTER[printer_agent.py<br/>3D Printing + OrcaSlicer]
KASA[kasa_agent.py<br/>Smart Home]
AUTH[authenticator.py<br/>MediaPipe Face Auth]
PM[project_manager.py<br/>Project Context]
end
UI --> SOCKET_C
SOCKET_C <--> SERVER
SERVER --> JARVIS
JARVIS --> WEB
JARVIS --> CAD
JARVIS --> KASA
SERVER --> AUTH
SERVER --> PM
SERVER --> PRINTER
CAD -->|STL file| THREE
CAD -->|STL file| PRINTER
Click to expand quick setup commands
# 1. Clone and enter
git clone https://github.com/vamsiindugu/jarvis_v2.git && cd jarvis_v2
# 2. Create Python environment (Python 3.11)
conda create -n jarvis_v2 python=3.11 -y && conda activate jarvis_v2
brew install portaudio # macOS only (for PyAudio)
pip install -r requirements.txt
playwright install chromium
# 3. Setup frontend
npm install
# 4. Create .env file
echo "GEMINI_API_KEY=your_key_here" > .env
# 5. Run!
conda activate jarvis_v2 && npm run devIf you have never coded before, follow these steps first!
- Download and install VS Code. This is where you will write code and run commands.
- Download Miniconda (a lightweight version of Anaconda).
- This tool allows us to create isolated "playgrounds" (environments) for our code so different projects don't break each other.
- Windows Users: During install, check "Add Anaconda to my PATH environment variable" (even if it says not recommended, it makes things easier for beginners).
- Windows: Download Git for Windows.
- Mac: Open the "Terminal" app (Cmd+Space, type Terminal) and type
git. If not installed, it will ask to install developer tools — say yes.
-
Open your terminal (or Command Prompt on Windows).
-
Type this command and hit Enter:
git clone https://github.com/vamsiindugu/jarvis_v2.git
-
This creates a folder named
jarvis_v2.
- Open VS Code.
- Go to File > Open Folder.
- Select the
jarvis_v2folder you just downloaded. - Open the internal terminal: Press
Ctrl + ~(tilde) or go to Terminal > New Terminal.
Once you have the basics above, continue here.
MacOS:
# Audio Input/Output support (PyAudio)
brew install portaudioWindows:
- No additional system dependencies required!
Create a single Python 3.11 environment:
conda create -n jarvis_v2 python=3.11
conda activate jarvis_v2
# Install all dependencies
pip install -r requirements.txt
# Install Playwright browsers
playwright install chromiumRequires Node.js 18+ and npm. Download from nodejs.org if not installed.
# Verify Node is installed
node --version # Should show v18.x or higher
# Install frontend dependencies
npm installTo use the secure voice features, JARVIS needs to know what you look like.
- Take a clear photo of your face (or use an existing one).
- Rename the file to
reference.jpg. - Drag and drop this file into the
jarvis_v2/backendfolder. - (Optional) You can toggle this feature on/off in
settings.jsonby changing"face_auth_enabled": true/false.
The system creates a settings.json file on first run. You can modify this to change behavior:
| Key | Type | Description |
|---|---|---|
face_auth_enabled |
bool |
If true, AI interaction is blocked. |
tool_permissions |
obj |
Controls manual approval for tools. |
tool_permissions.generate_cad |
bool |
Requires "Confirm" on UI. |
tool_permissions.run_web_agent |
bool |
Requires browser confirmation. |
tool_permissions.write_file |
bool |
Requires write confirmation. |
JARVIS V2 can slice STL files and send them directly to your 3D printer.
Supported Hardware:
- Klipper/Moonraker (Creality K1, Voron, etc.)
- OctoPrint instances
- PrusaLink (Experimental)
JARVIS uses OrcaSlicer (recommended) or PrusaSlicer to generate G-code.
- Download and install OrcaSlicer.
- Run it once to ensure profiles are created.
- JARVIS automatically detects the installation path.
- Ensure your printer and computer are on the same Wi-Fi network.
- Open the Printer Window in JARVIS (Cube icon).
- JARVIS automatically scans for printers using mDNS.
- Manual Connection: If your printer isn't found, use the "Add Printer" button and enter the IP address (e.g.,
192.168.1.50).
JARVIS uses Google's Gemini API for voice and intelligence. You need a free API key.
-
Go to Google AI Studio.
-
Sign in with your Google account.
-
Click "Create API Key" and copy the generated key.
-
Create a file named
.envin thejarvis_v2folder (same level asREADME.md). -
Add this line to the file:
GEMINI_API_KEY=your_api_key_here
-
Replace
your_api_key_herewith the key you copied.
Note: Keep this key private! Never commit your
.envfile to Git.
You have two options to run the app. Ensure your jarvis_v2 environment is active!
The app is smart enough to start the backend for you.
-
Open your terminal in the
jarvis_v2folder. -
Activate your environment:
conda activate jarvis_v2 -
Run:
npm run dev
-
The backend will start automatically in the background.
Use this if you want to see the Python logs (recommended for debugging).
Terminal 1 (Backend):
conda activate jarvis_v2
python backend/server.pyTerminal 2 (Frontend):
# Environment doesn't matter here, but keep it simple
npm run dev- Voice Check: Say "Hello Jarvis". He should respond.
- Vision Check: Look at the camera. If Face Auth is on, the lock screen should unlock.
- CAD Check: Open the CAD window and say "Create a cube". Watch the logs.
- Web Check: Open the Browser window and say "Go to Google".
- Smart Home: If you have Kasa devices, say "Turn on the lights".
- "Switch project to [Name]"
- "Create a new project called [Name]"
- "Turn on the [Room] light"
- "Make the light [Color]"
- "Pause audio" / "Stop audio"
- Prompt: "Create a 3D model of a hex bolt."
- Iterate: "Make the head thinner." (Requires previous context)
- Files: Saves to
projects/[ProjectName]/output.stl.
- Prompt: "Go to Amazon and find a USB-C cable under $10."
- Note: The agent will auto-scroll, click, and type. Do not interfere with the browser window while it runs.
- Auto-Discovery: JARVIS automatically finds printers on your network.
- Slicing: Click "Slice & Print" on any generated 3D model.
- Profiles: JARVIS intelligently selects correct OrcaSlicer profile.
Click to view common issues and solutions
Symptoms: Error about camera access, or video feed shows black.
Solution:
- Go to System Preferences > Privacy & Security > Camera.
- Ensure your terminal app (e.g., Terminal, iTerm, VS Code) has camera access enabled.
- Restart the app after granting permission.
Symptoms: Backend crashes on startup with "API key not found".
Solution:
- Make sure your
.envfile is in the rootjarvis_v2folder (not insidebackend/). - Verify format is exactly:
GEMINI_API_KEY=your_key(no quotes, no spaces). - Restart the backend after editing the file.
Symptoms: websockets.exceptions.ConnectionClosedError: 1011 (internal error).
Solution:
This is a server-side issue from the Gemini API. Simply reconnect by clicking the connect button or saying "Hello Jarvis" again. If it persists, check your internet connection or try again later.
Coming soon! Screenshots and demo videos will be added here.
Click to view file tree
jarvis_v2/
├── backend/ # Python server & AI logic
│ ├── jarvis.py # Gemini Live API integration
│ ├── server.py # FastAPI + Socket.IO server
│ ├── cad_agent.py # CAD generation orchestrator
│ ├── printer_agent.py # 3D printer discovery & slicing
│ ├── web_agent.py # Playwright browser automation
│ ├── kasa_agent.py # TP-Link smart home control
│ ├── authenticator.py # MediaPipe face auth logic
│ ├── project_manager.py # Project context management
│ ├── tools.py # Tool definitions for Gemini
│ └── reference.jpg # Your face photo (add this!)
├── src/ # React frontend
│ ├── App.jsx # Main application component
│ ├── components/ # UI components (11 files)
│ └── index.css # Global styles
├── electron/ # Electron main process
│ └── main.js # Window & IPC setup
├── projects/ # User project data (auto-created)
├── .env # API keys (create this!)
├── requirements.txt # Python dependencies
├── package.json # Node.js dependencies
└── README.md # You are here!
| Limitation | Details |
|---|---|
| macOS & Windows | Tested on macOS 14+ and Windows 10/11. Soon in Linux. |
| Camera Required | Face auth and gesture control need a working webcam. |
| Gemini API Quota | It has rate limits; heavy CAD usage may hit limits. |
| Network Dependency | Requires internet for Gemini API (no offline mode). |
| Single User | Face auth recognizes one person (the reference.jpg). |
Contributions are welcome! Here's how:
- Fork the repository.
- Create a branch:
git checkout -b feature/amazing-feature - Commit your changes:
git commit -m 'Add amazing feature' - Push to the branch:
git push origin feature/amazing-feature - Open a Pull Request with a clear description.
- Run backend separately (
python backend/server.py) to see Python logs. - Use
npm run devwithout Electron during frontend development. - The
projects/folder contains user data; do not commit it to Git.
| Aspect | Implementation |
|---|---|
| API Keys | Stored in .env, never committed to Git. |
| Face Data | Processed locally, never uploaded. |
| Tool Confirmations | Write/CAD/Web actions can require user approval. |
| No Cloud Storage | All project data stays on your machine. |
Warning
Never share your .env file or reference.jpg. These contain sensitive credentials and biometric data.
- Google Gemini — Native Audio API for real-time voice
- build123d — Modern parametric CAD library
- MediaPipe — Hand tracking, gesture recognition, and face authentication
- Playwright — Reliable browser automation
This project is licensed under the MIT License — see the LICENSE file for details.
Built by Vamsi Indugu Bridging AI, CAD, and Vision in a Single Interface