Create initial Dev Docs

.github/workflows/wiki.yml

@@ -0,0 +1,27 @@
+name: Publish Wiki
+
+on:
+  push:
+    branches: [develop]
+    paths:
+      - 'docs/**'
+  workflow_dispatch:
+
+concurrency:
+  group: publish-wiki
+  cancel-in-progress: true
+
+permissions:
+  contents: write
+
+jobs:
+  publish-wiki:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Sync docs to GitHub Wiki
+        uses: Andrew-Chen-Wang/github-wiki-action@v5
+        with:
+          path: docs/

docs/Getting-Started.md

@@ -0,0 +1,134 @@
+# Getting Started with API Tools
+
+This guide walks you through setting up your local development environment, configuring environment variables, compiling the CLI binary, and running tests.
+
+## Prerequisites
+
+Ensure you have the following installed:
+
+- Git - For version control
+  - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop).
+- Go - For building and running
+  - You can install from the [Go website](https://go.dev/dl/), or from [Homebrew](https://brew.sh/) or another package manager for automatic updates
+- Make _(Linux/macOS)_ - For build automation
+  - Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`)
+- Google Chrome or Chromium - For headless browser scraping
+- Docker _(Optional)_ - For running containerized runners locally
+
+**Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run.
+
+If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file. When you see any command starting with `make`, you can instead use `.\build.bat`.
+For example instead of `make setup`, you can run `.\build.bat setup`.
+
+## Local Setup
+
+### Clone the Repository
+
+Clone the repository with `git clone` and `cd` into the project directory or open it in your code editor
+
+**HTTPS:**
+
+```bash
+git clone https://github.com/UTDNebula/api-tools.git
+cd api-tools
+```
+
+or
+
+**SSH:**
+
+```bash
+git clone git@github.com:UTDNebula/api-tools.git
+cd api-tools
+```
+
+Now open the cloned project in your code editor!
+
+### Install Development Tooling
+
+Install the Go static analysis and formatting tools (`staticcheck` and `goimports`):
+
+```bash
+make setup
+```
+
+or
+
+```cmd
+.\build.bat setup
+```
+
+If you installed **Go** with [**Homebrew**](https://brew.sh/), you need to add Go tools to your path for tools like `staticcheck` and others to work.
+
+- **For zsh** (the default shell on MacOS)
+
+  ```bash
+  echo 'export PATH=${PATH}:`go env GOPATH`/bin' >> ~/.zshrc && source ~/.zshrc
+  ```
+
+- **For bash** (the default shell on most Linux distributions)
+
+  ```bash
+  echo 'export PATH=${PATH}:`go env GOPATH`/bin' >> ~/.bashrc && source ~/.bashrc
+  ```
+
+- **For fish**
+
+  ```bash
+  echo 'fish_add_path (go env GOPATH)/bin' >> ~/.config/fish/config.fish
+  ```
+
+### Configure Environment Variables
+
+Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env`
+If you're not sure what to put, please try contacting one of our members through discord.
+
+### Run Code Verification & Formatting
+
+Check your code with:
+
+```bash
+make check
+```
+
+or
+
+```cmd
+.\build.bat check
+```
+
+You'll want to run this frequently while developing
+
+### Build the CLI Executable
+
+Compile the Go source code into a runnable binary (`./api-tools` on Linux/macOS, `api-tools.exe` on Windows):
+
+```bash
+make build
+```
+
+or
+
+```cmd
+.\build.bat build
+```
+
+### Verify with Automated Tests
+
+Run the test suite to confirm your environment is ready:
+
+```bash
+make test
+```
+
+or
+
+```cmd
+.\build.bat test
+```
+
+If all tests pass, congratulations! Your local development environment is ready.
+
+## Next Step
+
+Now that your environment is running, it's time to dive deeper. Check out [Project-Architecture.md](Project-Architecture.md)

docs/Home.md

@@ -0,0 +1,17 @@
+# API Tools Developer Documentation
+
+Welcome to the **API Tools Developer Documentation**!
+
+These docs are intended for developers of `api-tools` rather than users. For usage instructions, see [README.md](../README.md).
+
+If you are a new developer, check out the pages under "Getting Started" in the Sidebar!
+
+## Need Help?
+
+Join [our community on Discord](https://discord.utdnebula.com) and ask questions in the `#api-help` or development channels. We are always happy to help!
+
+## Found something you want to change on these docs?
+
+Don't edit these docs manually on Github. Instead, suggest a change by [creating an issue](../issues), or [opening a pull request](../pulls) with your changes.
+
+These docs are automatically generated from the `docs/` folder of this repository.

docs/How-to-Contribute.md

@@ -0,0 +1,60 @@
+# How to Contribute
+
+Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more.
+Don't worry if you don't quite know what you're doing, we're here to help. We don't expect perfection and appreciate anything you can do to help!
+
+## Find something to work on
+
+Look at our current issues, or make your own on our [issues page](../issues).
+
+For beginner issues, please write the code yourself rather than using AI to generate it. These issues are designed to help new members learn the codebase and build familiarity with the project. We allow AI in Nebula Labs, but review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy) for more information on acceptable uses of AI.
+
+Once you find an issue, write a comment asking if you can work on it.
+
+If you're unsure about something, **don't hesitate to ask for help**. We're happy to answer questions and help you work through issues. You can reach out on GitHub, Discord, or in person.
+
+## Create a branch
+
+Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork as they do not have permission to make a branch.
+
+## Make Your Changes
+
+It's time to code!
+
+Don't forget to format and check your code. We have helper scripts with **Make** and **build.bat**:
+
+Check your code with
+
+```bash
+make check
+```
+
+or
+
+```cmd
+.\build.bat check
+```
+
+Run tests with
+
+```bash
+make test
+```
+
+or
+
+```cmd
+.\build.bat test
+```
+
+We would appreciate if you make a draft Pull Request as your working on it, so we can see your progress and help you out!
+
+## Making a Pull Request (PR)
+
+Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft.
+
+Maintainers will review your PR and may provide suggestions or request changes before approval.
+
+## Next Step
+
+You have everything you need to start, but feel free to check out other pages first.

docs/Project-Architecture.md

@@ -0,0 +1,74 @@
+# Project-Architecture
+
+`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded to the Nebula API.
+
+The data comes from various UTD data sources including courses, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them.
+
+Tools are written in Go. If you're new to Go (or just rusty), two good places to start are [A Tour of Go](https://go.dev/tour/list), which is interactive, runs in your browser, and needs no setup,
+and [Effective Go](https://go.dev/doc/effective_go), which covers how to write Go the idiomatic way and is worth skimming before diving into `scrapers/`, `parser/`, or `uploader/`.
+
+## Project Pipeline
+
+Data moves through `api-tools` in three stages:
+
+```mermaid
+flowchart LR
+    A[UTD Data Sources<br/>Websites, APIs] --> B
+
+    subgraph STAGE["Three Stages"]
+        direction LR
+        B[Scraper] --> C[Parser] --> D[Uploader]
+    end
+
+    D --> E[(Nebula API Database)]
+```
+
+### Scrapers (`scrapers/`)
+
+Scrapers connect to websites, APIs, and portals to download raw information. They capture raw data in `.html`, `.json`, or other formats, and store it in the `data/` directory. For example, a scrapped course would be found at `data/24f/cp_cs/cs1337.001.24f.html` after running the scraper.
+
+For most scraping we use Go's built-in `net/http` library to directly make API requests and get our data. Sometimes we need a browser to recreate more complex actions. For automating browser actions, we use [ChromeDP](https://github.com/chromedp/chromedp) which uses Chrome DevTools. For extra scraping functionality not already covered by ChromeDP, we use [cdproto](https://github.com/chromedp/cdproto).
+
+### Parsers (`parser/`)
+
+Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then convert this data into a more machine-readable format.
+
+For HTML documents like courses, sections, professors, degree requirements, and discounts, we use [goquery](https://github.com/PuerkitoBio/goquery). Goquery lets us navigate our `.html` files programmatically with selectors. For more unstructured documents like calendars and budgets, we use [Google Gemini](https://pkg.go.dev/google.golang.org/genai).
+
+Everything gets structured and validated against the data models in [nebula-api/api](https://github.com/UTDNebula/nebula-api) (`api/schema`) via `parser/validator.go`.
+
+### Uploaders (`uploader/`)
+
+Uploaders take parsed data and upload it to the Nebula API MongoDB database.
+
+We use the official [MongoDB driver](https://pkg.go.dev/go.mongodb.org/mongo-driver) to connect to the database, turn Go structs into BSON documents, write them in bulk, and run aggregation queries. The shared schema definitions come from [nebula-api/api](https://github.com/UTDNebula/nebula-api), so what we store always matches what the API expects to find.
+
+## Cloud Automation
+
+Most data sources are updated automatically through shell scripts in `runners/`. These scripts run in containerized environments in Google Cloud.
+
+We use [Docker](https://www.docker.com/) to create a consistent environment to run `api-tools` in the cloud. We use [Google Cloud Build](https://cloud.google.com/build) to build our docker image, and [Google Cloud Scheduler](https://docs.cloud.google.com/scheduler/docs) to schedule our cloud pipeline runs.
+
+## Other Important Tools
+
+We use several tools while developing the API.
+
+Tests use Go's standard `testing` package for unit tests across the project.
+
+We use [Sentry](https://github.com/getsentry/sentry-go) for error tracking.
+
+We use [GitHub Actions](https://github.com/features/actions) to check Go code for PRs, deploy our Docker image, and deploy this Wiki!
+
+## Supporting Libraries
+
+We use several supporting libraries for more minor things. Here is a list of most of them, and their purpose
+
+- [fastjson](https://github.com/valyala/fastjson) - lets us read huge JSON payloads in scrapers without loading the whole thing into memory at once. It walks through the data piece by piece, which is how we handle things like the Astra room scheduling events.
+- [dongri/phonenumber](https://github.com/dongri/phonenumber) - cleans up phone numbers from student discount programs. They show up in every format imaginable, so this gives us one consistent shape to store.
+- [golang.org/x/text](https://pkg.go.dev/golang.org/x/text) - handles Unicode transformations and casing rules, which keeps event names and titles from getting mangled when they contain non-English characters.
+- [google/go-cmp](https://github.com/google/go-cmp) - compares structs for our unit and regression tests, so when something breaks we can see exactly which field changed instead of just "not equal."
+- [godotenv](https://github.com/joho/godotenv) - reads our `.env` files and loads things like URIs and credentials when the program starts, so we don't have to hardcode secrets or pass them on the command line.
+
+## Next Step
+
+See [Project-Structure.md](Project-Structure.md)

docs/Project-Structure.md

@@ -0,0 +1,43 @@
+# Project Structure & Codebase Map
+
+This document provides an overview of the `api-tools` directory layout, explaining what each folder contains and guiding you on where to make changes when adding features or fixing bugs.
+
+## Project Structure
+
+```text
+api-tools/
+├── data/                       # Files from scrapers and parsers
+├── docs/                       # Source of wiki pages
+├── logs/                       # Timestamped log files from runtime executions
+├── parser/                     # Parsing, data normalization, and schema validation
+│   ├── parser.go               # Central parsing orchestrator
+│   ├── <domain>Parser.go       # Domain-specific parsers (courses, sections, maps, ASTRA, etc.)
+│   ├── gradeLoader.go          # Loads historical grade distributions
+│   ├── requisiteParser.go      # Parses prerequisite/corequisite strings
+│   ├── validator.go            # Validates parsed structs before upload
+│   └── testdata/               # HTML fixtures and expected JSON for regression tests
+├── runners/                    # Shell automation scripts run on Google Cloud
+├── scrapers/                   # Web scrapers for UTD websites, portals, and APIs
+├── static-data/                # Static data that isn't scraped
+│   ├── grades/                 # Historical grade-distribution CSVs by semester
+│   └── budgets/                # Historical budget PDFs
+├── uploader/                   # Database ingestion modules for Nebula API / MongoDB
+│   ├── database.go             # Establishes MongoDB connections
+│   ├── uploader.go             # Coursebook uploads and merge/replace strategies
+│   └── <domain>Uploader.go     # Domain-specific uploaders
+├── utils/                      # Shared utilities
+│   ├── methods.go              # ChromeDP, environment variables, headless mode, tokens
+│   ├── logger.go               # Runtime logging to stdout and timestamped files
+│   └── regexes.go              # Shared parsing and extraction regexes
+├── .env.template               # Template for required environment variables
+├── Dockerfile                  # Multi-stage container definition
+├── Makefile                    # Build, check, and test tasks for Linux/macOS
+├── build.bat                   # Build, check, and test tasks for Windows
+├── cloudbuild.yaml             # Google Cloud Build deployment configuration
+├── go.mod / go.sum             # Go dependency manifests
+└── main.go                     # Application entrypoint and CLI flag definitions
+```
+
+## Next Step
+
+See [How-to-Contribute.md](How-to-Contribute.md)

docs/Troubleshooting.md

@@ -0,0 +1,27 @@
+# Troubleshooting
+
+This guide gives solutions to common problems developers or users of this project may run into.
+
+## Enabling Verbose Debug Logging
+
+If a scraper or parser is behaving unexpectedly, re-run the command with the `-verbose` flag:
+
+For example:
+
+```bash
+./api-tools -verbose -scrape -coursebook -term 24F
+```
+
+This will give more logs, and can help you better understand what's going on.
+
+## "exec: 'chromium': executable file not found in $PATH" or Chrome fails to launch
+
+ChromeDP cannot find a Chromium or Google Chrome binary on your operating system. You'll need to install it.
+
+## KEY is missing from .env
+
+Ensure you have filled environment variables in .env. If you need help, ask for help!
+
+## Still Stuck?
+
+Ask for help on [Discord](https://discord.utdnebula.com)!