From 5bca445848d27bf2d3c7e9396aa5f1a8c33724c2 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 2 Sep 2026 15:34:40 -0500 Subject: [PATCH 01/42] Add initial docs --- README.md | 189 ++++++++++++++++++++--------------- docs/Docker-Guide.md | 90 +++++++++++++++++ docs/Getting-Started.md | 92 +++++++++++++++++ docs/Guides.md | 27 +++++ docs/Home.md | 25 +++++ docs/How-to-Contribute.md | 66 ++++++++++++ docs/Project-Architecture.md | 100 ++++++++++++++++++ docs/Project-Structure.md | 62 ++++++++++++ docs/Troubleshooting.md | 96 ++++++++++++++++++ 9 files changed, 667 insertions(+), 80 deletions(-) create mode 100644 docs/Docker-Guide.md create mode 100644 docs/Getting-Started.md create mode 100644 docs/Guides.md create mode 100644 docs/Home.md create mode 100644 docs/How-to-Contribute.md create mode 100644 docs/Project-Architecture.md create mode 100644 docs/Project-Structure.md create mode 100644 docs/Troubleshooting.md diff --git a/README.md b/README.md index d78eae0..1a9bb60 100644 --- a/README.md +++ b/README.md @@ -1,116 +1,145 @@ # API Tools -_A CLI to scrape some really useful UTD data, parse it, and upload it to the [Nebula API](https://github.com/utdnebula/nebula-api) for community use._ +_A CLI toolchain to scrape, parse, validate, and upload UTD university data to the [Nebula API](https://github.com/utdnebula/nebula-api) database._ Project maintained by [Nebula Labs](https://about.utdnebula.com). -### Design +> 📚 **Developer Wiki**: If you are interested in developing this project, head over to the [**Developer Wiki**](docs/Home.md)! -#### - The `scrapers` directory contains the scrapers for various UTD data sources. This is where the data pipeline begins. - - The scrapers are concerned solely with data collection, not necessarily validation or processing of said data. Those responsibilities are left to the parsing stage. -#### - The `parser` directory contains the files and methods that parse the scraped data. This is the 'middle man' of the data pipeline. - - The parsing stage is responsible for 'making sense' of the scraped data; this consists of reading, validating, and merging/intermixing of various data sources. - - The input data is considered **immutable** by the parsing stage. This means the parsers should never modify the data being fed into them. -#### - The `uploader` directory contains the uploader that sends the parsed data to the Nebula API MongoDB database. This is the final stage of the data pipeline. - - The uploader(s) are concerned solely with pushing parsed data to the database. Data, at this point, is assumed to be valid and ready for use. -#### - The `static-data/grades` directory contains .csv files of UTD grade data. - - Files are named by year and semester, with a suffix of `S`, `U`, or `F` denoting Spring, Summer, and Fall semesters, respectively. - - This means that, for example, `22F.csv` corresponds to the 2022 Fall semester, whereas `18U.csv` corresponds with the 2018 Summer semester. - - This grade data is collected independently from the scrapers, and is used during the parsing process. -#### - The `static-data/budgets` directory contains .pdf files of UTD budget data. - - Files are named by fiscal year. - - This budget data is used as a backup of scraped data as some years have been removed from the website. +--- -### Contributing +## Quick Navigation -Please visit our [Discord](https://discord.utdnebula.com) and talk to us if you'd like to contribute! +- [Developer Wiki](docs/Home.md) +- [Getting Started Guide](docs/getting-started.md) +- [Pipeline Architecture (Scraper vs Parser vs Uploader vs Runner)](docs/architecture.md) +- [Project Structure & Codebase Map](docs/project-structure.md) +- [Docker & Container Guide](docs/docker-guide.md) +- [Contributing Guide](docs/contributing.md) +- [Troubleshooting & FAQ](docs/troubleshooting.md) +- [Discord Community](https://discord.utdnebula.com) -### Prerequisites +--- -- Golang 1.24 (or higher) +## Prerequisites & Building -### Build +- **Go 1.24+** +- **Make** (Linux/macOS) -To build the project, simply clone the repository and then either: - - Run `make` in the root (top level) directory (for systems with `make` installed, i.e. most Linux distros, MacOS) - - Run `build.bat` on Windows systems (unless you want to deal with getting `make` to work on Windows :P) +To build the executable binary: -The build process will output an executable file named `api-tools`; this executable is the CLI and can be ran in your terminal! +```bash +# Linux / macOS +make setup # Install linter tools (staticcheck, goimports) +make check # Verify formatting, vetting, and linting +make build # Compiles ./api-tools binary +make test # Run automated test suite -Additionally, you can run build (on Windows) and make (on MacOS/Linux) with the following arguments: - - `setup`: Installs required dependencies for the tools. - - `check`: Verifies prequisites and ensures the executable can be built. - - `test`: Test run to see if the executable works after building - - `build`: Builds the executble and makes it ready for use. +# Windows +build.bat setup +build.bat check +build.bat build # Compiles api-tools.exe +build.bat test +``` -### Usage +--- -The `api-tools` command line interface supports three main modes: scraping, parsing and uploading data to the Nebula API. +## CLI Usage Reference -#### Environment Variables -Before being able to use the tool, configure the `.env` file by following these steps: - 1. Find the `.env.template` file and rename it to `.env` - 2. Specify the required credentials for your use case as a string ("") following the variable name. - Example: LOGIN_NETID="ABC123456" +The `api-tools` command line interface supports three main modes: **scraping**, **parsing**, and **uploading**. -#### Basic Usage +### Global Flags -Run the tool by changing directory using `cd` to the `api-tools` directory and running the executable with the appropriate flags in the command line. To see all available options with the tool, run: `./api-tools`. To enable logging for debugging, use the verbose flag: `./api-tools -verbose`. Find available flags for each mode in the following tables. +| Flag | Default | Description | +|---|---|---| +| `-i ` | `./data` | Input directory to read data from | +| `-o ` | `./data` | Output directory to write scraped/parsed data to | +| `-l ` | `./logs` | Directory to write timestamped log files to | +| `--verbose` | `false` | Enables verbose debug logging with file names and microsecond timestamps | +| `--headless`| `false` | Runs ChromeDP browser automation in background headless mode | --- -### Scraping Mode +### 1. Scraping Mode (`--scrape`) + +Scrapes raw data from UTD sources and saves it to `./data` (or the directory specified by `-o`). | Command | Description | -|---------|-------------| -| `./api-tools -scrape -academicCalendars` | Scrapes academic calendar PDFs. | -| `./api-tools -scrape -astra` | Scrapes Astra data. | -| `./api-tools -scrape -cometCalendar` | Scrapes Comet Calendar data. | -| `./api-tools -scrape -coursebook -term 24F` | Scrapes coursebook data for Fall 2024.
• Use `-resume` to continue from last prefix.
• Use `-startprefix [prefix]` to begin at a specific course prefix. | -| `./api-tools -scrape -discounts` | Scrapes discount programs. | -| `./api-tools -scrape -degrees` | Scrapes degrees data | -| `./api-tools -scrape -map` | Scrapes UTD Map data. | -| `./api-tools -scrape -mazevo` | Scrapes Mazevo data. | -| `./api-tools -scrape -profiles` | Scrapes UTD professor profiles. | -| `./api-tools -scrape -headless` | Runs ChromeDP in headless mode. | -| `./api-tools -o [directory]` | Sets output directory (default: `./data`). | - -### Parsing Mode: +|---|---| +| `./api-tools --scrape --coursebook --term 24F` | Scrapes CourseBook data for Fall 2024.
• `--resume`: Resume from last completed course prefix.
• `--startprefix `: Begin at a specific course prefix (e.g. `cp_cs`). | +| `./api-tools --scrape --evals --term 24F` | Scrapes CourseBook course evaluation data for the specified term. | +| `./api-tools --scrape --profiles` | Scrapes faculty and professor directory profiles. | +| `./api-tools --scrape --astra` | Scrapes Astra schedule and room reservation data. | +| `./api-tools --scrape --mazevo` | Scrapes Mazevo event reservation data via API. | +| `./api-tools --scrape --cometCalendar` | Scrapes public UTD Comet Calendar events. | +| `./api-tools --scrape --map` | Scrapes UTD campus map building and parking coordinates. | +| `./api-tools --scrape --degrees` | Scrapes academic degrees and catalog majors. | +| `./api-tools --scrape --discounts` | Scrapes student discount program listings. | +| `./api-tools --scrape --academicCalendars` | Scrapes academic calendar PDF documents. | +| `./api-tools --scrape --budgets` | Scrapes university budget reports. | + +--- + +### 2. Parsing Mode (`-parse`) + +Reads raw files from `./data` (or `-i`), normalizes records against `static-data/`, validates data models, and outputs structured schemas. | Command | Description | -|---------|-------------| -| `./api-tools -parse -academicCalendars` | Parses academic calendar PDFs. | -| `./api-tools -parse -astra` | Parses Astra data. | -| `./api-tools -parse -cometCalendar` | Parses Comet Calendar data. | -| `./api-tools -parse -csv [directory]` | Outputs grade data CSVs (default: `./static-data/grades`). | -| `./api-tools -parse -discounts` | Parses discount programs HTML. | -| `./api-tools -parse -degrees` | Parses degrees from HTML. | -| `./api-tools -parse -map` | Parses UTD Map data. | -| `./api-tools -parse -mazevo` | Parses Mazevo data. | -| `./api-tools -parse -skipv` | Skips post-parse validation (**use with caution**). | - - -### Upload Mode: +|---|---| +| `./api-tools -parse` | Runs full coursebook parsing pipeline across all scraped prefixes. | +| `./api-tools -parse -gradesDir ` | Specifies custom path for grade distribution CSVs (default: `./static-data/grades`). | +| `./api-tools -parse -astra` | Parses scraped Astra scheduling data. | +| `./api-tools -parse -mazevo` | Parses scraped Mazevo event data. | +| `./api-tools -parse -cometCalendar` | Parses scraped Comet Calendar events. | +| `./api-tools -parse -map` | Parses campus map data. | +| `./api-tools -parse -degrees` | Parses degree plans from HTML. | +| `./api-tools -parse -discounts` | Parses student discount listings from HTML. | +| `./api-tools -parse -academicCalendars` | Parses academic calendar dates from PDFs. | +| `./api-tools -parse -budgets` | Parses budget reports from PDFs (use `-useBackupBudgets` to include `static-data/budgets`). | +| `./api-tools -parse -skipv` | Skips post-parse schema validation (**use with caution**). | + +--- + +### 3. Upload Mode (`-upload`) + +Pushes validated data to the MongoDB database specified in `MONGODB_URI`. + | Command | Description | -|---------|-------------| -| `./api-tools -upload -academicCalendars` | Uploads academic calendars. | +|---|---| +| `./api-tools -upload` | Uploads parsed coursebook data (merges into existing records by default). | +| `./api-tools -upload -replace` | Overwrites/replaces existing records instead of merging. | +| `./api-tools -upload -static` | Uploads static aggregations only. | +| `./api-tools -upload -events` | Uploads Astra, Mazevo, and Comet Calendar event datasets. | +| `./api-tools -upload -map` | Uploads campus map locations. | +| `./api-tools -upload -degrees` | Uploads degree plans. | | `./api-tools -upload -discounts` | Uploads discount programs. | -| `./api-tools -upload -events` | Uploads Astra, Mazevo, and Comet Calendar data. | -| `./api-tools -upload -map` | Uploads UTD Map data. | -| `./api-tools -upload -replace` | Replaces old data instead of merging. | -| `./api-tools -upload -static` | Uploads only static aggregations. | +| `./api-tools -upload -academicCalendars` | Uploads academic calendar dates. | +| `./api-tools -upload -budgets` | Uploads budget data. | -Additionally, you can use the `-i [directory]` flag to specify where to read data from (default: `./data`) and the `-l [directory]` flag to specify where logs must be dumped (default: `./logs`). +--- -#### Docker +## Docker & Automated Runners -Docker is used for automated running on Google Cloud Platform. More info [here](https://nebula-labs.atlassian.net/wiki/x/AYBjFw). +Automated scheduled scraping is executed via Docker containers and cloud cron jobs: -To build the container for local testing first make sure all scripts in the `runners` folder have LF line endings then run: -``` +```bash +# Build the container for local testing docker build --target local -t my-runner:local . + +# Run daily event tasks docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=daily.sh my-runner:local + +# Run weekly catalog tasks +docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=weekly.sh my-runner:local + +# Run monthly map & budget tasks +docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=monthly.sh my-runner:local ``` -## Questions? -Reach out to the team on [Discord](https://discord.utdnebula.com) and with any questions you may have! +For more details on Docker execution and cloud configuration, see the [Docker Guide](docs/docker-guide.md). + +--- + +## Community & Support + +Have questions, suggestions, or want to contribute? Reach out on our [Discord](https://discord.utdnebula.com)! diff --git a/docs/Docker-Guide.md b/docs/Docker-Guide.md new file mode 100644 index 0000000..9e465d4 --- /dev/null +++ b/docs/Docker-Guide.md @@ -0,0 +1,90 @@ +# Docker & Containerization Guide + +This guide explains how Docker is used in `api-tools`, how our multi-stage Docker build works, and how you can run automated tasks inside a local Docker container. + +--- + +## Why Do We Use Docker? + +Running web scrapers and PDF parsers requires several system-level dependencies: +- **Chromium / Google Chrome**: Needed by ChromeDP for headless browser automation. +- **Poppler Utilities (`pdftotext`)**: Needed for converting academic calendar PDFs into raw text. +- **Go Runtime**: Needed to compile the application binary. +- **Google Cloud SDK (`gcloud`)**: Needed in production to access cloud secrets. + +Instead of requiring every contributor to manually install and configure these system packages, Docker packages everything into a standardized, lightweight Linux container. This ensures that scripts execute identically on developer laptops (Linux, macOS, Windows) and in production (Google Cloud Platform). + +> [!NOTE] +> **New to Docker?** Check out the official [Docker 101 Tutorial](https://www.docker.com/101-tutorial/) or the [Docker Get Started Guide](https://docs.docker.com/get-started/) to learn the basics of images and containers. + +--- + +## Understanding the Multi-Stage Dockerfile + +Our [`Dockerfile`](file:///var/home/justin/Documents/Projects/api-tools/Dockerfile) uses Docker's multi-stage build pattern to keep production images secure, fast, and minimal. + +1. **`builder` Stage (`golang:1.26`)**: + - Copies Go source code. + - Runs `make setup`, `make check` (vetting and linting), and `make build`. + - Produces the compiled `api-tools` binary. + +2. **`base` Stage (`debian:12-slim`)**: + - A lightweight Debian base image. + - Installs runtime dependencies: `chromium`, `poppler-utils`, and `google-cloud-sdk`. + - Copies the compiled `api-tools` binary from the `builder` stage. + - Copies `runners/` and `static-data/`. + - Sets the entrypoint to [`runners/setup.sh`](file:///var/home/justin/Documents/Projects/api-tools/runners/setup.sh). + +3. **`local` Stage (`FROM base AS local`)**: + - Inherits from `base` and copies your local `.env` file into the container. + - Ideal for local testing and debugging. + +--- + +## Running Docker Locally + +### Step 1: Ensure Your `.env` File Exists +Make sure you have copied `.env.template` to `.env` in the project root: +```bash +cp .env.template .env +``` + +### Step 2: Build the Local Docker Image +Build using the `--target local` flag to include your local `.env` file: +```bash +docker build --target local -t my-runner:local . +``` + +### Step 3: Run a Runner Script +Run the container by specifying the environment mode and target script name: + +```bash +docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=daily.sh my-runner:local +``` + +You can replace `daily.sh` with any runner in the `runners/` directory: +- `daily.sh` — Scrapes, parses, and uploads events (Astra, Mazevo, Comet Calendar). +- `weekly.sh` — Scrapes, parses, and uploads academic calendars, discounts, and degrees. +- `monthly.sh` — Scrapes, parses, and uploads map locations and budgets. + +--- + +## Production vs. Local Execution (`runners/setup.sh`) + +When the Docker container starts, it executes [`runners/setup.sh`](file:///var/home/justin/Documents/Projects/api-tools/runners/setup.sh): + +- **Local Mode (`ENVIRONMENT=local`)**: + - Skips cloud authentication. + - Directly uses the `.env` file baked into the local image. + - Runs `/app/runners/$RUNNER_SCRIPT_NAME`. + +- **Production Mode (`ENVIRONMENT=gcp`)**: + - Authenticates with Google Cloud Secret Manager using service account keys. + - Dynamically fetches secrets and creates the `.env` file at runtime. + - Runs `/app/runners/$RUNNER_SCRIPT_NAME`. + +--- + +## Next Steps +- Learn how to contribute code in the [**Contributing Guide**](contributing.md). +- Resolve any container or browser errors in the [**Troubleshooting Guide**](troubleshooting.md). diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md new file mode 100644 index 0000000..1c243e6 --- /dev/null +++ b/docs/Getting-Started.md @@ -0,0 +1,92 @@ +# Getting Started with API Tools + +This guide walks you through setting up your local development environment, configuring environment variables, compiling the CLI binary, and running tests. + +## Prerequisites + +Ensure you have the following installed: + + +- Git + - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop). +- Go + - You can install from the [Go website](https://go.dev/dl/), or from [Homebrew](https://brew.sh/) or another package manager for automatic updates +- **Make** *(Linux/macOS)* — Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) +- **Google Chrome** or **Chromium** — Required for headless browser scraping +- **Docker** *(Optional)* — For running containerized runners locally + +**Make** is a build automation tool. Feel free to check out [Makefile](../Makefile) to see exactly what's being run. + +If you're using Windows, instead of using `make`, you can also use our `.\build.bat` file. When you see any command starting with `make`, you can instead use `.\build.bat`. For example instead of `make setup`, you can run `.\build.bat setup`. + +## Local Setup + +### Step 1: Clone the Repository + +Clone the repository with `git clone` and `cd` into the project directory or open it in your code editor + +### Step 2: Install Development Tooling + +Install the Go static analysis and formatting tools (`staticcheck` and `goimports`): + +```bash +make setup +``` + +If you installed **Go** with [**Homebrew**](https://brew.sh/), you need to add Go tools to your path for tools like `staticcheck` and others to work. + +- **For zsh** (the default shell on MacOS) + + ```bash + echo 'export PATH=${PATH}:`go env GOPATH`/bin' >> ~/.zshrc && source ~/.zshrc + ``` + +- **For bash** (the default shell on most Linux distributions) + + ```bash + echo 'export PATH=${PATH}:`go env GOPATH`/bin' >> ~/.bashrc && source ~/.bashrc + ``` + +- **For fish** + + ```bash + echo 'fish_add_path (go env GOPATH)/bin' >> ~/.config/fish/config.fish + ``` + +### Step 3: Configure Environment Variables + +Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env`. If you're not sure what to put, ask for help. + + + +### Step 4: Run Code Verification & Formatting + +Check your code with: + +```bash +make check +``` + +You'll want to run this frequently while developing + +### Step 5: Build the CLI Executable + +Compile the Go source code into a runnable binary (`./api-tools` on Linux/macOS, `api-tools.exe` on Windows): + +```bash +make build +``` + +### Step 6: Verify with Automated Tests + +Run the test suite to confirm your environment is ready: + +```bash +make test +``` + +If all tests pass, congratulations! Your local development environment is ready. + +## Next Step + +Now that your environment is running, it's time to dive deeper. Check out [Project-Architecture.md](Project-Architecture.md) diff --git a/docs/Guides.md b/docs/Guides.md new file mode 100644 index 0000000..4c97890 --- /dev/null +++ b/docs/Guides.md @@ -0,0 +1,27 @@ +# Guides + +There's a lot of guides in here. You do **not** need to master everything before getting started! Here is a curated list of resources to help you learn the fundamentals. + +### Git & GitHub + +> **TODO:** [Link to Nebula Git Workshop Guide here] + +[GitHub Skills Git Introduction](https://skills.github.com/) + + +### Go (Golang) +- [A Tour of Go](https://go.dev/tour/) — Official interactive introduction to Go syntax, data structures, and concurrency. +- [Go by Example](https://gobyexample.com/) — Hands-on, concise Go examples covering real-world concepts from structs to JSON. +- [Learn Go in Y Minutes](https://learnxinyminutes.com/docs/go/) — Quick visual cheat sheet for developers who want a fast syntax refresher. + +### Docker & Containers +- [Docker 101 Tutorial](https://www.docker.com/101-tutorial/) — Learn how containers package code and dependencies for reliable execution. +- [Docker Documentation: Orientation and Setup](https://docs.docker.com/get-started/) — Official step-by-step introduction to images and containers. + +### Web Basics & Scraping +- [MDN Web Docs: HTML & DOM Basics](https://developer.mozilla.org/en-US/docs/Learn/HTML) — Learn how web pages are structured with elements, tags, and attributes. +- [MDN Web Docs: CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors) — Understand class, ID, and attribute selectors used to find page elements. +- [ChromeDP Documentation & Examples](https://github.com/chromedp/examples) — Examples for browser automation and scraping dynamic JavaScript pages in Go. + +### Command Line & Shell +- [Bash / CLI Crash Course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line) — Beginner-friendly guide to navigating files, running scripts, and managing processes in the terminal. diff --git a/docs/Home.md b/docs/Home.md new file mode 100644 index 0000000..43c1981 --- /dev/null +++ b/docs/Home.md @@ -0,0 +1,25 @@ +# API Tools Developer Documentation + +Welcome to the **API Tools Developer Documentation**! + +These docs are intended for developers of `api-tools` rather than users. For usage instructions, see [README.md](../README.md). + +--- + +If you are a new developer, we recommend checking out these pages in this order: + +1. [Getting-Started.md](Getting-Started.md) +1. [Project-Architecture.md](Project-Architecture.md) +1. [Project-Structure.md](Project-Structure.md) +1. [How-to-Contribute.md](How-to-Contribute.md) +1. [Guides.md](Guides.md) + +## Need Help? + +Join [our community on Discord](https://discord.utdnebula.com) and ask questions in the `#api-help` or development channels. We are always happy to help! + +## Found something you want to change on these docs? + +Don't edit these docs manually on Github. Instead, suggest a change by [creating an issue](https://github.com/UTDNebula/api-tools/issues), or [opening a pull request](https://github.com/UTDNebula/api-tools/pulls) with your changes. + +These docs are automatically generated from the `docs/` folder of this repository. \ No newline at end of file diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md new file mode 100644 index 0000000..cf33762 --- /dev/null +++ b/docs/How-to-Contribute.md @@ -0,0 +1,66 @@ +# How to Contribute + +Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more + +## 1. Find something to work on + + + + + + + +## 2. Create a branch + +Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork + +Our branch naming convention is `-` for example, `738-getting-started-docs` + +## 3. Make Your Changes + +Don't forget to format and check your code. We have helper scripts: + +```bash +make check +``` +or +```cmd +.\build.bat check +``` + +This executes several commands to check and format your code + +Run the test suite to ensure no regressions were introduced: + +- **Linux / macOS**: + ```bash + make test + ``` +- **Windows**: + ```cmd + build.bat test + ``` + + +## 4. Submitting a Pull Request (PR) + +1. **Commit Your Work**: Write concise, descriptive commit messages: + ```bash + git commit -m "feat(scrapers): add degree scraper for catalog 2026" + ``` +2. **Push to GitHub**: + ```bash + git push origin feat/your-feature-name + ``` +3. **Open a PR**: + - Navigate to the repository on GitHub and click **Compare & pull request**. + - Provide a clear summary of what changes were made and why. + - Mention any related issue numbers (e.g., `Closes #12`). + - Confirm that `make check` and `make test` pass cleanly. +4. **Address Review Feedback**: Maintainers will review your PR and provide suggestions or approval. + +--- + +## Next Step + +See [Guides.md](Guides.md) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md new file mode 100644 index 0000000..0fab8e3 --- /dev/null +++ b/docs/Project-Architecture.md @@ -0,0 +1,100 @@ +# Pipeline Architecture & Core Concepts + +`api-tools` is built around a unidirectional, staged data pipeline. Each stage has a single, well-defined responsibility. Understanding this separation of concerns is essential before modifying or adding code. + +--- + +## High-Level Data Flow + +The lifecycle of data moving through `api-tools` follows a four-stage progression: + +```mermaid +flowchart LR + A[UTD Data Sources\nWebsites, APIs, PDFs] -->|1. Scraper| B[(Raw Data Files\ndata/)] + B -->|2. Parser| C{Validator\nvalidator.go} + D[(Static Data\nstatic-data/)] -->|2. Parser| C + C -->|Validated Models| E[3. Uploader] + E -->|4. MongoDB Insert/Merge| F[(Nebula API Database)] + + subgraph Automation + G[Runners\nrunners/*.sh] -.->|Orchestrates| A + G -.->|Orchestrates| B + G -.->|Orchestrates| E + end +``` + +--- + +## The Four Core Components + +### 1. Scrapers (`scrapers/`) — *Data Acquisition* +- **What they do**: Connect to external UTD web servers, APIs, and portals to download raw information. +- **How they work**: They use HTTP requests (via standard Go `net/http`) or headless browser automation with [ChromeDP](https://github.com/chromedp/chromedp) to navigate pages, fill forms, trigger clicks, and capture raw HTML, JSON, or PDF files. +- **Rule of Thumb**: **Scrapers do NOT validate or parse data.** Their sole job is to faithfully capture raw data from the source and save it to disk in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). +- **External Resources**: + - [MDN Web Docs: Introduction to the DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction) + - [ChromeDP Examples](https://github.com/chromedp/examples) + +--- + +### 2. Parsers (`parser/`) — *Data Transformation & Validation* +- **What they do**: Read raw files produced by scrapers alongside static reference files, extract meaningful fields, normalize inconsistent formatting, and assemble validated Go data structures. +- **How they work**: + - They parse HTML using Go HTML tokenizers and CSS query selectors (via `golang.org/x/net/html`). + - They cross-reference scraped records with static datasets (such as grade distribution CSVs in `static-data/grades` or budget PDFs in `static-data/budgets`). + - They run the parsed structs through validation checks in [`parser/validator.go`](file:///var/home/justin/Documents/Projects/api-tools/parser/validator.go) to ensure all required fields conform to the Nebula API schema. +- **Core Principle: Immutability**: Input files in `data/` are strictly **immutable**. Parsers must treat input files as read-only and never modify raw scraped data. +- **External Resources**: + - [Go `html` package documentation](https://pkg.go.dev/golang.org/x/net/html) + - [Go Regular Expressions (`regexp`) Guide](https://pkg.go.dev/regexp) + +--- + +### 3. Uploaders (`uploader/`) — *Database Synchronization* +- **What they do**: Take validated data models and push them to the [Nebula API](https://github.com/utdnebula/nebula-api) database (MongoDB). +- **How they work**: + - Connect to MongoDB using the official Go Mongo driver (`go.mongodb.org/mongo-driver`). + - Support both **merge** operations (updating existing records with new fields) and **replace** operations (overwriting outdated datasets). + - Compute static aggregation metrics where required before saving. +- **Rule of Thumb**: Uploaders assume data passed to them has already been parsed and validated. +- **External Resources**: + - [MongoDB Go Driver Tutorial](https://www.mongodb.com/docs/drivers/go/current/) + +--- + +### 4. Runners (`runners/`) — *Automated Orchestration* +- **What they do**: Shell scripts (`.sh`) that coordinate automated end-to-end execution of the pipeline on a scheduled cron cadence. +- **How they work**: + - Instead of manually typing CLI commands, runners chain together scraping, parsing, and uploading steps for specific domains. + - Examples in `runners/`: + - `daily.sh`: Daily scrape/parse/upload for volatile event data (Astra, Mazevo, Comet Calendar). + - `weekly.sh`: Weekly scrape/parse/upload for academic calendars, discount programs, and degree plans. + - `monthly.sh`: Monthly updates for campus map locations and budget reports. + - `setup.sh`: Docker entrypoint script that retrieves secrets from Google Cloud Secret Manager (in production) before launching a specified runner. +- **External Resources**: + - [Bash Scripting Tutorial for Beginners](https://linuxconfig.org/bash-scripting-tutorial-for-beginners) + +--- + +## Component Comparison + +| Component | Directory | Input | Output | Mutation / Side Effects | Key Goal | +|---|---|---|---|---|---| +| **Scraper** | `scrapers/` | UTD Websites & APIs | Raw files in `data/` (`.html`, `.json`, `.pdf`) | Writes raw data to disk | Reliable network extraction | +| **Parser** | `parser/` | `data/` + `static-data/` | Validated Go data models | **None** (inputs are read-only) | Structure & schema validation | +| **Uploader** | `uploader/` | Validated Go data models | MongoDB Collections | Inserts / updates database records | Database persistence | +| **Runner** | `runners/` | Scheduled Cron / Trigger | Executes CLI binary sequence | Runs multiple stages in sequence | End-to-end pipeline automation | + +--- + +## Why Separate These Concerns? + +1. **Testability & Offline Development**: Because scrapers save raw data to `data/`, parsers can be tested and developed offline without repeatedly pinging university servers or needing internet access. +2. **Resilience**: If a university website layout changes, only the scraper or selector logic needs updating; the uploader and downstream database logic remain completely unaffected. +3. **Reproducibility**: If an issue occurs during parsing or uploading, the raw input data is preserved on disk, allowing developers to inspect exact HTML responses and write reproducible test fixtures. + +--- + +## Next Step +See [Project-Structure.md](Project-Structure.md) + diff --git a/docs/Project-Structure.md b/docs/Project-Structure.md new file mode 100644 index 0000000..42ca8ea --- /dev/null +++ b/docs/Project-Structure.md @@ -0,0 +1,62 @@ +# Project Structure & Codebase Map + +This document provides an overview of the `api-tools` directory layout, explaining what each folder contains and guiding you on where to make changes when adding features or fixing bugs. + +## Directory Overview + +```text +api-tools/ +├── data/ # Files from scrapers and parsers +├── docs/ # Source of wiki pages +├── logs/ # Timestamped log files from runtime executions +├── parser/ # Parsing, data normalization, and schema validation +│ └── testdata/ # HTML input fixtures and expected JSON, used for tests +├── runners/ # Shell automation scripts ran on Google Cloud (daily, weekly, monthly) +├── scrapers/ # Web scrapers for UTD websites, portals, and APIs +├── static-data/ # Static data that isn't scraped +├── uploader/ # Database ingestion modules for Nebula API / MongoDB +├── utils/ # Shared utilities (ChromeDP helpers, logger, regexes) +├── .env.template # Template for required environment variables +├── Dockerfile # Multi-stage container definition +├── Makefile # Task runner for Linux and macOS (build, check, test) +├── build.bat # Task runner for Windows (build, check, test) +├── cloudbuild.yaml # Google Cloud Build deployment configuration +├── go.mod / go.sum # Go dependency manifests +└── main.go # Application entrypoint and CLI flag definitions +``` + +## Directory Breakdown + +### `parser/` +Responsible for interpreting raw files, sanitizing fields, resolving cross-references, and ensuring schema compliance: +- `Parser.go`: Dedicated parser files corresponding to each scraper (e.g., `courseParser.go`, `sectionParser.go`, `mapParser.go`, `astraParser.go`). +- `parser.go`: The central orchestrator that coordinates parsing of scraped coursebook files with static grade and profile data. +- `gradeLoader.go`: Reads historical grade distribution CSVs from `static-data/grades`. +- `requisiteParser.go`: Parses prerequisite and corequisite strings into structured dependency trees. +- `validator.go`: Validates that parsed Go structs contain all non-empty mandatory fields before upload. +- `testdata/`: Test suites containing realistic sample input files (`input.html`) and expected JSON outputs (`course.json`, `section.json`, `professors.json`) for automated regression testing. + +### `uploader/` +Handles pushing structured, validated data into MongoDB: +- `database.go`: Establishes MongoDB client connections using `MONGODB_URI`. +- `uploader.go`: Core coursebook data uploader with merge/replace strategies. +- `Uploader.go`: Specialized uploaders for events, maps, discounts, degrees, and academic calendars. + +### `static-data/` +Persistent reference datasets that cannot be scraped on-demand or serve as historical baselines: +- `static-data/grades/`: CSV files organized by semester (e.g., `22F.csv` for Fall 2022, `23S.csv` for Spring 2023, `23U.csv` for Summer 2023). +- `static-data/budgets/`: Historical PDF budget files for fiscal years where web links are no longer available. + +### `utils/` +Shared helper functions used across scrapers, parsers, and uploaders: +- `methods.go`: ChromeDP initialization, headless settings, environment variable readers (`GetEnv`), and token refresh helpers. +- `logger.go`: Dual logger implementation (`SplitWriter`) that writes output simultaneously to `stdout` and timestamped files in `logs/`. +- `regexes.go`: Compiled regular expressions for text parsing and course extraction. + +### Root Configuration Files +- `main.go`: The entrypoint of the CLI application. Parses command line flags (like `-scrape`, `-parse`, `-upload`, `-verbose`, `-headless`) and routes execution to the appropriate package functions. +- `Makefile` / `build.bat`: Scripts to automate setup, formatting checks (`staticcheck`, `gofmt`, `goimports`, `go vet`), compiling the binary, and running unit tests. +- `Dockerfile`: Defines the multi-stage Docker build for local testing and production deployment. + +## Next Step +See [How-to-Contribute.md](How-to-Contribute.md) diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md new file mode 100644 index 0000000..ec6618e --- /dev/null +++ b/docs/Troubleshooting.md @@ -0,0 +1,96 @@ +# Troubleshooting & FAQ + +This guide provides fast, actionable solutions to common issues encountered when running or developing `api-tools`. + +--- + +## 1. Chromium & ChromeDP Issues + +### Issue: "exec: 'chromium': executable file not found in $PATH" or Chrome fails to launch +- **Cause**: ChromeDP cannot find a Chromium or Google Chrome binary on your operating system. +- **Solutions**: + - **Linux**: Install Chromium via your package manager: + ```bash + # Ubuntu / Debian + sudo apt update && sudo apt install -y chromium-browser + # Fedora / RHEL + sudo dnf install chromium + # Arch Linux + sudo pacman -S chromium + ``` + - **macOS / Windows**: Ensure [Google Chrome](https://www.google.com/chrome/) is installed in the standard applications directory. + - **Environment Variables**: If Chrome is installed in a non-standard location, specify its path explicitly before running `api-tools`: + ```bash + export CHROMIUM_BIN="/path/to/custom/chrome" + export GOOGLE_CHROME_BIN="/path/to/custom/chrome" + ``` + +### Issue: Chrome window opens during scraping and interferes with work +- **Solution**: Pass the `-headless` flag to run ChromeDP invisibly in background headless mode: + ```bash + ./api-tools -scrape -coursebook -term 24F -headless + ``` + +--- + +## 2. Environment Variables & MongoDB Connection Issues + +### Issue: Panic: ` is missing from .env!` +- **Cause**: An authenticated scraper or uploader attempted to read a required environment variable from `.env` via `utils.GetEnv`, but the key was either missing or empty. +- **Solutions**: + 1. Ensure you have copied `.env.template` to `.env`: + ```bash + cp .env.template .env + ``` + 2. For Coursebook, Astra, or Mazevo scrapers, populate the required credentials: + - `LOGIN_NETID` / `LOGIN_PASSWORD` (UTD NetID login) + - `LOGIN_ASTRA_USERNAME` / `LOGIN_ASTRA_PASSWORD` + - `MAZEVO_API_KEY` + 3. If you only want to parse existing local data or run tests, you do **not** need these credentials; avoid calling authenticated scraping flags. + +### Issue: MongoDB connection timed out / "server selection error" +- **Cause**: The uploader could not establish a connection to the MongoDB instance specified in `MONGODB_URI`. +- **Solutions**: + - **Verify Connection String**: Check that `MONGODB_URI` in `.env` is formatted correctly: + ```env + MONGODB_URI="mongodb+srv://:@cluster0.abcde.mongodb.net/?retryWrites=true&w=majority" + ``` + - **Check Network & IP Whitelist**: If using MongoDB Atlas, ensure your current IP address is whitelisted under Atlas Network Access. + - **Local Development**: If testing locally with a local MongoDB container or service, verify MongoDB is running: + ```bash + # Test local MongoDB connection + mongosh "mongodb://localhost:27017" + ``` + +--- + +## 3. Locating & Inspecting Log Files + +### How Logging Works +When `api-tools` executes, `utils.NewSplitWriter` writes all log output to two destinations simultaneously: +1. **Console (`stdout`)**: Live output in your terminal window. +2. **Log File (`logs/*.log`)**: A persistent log file saved to the `logs/` directory named with the execution timestamp: + ``` + logs/8-31-2026T12-30-15.log + ``` + +### Enabling Verbose Debug Logging +If a scraper or parser is behaving unexpectedly, re-run the command with the `-verbose` flag: + +```bash +./api-tools -verbose -scrape -coursebook -term 24F +``` + +Verbose mode enables: +- Microsecond timestamp precision (`log.Lmicroseconds`). +- Source code filename and line number for every log statement (`log.Lshortfile`). +- Additional internal debug logs (`utils.Lverbose`). + +--- + +## Still Stuck? + +Reach out to the team on [Discord](https://discord.utdnebula.com) with: +1. The exact command you ran. +2. The relevant lines from your latest log file in `logs/`. +3. Your operating system and Go version (`go version`). From 38fd1cb301f50b2bc874f67b032dfd6a4305d41d Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 2 Sep 2026 16:05:07 -0500 Subject: [PATCH 02/42] Update docs formatting --- docs/Getting-Started.md | 5 +---- docs/Home.md | 8 ++++---- 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index 1c243e6..bfe6856 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -6,7 +6,6 @@ This guide walks you through setting up your local development environment, conf Ensure you have the following installed: - - Git - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop). - Go @@ -15,7 +14,7 @@ Ensure you have the following installed: - **Google Chrome** or **Chromium** — Required for headless browser scraping - **Docker** *(Optional)* — For running containerized runners locally -**Make** is a build automation tool. Feel free to check out [Makefile](../Makefile) to see exactly what's being run. +**Make** is a build automation tool. Feel free to check out [Makefile](../Makefile) to see exactly what's being run. If you're using Windows, instead of using `make`, you can also use our `.\build.bat` file. When you see any command starting with `make`, you can instead use `.\build.bat`. For example instead of `make setup`, you can run `.\build.bat setup`. @@ -57,8 +56,6 @@ If you installed **Go** with [**Homebrew**](https://brew.sh/), you need to add G Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env`. If you're not sure what to put, ask for help. - - ### Step 4: Run Code Verification & Formatting Check your code with: diff --git a/docs/Home.md b/docs/Home.md index 43c1981..6e37190 100644 --- a/docs/Home.md +++ b/docs/Home.md @@ -1,6 +1,6 @@ # API Tools Developer Documentation -Welcome to the **API Tools Developer Documentation**! +Welcome to the **API Tools Developer Documentation**! These docs are intended for developers of `api-tools` rather than users. For usage instructions, see [README.md](../README.md). @@ -16,10 +16,10 @@ If you are a new developer, we recommend checking out these pages in this order: ## Need Help? -Join [our community on Discord](https://discord.utdnebula.com) and ask questions in the `#api-help` or development channels. We are always happy to help! +Join [our community on Discord](https://discord.utdnebula.com) and ask questions in the `#api-help` or development channels. We are always happy to help! ## Found something you want to change on these docs? -Don't edit these docs manually on Github. Instead, suggest a change by [creating an issue](https://github.com/UTDNebula/api-tools/issues), or [opening a pull request](https://github.com/UTDNebula/api-tools/pulls) with your changes. +Don't edit these docs manually on Github. Instead, suggest a change by [creating an issue](../issues), or [opening a pull request](../pulls) with your changes. -These docs are automatically generated from the `docs/` folder of this repository. \ No newline at end of file +These docs are automatically generated from the `docs/` folder of this repository. From 0f50a42cd3ed04b666f1625f39542ea12e45b0ea Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 2 Sep 2026 16:05:14 -0500 Subject: [PATCH 03/42] Update how-to-contribute --- docs/How-to-Contribute.md | 64 ++++++++++++++------------------------- 1 file changed, 22 insertions(+), 42 deletions(-) diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index cf33762..7dd426d 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -1,65 +1,45 @@ # How to Contribute -Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more +Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more. Don't worry if you don't quite know what you're doing, we're here to help. We don't expect perfection and appreciate anything you can do to help! ## 1. Find something to work on - - - +Look at our current issues, or make your own on our [issues page](../issues). +Write a comment asking if you can work on a particular issue, or assign it to yourself. You can work collaboratively, or individually on an issue. +Be sure to **discuss the issue frequently** especially if you don't quite understand. You can do this on GitHub, Discord, or in person. ## 2. Create a branch -Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork +Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork as they do not have permission to make a branch. -Our branch naming convention is `-` for example, `738-getting-started-docs` +Our branch naming convention is `-` for example, `738-new-developer-docs`. GitHub can automatically create a branch for you following this convention if you select "Create a branch" on the right sidebar under "Development" ## 3. Make Your Changes -Don't forget to format and check your code. We have helper scripts: +It's time to code! + +Don't forget to format and check your code. We have helper scripts with **Make** and **build.bat**: ```bash make check ``` -or -```cmd -.\build.bat check + +Run tests with +```bash +make test ``` -This executes several commands to check and format your code - -Run the test suite to ensure no regressions were introduced: - -- **Linux / macOS**: - ```bash - make test - ``` -- **Windows**: - ```cmd - build.bat test - ``` - - -## 4. Submitting a Pull Request (PR) - -1. **Commit Your Work**: Write concise, descriptive commit messages: - ```bash - git commit -m "feat(scrapers): add degree scraper for catalog 2026" - ``` -2. **Push to GitHub**: - ```bash - git push origin feat/your-feature-name - ``` -3. **Open a PR**: - - Navigate to the repository on GitHub and click **Compare & pull request**. - - Provide a clear summary of what changes were made and why. - - Mention any related issue numbers (e.g., `Closes #12`). - - Confirm that `make check` and `make test` pass cleanly. -4. **Address Review Feedback**: Maintainers will review your PR and provide suggestions or approval. - ---- +Discuss your progress frequently, and push your commits to GitHub! If you get stuck anywhere we're here to help! + +We'd love to see you make a draft Pull Request, so we can see your progress and help you out! + +## 4. Making a Pull Request (PR) + +Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft. + +Maintainers will review your PR and provide suggestions or approval. Don't take suggestions personally, we just want to help! ## Next Step From 94e7a5ba997193ea9de85639099bb69d0ffe5f78 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 2 Sep 2026 16:13:09 -0500 Subject: [PATCH 04/42] Add Wiki GitHub Action --- .github/workflows/wiki.yml | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 .github/workflows/wiki.yml diff --git a/.github/workflows/wiki.yml b/.github/workflows/wiki.yml new file mode 100644 index 0000000..38d21af --- /dev/null +++ b/.github/workflows/wiki.yml @@ -0,0 +1,27 @@ +name: Publish Wiki + +on: + push: + branches: [develop] + paths: + - 'docs/**' + workflow_dispatch: + +concurrency: + group: publish-wiki + cancel-in-progress: true + +permissions: + contents: write + +jobs: + publish-wiki: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Sync docs to GitHub Wiki + uses: Andrew-Chen-Wang/github-wiki-action@v5 + with: + path: docs/ \ No newline at end of file From 37305e5324ab7e2299cbc294104b7c24343fbde5 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 2 Sep 2026 20:49:55 -0500 Subject: [PATCH 05/42] issue -> contribution in how to contribute --- docs/How-to-Contribute.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index 7dd426d..7a78ab2 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -31,7 +31,7 @@ Run tests with make test ``` -Discuss your progress frequently, and push your commits to GitHub! If you get stuck anywhere we're here to help! +Discuss your progress frequently, and push your commits to GitHub! If you get stuck anywhere we're here to help! Avoid using AI to write code for your first contribution. We'd love to see you make a draft Pull Request, so we can see your progress and help you out! From 73cb0f43b4733143192fc00ace0c6c3ff5db1187 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 3 Sep 2026 08:56:56 -0500 Subject: [PATCH 06/42] Add build.bat to Getting-Started.md --- docs/Getting-Started.md | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index bfe6856..ff0d536 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -14,9 +14,9 @@ Ensure you have the following installed: - **Google Chrome** or **Chromium** — Required for headless browser scraping - **Docker** *(Optional)* — For running containerized runners locally -**Make** is a build automation tool. Feel free to check out [Makefile](../Makefile) to see exactly what's being run. +**Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run. -If you're using Windows, instead of using `make`, you can also use our `.\build.bat` file. When you see any command starting with `make`, you can instead use `.\build.bat`. For example instead of `make setup`, you can run `.\build.bat setup`. +If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file. When you see any command starting with `make`, you can instead use `.\build.bat`. For example instead of `make setup`, you can run `.\build.bat setup`. ## Local Setup @@ -32,6 +32,12 @@ Install the Go static analysis and formatting tools (`staticcheck` and `goimport make setup ``` +or + +```cmd +.\build.bat setup +``` + If you installed **Go** with [**Homebrew**](https://brew.sh/), you need to add Go tools to your path for tools like `staticcheck` and others to work. - **For zsh** (the default shell on MacOS) @@ -64,6 +70,12 @@ Check your code with: make check ``` +or + +```cmd +.\build.bat check +``` + You'll want to run this frequently while developing ### Step 5: Build the CLI Executable @@ -74,6 +86,12 @@ Compile the Go source code into a runnable binary (`./api-tools` on Linux/macOS, make build ``` +or + +```cmd +.\build.bat build +``` + ### Step 6: Verify with Automated Tests Run the test suite to confirm your environment is ready: @@ -82,6 +100,12 @@ Run the test suite to confirm your environment is ready: make test ``` +or + +```cmd +.\build.bat test +``` + If all tests pass, congratulations! Your local development environment is ready. ## Next Step From 0c4653c8c6dc1534f0b20646fc00e2c012465f61 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 3 Sep 2026 09:28:23 -0500 Subject: [PATCH 07/42] Update docs formatting and more build.bat --- docs/Getting-Started.md | 12 ++++++------ docs/How-to-Contribute.md | 21 +++++++++++++++++---- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index ff0d536..95b1bbb 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -20,11 +20,11 @@ If you're using Windows, instead of using `make`, you can also use our [`build.b ## Local Setup -### Step 1: Clone the Repository +### Clone the Repository Clone the repository with `git clone` and `cd` into the project directory or open it in your code editor -### Step 2: Install Development Tooling +### Install Development Tooling Install the Go static analysis and formatting tools (`staticcheck` and `goimports`): @@ -58,11 +58,11 @@ If you installed **Go** with [**Homebrew**](https://brew.sh/), you need to add G echo 'fish_add_path (go env GOPATH)/bin' >> ~/.config/fish/config.fish ``` -### Step 3: Configure Environment Variables +### Configure Environment Variables Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env`. If you're not sure what to put, ask for help. -### Step 4: Run Code Verification & Formatting +### Run Code Verification & Formatting Check your code with: @@ -78,7 +78,7 @@ or You'll want to run this frequently while developing -### Step 5: Build the CLI Executable +### Build the CLI Executable Compile the Go source code into a runnable binary (`./api-tools` on Linux/macOS, `api-tools.exe` on Windows): @@ -92,7 +92,7 @@ or .\build.bat build ``` -### Step 6: Verify with Automated Tests +### Verify with Automated Tests Run the test suite to confirm your environment is ready: diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index 7a78ab2..06b5c1a 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -2,7 +2,7 @@ Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more. Don't worry if you don't quite know what you're doing, we're here to help. We don't expect perfection and appreciate anything you can do to help! -## 1. Find something to work on +## Find something to work on Look at our current issues, or make your own on our [issues page](../issues). @@ -10,13 +10,13 @@ Write a comment asking if you can work on a particular issue, or assign it to yo Be sure to **discuss the issue frequently** especially if you don't quite understand. You can do this on GitHub, Discord, or in person. -## 2. Create a branch +## Create a branch Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork as they do not have permission to make a branch. Our branch naming convention is `-` for example, `738-new-developer-docs`. GitHub can automatically create a branch for you following this convention if you select "Create a branch" on the right sidebar under "Development" -## 3. Make Your Changes +## Make Your Changes It's time to code! @@ -26,16 +26,29 @@ Don't forget to format and check your code. We have helper scripts with **Make** make check ``` +or + +```cmd +.\build.bat check +``` + Run tests with + ```bash make test ``` +or + +```cmd +.\build.bat test +``` + Discuss your progress frequently, and push your commits to GitHub! If you get stuck anywhere we're here to help! Avoid using AI to write code for your first contribution. We'd love to see you make a draft Pull Request, so we can see your progress and help you out! -## 4. Making a Pull Request (PR) +## Making a Pull Request (PR) Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft. From 662c3502086106ff6a899a0befb29fac76256da1 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Thu, 3 Sep 2026 10:19:38 -0500 Subject: [PATCH 08/42] Update Getting-Started.md --- docs/Getting-Started.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index 95b1bbb..e5cd85b 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -10,13 +10,14 @@ Ensure you have the following installed: - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop). - Go - You can install from the [Go website](https://go.dev/dl/), or from [Homebrew](https://brew.sh/) or another package manager for automatic updates -- **Make** *(Linux/macOS)* — Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) +- **Make** _(Linux/macOS)_ — Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) - **Google Chrome** or **Chromium** — Required for headless browser scraping -- **Docker** *(Optional)* — For running containerized runners locally +- **Docker** _(Optional)_ — For running containerized runners locally **Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run. -If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file. When you see any command starting with `make`, you can instead use `.\build.bat`. For example instead of `make setup`, you can run `.\build.bat setup`. +If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file.When you see any command starting with `make`, you can instead use `.\build.bat`. +For example instead of `make setup`, you can run `.\build.bat setup`. ## Local Setup @@ -24,6 +25,11 @@ If you're using Windows, instead of using `make`, you can also use our [`build.b Clone the repository with `git clone` and `cd` into the project directory or open it in your code editor +```bash +git clone https://github.com/UTDNebula/api-tools.git +cd api-tools +``` + ### Install Development Tooling Install the Go static analysis and formatting tools (`staticcheck` and `goimports`): @@ -60,7 +66,8 @@ If you installed **Go** with [**Homebrew**](https://brew.sh/), you need to add G ### Configure Environment Variables -Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env`. If you're not sure what to put, ask for help. +Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env` +If you're not sure what to put, please try contacting one of our members through discord. ### Run Code Verification & Formatting From 9be53f5925ddd6f5e31b3d38bab015126e43cf63 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Thu, 3 Sep 2026 10:57:50 -0500 Subject: [PATCH 09/42] Update Project-Architecture.md --- docs/Project-Architecture.md | 59 ++++++++++++++++++------------------ 1 file changed, 29 insertions(+), 30 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 0fab8e3..df397d0 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -1,10 +1,11 @@ # Pipeline Architecture & Core Concepts -`api-tools` is built around a unidirectional, staged data pipeline. Each stage has a single, well-defined responsibility. Understanding this separation of concerns is essential before modifying or adding code. +`api-tools` is built around a unidirectional, staged data pipeline. Each stage has a single, well-defined responsibility. +Understanding this separation of concerns is essential before modifying or adding code. --- -## High-Level Data Flow +## Project Pipeline The lifecycle of data moving through `api-tools` follows a four-stage progression: @@ -15,7 +16,7 @@ flowchart LR D[(Static Data\nstatic-data/)] -->|2. Parser| C C -->|Validated Models| E[3. Uploader] E -->|4. MongoDB Insert/Merge| F[(Nebula API Database)] - + subgraph Automation G[Runners\nrunners/*.sh] -.->|Orchestrates| A G -.->|Orchestrates| B @@ -27,42 +28,48 @@ flowchart LR ## The Four Core Components -### 1. Scrapers (`scrapers/`) — *Data Acquisition* +### 1. Scrapers (`scrapers/`) + - **What they do**: Connect to external UTD web servers, APIs, and portals to download raw information. -- **How they work**: They use HTTP requests (via standard Go `net/http`) or headless browser automation with [ChromeDP](https://github.com/chromedp/chromedp) to navigate pages, fill forms, trigger clicks, and capture raw HTML, JSON, or PDF files. -- **Rule of Thumb**: **Scrapers do NOT validate or parse data.** Their sole job is to faithfully capture raw data from the source and save it to disk in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). +- **How they work**: Scrapers fetch data via: + - Standard HTTP requests (`net/http`) + - Headless browser automation ([ChromeDP](https://github.com/chromedp/chromedp)) for JavaScript-heavy pages +- **Key Principle**: **Scrapers do NOT validate or parse data.** Their sole job is to faithfully capture raw data from the source and save it to disk in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). - **External Resources**: - [MDN Web Docs: Introduction to the DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction) - [ChromeDP Examples](https://github.com/chromedp/examples) --- -### 2. Parsers (`parser/`) — *Data Transformation & Validation* -- **What they do**: Read raw files produced by scrapers alongside static reference files, extract meaningful fields, normalize inconsistent formatting, and assemble validated Go data structures. +### 2. Parsers (`parser/`) + +- **What they do**: Read raw files produced by scrapers alongside static reference files, extract meaningful fields, normalize inconsistent formatting, and assemble validated Go data structures.to - **How they work**: - - They parse HTML using Go HTML tokenizers and CSS query selectors (via `golang.org/x/net/html`). - - They cross-reference scraped records with static datasets (such as grade distribution CSVs in `static-data/grades` or budget PDFs in `static-data/budgets`). - - They run the parsed structs through validation checks in [`parser/validator.go`](file:///var/home/justin/Documents/Projects/api-tools/parser/validator.go) to ensure all required fields conform to the Nebula API schema. -- **Core Principle: Immutability**: Input files in `data/` are strictly **immutable**. Parsers must treat input files as read-only and never modify raw scraped data. + - Parse HTML via Go tokenizers and CSS selectors (`golang.org/x/net/html`) + - Cross-reference scraped data with static datasets (grade CSVs, budget PDFs) + - Validate structs against the Nebula API schema via [`parser/validator.go`](file:///var/home/justin/Documents/Projects/api-tools/parser/validator.go) +- **Key Principle**: Input files in `data/` are strictly **immutable**. Parsers must treat input files as read-only and never modify raw scraped data. - **External Resources**: - [Go `html` package documentation](https://pkg.go.dev/golang.org/x/net/html) - [Go Regular Expressions (`regexp`) Guide](https://pkg.go.dev/regexp) --- -### 3. Uploaders (`uploader/`) — *Database Synchronization* +### 3. Uploaders (`uploader/`) + - **What they do**: Take validated data models and push them to the [Nebula API](https://github.com/utdnebula/nebula-api) database (MongoDB). - **How they work**: - Connect to MongoDB using the official Go Mongo driver (`go.mongodb.org/mongo-driver`). - Support both **merge** operations (updating existing records with new fields) and **replace** operations (overwriting outdated datasets). - Compute static aggregation metrics where required before saving. -- **Rule of Thumb**: Uploaders assume data passed to them has already been parsed and validated. +- **Key Principle**: Uploaders assume data passed to them has already been parsed and validated. - **External Resources**: - [MongoDB Go Driver Tutorial](https://www.mongodb.com/docs/drivers/go/current/) --- -### 4. Runners (`runners/`) — *Automated Orchestration* +### 4. Runners (`runners/`) + - **What they do**: Shell scripts (`.sh`) that coordinate automated end-to-end execution of the pipeline on a scheduled cron cadence. - **How they work**: - Instead of manually typing CLI commands, runners chain together scraping, parsing, and uploading steps for specific domains. @@ -78,23 +85,15 @@ flowchart LR ## Component Comparison -| Component | Directory | Input | Output | Mutation / Side Effects | Key Goal | -|---|---|---|---|---|---| -| **Scraper** | `scrapers/` | UTD Websites & APIs | Raw files in `data/` (`.html`, `.json`, `.pdf`) | Writes raw data to disk | Reliable network extraction | -| **Parser** | `parser/` | `data/` + `static-data/` | Validated Go data models | **None** (inputs are read-only) | Structure & schema validation | -| **Uploader** | `uploader/` | Validated Go data models | MongoDB Collections | Inserts / updates database records | Database persistence | -| **Runner** | `runners/` | Scheduled Cron / Trigger | Executes CLI binary sequence | Runs multiple stages in sequence | End-to-end pipeline automation | - ---- - -## Why Separate These Concerns? - -1. **Testability & Offline Development**: Because scrapers save raw data to `data/`, parsers can be tested and developed offline without repeatedly pinging university servers or needing internet access. -2. **Resilience**: If a university website layout changes, only the scraper or selector logic needs updating; the uploader and downstream database logic remain completely unaffected. -3. **Reproducibility**: If an issue occurs during parsing or uploading, the raw input data is preserved on disk, allowing developers to inspect exact HTML responses and write reproducible test fixtures. +| Component | Directory | Input | Output | Side Effects | Key Goal | +| ------------ | ----------- | ------------------------ | ------------------------------- | ----------------------- | --------------------------------------------- | +| **Scraper** | `scrapers/` | UTD websites, APIs | Raw `.html`, `.json` in `data/` | Writes files to disk | Capture raw data from UTD websites | +| **Parser** | `parser/` | `data/` + `static-data/` | Validated Go structs | **None** (read-only) | Extract & validate structured data in `data/` | +| **Uploader** | `uploader/` | Validated Go structs | MongoDB documents | DB inserts/updates | Persist data to API database | +| **Runner** | `runners/` | Cron schedule | Pipeline execution | Orchestrates all stages | Automate scheduled pipeline execution | --- ## Next Step -See [Project-Structure.md](Project-Structure.md) +See [Project-Structure.md](Project-Structure.md) From aea701f77bd51d94db011df681590cde1fb19841 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Thu, 3 Sep 2026 11:03:56 -0500 Subject: [PATCH 10/42] Update mermaid.js diagram in Project-Architecture.md --- docs/Project-Architecture.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index df397d0..ea4c14b 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -17,11 +17,13 @@ flowchart LR C -->|Validated Models| E[3. Uploader] E -->|4. MongoDB Insert/Merge| F[(Nebula API Database)] - subgraph Automation - G[Runners\nrunners/*.sh] -.->|Orchestrates| A + subgraph Automation["Automation"] + G["Runners\nrunners/*.sh"] + end + + G -.->|Orchestrates| A G -.->|Orchestrates| B G -.->|Orchestrates| E - end ``` --- From b64be302c40c473df4f68b2311426ed4d826afa8 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Fri, 4 Sep 2026 00:32:56 -0500 Subject: [PATCH 11/42] Finalize Docs for Project-Architecture.md --- docs/Project-Architecture.md | 74 +++++++++++++----------------------- 1 file changed, 27 insertions(+), 47 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index ea4c14b..c1cf95c 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -11,39 +11,38 @@ The lifecycle of data moving through `api-tools` follows a four-stage progressio ```mermaid flowchart LR - A[UTD Data Sources\nWebsites, APIs, PDFs] -->|1. Scraper| B[(Raw Data Files\ndata/)] - B -->|2. Parser| C{Validator\nvalidator.go} - D[(Static Data\nstatic-data/)] -->|2. Parser| C - C -->|Validated Models| E[3. Uploader] - E -->|4. MongoDB Insert/Merge| F[(Nebula API Database)] - - subgraph Automation["Automation"] - G["Runners\nrunners/*.sh"] - end - - G -.->|Orchestrates| A - G -.->|Orchestrates| B - G -.->|Orchestrates| E + A[UTD Data Sources\nUTD Websites] -->|Scraper| B[(Raw Data Files\ndata/)] + B -->|Parser| C{Validator\nvalidator.go} + D[(Static Data\nstatic-data/)] -->|Parser| C + C -->|Validated Models| E[Uploader] + E -->|MongoDB Insert/Merge| F[(Nebula API Database)] ``` --- -## The Four Core Components +## Component Comparison + +| Component | Directory | Input | Output | Side Effects | Key Goal | +| ------------ | ----------- | ------------------------ | ------------------------------- | -------------------- | --------------------------------------------- | +| **Scraper** | `scrapers/` | UTD websites, APIs | Raw `.html`, `.json` in `data/` | Writes files to disk | Capture raw data from UTD websites | +| **Parser** | `parser/` | `data/` + `static-data/` | Validated Go structs | **None** (read-only) | Extract & validate structured data in `data/` | +| **Uploader** | `uploader/` | Validated Go structs | MongoDB documents | DB inserts/updates | Persist data to API database | + +--- + +## The Three Core Components -### 1. Scrapers (`scrapers/`) +### Scrapers (`scrapers/`) - **What they do**: Connect to external UTD web servers, APIs, and portals to download raw information. - **How they work**: Scrapers fetch data via: - Standard HTTP requests (`net/http`) - - Headless browser automation ([ChromeDP](https://github.com/chromedp/chromedp)) for JavaScript-heavy pages + - Web scraping via ([ChromeDP](https://github.com/chromedp/chromedp)) - **Key Principle**: **Scrapers do NOT validate or parse data.** Their sole job is to faithfully capture raw data from the source and save it to disk in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). -- **External Resources**: - - [MDN Web Docs: Introduction to the DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction) - - [ChromeDP Examples](https://github.com/chromedp/examples) --- -### 2. Parsers (`parser/`) +### Parsers (`parser/`) - **What they do**: Read raw files produced by scrapers alongside static reference files, extract meaningful fields, normalize inconsistent formatting, and assemble validated Go data structures.to - **How they work**: @@ -51,13 +50,10 @@ flowchart LR - Cross-reference scraped data with static datasets (grade CSVs, budget PDFs) - Validate structs against the Nebula API schema via [`parser/validator.go`](file:///var/home/justin/Documents/Projects/api-tools/parser/validator.go) - **Key Principle**: Input files in `data/` are strictly **immutable**. Parsers must treat input files as read-only and never modify raw scraped data. -- **External Resources**: - - [Go `html` package documentation](https://pkg.go.dev/golang.org/x/net/html) - - [Go Regular Expressions (`regexp`) Guide](https://pkg.go.dev/regexp) --- -### 3. Uploaders (`uploader/`) +### Uploaders (`uploader/`) - **What they do**: Take validated data models and push them to the [Nebula API](https://github.com/utdnebula/nebula-api) database (MongoDB). - **How they work**: @@ -65,34 +61,18 @@ flowchart LR - Support both **merge** operations (updating existing records with new fields) and **replace** operations (overwriting outdated datasets). - Compute static aggregation metrics where required before saving. - **Key Principle**: Uploaders assume data passed to them has already been parsed and validated. -- **External Resources**: - - [MongoDB Go Driver Tutorial](https://www.mongodb.com/docs/drivers/go/current/) --- -### 4. Runners (`runners/`) +## Automation -- **What they do**: Shell scripts (`.sh`) that coordinate automated end-to-end execution of the pipeline on a scheduled cron cadence. -- **How they work**: - - Instead of manually typing CLI commands, runners chain together scraping, parsing, and uploading steps for specific domains. - - Examples in `runners/`: - - `daily.sh`: Daily scrape/parse/upload for volatile event data (Astra, Mazevo, Comet Calendar). - - `weekly.sh`: Weekly scrape/parse/upload for academic calendars, discount programs, and degree plans. - - `monthly.sh`: Monthly updates for campus map locations and budget reports. - - `setup.sh`: Docker entrypoint script that retrieves secrets from Google Cloud Secret Manager (in production) before launching a specified runner. -- **External Resources**: - - [Bash Scripting Tutorial for Beginners](https://linuxconfig.org/bash-scripting-tutorial-for-beginners) - ---- - -## Component Comparison +It is important to note that api-tools is mostly automated. Most data sources run via Shell scripts (`.sh`) that coordinate automated end-to-end execution of the pipeline on a scheduled cron job. +Our Scripts are in `runners/`: -| Component | Directory | Input | Output | Side Effects | Key Goal | -| ------------ | ----------- | ------------------------ | ------------------------------- | ----------------------- | --------------------------------------------- | -| **Scraper** | `scrapers/` | UTD websites, APIs | Raw `.html`, `.json` in `data/` | Writes files to disk | Capture raw data from UTD websites | -| **Parser** | `parser/` | `data/` + `static-data/` | Validated Go structs | **None** (read-only) | Extract & validate structured data in `data/` | -| **Uploader** | `uploader/` | Validated Go structs | MongoDB documents | DB inserts/updates | Persist data to API database | -| **Runner** | `runners/` | Cron schedule | Pipeline execution | Orchestrates all stages | Automate scheduled pipeline execution | +- `daily.sh`: Daily scrape/parse/upload for volatile event data (Astra, Mazevo, Comet Calendar). +- `weekly.sh`: Weekly scrape/parse/upload for academic calendars, discount programs, and degree plans. +- `monthly.sh`: Monthly updates for campus map locations and budget reports. +- `setup.sh`: Docker entrypoint script that retrieves secrets from Google Cloud Secret Manager (in production) before launching a specified runner. --- From 2ae7476f1f9b41d3b1e168e698bcf25236fc80bc Mon Sep 17 00:00:00 2001 From: LazyFork Date: Fri, 4 Sep 2026 00:44:42 -0500 Subject: [PATCH 12/42] Update Project-Structure.md --- docs/Project-Structure.md | 84 +++++++++++++++------------------------ 1 file changed, 33 insertions(+), 51 deletions(-) diff --git a/docs/Project-Structure.md b/docs/Project-Structure.md index 42ca8ea..5236467 100644 --- a/docs/Project-Structure.md +++ b/docs/Project-Structure.md @@ -5,58 +5,40 @@ This document provides an overview of the `api-tools` directory layout, explaini ## Directory Overview ```text + api-tools/ -├── data/ # Files from scrapers and parsers -├── docs/ # Source of wiki pages -├── logs/ # Timestamped log files from runtime executions -├── parser/ # Parsing, data normalization, and schema validation -│ └── testdata/ # HTML input fixtures and expected JSON, used for tests -├── runners/ # Shell automation scripts ran on Google Cloud (daily, weekly, monthly) -├── scrapers/ # Web scrapers for UTD websites, portals, and APIs -├── static-data/ # Static data that isn't scraped -├── uploader/ # Database ingestion modules for Nebula API / MongoDB -├── utils/ # Shared utilities (ChromeDP helpers, logger, regexes) -├── .env.template # Template for required environment variables -├── Dockerfile # Multi-stage container definition -├── Makefile # Task runner for Linux and macOS (build, check, test) -├── build.bat # Task runner for Windows (build, check, test) -├── cloudbuild.yaml # Google Cloud Build deployment configuration -├── go.mod / go.sum # Go dependency manifests -└── main.go # Application entrypoint and CLI flag definitions +├── data/ # Files from scrapers and parsers +├── docs/ # Source of wiki pages +├── logs/ # Timestamped log files from runtime executions +├── parser/ # Parsing, data normalization, and schema validation +│ ├── parser.go # Central parsing orchestrator +│ ├── Parser.go # Domain-specific parsers (courses, sections, maps, ASTRA, etc.) +│ ├── gradeLoader.go # Loads historical grade distributions +│ ├── requisiteParser.go # Parses prerequisite/corequisite strings +│ ├── validator.go # Validates parsed structs before upload +│ └── testdata/ # HTML fixtures and expected JSON for regression tests +├── runners/ # Shell automation scripts run on Google Cloud +├── scrapers/ # Web scrapers for UTD websites, portals, and APIs +├── static-data/ # Static data that isn't scraped +│ ├── grades/ # Historical grade-distribution CSVs by semester +│ └── budgets/ # Historical budget PDFs +├── uploader/ # Database ingestion modules for Nebula API / MongoDB +│ ├── database.go # Establishes MongoDB connections +│ ├── uploader.go # Coursebook uploads and merge/replace strategies +│ └── Uploader.go # Domain-specific uploaders +├── utils/ # Shared utilities +│ ├── methods.go # ChromeDP, environment variables, headless mode, tokens +│ ├── logger.go # Runtime logging to stdout and timestamped files +│ └── regexes.go # Shared parsing and extraction regexes +├── .env.template # Template for required environment variables +├── Dockerfile # Multi-stage container definition +├── Makefile # Build, check, and test tasks for Linux/macOS +├── build.bat # Build, check, and test tasks for Windows +├── cloudbuild.yaml # Google Cloud Build deployment configuration +├── go.mod / go.sum # Go dependency manifests +└── main.go # Application entrypoint and CLI flag definitions ``` -## Directory Breakdown - -### `parser/` -Responsible for interpreting raw files, sanitizing fields, resolving cross-references, and ensuring schema compliance: -- `Parser.go`: Dedicated parser files corresponding to each scraper (e.g., `courseParser.go`, `sectionParser.go`, `mapParser.go`, `astraParser.go`). -- `parser.go`: The central orchestrator that coordinates parsing of scraped coursebook files with static grade and profile data. -- `gradeLoader.go`: Reads historical grade distribution CSVs from `static-data/grades`. -- `requisiteParser.go`: Parses prerequisite and corequisite strings into structured dependency trees. -- `validator.go`: Validates that parsed Go structs contain all non-empty mandatory fields before upload. -- `testdata/`: Test suites containing realistic sample input files (`input.html`) and expected JSON outputs (`course.json`, `section.json`, `professors.json`) for automated regression testing. - -### `uploader/` -Handles pushing structured, validated data into MongoDB: -- `database.go`: Establishes MongoDB client connections using `MONGODB_URI`. -- `uploader.go`: Core coursebook data uploader with merge/replace strategies. -- `Uploader.go`: Specialized uploaders for events, maps, discounts, degrees, and academic calendars. - -### `static-data/` -Persistent reference datasets that cannot be scraped on-demand or serve as historical baselines: -- `static-data/grades/`: CSV files organized by semester (e.g., `22F.csv` for Fall 2022, `23S.csv` for Spring 2023, `23U.csv` for Summer 2023). -- `static-data/budgets/`: Historical PDF budget files for fiscal years where web links are no longer available. - -### `utils/` -Shared helper functions used across scrapers, parsers, and uploaders: -- `methods.go`: ChromeDP initialization, headless settings, environment variable readers (`GetEnv`), and token refresh helpers. -- `logger.go`: Dual logger implementation (`SplitWriter`) that writes output simultaneously to `stdout` and timestamped files in `logs/`. -- `regexes.go`: Compiled regular expressions for text parsing and course extraction. - -### Root Configuration Files -- `main.go`: The entrypoint of the CLI application. Parses command line flags (like `-scrape`, `-parse`, `-upload`, `-verbose`, `-headless`) and routes execution to the appropriate package functions. -- `Makefile` / `build.bat`: Scripts to automate setup, formatting checks (`staticcheck`, `gofmt`, `goimports`, `go vet`), compiling the binary, and running unit tests. -- `Dockerfile`: Defines the multi-stage Docker build for local testing and production deployment. - ## Next Step -See [How-to-Contribute.md](How-to-Contribute.md) + +See [How-to-Contribute.md](How-to-Contribute.md) From 1bd3bec2fc96ecd7bb7554bc488d70242ebf0c48 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Fri, 4 Sep 2026 00:55:25 -0500 Subject: [PATCH 13/42] Update How-to-Contribute.md --- docs/How-to-Contribute.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index 06b5c1a..bb7e4af 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -1,20 +1,21 @@ # How to Contribute -Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more. Don't worry if you don't quite know what you're doing, we're here to help. We don't expect perfection and appreciate anything you can do to help! +Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more. +Don't worry if you don't quite know what you're doing, we're here to help. We don't expect perfection and appreciate anything you can do to help! ## Find something to work on Look at our current issues, or make your own on our [issues page](../issues). -Write a comment asking if you can work on a particular issue, or assign it to yourself. You can work collaboratively, or individually on an issue. +Once you find an issue, write a comment asking if you can work on a particular issue. -Be sure to **discuss the issue frequently** especially if you don't quite understand. You can do this on GitHub, Discord, or in person. +Be sure to reference [[getting-started-with-api-tools|Getting Started.md]] to create the project. ## Create a branch Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork as they do not have permission to make a branch. -Our branch naming convention is `-` for example, `738-new-developer-docs`. GitHub can automatically create a branch for you following this convention if you select "Create a branch" on the right sidebar under "Development" +Our branch naming convention is `-` for example, `738-new-developer-docs`. ## Make Your Changes @@ -44,15 +45,13 @@ or .\build.bat test ``` -Discuss your progress frequently, and push your commits to GitHub! If you get stuck anywhere we're here to help! Avoid using AI to write code for your first contribution. - -We'd love to see you make a draft Pull Request, so we can see your progress and help you out! +We would appreciate if you make a draft Pull Request as your working on it, so we can see your progress and help you out! ## Making a Pull Request (PR) Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft. -Maintainers will review your PR and provide suggestions or approval. Don't take suggestions personally, we just want to help! +Maintainers will review your PR and provide suggestions or approval. ## Next Step From f94d457d2cf9cd7bcafef540accb34c603b51eb2 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Fri, 4 Sep 2026 00:55:50 -0500 Subject: [PATCH 14/42] Update Project-Architecture.md --- docs/Project-Architecture.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index c1cf95c..fb9281d 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -66,7 +66,8 @@ flowchart LR ## Automation -It is important to note that api-tools is mostly automated. Most data sources run via Shell scripts (`.sh`) that coordinate automated end-to-end execution of the pipeline on a scheduled cron job. +It is important to note that api-tools is mostly automated. +Most data sources run via Shell scripts (`.sh`) that coordinate automated end-to-end execution of the pipeline on a scheduled cron job in Google Cloud. Our Scripts are in `runners/`: - `daily.sh`: Daily scrape/parse/upload for volatile event data (Astra, Mazevo, Comet Calendar). From e84316c1e8ee05843fda615ef6b5a534106484e7 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Fri, 4 Sep 2026 10:53:03 -0500 Subject: [PATCH 15/42] final polish, first draft ready --- docs/Getting-Started.md | 6 +++--- docs/Guides.md | 30 +++++++++++++++++------------- docs/Project-Architecture.md | 8 ++++---- docs/Project-Structure.md | 1 - 4 files changed, 24 insertions(+), 21 deletions(-) diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index e5cd85b..ebbc50c 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -10,9 +10,9 @@ Ensure you have the following installed: - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop). - Go - You can install from the [Go website](https://go.dev/dl/), or from [Homebrew](https://brew.sh/) or another package manager for automatic updates -- **Make** _(Linux/macOS)_ — Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) -- **Google Chrome** or **Chromium** — Required for headless browser scraping -- **Docker** _(Optional)_ — For running containerized runners locally +- **Make** _(Linux/macOS)_ - Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) +- **Google Chrome** or **Chromium** - Required for headless browser scraping +- **Docker** _(Optional)_ - For running containerized runners locally **Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run. diff --git a/docs/Guides.md b/docs/Guides.md index 4c97890..b60d497 100644 --- a/docs/Guides.md +++ b/docs/Guides.md @@ -2,26 +2,30 @@ There's a lot of guides in here. You do **not** need to master everything before getting started! Here is a curated list of resources to help you learn the fundamentals. -### Git & GitHub - -> **TODO:** [Link to Nebula Git Workshop Guide here] +## Git & GitHub -[GitHub Skills Git Introduction](https://skills.github.com/) +- [Nebula GitHub Workshop](https://github.com/UTDNebula/git-workshop) +- [GitHub Skills Git Introduction](https://skills.github.com/) -### Go (Golang) -- [A Tour of Go](https://go.dev/tour/) — Official interactive introduction to Go syntax, data structures, and concurrency. -- [Go by Example](https://gobyexample.com/) — Hands-on, concise Go examples covering real-world concepts from structs to JSON. -- [Learn Go in Y Minutes](https://learnxinyminutes.com/docs/go/) — Quick visual cheat sheet for developers who want a fast syntax refresher. -### Docker & Containers -- [Docker 101 Tutorial](https://www.docker.com/101-tutorial/) — Learn how containers package code and dependencies for reliable execution. +## Go (Golang) + +- [A Tour of Go](https://go.dev/tour/) - Official interactive introduction to Go syntax, data structures, and concurrency. +- [Go by Example](https://gobyexample.com/) - Hands-on, concise Go examples. Very good for quick reference while working on issues +- [Learn Go in Y Minutes](https://learnxinyminutes.com/docs/go/) - Quick visual cheat sheet for developers who want a fast syntax refresher. + +## Docker & Containers + - [Docker Documentation: Orientation and Setup](https://docs.docker.com/get-started/) — Official step-by-step introduction to images and containers. -### Web Basics & Scraping +## Web Basics & Scraping + - [MDN Web Docs: HTML & DOM Basics](https://developer.mozilla.org/en-US/docs/Learn/HTML) — Learn how web pages are structured with elements, tags, and attributes. - [MDN Web Docs: CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors) — Understand class, ID, and attribute selectors used to find page elements. +- [Headless Browser Tutorial](https://www.zenrows.com/blog/chromedp) - Good introduction to ChromeDP - [ChromeDP Documentation & Examples](https://github.com/chromedp/examples) — Examples for browser automation and scraping dynamic JavaScript pages in Go. -### Command Line & Shell -- [Bash / CLI Crash Course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line) — Beginner-friendly guide to navigating files, running scripts, and managing processes in the terminal. +## Command Line & Shell + +- [Bash / CLI Crash Course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line) - Beginner-friendly guide to Bash diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index fb9281d..5ae9673 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -7,7 +7,7 @@ Understanding this separation of concerns is essential before modifying or addin ## Project Pipeline -The lifecycle of data moving through `api-tools` follows a four-stage progression: +The lifecycle of data moving through `api-tools` follows a three-stage progression: ```mermaid flowchart LR @@ -57,7 +57,7 @@ flowchart LR - **What they do**: Take validated data models and push them to the [Nebula API](https://github.com/utdnebula/nebula-api) database (MongoDB). - **How they work**: - - Connect to MongoDB using the official Go Mongo driver (`go.mongodb.org/mongo-driver`). + - Connect to MongoDB using the official [Go Mongo driver](`go.mongodb.org/mongo-driver`). - Support both **merge** operations (updating existing records with new fields) and **replace** operations (overwriting outdated datasets). - Compute static aggregation metrics where required before saving. - **Key Principle**: Uploaders assume data passed to them has already been parsed and validated. @@ -66,8 +66,8 @@ flowchart LR ## Automation -It is important to note that api-tools is mostly automated. -Most data sources run via Shell scripts (`.sh`) that coordinate automated end-to-end execution of the pipeline on a scheduled cron job in Google Cloud. +It is important to note every scraper api-tools is automated except for `coursebook.go`. +Most data sources are updated by Shell scripts (.sh) that run web scrapers automatically. A cron job schedules these scripts to run regularly in Google Cloud. Our Scripts are in `runners/`: - `daily.sh`: Daily scrape/parse/upload for volatile event data (Astra, Mazevo, Comet Calendar). diff --git a/docs/Project-Structure.md b/docs/Project-Structure.md index 5236467..6cc83ef 100644 --- a/docs/Project-Structure.md +++ b/docs/Project-Structure.md @@ -5,7 +5,6 @@ This document provides an overview of the `api-tools` directory layout, explaini ## Directory Overview ```text - api-tools/ ├── data/ # Files from scrapers and parsers ├── docs/ # Source of wiki pages From e27985dd7d4db68ede24e48b04602673fdc6ef7c Mon Sep 17 00:00:00 2001 From: LazyFork Date: Fri, 4 Sep 2026 23:09:53 -0500 Subject: [PATCH 16/42] remove em dashes from Guides.md --- docs/Guides.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Guides.md b/docs/Guides.md index b60d497..07c7f74 100644 --- a/docs/Guides.md +++ b/docs/Guides.md @@ -21,10 +21,10 @@ There's a lot of guides in here. You do **not** need to master everything before ## Web Basics & Scraping -- [MDN Web Docs: HTML & DOM Basics](https://developer.mozilla.org/en-US/docs/Learn/HTML) — Learn how web pages are structured with elements, tags, and attributes. -- [MDN Web Docs: CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors) — Understand class, ID, and attribute selectors used to find page elements. +- [MDN Web Docs: HTML & DOM Basics](https://developer.mozilla.org/en-US/docs/Learn/HTML) - Learn how web pages are structured with elements, tags, and attributes. +- [MDN Web Docs: CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors)- Understand class, ID, and attribute selectors used to find page elements. - [Headless Browser Tutorial](https://www.zenrows.com/blog/chromedp) - Good introduction to ChromeDP -- [ChromeDP Documentation & Examples](https://github.com/chromedp/examples) — Examples for browser automation and scraping dynamic JavaScript pages in Go. +- [ChromeDP Documentation & Examples](https://github.com/chromedp/examples) - Examples for browser automation and scraping dynamic JavaScript pages in Go. ## Command Line & Shell From e3f74b70f9edc5c6316787e35f4529a76bfbedd0 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Fri, 4 Sep 2026 23:15:49 -0500 Subject: [PATCH 17/42] Upade Project-Architecture.md --- docs/Project-Architecture.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 5ae9673..ad0e01d 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -11,11 +11,10 @@ The lifecycle of data moving through `api-tools` follows a three-stage progressi ```mermaid flowchart LR - A[UTD Data Sources\nUTD Websites] -->|Scraper| B[(Raw Data Files\ndata/)] - B -->|Parser| C{Validator\nvalidator.go} - D[(Static Data\nstatic-data/)] -->|Parser| C - C -->|Validated Models| E[Uploader] - E -->|MongoDB Insert/Merge| F[(Nebula API Database)] + A[UTD Data Sources
Websites, APIs, PDFs] -->|Scrape| B[Scraper] + B -->|Parse| C[Parser] + C -->|Upload| D[Uploader] + D -->|Insert / Merge| E[(Nebula API Database)] ``` --- From 111cfdfbcb3ed7725de9e641fcde42a1de3eed36 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Fri, 4 Sep 2026 23:19:57 -0500 Subject: [PATCH 18/42] Update Project-Architecture.md --- docs/Project-Architecture.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index ad0e01d..8fd7d1b 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -11,9 +11,17 @@ The lifecycle of data moving through `api-tools` follows a three-stage progressi ```mermaid flowchart LR - A[UTD Data Sources
Websites, APIs, PDFs] -->|Scrape| B[Scraper] - B -->|Parse| C[Parser] - C -->|Upload| D[Uploader] + A[UTD Data Sources
Websites, APIs] -->|Scrape| B + + subgraph STAGE["Three-Stages"] + direction LR + B[Scraper] + C[Parser] + D[Uploader] + B -->|Parse| C + C -->|Upload| D + end + D -->|Insert / Merge| E[(Nebula API Database)] ``` From 1a1d70d4cf226a5c7a7d5dddfda507a3e571b8a4 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Sat, 5 Sep 2026 10:03:45 -0500 Subject: [PATCH 19/42] Update README.md --- README.md | 108 +++++++++++++++++++++++++++--------------------------- 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/README.md b/README.md index 1a9bb60..27f1fa9 100644 --- a/README.md +++ b/README.md @@ -4,19 +4,19 @@ _A CLI toolchain to scrape, parse, validate, and upload UTD university data to t Project maintained by [Nebula Labs](https://about.utdnebula.com). -> 📚 **Developer Wiki**: If you are interested in developing this project, head over to the [**Developer Wiki**](docs/Home.md)! +> **Developer Wiki**: If you are interested in developing this project, head over to the [**Developer Wiki**](docs/Home.md)! --- ## Quick Navigation - [Developer Wiki](docs/Home.md) -- [Getting Started Guide](docs/getting-started.md) -- [Pipeline Architecture (Scraper vs Parser vs Uploader vs Runner)](docs/architecture.md) -- [Project Structure & Codebase Map](docs/project-structure.md) -- [Docker & Container Guide](docs/docker-guide.md) -- [Contributing Guide](docs/contributing.md) -- [Troubleshooting & FAQ](docs/troubleshooting.md) +- [Getting Started Guide](/docs/Getting-Started.md) +- [Pipeline Architecture (Scraper -> Parser -> Uploader)](/docs/Project-Architecture.md) +- [Project Structure & Codebase Map](/docs/Project-Structure.md) +- [Docker & Container Guide](/docs/Docker-Guide.md) +- [Contributing Guide](/docs/How-to-Contribute.md) +- [Troubleshooting & FAQ](/docs/Troubleshooting.md) - [Discord Community](https://discord.utdnebula.com) --- @@ -50,71 +50,71 @@ The `api-tools` command line interface supports three main modes: **scraping**, ### Global Flags -| Flag | Default | Description | -|---|---|---| -| `-i ` | `./data` | Input directory to read data from | -| `-o ` | `./data` | Output directory to write scraped/parsed data to | -| `-l ` | `./logs` | Directory to write timestamped log files to | -| `--verbose` | `false` | Enables verbose debug logging with file names and microsecond timestamps | -| `--headless`| `false` | Runs ChromeDP browser automation in background headless mode | +| Flag | Default | Description | +| ------------ | -------- | ------------------------------------------------------------------------ | +| `-i ` | `./data` | Input directory to read data from | +| `-o ` | `./data` | Output directory to write scraped/parsed data to | +| `-l ` | `./logs` | Directory to write timestamped log files to | +| `--verbose` | `false` | Enables verbose debug logging with file names and microsecond timestamps | +| `--headless` | `false` | Runs ChromeDP browser automation in background headless mode | --- -### 1. Scraping Mode (`--scrape`) +### Scraping Mode (`--scrape`) Scrapes raw data from UTD sources and saves it to `./data` (or the directory specified by `-o`). -| Command | Description | -|---|---| +| Command | Description | +| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `./api-tools --scrape --coursebook --term 24F` | Scrapes CourseBook data for Fall 2024.
• `--resume`: Resume from last completed course prefix.
• `--startprefix `: Begin at a specific course prefix (e.g. `cp_cs`). | -| `./api-tools --scrape --evals --term 24F` | Scrapes CourseBook course evaluation data for the specified term. | -| `./api-tools --scrape --profiles` | Scrapes faculty and professor directory profiles. | -| `./api-tools --scrape --astra` | Scrapes Astra schedule and room reservation data. | -| `./api-tools --scrape --mazevo` | Scrapes Mazevo event reservation data via API. | -| `./api-tools --scrape --cometCalendar` | Scrapes public UTD Comet Calendar events. | -| `./api-tools --scrape --map` | Scrapes UTD campus map building and parking coordinates. | -| `./api-tools --scrape --degrees` | Scrapes academic degrees and catalog majors. | -| `./api-tools --scrape --discounts` | Scrapes student discount program listings. | -| `./api-tools --scrape --academicCalendars` | Scrapes academic calendar PDF documents. | -| `./api-tools --scrape --budgets` | Scrapes university budget reports. | +| `./api-tools --scrape --evals --term 24F` | Scrapes CourseBook course evaluation data for the specified term. | +| `./api-tools --scrape --profiles` | Scrapes faculty and professor directory profiles. | +| `./api-tools --scrape --astra` | Scrapes Astra schedule and room reservation data. | +| `./api-tools --scrape --mazevo` | Scrapes Mazevo event reservation data via API. | +| `./api-tools --scrape --cometCalendar` | Scrapes public UTD Comet Calendar events. | +| `./api-tools --scrape --map` | Scrapes UTD campus map building and parking coordinates. | +| `./api-tools --scrape --degrees` | Scrapes academic degrees and catalog majors. | +| `./api-tools --scrape --discounts` | Scrapes student discount program listings. | +| `./api-tools --scrape --academicCalendars` | Scrapes academic calendar PDF documents. | +| `./api-tools --scrape --budgets` | Scrapes university budget reports. | --- -### 2. Parsing Mode (`-parse`) +### Parsing Mode (`-parse`) Reads raw files from `./data` (or `-i`), normalizes records against `static-data/`, validates data models, and outputs structured schemas. -| Command | Description | -|---|---| -| `./api-tools -parse` | Runs full coursebook parsing pipeline across all scraped prefixes. | -| `./api-tools -parse -gradesDir ` | Specifies custom path for grade distribution CSVs (default: `./static-data/grades`). | -| `./api-tools -parse -astra` | Parses scraped Astra scheduling data. | -| `./api-tools -parse -mazevo` | Parses scraped Mazevo event data. | -| `./api-tools -parse -cometCalendar` | Parses scraped Comet Calendar events. | -| `./api-tools -parse -map` | Parses campus map data. | -| `./api-tools -parse -degrees` | Parses degree plans from HTML. | -| `./api-tools -parse -discounts` | Parses student discount listings from HTML. | -| `./api-tools -parse -academicCalendars` | Parses academic calendar dates from PDFs. | -| `./api-tools -parse -budgets` | Parses budget reports from PDFs (use `-useBackupBudgets` to include `static-data/budgets`). | -| `./api-tools -parse -skipv` | Skips post-parse schema validation (**use with caution**). | +| Command | Description | +| --------------------------------------- | ------------------------------------------------------------------------------------------- | +| `./api-tools -parse` | Runs full coursebook parsing pipeline across all scraped prefixes. | +| `./api-tools -parse -gradesDir ` | Specifies custom path for grade distribution CSVs (default: `./static-data/grades`). | +| `./api-tools -parse -astra` | Parses scraped Astra scheduling data. | +| `./api-tools -parse -mazevo` | Parses scraped Mazevo event data. | +| `./api-tools -parse -cometCalendar` | Parses scraped Comet Calendar events. | +| `./api-tools -parse -map` | Parses campus map data. | +| `./api-tools -parse -degrees` | Parses degree plans from HTML. | +| `./api-tools -parse -discounts` | Parses student discount listings from HTML. | +| `./api-tools -parse -academicCalendars` | Parses academic calendar dates from PDFs. | +| `./api-tools -parse -budgets` | Parses budget reports from PDFs (use `-useBackupBudgets` to include `static-data/budgets`). | +| `./api-tools -parse -skipv` | Skips post-parse schema validation (**use with caution**). | --- -### 3. Upload Mode (`-upload`) +### Upload Mode (`-upload`) Pushes validated data to the MongoDB database specified in `MONGODB_URI`. -| Command | Description | -|---|---| -| `./api-tools -upload` | Uploads parsed coursebook data (merges into existing records by default). | -| `./api-tools -upload -replace` | Overwrites/replaces existing records instead of merging. | -| `./api-tools -upload -static` | Uploads static aggregations only. | -| `./api-tools -upload -events` | Uploads Astra, Mazevo, and Comet Calendar event datasets. | -| `./api-tools -upload -map` | Uploads campus map locations. | -| `./api-tools -upload -degrees` | Uploads degree plans. | -| `./api-tools -upload -discounts` | Uploads discount programs. | -| `./api-tools -upload -academicCalendars` | Uploads academic calendar dates. | -| `./api-tools -upload -budgets` | Uploads budget data. | +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------- | +| `./api-tools -upload` | Uploads parsed coursebook data (merges into existing records by default). | +| `./api-tools -upload -replace` | Overwrites/replaces existing records instead of merging. | +| `./api-tools -upload -static` | Uploads static aggregations only. | +| `./api-tools -upload -events` | Uploads Astra, Mazevo, and Comet Calendar event datasets. | +| `./api-tools -upload -map` | Uploads campus map locations. | +| `./api-tools -upload -degrees` | Uploads degree plans. | +| `./api-tools -upload -discounts` | Uploads discount programs. | +| `./api-tools -upload -academicCalendars` | Uploads academic calendar dates. | +| `./api-tools -upload -budgets` | Uploads budget data. | --- @@ -136,7 +136,7 @@ docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=weekly.sh my-runner:l docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=monthly.sh my-runner:local ``` -For more details on Docker execution and cloud configuration, see the [Docker Guide](docs/docker-guide.md). +For more details on Docker execution and cloud configuration, see the [Docker Guide](/docs/Docker-Guide.md). --- From 108e08b85df81345cc985848942bc6476664fd41 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Sat, 5 Sep 2026 10:03:54 -0500 Subject: [PATCH 20/42] Update Docker-Guide.md --- docs/Docker-Guide.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/Docker-Guide.md b/docs/Docker-Guide.md index 9e465d4..817410e 100644 --- a/docs/Docker-Guide.md +++ b/docs/Docker-Guide.md @@ -6,13 +6,15 @@ This guide explains how Docker is used in `api-tools`, how our multi-stage Docke ## Why Do We Use Docker? -Running web scrapers and PDF parsers requires several system-level dependencies: +Running web scrapers requires several system-level dependencies: + - **Chromium / Google Chrome**: Needed by ChromeDP for headless browser automation. - **Poppler Utilities (`pdftotext`)**: Needed for converting academic calendar PDFs into raw text. - **Go Runtime**: Needed to compile the application binary. - **Google Cloud SDK (`gcloud`)**: Needed in production to access cloud secrets. -Instead of requiring every contributor to manually install and configure these system packages, Docker packages everything into a standardized, lightweight Linux container. This ensures that scripts execute identically on developer laptops (Linux, macOS, Windows) and in production (Google Cloud Platform). +Instead of requiring every contributor to manually install and configure these system packages, Docker packages everything into a standardized, lightweight Linux container. +This ensures that scripts execute identically on developer laptops (Linux, macOS, Windows) and in production (Google Cloud Platform). > [!NOTE] > **New to Docker?** Check out the official [Docker 101 Tutorial](https://www.docker.com/101-tutorial/) or the [Docker Get Started Guide](https://docs.docker.com/get-started/) to learn the basics of images and containers. @@ -44,18 +46,23 @@ Our [`Dockerfile`](file:///var/home/justin/Documents/Projects/api-tools/Dockerfi ## Running Docker Locally ### Step 1: Ensure Your `.env` File Exists + Make sure you have copied `.env.template` to `.env` in the project root: + ```bash cp .env.template .env ``` ### Step 2: Build the Local Docker Image + Build using the `--target local` flag to include your local `.env` file: + ```bash docker build --target local -t my-runner:local . ``` ### Step 3: Run a Runner Script + Run the container by specifying the environment mode and target script name: ```bash @@ -63,6 +70,7 @@ docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=daily.sh my-runner:lo ``` You can replace `daily.sh` with any runner in the `runners/` directory: + - `daily.sh` — Scrapes, parses, and uploads events (Astra, Mazevo, Comet Calendar). - `weekly.sh` — Scrapes, parses, and uploads academic calendars, discounts, and degrees. - `monthly.sh` — Scrapes, parses, and uploads map locations and budgets. @@ -83,8 +91,4 @@ When the Docker container starts, it executes [`runners/setup.sh`](file:///var/h - Dynamically fetches secrets and creates the `.env` file at runtime. - Runs `/app/runners/$RUNNER_SCRIPT_NAME`. ---- - -## Next Steps -- Learn how to contribute code in the [**Contributing Guide**](contributing.md). -- Resolve any container or browser errors in the [**Troubleshooting Guide**](troubleshooting.md). +- Resolve any container or browser errors in the [**Troubleshooting Guide**](/docs/Troubleshooting.md). From 3ff5334d829b94a4a872336de039376952435773 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Sat, 5 Sep 2026 12:11:31 -0500 Subject: [PATCH 21/42] Update Troubleshooting.md --- docs/Troubleshooting.md | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md index ec6618e..4677b21 100644 --- a/docs/Troubleshooting.md +++ b/docs/Troubleshooting.md @@ -4,12 +4,14 @@ This guide provides fast, actionable solutions to common issues encountered when --- -## 1. Chromium & ChromeDP Issues +## Chromium & ChromeDP Issues ### Issue: "exec: 'chromium': executable file not found in $PATH" or Chrome fails to launch + - **Cause**: ChromeDP cannot find a Chromium or Google Chrome binary on your operating system. - **Solutions**: - **Linux**: Install Chromium via your package manager: + ```bash # Ubuntu / Debian sudo apt update && sudo apt install -y chromium-browser @@ -18,30 +20,37 @@ This guide provides fast, actionable solutions to common issues encountered when # Arch Linux sudo pacman -S chromium ``` + - **macOS / Windows**: Ensure [Google Chrome](https://www.google.com/chrome/) is installed in the standard applications directory. - **Environment Variables**: If Chrome is installed in a non-standard location, specify its path explicitly before running `api-tools`: + ```bash export CHROMIUM_BIN="/path/to/custom/chrome" export GOOGLE_CHROME_BIN="/path/to/custom/chrome" ``` ### Issue: Chrome window opens during scraping and interferes with work + - **Solution**: Pass the `-headless` flag to run ChromeDP invisibly in background headless mode: + ```bash ./api-tools -scrape -coursebook -term 24F -headless ``` --- -## 2. Environment Variables & MongoDB Connection Issues +## Environment Variables & MongoDB Connection Issues ### Issue: Panic: ` is missing from .env!` + - **Cause**: An authenticated scraper or uploader attempted to read a required environment variable from `.env` via `utils.GetEnv`, but the key was either missing or empty. - **Solutions**: 1. Ensure you have copied `.env.template` to `.env`: + ```bash cp .env.template .env ``` + 2. For Coursebook, Astra, or Mazevo scrapers, populate the required credentials: - `LOGIN_NETID` / `LOGIN_PASSWORD` (UTD NetID login) - `LOGIN_ASTRA_USERNAME` / `LOGIN_ASTRA_PASSWORD` @@ -49,14 +58,18 @@ This guide provides fast, actionable solutions to common issues encountered when 3. If you only want to parse existing local data or run tests, you do **not** need these credentials; avoid calling authenticated scraping flags. ### Issue: MongoDB connection timed out / "server selection error" + - **Cause**: The uploader could not establish a connection to the MongoDB instance specified in `MONGODB_URI`. - **Solutions**: - **Verify Connection String**: Check that `MONGODB_URI` in `.env` is formatted correctly: + ```env MONGODB_URI="mongodb+srv://:@cluster0.abcde.mongodb.net/?retryWrites=true&w=majority" ``` + - **Check Network & IP Whitelist**: If using MongoDB Atlas, ensure your current IP address is whitelisted under Atlas Network Access. - **Local Development**: If testing locally with a local MongoDB container or service, verify MongoDB is running: + ```bash # Test local MongoDB connection mongosh "mongodb://localhost:27017" @@ -64,17 +77,21 @@ This guide provides fast, actionable solutions to common issues encountered when --- -## 3. Locating & Inspecting Log Files +## Locating & Inspecting Log Files ### How Logging Works + When `api-tools` executes, `utils.NewSplitWriter` writes all log output to two destinations simultaneously: + 1. **Console (`stdout`)**: Live output in your terminal window. 2. **Log File (`logs/*.log`)**: A persistent log file saved to the `logs/` directory named with the execution timestamp: + ``` logs/8-31-2026T12-30-15.log ``` ### Enabling Verbose Debug Logging + If a scraper or parser is behaving unexpectedly, re-run the command with the `-verbose` flag: ```bash @@ -82,6 +99,7 @@ If a scraper or parser is behaving unexpectedly, re-run the command with the `-v ``` Verbose mode enables: + - Microsecond timestamp precision (`log.Lmicroseconds`). - Source code filename and line number for every log statement (`log.Lshortfile`). - Additional internal debug logs (`utils.Lverbose`). @@ -91,6 +109,7 @@ Verbose mode enables: ## Still Stuck? Reach out to the team on [Discord](https://discord.utdnebula.com) with: + 1. The exact command you ran. 2. The relevant lines from your latest log file in `logs/`. 3. Your operating system and Go version (`go version`). From 80cd2cd9c17ae90cf2e7a528905ffc1a1b659a9b Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 10:53:13 -0500 Subject: [PATCH 22/42] Requested changes to How-to-Contribute.md --- docs/How-to-Contribute.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index bb7e4af..89649b3 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -7,7 +7,9 @@ Don't worry if you don't quite know what you're doing, we're here to help. We do Look at our current issues, or make your own on our [issues page](../issues). -Once you find an issue, write a comment asking if you can work on a particular issue. +Once you find an issue, write a comment asking if you can work on it. + +If you're unsure about something, **don't hesitate to ask for help**. We're happy to answer questions and help you work through issues. You can reach out on GitHub, Discord, or in person. Be sure to reference [[getting-started-with-api-tools|Getting Started.md]] to create the project. @@ -15,8 +17,6 @@ Be sure to reference [[getting-started-with-api-tools|Getting Started.md]] to cr Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork as they do not have permission to make a branch. -Our branch naming convention is `-` for example, `738-new-developer-docs`. - ## Make Your Changes It's time to code! @@ -51,7 +51,7 @@ We would appreciate if you make a draft Pull Request as your working on it, so w Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft. -Maintainers will review your PR and provide suggestions or approval. +Maintainers will review your PR and may provide suggestions or request changes before approval. ## Next Step From c0f2a65271e221f65320dc9b3ccd29d1b1d07612 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 11:03:21 -0500 Subject: [PATCH 23/42] Requested Changes in Project-Architecture.md --- docs/Project-Architecture.md | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 8fd7d1b..4a279bc 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -27,16 +27,6 @@ flowchart LR --- -## Component Comparison - -| Component | Directory | Input | Output | Side Effects | Key Goal | -| ------------ | ----------- | ------------------------ | ------------------------------- | -------------------- | --------------------------------------------- | -| **Scraper** | `scrapers/` | UTD websites, APIs | Raw `.html`, `.json` in `data/` | Writes files to disk | Capture raw data from UTD websites | -| **Parser** | `parser/` | `data/` + `static-data/` | Validated Go structs | **None** (read-only) | Extract & validate structured data in `data/` | -| **Uploader** | `uploader/` | Validated Go structs | MongoDB documents | DB inserts/updates | Persist data to API database | - ---- - ## The Three Core Components ### Scrapers (`scrapers/`) @@ -73,14 +63,7 @@ flowchart LR ## Automation -It is important to note every scraper api-tools is automated except for `coursebook.go`. -Most data sources are updated by Shell scripts (.sh) that run web scrapers automatically. A cron job schedules these scripts to run regularly in Google Cloud. -Our Scripts are in `runners/`: - -- `daily.sh`: Daily scrape/parse/upload for volatile event data (Astra, Mazevo, Comet Calendar). -- `weekly.sh`: Weekly scrape/parse/upload for academic calendars, discount programs, and degree plans. -- `monthly.sh`: Monthly updates for campus map locations and budget reports. -- `setup.sh`: Docker entrypoint script that retrieves secrets from Google Cloud Secret Manager (in production) before launching a specified runner. +Most data sources are updated automatically by web scrapers run through shell scripts (.sh). These scripts are scheduled to run regularly via cron jobs in Google Cloud. The scripts are located in `runners/`. --- From 99379a209267cff04c79c5cb1702bd4100be4680 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 11:03:34 -0500 Subject: [PATCH 24/42] Requested changes in Project-Structure.md --- docs/Project-Structure.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/Project-Structure.md b/docs/Project-Structure.md index 6cc83ef..7682e51 100644 --- a/docs/Project-Structure.md +++ b/docs/Project-Structure.md @@ -10,25 +10,25 @@ api-tools/ ├── docs/ # Source of wiki pages ├── logs/ # Timestamped log files from runtime executions ├── parser/ # Parsing, data normalization, and schema validation -│ ├── parser.go # Central parsing orchestrator -│ ├── Parser.go # Domain-specific parsers (courses, sections, maps, ASTRA, etc.) -│ ├── gradeLoader.go # Loads historical grade distributions -│ ├── requisiteParser.go # Parses prerequisite/corequisite strings -│ ├── validator.go # Validates parsed structs before upload -│ └── testdata/ # HTML fixtures and expected JSON for regression tests +│ ├── parser.go # Central parsing orchestrator +│ ├── Parser.go # Domain-specific parsers (courses, sections, maps, ASTRA, etc.) +│ ├── gradeLoader.go # Loads historical grade distributions +│ ├── requisiteParser.go # Parses prerequisite/corequisite strings +│ ├── validator.go # Validates parsed structs before upload +│ └── testdata/ # HTML fixtures and expected JSON for regression tests ├── runners/ # Shell automation scripts run on Google Cloud ├── scrapers/ # Web scrapers for UTD websites, portals, and APIs ├── static-data/ # Static data that isn't scraped -│ ├── grades/ # Historical grade-distribution CSVs by semester -│ └── budgets/ # Historical budget PDFs +│ ├── grades/ # Historical grade-distribution CSVs by semester +│ └── budgets/ # Historical budget PDFs ├── uploader/ # Database ingestion modules for Nebula API / MongoDB -│ ├── database.go # Establishes MongoDB connections -│ ├── uploader.go # Coursebook uploads and merge/replace strategies -│ └── Uploader.go # Domain-specific uploaders +│ ├── database.go # Establishes MongoDB connections +│ ├── uploader.go # Coursebook uploads and merge/replace strategies +│ └── Uploader.go # Domain-specific uploaders ├── utils/ # Shared utilities -│ ├── methods.go # ChromeDP, environment variables, headless mode, tokens -│ ├── logger.go # Runtime logging to stdout and timestamped files -│ └── regexes.go # Shared parsing and extraction regexes +│ ├── methods.go # ChromeDP, environment variables, headless mode, tokens +│ ├── logger.go # Runtime logging to stdout and timestamped files +│ └── regexes.go # Shared parsing and extraction regexes ├── .env.template # Template for required environment variables ├── Dockerfile # Multi-stage container definition ├── Makefile # Build, check, and test tasks for Linux/macOS From b953531f3ca8f0a9f400273bf5ef8d2e342c54e7 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 11:08:14 -0500 Subject: [PATCH 25/42] Requested changes in Docker-Guide.md --- docs/Docker-Guide.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/Docker-Guide.md b/docs/Docker-Guide.md index 817410e..9ded565 100644 --- a/docs/Docker-Guide.md +++ b/docs/Docker-Guide.md @@ -14,7 +14,6 @@ Running web scrapers requires several system-level dependencies: - **Google Cloud SDK (`gcloud`)**: Needed in production to access cloud secrets. Instead of requiring every contributor to manually install and configure these system packages, Docker packages everything into a standardized, lightweight Linux container. -This ensures that scripts execute identically on developer laptops (Linux, macOS, Windows) and in production (Google Cloud Platform). > [!NOTE] > **New to Docker?** Check out the official [Docker 101 Tutorial](https://www.docker.com/101-tutorial/) or the [Docker Get Started Guide](https://docs.docker.com/get-started/) to learn the basics of images and containers. From cd35a75055ba73c537f32457fff4acd17ee3a86a Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 11:43:12 -0500 Subject: [PATCH 26/42] Requested changes on Guides.md --- docs/Guides.md | 58 +++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 48 insertions(+), 10 deletions(-) diff --git a/docs/Guides.md b/docs/Guides.md index 07c7f74..a89a35a 100644 --- a/docs/Guides.md +++ b/docs/Guides.md @@ -2,30 +2,68 @@ There's a lot of guides in here. You do **not** need to master everything before getting started! Here is a curated list of resources to help you learn the fundamentals. -## Git & GitHub +## General Guides + +### Git & GitHub + +Learn how to manage code changes, work with branches, and collaborate using GitHub. - [Nebula GitHub Workshop](https://github.com/UTDNebula/git-workshop) - [GitHub Skills Git Introduction](https://skills.github.com/) -## Go (Golang) +### Go (Golang) + +Learn the Go programming language and the fundamentals needed to work on our codebase. - [A Tour of Go](https://go.dev/tour/) - Official interactive introduction to Go syntax, data structures, and concurrency. -- [Go by Example](https://gobyexample.com/) - Hands-on, concise Go examples. Very good for quick reference while working on issues +- [Go by Example](https://gobyexample.com/) - Hands-on, concise Go examples. Very good for quick reference while working on issues. - [Learn Go in Y Minutes](https://learnxinyminutes.com/docs/go/) - Quick visual cheat sheet for developers who want a fast syntax refresher. -## Docker & Containers +### Docker & Containers + +Learn how to build, run, and manage applications using Docker containers. + +- [Docker Documentation: Orientation and Setup](https://docs.docker.com/get-started/) - Official step-by-step introduction to images and containers. -- [Docker Documentation: Orientation and Setup](https://docs.docker.com/get-started/) — Official step-by-step introduction to images and containers. +### Web Basics -## Web Basics & Scraping +Learn how websites are structured and how HTML and CSS work together. - [MDN Web Docs: HTML & DOM Basics](https://developer.mozilla.org/en-US/docs/Learn/HTML) - Learn how web pages are structured with elements, tags, and attributes. -- [MDN Web Docs: CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors)- Understand class, ID, and attribute selectors used to find page elements. -- [Headless Browser Tutorial](https://www.zenrows.com/blog/chromedp) - Good introduction to ChromeDP +- [MDN Web Docs: CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors) - Understand class, ID, and attribute selectors used to find page elements. + +## Scraper Libraries + +ChromeDP lets us control a headless Chrome browser to interact with and scrape dynamic websites. + +### ChromeDP + +- [Headless Browser Tutorial](https://www.zenrows.com/blog/chromedp) - Good introduction to ChromeDP. - [ChromeDP Documentation & Examples](https://github.com/chromedp/examples) - Examples for browser automation and scraping dynamic JavaScript pages in Go. -## Command Line & Shell +## Parser Libraries + +### Goquery + +Goquery lets us parse and query HTML, making it easier to extract data from web pages. + +- [Goquery Guide](https://www.zenrows.com/blog/goquery) - Excellent follow-up to the ChromeDP Browser Tutorial. +- [Goquery Guide](https://zetcode.com/golang/goquery/) - More in-depth guide to Goquery. + +## Uploader Libraries + +### MongoDB Go Driver + +The MongoDB Go Driver lets us store and retrieve scraped data from MongoDB using Go. + +- [MongoDB Go Driver Official Docs](https://www.mongodb.com/docs/drivers/go/current/) - Official documentation for using MongoDB with go + +## Automation + +### Bash + +Learn the Go programming language and the fundamentals needed to work on our codebase. -- [Bash / CLI Crash Course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line) - Beginner-friendly guide to Bash +- [Bash / CLI Crash Course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line) - Beginner-friendly guide to Bash. From ecaae21a17b04b5155f9852e3a1693eb2499cc26 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 11:50:52 -0500 Subject: [PATCH 27/42] Requested changes in Project-Architecture.md --- docs/Project-Architecture.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 4a279bc..b5cd67c 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -1,7 +1,7 @@ # Pipeline Architecture & Core Concepts -`api-tools` is built around a unidirectional, staged data pipeline. Each stage has a single, well-defined responsibility. -Understanding this separation of concerns is essential before modifying or adding code. +`api-tools` uses a simple, one-way data pipeline with several stages. Each stage has one clear job. +Understanding how these stages work together is important before adding or changing code. --- @@ -45,7 +45,7 @@ flowchart LR - **How they work**: - Parse HTML via Go tokenizers and CSS selectors (`golang.org/x/net/html`) - Cross-reference scraped data with static datasets (grade CSVs, budget PDFs) - - Validate structs against the Nebula API schema via [`parser/validator.go`](file:///var/home/justin/Documents/Projects/api-tools/parser/validator.go) + - Validate structs against the Nebula API schema via `parser/validator.go` - **Key Principle**: Input files in `data/` are strictly **immutable**. Parsers must treat input files as read-only and never modify raw scraped data. --- From 897509b40c59747dad123b02bd9d474709f45721 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 11:57:58 -0500 Subject: [PATCH 28/42] Add AI policy mention --- docs/How-to-Contribute.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index 89649b3..9aeb995 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -47,6 +47,8 @@ or We would appreciate if you make a draft Pull Request as your working on it, so we can see your progress and help you out! +If you use AI tools while contributing, please review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy). + ## Making a Pull Request (PR) Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft. From 7cc300202a3a419a1410bc116b56484296bdbc01 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 12:02:10 -0500 Subject: [PATCH 29/42] Update AI mention on How-to-Contribute.md --- docs/How-to-Contribute.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index 9aeb995..892cf8d 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -7,6 +7,9 @@ Don't worry if you don't quite know what you're doing, we're here to help. We do Look at our current issues, or make your own on our [issues page](../issues). +For any issue, please write the code yourself rather than using AI to generate it. +Please review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy) for more details on acceptable uses of AI. + Once you find an issue, write a comment asking if you can work on it. If you're unsure about something, **don't hesitate to ask for help**. We're happy to answer questions and help you work through issues. You can reach out on GitHub, Discord, or in person. @@ -47,8 +50,6 @@ or We would appreciate if you make a draft Pull Request as your working on it, so we can see your progress and help you out! -If you use AI tools while contributing, please review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy). - ## Making a Pull Request (PR) Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft. From bb840868fa8d7bc1158b0c1232baaba8d5b70379 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Mon, 7 Sep 2026 12:06:03 -0500 Subject: [PATCH 30/42] update mermaid diagram in Project-Architecture.md --- docs/Project-Architecture.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index b5cd67c..a886e9d 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -11,18 +11,14 @@ The lifecycle of data moving through `api-tools` follows a three-stage progressi ```mermaid flowchart LR - A[UTD Data Sources
Websites, APIs] -->|Scrape| B + A[UTD Data Sources
Websites, APIs] --> B - subgraph STAGE["Three-Stages"] + subgraph STAGE["Three Stages"] direction LR - B[Scraper] - C[Parser] - D[Uploader] - B -->|Parse| C - C -->|Upload| D + B[Scraper] --> C[Parser] --> D[Uploader] end - D -->|Insert / Merge| E[(Nebula API Database)] + D --> E[(Nebula API Database)] ``` --- From 0ff37fc8da327397ad99a4f783c918704e52a2c9 Mon Sep 17 00:00:00 2001 From: Yogeb Date: Tue, 8 Sep 2026 14:57:50 -0500 Subject: [PATCH 31/42] Address requested changes --- docs/How-to-Contribute.md | 6 +++--- docs/Project-Architecture.md | 7 ++++--- docs/Project-Structure.md | 2 +- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index 892cf8d..cd65251 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -7,14 +7,14 @@ Don't worry if you don't quite know what you're doing, we're here to help. We do Look at our current issues, or make your own on our [issues page](../issues). -For any issue, please write the code yourself rather than using AI to generate it. -Please review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy) for more details on acceptable uses of AI. +For beginner issues, please write the code yourself rather than using AI to generate it. These issues are designed to help new members learn the codebase and build familiarity with the project. +For all other issues, please review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy) for more information on acceptable uses of AI. Once you find an issue, write a comment asking if you can work on it. If you're unsure about something, **don't hesitate to ask for help**. We're happy to answer questions and help you work through issues. You can reach out on GitHub, Discord, or in person. -Be sure to reference [[getting-started-with-api-tools|Getting Started.md]] to create the project. +Be sure to reference [Getting Started.md](/docs/Getting-Started.md) to create the project. ## Create a branch diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index a886e9d..4b223bc 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -1,13 +1,14 @@ -# Pipeline Architecture & Core Concepts +# Pipeline Architecture -`api-tools` uses a simple, one-way data pipeline with several stages. Each stage has one clear job. +`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api) to collect, process, validate, and upload data from various UTD data sources into the Nebula API. +`api-tools` uses a one-way data pipeline with several stages. Each stage has one clear job. Understanding how these stages work together is important before adding or changing code. --- ## Project Pipeline -The lifecycle of data moving through `api-tools` follows a three-stage progression: +The lifecycle of data moving through `api-tools` follows a three-stage process: ```mermaid flowchart LR diff --git a/docs/Project-Structure.md b/docs/Project-Structure.md index 7682e51..6b72580 100644 --- a/docs/Project-Structure.md +++ b/docs/Project-Structure.md @@ -2,7 +2,7 @@ This document provides an overview of the `api-tools` directory layout, explaining what each folder contains and guiding you on where to make changes when adding features or fixing bugs. -## Directory Overview +## Project Structure ```text api-tools/ From 0a37658e595146c9569f8439337d60a47787eee8 Mon Sep 17 00:00:00 2001 From: Yogeb Date: Tue, 8 Sep 2026 15:05:06 -0500 Subject: [PATCH 32/42] Update Project-Architecture.md --- docs/Project-Architecture.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 4b223bc..d53d5bb 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -1,7 +1,9 @@ # Pipeline Architecture `api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api) to collect, process, validate, and upload data from various UTD data sources into the Nebula API. -`api-tools` uses a one-way data pipeline with several stages. Each stage has one clear job. +Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. + +`api-tools` follows a one-way data pipeline with several stages. Each stage has one clear job. Understanding how these stages work together is important before adding or changing code. --- From 18ed8385f10510dfb9dc3cf8a003cc78ac99bb0c Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 9 Sep 2026 16:25:05 -0500 Subject: [PATCH 33/42] Add ssh and more specifics to getting-started.md --- docs/Getting-Started.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index ebbc50c..a87d0f9 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -6,13 +6,14 @@ This guide walks you through setting up your local development environment, conf Ensure you have the following installed: -- Git +- Git — For version control - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop). -- Go +- Go — For building and running - You can install from the [Go website](https://go.dev/dl/), or from [Homebrew](https://brew.sh/) or another package manager for automatic updates -- **Make** _(Linux/macOS)_ - Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) -- **Google Chrome** or **Chromium** - Required for headless browser scraping -- **Docker** _(Optional)_ - For running containerized runners locally +- Make *(Linux/macOS)* — For build automation + - Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) +- Google Chrome or Chromium — For headless browser scraping +- Docker *(Optional)* — For running containerized runners locally **Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run. @@ -25,11 +26,22 @@ For example instead of `make setup`, you can run `.\build.bat setup`. Clone the repository with `git clone` and `cd` into the project directory or open it in your code editor +**HTTPS:** + ```bash git clone https://github.com/UTDNebula/api-tools.git cd api-tools ``` +or + +**SSH:** + +```bash +git clone git@github.com:UTDNebula/api-tools.git +cd api-tools +``` + ### Install Development Tooling Install the Go static analysis and formatting tools (`staticcheck` and `goimports`): From 5ae93681280df525a777c4594aba722084570e08 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 9 Sep 2026 17:13:38 -0500 Subject: [PATCH 34/42] Add more libraries to Proejct-Architecture --- docs/Project-Architecture.md | 74 ++++++++++++++++++++---------------- 1 file changed, 41 insertions(+), 33 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index d53d5bb..8b245b0 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -1,12 +1,9 @@ -# Pipeline Architecture +# Project Architecture -`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api) to collect, process, validate, and upload data from various UTD data sources into the Nebula API. -Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. +`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded into the Nebula API. The data comes from various UTD data sources including coursebook, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. -`api-tools` follows a one-way data pipeline with several stages. Each stage has one clear job. -Understanding how these stages work together is important before adding or changing code. - ---- +Tools are written in Go. ADD GO REFERENCE! + ## Project Pipeline @@ -24,47 +21,58 @@ flowchart LR D --> E[(Nebula API Database)] ``` ---- - -## The Three Core Components - ### Scrapers (`scrapers/`) -- **What they do**: Connect to external UTD web servers, APIs, and portals to download raw information. -- **How they work**: Scrapers fetch data via: - - Standard HTTP requests (`net/http`) - - Web scraping via ([ChromeDP](https://github.com/chromedp/chromedp)) -- **Key Principle**: **Scrapers do NOT validate or parse data.** Their sole job is to faithfully capture raw data from the source and save it to disk in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). +Scrapers connect to external UTD web servers, APIs, and portals to download raw information. They capture raw data (`.html`, `.json`, etc.) and store it in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). ---- +Libraries used: + +- `net/http`: Go Standard library HTTP client for fetching static pages and making REST API calls. +- [ChromeDP](https://github.com/chromedp/chromedp): Headless browser automation via the Chrome DevTools Protocol for scraping dynamic, JavaScript-rendered, or authenticated pages (WAT NetID login, Coursebook, Astra Schedule). +- [cdproto](https://github.com/chromedp/cdproto): Chrome DevTools Protocol definitions used for low-level network event handling, cookie extraction, and DOM interaction. +- [fastjson](https://github.com/valyala/fastjson): High-performance, zero-allocation JSON parser for reading large raw payloads (e.g., Astra room scheduling events). ### Parsers (`parser/`) -- **What they do**: Read raw files produced by scrapers alongside static reference files, extract meaningful fields, normalize inconsistent formatting, and assemble validated Go data structures.to -- **How they work**: - - Parse HTML via Go tokenizers and CSS selectors (`golang.org/x/net/html`) - - Cross-reference scraped data with static datasets (grade CSVs, budget PDFs) - - Validate structs against the Nebula API schema via `parser/validator.go` -- **Key Principle**: Input files in `data/` are strictly **immutable**. Parsers must treat input files as read-only and never modify raw scraped data. +Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then extract meaningful fields, fix inconsistent formatting, and turn raw data into validated Go data structures matching our schema. Parsers do not modify input data. + +Libraries used: ---- +- [goquery](https://github.com/PuerkitoBio/goquery): jQuery-like DOM traversal and CSS selector querying for HTML documents (courses, sections, professors, degree requirements, discounts). +- [golang.org/x/net](https://pkg.go.dev/golang.org/x/net): Low-level HTML tokenization and atom lookup (`html/atom`) during section parsing. +- [google.golang.org/genai](https://pkg.go.dev/google.golang.org/genai): Google Gemini LLM client (Vertex AI / Gemini API) for structured schema-based extraction from unstructured documents (academic calendars and university budgets). +- [nebula-api/api](https://github.com/UTDNebula/nebula-api): Canonical data models (`api/schema`) used to structure and validate parsed data via `parser/validator.go`. +- [dongri/phonenumber](https://github.com/dongri/phonenumber): Phone number parsing and normalization for student discount programs. +- [golang.org/x/text](https://pkg.go.dev/golang.org/x/text): Unicode text transformation and casing rules (`cases`, `language`) for event names and titles. +- [google/go-cmp](https://github.com/google/go-cmp): Deep equality comparisons and struct diffing across unit and regression test suites. ### Uploaders (`uploader/`) -- **What they do**: Take validated data models and push them to the [Nebula API](https://github.com/utdnebula/nebula-api) database (MongoDB). -- **How they work**: - - Connect to MongoDB using the official [Go Mongo driver](`go.mongodb.org/mongo-driver`). - - Support both **merge** operations (updating existing records with new fields) and **replace** operations (overwriting outdated datasets). - - Compute static aggregation metrics where required before saving. -- **Key Principle**: Uploaders assume data passed to them has already been parsed and validated. +Uploaders take validated data models and push them to the Nebula API MongoDB database. ---- +Libraries used: + +- [go.mongodb.org/mongo-driver](https://pkg.go.dev/go.mongodb.org/mongo-driver): Official MongoDB driver (`mongo`, `bson`) for database connection management, BSON document serialization, bulk upserts, and aggregation pipelines. +- [nebula-api/api](https://github.com/UTDNebula/nebula-api): Shared schema definitions ensuring stored documents match Nebula API collection expectations. ## Automation -Most data sources are updated automatically by web scrapers run through shell scripts (.sh). These scripts are scheduled to run regularly via cron jobs in Google Cloud. The scripts are located in `runners/`. +Most data sources are updated automatically through orchestration shell scripts that coordinate scraper, parser, and uploader execution. These scripts run in containerized environments scheduled by cron jobs in Google Cloud. The automation scripts are located in `runners/`. + +Tools and Libraries used: + +- **Bash (`runners/*.sh`)**: Command-line orchestration scripts that run end-to-end pipeline sequences for different data sources. +- **Docker**: Multi-stage container definitions providing reproducible execution environments across local and cloud runners. +- **Google Cloud Build & Cloud Scheduler**: Cloud CI/CD and cron scheduling services that trigger periodic pipeline executions. + +## Lifecycle and workflow + +We use several tools to aid in the development lifecycle and workflow: ---- +- `testing`: Go standard library testing package for unit tests across the project. +- **GitHub Actions**: Runs automated CI workflows for PRs, and deploys API-Tools to Google Artifact Registry and Google Cloud Run jobs. +- [godotenv](https://github.com/joho/godotenv): Parses `.env` files and loads variables such as URIs and credentials at startup. +- [getsentry/sentry-go](https://github.com/getsentry/sentry-go): Production error tracking, panic handling, and trace sampling. Initialized in `main.go`, it captures unhandled runtime failures during scraper, parser, and uploader runs and flushes diagnostics to Sentry before exit. ## Next Step From 77620f9e53c5317c1654bc93a50d76abbf24935e Mon Sep 17 00:00:00 2001 From: LazyFork Date: Wed, 9 Sep 2026 20:00:30 -0500 Subject: [PATCH 35/42] Slight rephrase in Automation section in Project-Architecture.md I found orchrestration could be replaced with simpler language --- docs/Project-Architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 8b245b0..0a6eb8e 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -57,7 +57,7 @@ Libraries used: ## Automation -Most data sources are updated automatically through orchestration shell scripts that coordinate scraper, parser, and uploader execution. These scripts run in containerized environments scheduled by cron jobs in Google Cloud. The automation scripts are located in `runners/`. +Most data sources are updated automatically through shell scripts that coordinate scraper, parser, and uploader execution. These scripts run in containerized environments scheduled by cron jobs in Google Cloud. The automation scripts are located in runners/. Tools and Libraries used: From 156e7eb94cb4877aff9d6278537b812bc06ffbb3 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Wed, 9 Sep 2026 20:15:15 -0500 Subject: [PATCH 36/42] Minor adjustments to achieve simple language in Project-Architecture.md --- docs/Project-Architecture.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 0a6eb8e..5570343 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -1,13 +1,13 @@ # Project Architecture -`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded into the Nebula API. The data comes from various UTD data sources including coursebook, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. +`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded to the Nebula API. The data comes from various UTD data sources including coursebook, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. Tools are written in Go. ADD GO REFERENCE! ## Project Pipeline -The lifecycle of data moving through `api-tools` follows a three-stage process: +Data moves through `api-tools` in three stages: ```mermaid flowchart LR @@ -57,7 +57,7 @@ Libraries used: ## Automation -Most data sources are updated automatically through shell scripts that coordinate scraper, parser, and uploader execution. These scripts run in containerized environments scheduled by cron jobs in Google Cloud. The automation scripts are located in runners/. +Most data sources are updated automatically through shell scripts that coordinate scraper, parser, and uploader execution. These scripts run in containerized environments scheduled by cron jobs in Google Cloud. The automation scripts are located in `runners/`. Tools and Libraries used: @@ -67,7 +67,7 @@ Tools and Libraries used: ## Lifecycle and workflow -We use several tools to aid in the development lifecycle and workflow: +We use several tools while developing the API: - `testing`: Go standard library testing package for unit tests across the project. - **GitHub Actions**: Runs automated CI workflows for PRs, and deploys API-Tools to Google Artifact Registry and Google Cloud Run jobs. From e0b8a1c24b41cfcd040e1e6e2fe6ed8c60e243d7 Mon Sep 17 00:00:00 2001 From: LazyFork Date: Wed, 9 Sep 2026 20:33:38 -0500 Subject: [PATCH 37/42] Added go references to Project-Architecture.md --- docs/Project-Architecture.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 5570343..282d4fb 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -2,8 +2,10 @@ `api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded to the Nebula API. The data comes from various UTD data sources including coursebook, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. -Tools are written in Go. ADD GO REFERENCE! - +Tools are written in Go. If you're new to Go (or just rusty), here are two good places to start: + +- [A Tour of Go](https://go.dev/tour/list): Interactive, runs in your browser, no setup required. Covers syntax, concurrency, and interfaces. +- [Effective Go](https://go.dev/doc/effective_go): How to write Go the idiomatic way. Worth skimming before diving into `scrapers/`, `parser/`, or `uploader/`. ## Project Pipeline From 67b9ff4707f9845b8b7ddbb732d2ef3ded5b761b Mon Sep 17 00:00:00 2001 From: LazyFork Date: Wed, 9 Sep 2026 20:35:04 -0500 Subject: [PATCH 38/42] minor change --- docs/Getting-Started.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index a87d0f9..6c6e616 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -6,14 +6,14 @@ This guide walks you through setting up your local development environment, conf Ensure you have the following installed: -- Git — For version control +- Git - For version control - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop). -- Go — For building and running +- Go - For building and running - You can install from the [Go website](https://go.dev/dl/), or from [Homebrew](https://brew.sh/) or another package manager for automatic updates -- Make *(Linux/macOS)* — For build automation +- Make _(Linux/macOS)_ - For build automation - Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) -- Google Chrome or Chromium — For headless browser scraping -- Docker *(Optional)* — For running containerized runners locally +- Google Chrome or Chromium - For headless browser scraping +- Docker _(Optional)_ - For running containerized runners locally **Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run. From 95dab35d3726faf8f8bf40579d96f2248fe27f7f Mon Sep 17 00:00:00 2001 From: LazyFork Date: Wed, 9 Sep 2026 22:34:09 -0500 Subject: [PATCH 39/42] Project-Architecture.md rewrite; more simple language + no bullet points --- docs/Project-Architecture.md | 79 +++++++++++++++++++----------------- 1 file changed, 42 insertions(+), 37 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 282d4fb..405670d 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -1,11 +1,11 @@ -# Project Architecture +# Project-Architecture -`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded to the Nebula API. The data comes from various UTD data sources including coursebook, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. +`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded to the Nebula API. -Tools are written in Go. If you're new to Go (or just rusty), here are two good places to start: +The data comes from various UTD data sources including coursebook, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. -- [A Tour of Go](https://go.dev/tour/list): Interactive, runs in your browser, no setup required. Covers syntax, concurrency, and interfaces. -- [Effective Go](https://go.dev/doc/effective_go): How to write Go the idiomatic way. Worth skimming before diving into `scrapers/`, `parser/`, or `uploader/`. +Tools are written in Go. If you're new to Go (or just rusty), two good places to start are [A Tour of Go](https://go.dev/tour/list), which is interactive, runs in your browser, and needs no setup, +and [Effective Go](https://go.dev/doc/effective_go), which covers how to write Go the idiomatic way and is worth skimming before diving into `scrapers/`, `parser/`, or `uploader/`. ## Project Pipeline @@ -25,57 +25,62 @@ flowchart LR ### Scrapers (`scrapers/`) -Scrapers connect to external UTD web servers, APIs, and portals to download raw information. They capture raw data (`.html`, `.json`, etc.) and store it in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). +Scrapers connect to external UTD web servers, APIs, and portals to download raw information. They capture raw data (`.html`, `.json`, etc.) +and store it in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). -Libraries used: - -- `net/http`: Go Standard library HTTP client for fetching static pages and making REST API calls. -- [ChromeDP](https://github.com/chromedp/chromedp): Headless browser automation via the Chrome DevTools Protocol for scraping dynamic, JavaScript-rendered, or authenticated pages (WAT NetID login, Coursebook, Astra Schedule). -- [cdproto](https://github.com/chromedp/cdproto): Chrome DevTools Protocol definitions used for low-level network event handling, cookie extraction, and DOM interaction. -- [fastjson](https://github.com/valyala/fastjson): High-performance, zero-allocation JSON parser for reading large raw payloads (e.g., Astra room scheduling events). +For most scraping we use Go's built-in `net/http` library to directly make requests, which covers static pages and REST API calls. +Sometimes we need a browser, and for that we use [ChromeDP](https://github.com/chromedp/chromedp), which handles headless browser automation via the Chrome DevTools Protocol +and lets us scrape dynamic, JavaScript-rendered, or authenticated pages like WAT NetID login, Coursebook, and Astra Schedule. +For extra scraping functionality not already covered by ChromeDP, we use [cdproto](https://github.com/chromedp/cdproto), ### Parsers (`parser/`) -Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then extract meaningful fields, fix inconsistent formatting, and turn raw data into validated Go data structures matching our schema. Parsers do not modify input data. - -Libraries used: +Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then extract meaningful fields, fix inconsistent formatting, +and turn raw data into validated Go data structures matching our schema. Parsers do not modify input data. -- [goquery](https://github.com/PuerkitoBio/goquery): jQuery-like DOM traversal and CSS selector querying for HTML documents (courses, sections, professors, degree requirements, discounts). -- [golang.org/x/net](https://pkg.go.dev/golang.org/x/net): Low-level HTML tokenization and atom lookup (`html/atom`) during section parsing. -- [google.golang.org/genai](https://pkg.go.dev/google.golang.org/genai): Google Gemini LLM client (Vertex AI / Gemini API) for structured schema-based extraction from unstructured documents (academic calendars and university budgets). -- [nebula-api/api](https://github.com/UTDNebula/nebula-api): Canonical data models (`api/schema`) used to structure and validate parsed data via `parser/validator.go`. -- [dongri/phonenumber](https://github.com/dongri/phonenumber): Phone number parsing and normalization for student discount programs. -- [golang.org/x/text](https://pkg.go.dev/golang.org/x/text): Unicode text transformation and casing rules (`cases`, `language`) for event names and titles. -- [google/go-cmp](https://github.com/google/go-cmp): Deep equality comparisons and struct diffing across unit and regression test suites. +For HTML documents like courses, sections, professors, degree requirements, and discounts, we use [goquery](https://github.com/PuerkitoBio/goquery), +which gives us jQuery-like DOM traversal and CSS selector querying. +When we need low-level HTML tokenization and atom lookup during section parsing, we use [golang.org/x/net](https://pkg.go.dev/golang.org/x/net). +For structured schema-based extraction from unstructured documents like academic calendars and university budgets, we use the Google Gemini LLM client +[google.golang.org/genai](https://pkg.go.dev/google.golang.org/genai) (Vertex AI / Gemini API). +Everything gets structured and validated against the canonical data models in [nebula-api/api](https://github.com/UTDNebula/nebula-api) (`api/schema`) via `parser/validator.go`. ### Uploaders (`uploader/`) Uploaders take validated data models and push them to the Nebula API MongoDB database. -Libraries used: +We use the official [MongoDB driver](https://pkg.go.dev/go.mongodb.org/mongo-driver) to connect to the database, turn Go structs into BSON documents, +write them in bulk, and run aggregation queries. The shared schema definitions come from [nebula-api/api](https://github.com/UTDNebula/nebula-api), +so what we store always matches what the API expects to find. -- [go.mongodb.org/mongo-driver](https://pkg.go.dev/go.mongodb.org/mongo-driver): Official MongoDB driver (`mongo`, `bson`) for database connection management, BSON document serialization, bulk upserts, and aggregation pipelines. -- [nebula-api/api](https://github.com/UTDNebula/nebula-api): Shared schema definitions ensuring stored documents match Nebula API collection expectations. +## Supporting libraries -## Automation +A few libraries don't fit neatly into any single stage of the pipeline, so they live here. -Most data sources are updated automatically through shell scripts that coordinate scraper, parser, and uploader execution. These scripts run in containerized environments scheduled by cron jobs in Google Cloud. The automation scripts are located in `runners/`. +- [fastjson](https://github.com/valyala/fastjson) - lets us read huge JSON payloads in scrapers without loading the whole thing into memory at once. It walks through the data piece by piece, which is how we handle things like the Astra room scheduling events. +- [dongri/phonenumber](https://github.com/dongri/phonenumber) - cleans up phone numbers from student discount programs. They show up in every format imaginable, so this gives us one consistent shape to store. +- [golang.org/x/text](https://pkg.go.dev/golang.org/x/text) - handles Unicode transformations and casing rules, which keeps event names and titles from getting mangled when they contain non-English characters. +- [google/go-cmp](https://github.com/google/go-cmp) - compares structs for our unit and regression tests, so when something breaks we can see exactly which field changed instead of just "not equal." +- [godotenv](https://github.com/joho/godotenv) - reads our `.env` files and loads things like URIs and credentials when the program starts, so we don't have to hardcode secrets or pass them on the command line. + +## Automation -Tools and Libraries used: +Most data sources are updated automatically through shell scripts that coordinate scraper, parser, and uploader execution. These scripts run in +containerized environments scheduled by cron jobs in Google Cloud. The automation scripts are located in `runners/`. -- **Bash (`runners/*.sh`)**: Command-line orchestration scripts that run end-to-end pipeline sequences for different data sources. -- **Docker**: Multi-stage container definitions providing reproducible execution environments across local and cloud runners. -- **Google Cloud Build & Cloud Scheduler**: Cloud CI/CD and cron scheduling services that trigger periodic pipeline executions. +The orchestration itself is plain Bash (`runners/*.sh`), which runs end-to-end pipeline sequences for different data sources. +**Docker** provides multi-stage container definitions for reproducible execution environments across local and cloud runners. +And **Google Cloud Build & Cloud Scheduler** handle the cloud CI/CD and cron scheduling that trigger periodic pipeline executions. ## Lifecycle and workflow -We use several tools while developing the API: +We use several tools while developing the API. -- `testing`: Go standard library testing package for unit tests across the project. -- **GitHub Actions**: Runs automated CI workflows for PRs, and deploys API-Tools to Google Artifact Registry and Google Cloud Run jobs. -- [godotenv](https://github.com/joho/godotenv): Parses `.env` files and loads variables such as URIs and credentials at startup. -- [getsentry/sentry-go](https://github.com/getsentry/sentry-go): Production error tracking, panic handling, and trace sampling. Initialized in `main.go`, it captures unhandled runtime failures during scraper, parser, and uploader runs and flushes diagnostics to Sentry before exit. +Tests use Go's standard `testing` package for unit tests across the project. **GitHub Actions** runs automated CI workflows for PRs +and deploys API-Tools to Google Artifact Registry and Google Cloud Run jobs. And [getsentry/sentry-go](https://github.com/getsentry/sentry-go) handles production error tracking, +panic handling, and trace sampling - it's initialized in `main.go`, captures unhandled runtime failures during scraper, parser, and uploader runs, +and flushes diagnostics to Sentry before exit. ## Next Step -See [Project-Structure.md](Project-Structure.md) +See [Project-Structure.md](/docs/Project-Structure.md) From a782c931d16d138827ba7274d17f3c3a0b9cd47b Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 10 Sep 2026 12:10:20 -0500 Subject: [PATCH 40/42] Made tweaks to Project-Architecture.md --- docs/Project-Architecture.md | 64 +++++++++++++++--------------------- 1 file changed, 26 insertions(+), 38 deletions(-) diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 405670d..17338d5 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -2,7 +2,7 @@ `api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded to the Nebula API. -The data comes from various UTD data sources including coursebook, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. +The data comes from various UTD data sources including courses, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. Tools are written in Go. If you're new to Go (or just rusty), two good places to start are [A Tour of Go](https://go.dev/tour/list), which is interactive, runs in your browser, and needs no setup, and [Effective Go](https://go.dev/doc/effective_go), which covers how to write Go the idiomatic way and is worth skimming before diving into `scrapers/`, `parser/`, or `uploader/`. @@ -25,61 +25,49 @@ flowchart LR ### Scrapers (`scrapers/`) -Scrapers connect to external UTD web servers, APIs, and portals to download raw information. They capture raw data (`.html`, `.json`, etc.) -and store it in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). +Scrapers connect to websites, APIs, and portals to download raw information. They capture raw data in `.html`, `.json`, or other formats, and store it in the `data/` directory. For example, a scrapped course would be found at `data/24f/cp_cs/cs1337.001.24f.html` after running the scraper. -For most scraping we use Go's built-in `net/http` library to directly make requests, which covers static pages and REST API calls. -Sometimes we need a browser, and for that we use [ChromeDP](https://github.com/chromedp/chromedp), which handles headless browser automation via the Chrome DevTools Protocol -and lets us scrape dynamic, JavaScript-rendered, or authenticated pages like WAT NetID login, Coursebook, and Astra Schedule. -For extra scraping functionality not already covered by ChromeDP, we use [cdproto](https://github.com/chromedp/cdproto), +For most scraping we use Go's built-in `net/http` library to directly make API requests and get our data. Sometimes we need a browser to recreate more complex actions. For automating browser actions, we use [ChromeDP](https://github.com/chromedp/chromedp) which uses Chrome DevTools. For extra scraping functionality not already covered by ChromeDP, we use [cdproto](https://github.com/chromedp/cdproto). ### Parsers (`parser/`) -Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then extract meaningful fields, fix inconsistent formatting, -and turn raw data into validated Go data structures matching our schema. Parsers do not modify input data. +Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then convert this data into a more machine-readable format. -For HTML documents like courses, sections, professors, degree requirements, and discounts, we use [goquery](https://github.com/PuerkitoBio/goquery), -which gives us jQuery-like DOM traversal and CSS selector querying. -When we need low-level HTML tokenization and atom lookup during section parsing, we use [golang.org/x/net](https://pkg.go.dev/golang.org/x/net). -For structured schema-based extraction from unstructured documents like academic calendars and university budgets, we use the Google Gemini LLM client -[google.golang.org/genai](https://pkg.go.dev/google.golang.org/genai) (Vertex AI / Gemini API). -Everything gets structured and validated against the canonical data models in [nebula-api/api](https://github.com/UTDNebula/nebula-api) (`api/schema`) via `parser/validator.go`. +For HTML documents like courses, sections, professors, degree requirements, and discounts, we use [goquery](https://github.com/PuerkitoBio/goquery). Goquery lets us navigate our `.html` files programmatically with selectors. For more unstructured documents like calendars and budgets, we use the Google Gemini [google.golang.org/genai](https://pkg.go.dev/google.golang.org/genai). + +Everything gets structured and validated against the data models in [nebula-api/api](https://github.com/UTDNebula/nebula-api) (`api/schema`) via `parser/validator.go`. ### Uploaders (`uploader/`) -Uploaders take validated data models and push them to the Nebula API MongoDB database. +Uploaders take parsed data and upload it to the Nebula API MongoDB database. -We use the official [MongoDB driver](https://pkg.go.dev/go.mongodb.org/mongo-driver) to connect to the database, turn Go structs into BSON documents, -write them in bulk, and run aggregation queries. The shared schema definitions come from [nebula-api/api](https://github.com/UTDNebula/nebula-api), -so what we store always matches what the API expects to find. +We use the official [MongoDB driver](https://pkg.go.dev/go.mongodb.org/mongo-driver) to connect to the database, turn Go structs into BSON documents, write them in bulk, and run aggregation queries. The shared schema definitions come from [nebula-api/api](https://github.com/UTDNebula/nebula-api), so what we store always matches what the API expects to find. -## Supporting libraries +## Cloud Automation -A few libraries don't fit neatly into any single stage of the pipeline, so they live here. +Most data sources are updated automatically through shell scripts in `runners/`. These scripts run in containerized environments in Google Cloud. -- [fastjson](https://github.com/valyala/fastjson) - lets us read huge JSON payloads in scrapers without loading the whole thing into memory at once. It walks through the data piece by piece, which is how we handle things like the Astra room scheduling events. -- [dongri/phonenumber](https://github.com/dongri/phonenumber) - cleans up phone numbers from student discount programs. They show up in every format imaginable, so this gives us one consistent shape to store. -- [golang.org/x/text](https://pkg.go.dev/golang.org/x/text) - handles Unicode transformations and casing rules, which keeps event names and titles from getting mangled when they contain non-English characters. -- [google/go-cmp](https://github.com/google/go-cmp) - compares structs for our unit and regression tests, so when something breaks we can see exactly which field changed instead of just "not equal." -- [godotenv](https://github.com/joho/godotenv) - reads our `.env` files and loads things like URIs and credentials when the program starts, so we don't have to hardcode secrets or pass them on the command line. +We use [Docker](https://www.docker.com/) to create a consistent environment to run `api-tools` in the cloud. We use [Google Cloud Build](https://cloud.google.com/build) to build our docker image, and [Google Cloud Scheduler](https://docs.cloud.google.com/scheduler/docs) to schedule our cloud pipeline runs. -## Automation +## Other Imporatnt tools -Most data sources are updated automatically through shell scripts that coordinate scraper, parser, and uploader execution. These scripts run in -containerized environments scheduled by cron jobs in Google Cloud. The automation scripts are located in `runners/`. +We use several tools while developing the API. -The orchestration itself is plain Bash (`runners/*.sh`), which runs end-to-end pipeline sequences for different data sources. -**Docker** provides multi-stage container definitions for reproducible execution environments across local and cloud runners. -And **Google Cloud Build & Cloud Scheduler** handle the cloud CI/CD and cron scheduling that trigger periodic pipeline executions. +Tests use Go's standard `testing` package for unit tests across the project. -## Lifecycle and workflow +We use [Sentry](https://github.com/getsentry/sentry-go) for error tracking. -We use several tools while developing the API. +We use [GitHub Actions](https://github.com/features/actions) to check Go code for PRs, deploy our Docker image, and deploy this Wiki! -Tests use Go's standard `testing` package for unit tests across the project. **GitHub Actions** runs automated CI workflows for PRs -and deploys API-Tools to Google Artifact Registry and Google Cloud Run jobs. And [getsentry/sentry-go](https://github.com/getsentry/sentry-go) handles production error tracking, -panic handling, and trace sampling - it's initialized in `main.go`, captures unhandled runtime failures during scraper, parser, and uploader runs, -and flushes diagnostics to Sentry before exit. +## Supporting libraries + +We use several supporting libraries for more minor things. Here is a list of most of them, and their purpose + +- [fastjson](https://github.com/valyala/fastjson) - lets us read huge JSON payloads in scrapers without loading the whole thing into memory at once. It walks through the data piece by piece, which is how we handle things like the Astra room scheduling events. +- [dongri/phonenumber](https://github.com/dongri/phonenumber) - cleans up phone numbers from student discount programs. They show up in every format imaginable, so this gives us one consistent shape to store. +- [golang.org/x/text](https://pkg.go.dev/golang.org/x/text) - handles Unicode transformations and casing rules, which keeps event names and titles from getting mangled when they contain non-English characters. +- [google/go-cmp](https://github.com/google/go-cmp) - compares structs for our unit and regression tests, so when something breaks we can see exactly which field changed instead of just "not equal." +- [godotenv](https://github.com/joho/godotenv) - reads our `.env` files and loads things like URIs and credentials when the program starts, so we don't have to hardcode secrets or pass them on the command line. ## Next Step From 8640481d3bd7f600c674612049938c15fd8b0f67 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 10 Sep 2026 12:19:00 -0500 Subject: [PATCH 41/42] Fix various types and word tweaks --- docs/Getting-Started.md | 2 +- docs/How-to-Contribute.md | 9 ++++----- docs/Project-Architecture.md | 6 +++--- 3 files changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index 6c6e616..c7495f7 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -17,7 +17,7 @@ Ensure you have the following installed: **Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run. -If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file.When you see any command starting with `make`, you can instead use `.\build.bat`. +If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file. When you see any command starting with `make`, you can instead use `.\build.bat`. For example instead of `make setup`, you can run `.\build.bat setup`. ## Local Setup diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index cd65251..3487469 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -7,15 +7,12 @@ Don't worry if you don't quite know what you're doing, we're here to help. We do Look at our current issues, or make your own on our [issues page](../issues). -For beginner issues, please write the code yourself rather than using AI to generate it. These issues are designed to help new members learn the codebase and build familiarity with the project. -For all other issues, please review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy) for more information on acceptable uses of AI. +For beginner issues, please write the code yourself rather than using AI to generate it. These issues are designed to help new members learn the codebase and build familiarity with the project. We allow AI in Nebula Labs, but review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy) for more information on acceptable uses of AI. Once you find an issue, write a comment asking if you can work on it. If you're unsure about something, **don't hesitate to ask for help**. We're happy to answer questions and help you work through issues. You can reach out on GitHub, Discord, or in person. -Be sure to reference [Getting Started.md](/docs/Getting-Started.md) to create the project. - ## Create a branch Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork as they do not have permission to make a branch. @@ -26,6 +23,8 @@ It's time to code! Don't forget to format and check your code. We have helper scripts with **Make** and **build.bat**: +Check your code with + ```bash make check ``` @@ -58,4 +57,4 @@ Maintainers will review your PR and may provide suggestions or request changes b ## Next Step -See [Guides.md](Guides.md) +You have everything you need to start, but feel free to check out other pages first. diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 17338d5..16fe47c 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -33,7 +33,7 @@ For most scraping we use Go's built-in `net/http` library to directly make API r Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then convert this data into a more machine-readable format. -For HTML documents like courses, sections, professors, degree requirements, and discounts, we use [goquery](https://github.com/PuerkitoBio/goquery). Goquery lets us navigate our `.html` files programmatically with selectors. For more unstructured documents like calendars and budgets, we use the Google Gemini [google.golang.org/genai](https://pkg.go.dev/google.golang.org/genai). +For HTML documents like courses, sections, professors, degree requirements, and discounts, we use [goquery](https://github.com/PuerkitoBio/goquery). Goquery lets us navigate our `.html` files programmatically with selectors. For more unstructured documents like calendars and budgets, we use [Google Gemini](https://pkg.go.dev/google.golang.org/genai). Everything gets structured and validated against the data models in [nebula-api/api](https://github.com/UTDNebula/nebula-api) (`api/schema`) via `parser/validator.go`. @@ -49,7 +49,7 @@ Most data sources are updated automatically through shell scripts in `runners/`. We use [Docker](https://www.docker.com/) to create a consistent environment to run `api-tools` in the cloud. We use [Google Cloud Build](https://cloud.google.com/build) to build our docker image, and [Google Cloud Scheduler](https://docs.cloud.google.com/scheduler/docs) to schedule our cloud pipeline runs. -## Other Imporatnt tools +## Other Important Tools We use several tools while developing the API. @@ -59,7 +59,7 @@ We use [Sentry](https://github.com/getsentry/sentry-go) for error tracking. We use [GitHub Actions](https://github.com/features/actions) to check Go code for PRs, deploy our Docker image, and deploy this Wiki! -## Supporting libraries +## Supporting Libraries We use several supporting libraries for more minor things. Here is a list of most of them, and their purpose From 392f4828af299a550a2e8fb7bd3902235f05bc71 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 10 Sep 2026 14:51:52 -0500 Subject: [PATCH 42/42] Add sidebar, remove bad pages, and make minor tweaks --- docs/Docker-Guide.md | 93 ---------------------------- docs/Getting-Started.md | 2 + docs/Guides.md | 69 --------------------- docs/Home.md | 10 +-- docs/Project-Architecture.md | 2 +- docs/Troubleshooting.md | 114 ++++------------------------------- docs/_Sidebar.md | 10 +++ 7 files changed, 27 insertions(+), 273 deletions(-) delete mode 100644 docs/Docker-Guide.md delete mode 100644 docs/Guides.md create mode 100644 docs/_Sidebar.md diff --git a/docs/Docker-Guide.md b/docs/Docker-Guide.md deleted file mode 100644 index 9ded565..0000000 --- a/docs/Docker-Guide.md +++ /dev/null @@ -1,93 +0,0 @@ -# Docker & Containerization Guide - -This guide explains how Docker is used in `api-tools`, how our multi-stage Docker build works, and how you can run automated tasks inside a local Docker container. - ---- - -## Why Do We Use Docker? - -Running web scrapers requires several system-level dependencies: - -- **Chromium / Google Chrome**: Needed by ChromeDP for headless browser automation. -- **Poppler Utilities (`pdftotext`)**: Needed for converting academic calendar PDFs into raw text. -- **Go Runtime**: Needed to compile the application binary. -- **Google Cloud SDK (`gcloud`)**: Needed in production to access cloud secrets. - -Instead of requiring every contributor to manually install and configure these system packages, Docker packages everything into a standardized, lightweight Linux container. - -> [!NOTE] -> **New to Docker?** Check out the official [Docker 101 Tutorial](https://www.docker.com/101-tutorial/) or the [Docker Get Started Guide](https://docs.docker.com/get-started/) to learn the basics of images and containers. - ---- - -## Understanding the Multi-Stage Dockerfile - -Our [`Dockerfile`](file:///var/home/justin/Documents/Projects/api-tools/Dockerfile) uses Docker's multi-stage build pattern to keep production images secure, fast, and minimal. - -1. **`builder` Stage (`golang:1.26`)**: - - Copies Go source code. - - Runs `make setup`, `make check` (vetting and linting), and `make build`. - - Produces the compiled `api-tools` binary. - -2. **`base` Stage (`debian:12-slim`)**: - - A lightweight Debian base image. - - Installs runtime dependencies: `chromium`, `poppler-utils`, and `google-cloud-sdk`. - - Copies the compiled `api-tools` binary from the `builder` stage. - - Copies `runners/` and `static-data/`. - - Sets the entrypoint to [`runners/setup.sh`](file:///var/home/justin/Documents/Projects/api-tools/runners/setup.sh). - -3. **`local` Stage (`FROM base AS local`)**: - - Inherits from `base` and copies your local `.env` file into the container. - - Ideal for local testing and debugging. - ---- - -## Running Docker Locally - -### Step 1: Ensure Your `.env` File Exists - -Make sure you have copied `.env.template` to `.env` in the project root: - -```bash -cp .env.template .env -``` - -### Step 2: Build the Local Docker Image - -Build using the `--target local` flag to include your local `.env` file: - -```bash -docker build --target local -t my-runner:local . -``` - -### Step 3: Run a Runner Script - -Run the container by specifying the environment mode and target script name: - -```bash -docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=daily.sh my-runner:local -``` - -You can replace `daily.sh` with any runner in the `runners/` directory: - -- `daily.sh` — Scrapes, parses, and uploads events (Astra, Mazevo, Comet Calendar). -- `weekly.sh` — Scrapes, parses, and uploads academic calendars, discounts, and degrees. -- `monthly.sh` — Scrapes, parses, and uploads map locations and budgets. - ---- - -## Production vs. Local Execution (`runners/setup.sh`) - -When the Docker container starts, it executes [`runners/setup.sh`](file:///var/home/justin/Documents/Projects/api-tools/runners/setup.sh): - -- **Local Mode (`ENVIRONMENT=local`)**: - - Skips cloud authentication. - - Directly uses the `.env` file baked into the local image. - - Runs `/app/runners/$RUNNER_SCRIPT_NAME`. - -- **Production Mode (`ENVIRONMENT=gcp`)**: - - Authenticates with Google Cloud Secret Manager using service account keys. - - Dynamically fetches secrets and creates the `.env` file at runtime. - - Runs `/app/runners/$RUNNER_SCRIPT_NAME`. - -- Resolve any container or browser errors in the [**Troubleshooting Guide**](/docs/Troubleshooting.md). diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index c7495f7..65c3ccb 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -42,6 +42,8 @@ git clone git@github.com:UTDNebula/api-tools.git cd api-tools ``` +Now open the cloned project in your code editor! + ### Install Development Tooling Install the Go static analysis and formatting tools (`staticcheck` and `goimports`): diff --git a/docs/Guides.md b/docs/Guides.md deleted file mode 100644 index a89a35a..0000000 --- a/docs/Guides.md +++ /dev/null @@ -1,69 +0,0 @@ -# Guides - -There's a lot of guides in here. You do **not** need to master everything before getting started! Here is a curated list of resources to help you learn the fundamentals. - -## General Guides - -### Git & GitHub - -Learn how to manage code changes, work with branches, and collaborate using GitHub. - -- [Nebula GitHub Workshop](https://github.com/UTDNebula/git-workshop) -- [GitHub Skills Git Introduction](https://skills.github.com/) - - - -### Go (Golang) - -Learn the Go programming language and the fundamentals needed to work on our codebase. - -- [A Tour of Go](https://go.dev/tour/) - Official interactive introduction to Go syntax, data structures, and concurrency. -- [Go by Example](https://gobyexample.com/) - Hands-on, concise Go examples. Very good for quick reference while working on issues. -- [Learn Go in Y Minutes](https://learnxinyminutes.com/docs/go/) - Quick visual cheat sheet for developers who want a fast syntax refresher. - -### Docker & Containers - -Learn how to build, run, and manage applications using Docker containers. - -- [Docker Documentation: Orientation and Setup](https://docs.docker.com/get-started/) - Official step-by-step introduction to images and containers. - -### Web Basics - -Learn how websites are structured and how HTML and CSS work together. - -- [MDN Web Docs: HTML & DOM Basics](https://developer.mozilla.org/en-US/docs/Learn/HTML) - Learn how web pages are structured with elements, tags, and attributes. -- [MDN Web Docs: CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors) - Understand class, ID, and attribute selectors used to find page elements. - -## Scraper Libraries - -ChromeDP lets us control a headless Chrome browser to interact with and scrape dynamic websites. - -### ChromeDP - -- [Headless Browser Tutorial](https://www.zenrows.com/blog/chromedp) - Good introduction to ChromeDP. -- [ChromeDP Documentation & Examples](https://github.com/chromedp/examples) - Examples for browser automation and scraping dynamic JavaScript pages in Go. - -## Parser Libraries - -### Goquery - -Goquery lets us parse and query HTML, making it easier to extract data from web pages. - -- [Goquery Guide](https://www.zenrows.com/blog/goquery) - Excellent follow-up to the ChromeDP Browser Tutorial. -- [Goquery Guide](https://zetcode.com/golang/goquery/) - More in-depth guide to Goquery. - -## Uploader Libraries - -### MongoDB Go Driver - -The MongoDB Go Driver lets us store and retrieve scraped data from MongoDB using Go. - -- [MongoDB Go Driver Official Docs](https://www.mongodb.com/docs/drivers/go/current/) - Official documentation for using MongoDB with go - -## Automation - -### Bash - -Learn the Go programming language and the fundamentals needed to work on our codebase. - -- [Bash / CLI Crash Course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line) - Beginner-friendly guide to Bash. diff --git a/docs/Home.md b/docs/Home.md index 6e37190..10c2b2a 100644 --- a/docs/Home.md +++ b/docs/Home.md @@ -4,15 +4,7 @@ Welcome to the **API Tools Developer Documentation**! These docs are intended for developers of `api-tools` rather than users. For usage instructions, see [README.md](../README.md). ---- - -If you are a new developer, we recommend checking out these pages in this order: - -1. [Getting-Started.md](Getting-Started.md) -1. [Project-Architecture.md](Project-Architecture.md) -1. [Project-Structure.md](Project-Structure.md) -1. [How-to-Contribute.md](How-to-Contribute.md) -1. [Guides.md](Guides.md) +If you are a new developer, check out the pages under "Getting Started" in the Sidebar! ## Need Help? diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 16fe47c..5eb73cd 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -71,4 +71,4 @@ We use several supporting libraries for more minor things. Here is a list of mos ## Next Step -See [Project-Structure.md](/docs/Project-Structure.md) +See [Project-Structure.md](Project-Structure.md) diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md index 4677b21..3c53059 100644 --- a/docs/Troubleshooting.md +++ b/docs/Troubleshooting.md @@ -1,115 +1,27 @@ -# Troubleshooting & FAQ +# Troubleshooting -This guide provides fast, actionable solutions to common issues encountered when running or developing `api-tools`. +This guide gives solutions to common problems developers or users of this project may run into. ---- - -## Chromium & ChromeDP Issues - -### Issue: "exec: 'chromium': executable file not found in $PATH" or Chrome fails to launch - -- **Cause**: ChromeDP cannot find a Chromium or Google Chrome binary on your operating system. -- **Solutions**: - - **Linux**: Install Chromium via your package manager: - - ```bash - # Ubuntu / Debian - sudo apt update && sudo apt install -y chromium-browser - # Fedora / RHEL - sudo dnf install chromium - # Arch Linux - sudo pacman -S chromium - ``` - - - **macOS / Windows**: Ensure [Google Chrome](https://www.google.com/chrome/) is installed in the standard applications directory. - - **Environment Variables**: If Chrome is installed in a non-standard location, specify its path explicitly before running `api-tools`: - - ```bash - export CHROMIUM_BIN="/path/to/custom/chrome" - export GOOGLE_CHROME_BIN="/path/to/custom/chrome" - ``` - -### Issue: Chrome window opens during scraping and interferes with work - -- **Solution**: Pass the `-headless` flag to run ChromeDP invisibly in background headless mode: - - ```bash - ./api-tools -scrape -coursebook -term 24F -headless - ``` - ---- - -## Environment Variables & MongoDB Connection Issues - -### Issue: Panic: ` is missing from .env!` - -- **Cause**: An authenticated scraper or uploader attempted to read a required environment variable from `.env` via `utils.GetEnv`, but the key was either missing or empty. -- **Solutions**: - 1. Ensure you have copied `.env.template` to `.env`: - - ```bash - cp .env.template .env - ``` - - 2. For Coursebook, Astra, or Mazevo scrapers, populate the required credentials: - - `LOGIN_NETID` / `LOGIN_PASSWORD` (UTD NetID login) - - `LOGIN_ASTRA_USERNAME` / `LOGIN_ASTRA_PASSWORD` - - `MAZEVO_API_KEY` - 3. If you only want to parse existing local data or run tests, you do **not** need these credentials; avoid calling authenticated scraping flags. - -### Issue: MongoDB connection timed out / "server selection error" - -- **Cause**: The uploader could not establish a connection to the MongoDB instance specified in `MONGODB_URI`. -- **Solutions**: - - **Verify Connection String**: Check that `MONGODB_URI` in `.env` is formatted correctly: - - ```env - MONGODB_URI="mongodb+srv://:@cluster0.abcde.mongodb.net/?retryWrites=true&w=majority" - ``` - - - **Check Network & IP Whitelist**: If using MongoDB Atlas, ensure your current IP address is whitelisted under Atlas Network Access. - - **Local Development**: If testing locally with a local MongoDB container or service, verify MongoDB is running: - - ```bash - # Test local MongoDB connection - mongosh "mongodb://localhost:27017" - ``` - ---- - -## Locating & Inspecting Log Files - -### How Logging Works - -When `api-tools` executes, `utils.NewSplitWriter` writes all log output to two destinations simultaneously: - -1. **Console (`stdout`)**: Live output in your terminal window. -2. **Log File (`logs/*.log`)**: A persistent log file saved to the `logs/` directory named with the execution timestamp: - - ``` - logs/8-31-2026T12-30-15.log - ``` - -### Enabling Verbose Debug Logging +## Enabling Verbose Debug Logging If a scraper or parser is behaving unexpectedly, re-run the command with the `-verbose` flag: +For example: + ```bash ./api-tools -verbose -scrape -coursebook -term 24F ``` -Verbose mode enables: +This will give more logs, and can help you better understand what's going on. -- Microsecond timestamp precision (`log.Lmicroseconds`). -- Source code filename and line number for every log statement (`log.Lshortfile`). -- Additional internal debug logs (`utils.Lverbose`). +## "exec: 'chromium': executable file not found in $PATH" or Chrome fails to launch ---- +ChromeDP cannot find a Chromium or Google Chrome binary on your operating system. You'll need to install it. -## Still Stuck? +## KEY is missing from .env + +Ensure you have filled environment variables in .env. If you need help, ask for help! -Reach out to the team on [Discord](https://discord.utdnebula.com) with: +## Still Stuck? -1. The exact command you ran. -2. The relevant lines from your latest log file in `logs/`. -3. Your operating system and Go version (`go version`). +Ask for help on [Discord](https://discord.utdnebula.com)! diff --git a/docs/_Sidebar.md b/docs/_Sidebar.md new file mode 100644 index 0000000..0fb1541 --- /dev/null +++ b/docs/_Sidebar.md @@ -0,0 +1,10 @@ +## Getting Started + +- [Getting Started](Getting-Started.md) +- [Project Architecture](Project-Architecture.md) +- [Project Structure](Project-Structure.md) +- [How to Contribute](How-to-Contribute.md) + +## References + +- [Troubleshooting](Troubleshooting.md) \ No newline at end of file