diff --git a/README.md b/README.md index 1a9bb60..27f1fa9 100644 --- a/README.md +++ b/README.md @@ -4,19 +4,19 @@ _A CLI toolchain to scrape, parse, validate, and upload UTD university data to t Project maintained by [Nebula Labs](https://about.utdnebula.com). -> 📚 **Developer Wiki**: If you are interested in developing this project, head over to the [**Developer Wiki**](docs/Home.md)! +> **Developer Wiki**: If you are interested in developing this project, head over to the [**Developer Wiki**](docs/Home.md)! --- ## Quick Navigation - [Developer Wiki](docs/Home.md) -- [Getting Started Guide](docs/getting-started.md) -- [Pipeline Architecture (Scraper vs Parser vs Uploader vs Runner)](docs/architecture.md) -- [Project Structure & Codebase Map](docs/project-structure.md) -- [Docker & Container Guide](docs/docker-guide.md) -- [Contributing Guide](docs/contributing.md) -- [Troubleshooting & FAQ](docs/troubleshooting.md) +- [Getting Started Guide](/docs/Getting-Started.md) +- [Pipeline Architecture (Scraper -> Parser -> Uploader)](/docs/Project-Architecture.md) +- [Project Structure & Codebase Map](/docs/Project-Structure.md) +- [Docker & Container Guide](/docs/Docker-Guide.md) +- [Contributing Guide](/docs/How-to-Contribute.md) +- [Troubleshooting & FAQ](/docs/Troubleshooting.md) - [Discord Community](https://discord.utdnebula.com) --- @@ -50,71 +50,71 @@ The `api-tools` command line interface supports three main modes: **scraping**, ### Global Flags -| Flag | Default | Description | -|---|---|---| -| `-i ` | `./data` | Input directory to read data from | -| `-o ` | `./data` | Output directory to write scraped/parsed data to | -| `-l ` | `./logs` | Directory to write timestamped log files to | -| `--verbose` | `false` | Enables verbose debug logging with file names and microsecond timestamps | -| `--headless`| `false` | Runs ChromeDP browser automation in background headless mode | +| Flag | Default | Description | +| ------------ | -------- | ------------------------------------------------------------------------ | +| `-i ` | `./data` | Input directory to read data from | +| `-o ` | `./data` | Output directory to write scraped/parsed data to | +| `-l ` | `./logs` | Directory to write timestamped log files to | +| `--verbose` | `false` | Enables verbose debug logging with file names and microsecond timestamps | +| `--headless` | `false` | Runs ChromeDP browser automation in background headless mode | --- -### 1. Scraping Mode (`--scrape`) +### Scraping Mode (`--scrape`) Scrapes raw data from UTD sources and saves it to `./data` (or the directory specified by `-o`). -| Command | Description | -|---|---| +| Command | Description | +| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `./api-tools --scrape --coursebook --term 24F` | Scrapes CourseBook data for Fall 2024.
• `--resume`: Resume from last completed course prefix.
• `--startprefix `: Begin at a specific course prefix (e.g. `cp_cs`). | -| `./api-tools --scrape --evals --term 24F` | Scrapes CourseBook course evaluation data for the specified term. | -| `./api-tools --scrape --profiles` | Scrapes faculty and professor directory profiles. | -| `./api-tools --scrape --astra` | Scrapes Astra schedule and room reservation data. | -| `./api-tools --scrape --mazevo` | Scrapes Mazevo event reservation data via API. | -| `./api-tools --scrape --cometCalendar` | Scrapes public UTD Comet Calendar events. | -| `./api-tools --scrape --map` | Scrapes UTD campus map building and parking coordinates. | -| `./api-tools --scrape --degrees` | Scrapes academic degrees and catalog majors. | -| `./api-tools --scrape --discounts` | Scrapes student discount program listings. | -| `./api-tools --scrape --academicCalendars` | Scrapes academic calendar PDF documents. | -| `./api-tools --scrape --budgets` | Scrapes university budget reports. | +| `./api-tools --scrape --evals --term 24F` | Scrapes CourseBook course evaluation data for the specified term. | +| `./api-tools --scrape --profiles` | Scrapes faculty and professor directory profiles. | +| `./api-tools --scrape --astra` | Scrapes Astra schedule and room reservation data. | +| `./api-tools --scrape --mazevo` | Scrapes Mazevo event reservation data via API. | +| `./api-tools --scrape --cometCalendar` | Scrapes public UTD Comet Calendar events. | +| `./api-tools --scrape --map` | Scrapes UTD campus map building and parking coordinates. | +| `./api-tools --scrape --degrees` | Scrapes academic degrees and catalog majors. | +| `./api-tools --scrape --discounts` | Scrapes student discount program listings. | +| `./api-tools --scrape --academicCalendars` | Scrapes academic calendar PDF documents. | +| `./api-tools --scrape --budgets` | Scrapes university budget reports. | --- -### 2. Parsing Mode (`-parse`) +### Parsing Mode (`-parse`) Reads raw files from `./data` (or `-i`), normalizes records against `static-data/`, validates data models, and outputs structured schemas. -| Command | Description | -|---|---| -| `./api-tools -parse` | Runs full coursebook parsing pipeline across all scraped prefixes. | -| `./api-tools -parse -gradesDir ` | Specifies custom path for grade distribution CSVs (default: `./static-data/grades`). | -| `./api-tools -parse -astra` | Parses scraped Astra scheduling data. | -| `./api-tools -parse -mazevo` | Parses scraped Mazevo event data. | -| `./api-tools -parse -cometCalendar` | Parses scraped Comet Calendar events. | -| `./api-tools -parse -map` | Parses campus map data. | -| `./api-tools -parse -degrees` | Parses degree plans from HTML. | -| `./api-tools -parse -discounts` | Parses student discount listings from HTML. | -| `./api-tools -parse -academicCalendars` | Parses academic calendar dates from PDFs. | -| `./api-tools -parse -budgets` | Parses budget reports from PDFs (use `-useBackupBudgets` to include `static-data/budgets`). | -| `./api-tools -parse -skipv` | Skips post-parse schema validation (**use with caution**). | +| Command | Description | +| --------------------------------------- | ------------------------------------------------------------------------------------------- | +| `./api-tools -parse` | Runs full coursebook parsing pipeline across all scraped prefixes. | +| `./api-tools -parse -gradesDir ` | Specifies custom path for grade distribution CSVs (default: `./static-data/grades`). | +| `./api-tools -parse -astra` | Parses scraped Astra scheduling data. | +| `./api-tools -parse -mazevo` | Parses scraped Mazevo event data. | +| `./api-tools -parse -cometCalendar` | Parses scraped Comet Calendar events. | +| `./api-tools -parse -map` | Parses campus map data. | +| `./api-tools -parse -degrees` | Parses degree plans from HTML. | +| `./api-tools -parse -discounts` | Parses student discount listings from HTML. | +| `./api-tools -parse -academicCalendars` | Parses academic calendar dates from PDFs. | +| `./api-tools -parse -budgets` | Parses budget reports from PDFs (use `-useBackupBudgets` to include `static-data/budgets`). | +| `./api-tools -parse -skipv` | Skips post-parse schema validation (**use with caution**). | --- -### 3. Upload Mode (`-upload`) +### Upload Mode (`-upload`) Pushes validated data to the MongoDB database specified in `MONGODB_URI`. -| Command | Description | -|---|---| -| `./api-tools -upload` | Uploads parsed coursebook data (merges into existing records by default). | -| `./api-tools -upload -replace` | Overwrites/replaces existing records instead of merging. | -| `./api-tools -upload -static` | Uploads static aggregations only. | -| `./api-tools -upload -events` | Uploads Astra, Mazevo, and Comet Calendar event datasets. | -| `./api-tools -upload -map` | Uploads campus map locations. | -| `./api-tools -upload -degrees` | Uploads degree plans. | -| `./api-tools -upload -discounts` | Uploads discount programs. | -| `./api-tools -upload -academicCalendars` | Uploads academic calendar dates. | -| `./api-tools -upload -budgets` | Uploads budget data. | +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------- | +| `./api-tools -upload` | Uploads parsed coursebook data (merges into existing records by default). | +| `./api-tools -upload -replace` | Overwrites/replaces existing records instead of merging. | +| `./api-tools -upload -static` | Uploads static aggregations only. | +| `./api-tools -upload -events` | Uploads Astra, Mazevo, and Comet Calendar event datasets. | +| `./api-tools -upload -map` | Uploads campus map locations. | +| `./api-tools -upload -degrees` | Uploads degree plans. | +| `./api-tools -upload -discounts` | Uploads discount programs. | +| `./api-tools -upload -academicCalendars` | Uploads academic calendar dates. | +| `./api-tools -upload -budgets` | Uploads budget data. | --- @@ -136,7 +136,7 @@ docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=weekly.sh my-runner:l docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=monthly.sh my-runner:local ``` -For more details on Docker execution and cloud configuration, see the [Docker Guide](docs/docker-guide.md). +For more details on Docker execution and cloud configuration, see the [Docker Guide](/docs/Docker-Guide.md). --- diff --git a/docs/Docker-Guide.md b/docs/Docker-Guide.md deleted file mode 100644 index 9e465d4..0000000 --- a/docs/Docker-Guide.md +++ /dev/null @@ -1,90 +0,0 @@ -# Docker & Containerization Guide - -This guide explains how Docker is used in `api-tools`, how our multi-stage Docker build works, and how you can run automated tasks inside a local Docker container. - ---- - -## Why Do We Use Docker? - -Running web scrapers and PDF parsers requires several system-level dependencies: -- **Chromium / Google Chrome**: Needed by ChromeDP for headless browser automation. -- **Poppler Utilities (`pdftotext`)**: Needed for converting academic calendar PDFs into raw text. -- **Go Runtime**: Needed to compile the application binary. -- **Google Cloud SDK (`gcloud`)**: Needed in production to access cloud secrets. - -Instead of requiring every contributor to manually install and configure these system packages, Docker packages everything into a standardized, lightweight Linux container. This ensures that scripts execute identically on developer laptops (Linux, macOS, Windows) and in production (Google Cloud Platform). - -> [!NOTE] -> **New to Docker?** Check out the official [Docker 101 Tutorial](https://www.docker.com/101-tutorial/) or the [Docker Get Started Guide](https://docs.docker.com/get-started/) to learn the basics of images and containers. - ---- - -## Understanding the Multi-Stage Dockerfile - -Our [`Dockerfile`](file:///var/home/justin/Documents/Projects/api-tools/Dockerfile) uses Docker's multi-stage build pattern to keep production images secure, fast, and minimal. - -1. **`builder` Stage (`golang:1.26`)**: - - Copies Go source code. - - Runs `make setup`, `make check` (vetting and linting), and `make build`. - - Produces the compiled `api-tools` binary. - -2. **`base` Stage (`debian:12-slim`)**: - - A lightweight Debian base image. - - Installs runtime dependencies: `chromium`, `poppler-utils`, and `google-cloud-sdk`. - - Copies the compiled `api-tools` binary from the `builder` stage. - - Copies `runners/` and `static-data/`. - - Sets the entrypoint to [`runners/setup.sh`](file:///var/home/justin/Documents/Projects/api-tools/runners/setup.sh). - -3. **`local` Stage (`FROM base AS local`)**: - - Inherits from `base` and copies your local `.env` file into the container. - - Ideal for local testing and debugging. - ---- - -## Running Docker Locally - -### Step 1: Ensure Your `.env` File Exists -Make sure you have copied `.env.template` to `.env` in the project root: -```bash -cp .env.template .env -``` - -### Step 2: Build the Local Docker Image -Build using the `--target local` flag to include your local `.env` file: -```bash -docker build --target local -t my-runner:local . -``` - -### Step 3: Run a Runner Script -Run the container by specifying the environment mode and target script name: - -```bash -docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=daily.sh my-runner:local -``` - -You can replace `daily.sh` with any runner in the `runners/` directory: -- `daily.sh` — Scrapes, parses, and uploads events (Astra, Mazevo, Comet Calendar). -- `weekly.sh` — Scrapes, parses, and uploads academic calendars, discounts, and degrees. -- `monthly.sh` — Scrapes, parses, and uploads map locations and budgets. - ---- - -## Production vs. Local Execution (`runners/setup.sh`) - -When the Docker container starts, it executes [`runners/setup.sh`](file:///var/home/justin/Documents/Projects/api-tools/runners/setup.sh): - -- **Local Mode (`ENVIRONMENT=local`)**: - - Skips cloud authentication. - - Directly uses the `.env` file baked into the local image. - - Runs `/app/runners/$RUNNER_SCRIPT_NAME`. - -- **Production Mode (`ENVIRONMENT=gcp`)**: - - Authenticates with Google Cloud Secret Manager using service account keys. - - Dynamically fetches secrets and creates the `.env` file at runtime. - - Runs `/app/runners/$RUNNER_SCRIPT_NAME`. - ---- - -## Next Steps -- Learn how to contribute code in the [**Contributing Guide**](contributing.md). -- Resolve any container or browser errors in the [**Troubleshooting Guide**](troubleshooting.md). diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index 95b1bbb..65c3ccb 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -6,17 +6,19 @@ This guide walks you through setting up your local development environment, conf Ensure you have the following installed: -- Git +- Git - For version control - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop). -- Go +- Go - For building and running - You can install from the [Go website](https://go.dev/dl/), or from [Homebrew](https://brew.sh/) or another package manager for automatic updates -- **Make** *(Linux/macOS)* — Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) -- **Google Chrome** or **Chromium** — Required for headless browser scraping -- **Docker** *(Optional)* — For running containerized runners locally +- Make _(Linux/macOS)_ - For build automation + - Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) +- Google Chrome or Chromium - For headless browser scraping +- Docker _(Optional)_ - For running containerized runners locally **Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run. -If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file. When you see any command starting with `make`, you can instead use `.\build.bat`. For example instead of `make setup`, you can run `.\build.bat setup`. +If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file. When you see any command starting with `make`, you can instead use `.\build.bat`. +For example instead of `make setup`, you can run `.\build.bat setup`. ## Local Setup @@ -24,6 +26,24 @@ If you're using Windows, instead of using `make`, you can also use our [`build.b Clone the repository with `git clone` and `cd` into the project directory or open it in your code editor +**HTTPS:** + +```bash +git clone https://github.com/UTDNebula/api-tools.git +cd api-tools +``` + +or + +**SSH:** + +```bash +git clone git@github.com:UTDNebula/api-tools.git +cd api-tools +``` + +Now open the cloned project in your code editor! + ### Install Development Tooling Install the Go static analysis and formatting tools (`staticcheck` and `goimports`): @@ -60,7 +80,8 @@ If you installed **Go** with [**Homebrew**](https://brew.sh/), you need to add G ### Configure Environment Variables -Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env`. If you're not sure what to put, ask for help. +Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env` +If you're not sure what to put, please try contacting one of our members through discord. ### Run Code Verification & Formatting diff --git a/docs/Guides.md b/docs/Guides.md deleted file mode 100644 index 4c97890..0000000 --- a/docs/Guides.md +++ /dev/null @@ -1,27 +0,0 @@ -# Guides - -There's a lot of guides in here. You do **not** need to master everything before getting started! Here is a curated list of resources to help you learn the fundamentals. - -### Git & GitHub - -> **TODO:** [Link to Nebula Git Workshop Guide here] - -[GitHub Skills Git Introduction](https://skills.github.com/) - - -### Go (Golang) -- [A Tour of Go](https://go.dev/tour/) — Official interactive introduction to Go syntax, data structures, and concurrency. -- [Go by Example](https://gobyexample.com/) — Hands-on, concise Go examples covering real-world concepts from structs to JSON. -- [Learn Go in Y Minutes](https://learnxinyminutes.com/docs/go/) — Quick visual cheat sheet for developers who want a fast syntax refresher. - -### Docker & Containers -- [Docker 101 Tutorial](https://www.docker.com/101-tutorial/) — Learn how containers package code and dependencies for reliable execution. -- [Docker Documentation: Orientation and Setup](https://docs.docker.com/get-started/) — Official step-by-step introduction to images and containers. - -### Web Basics & Scraping -- [MDN Web Docs: HTML & DOM Basics](https://developer.mozilla.org/en-US/docs/Learn/HTML) — Learn how web pages are structured with elements, tags, and attributes. -- [MDN Web Docs: CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors) — Understand class, ID, and attribute selectors used to find page elements. -- [ChromeDP Documentation & Examples](https://github.com/chromedp/examples) — Examples for browser automation and scraping dynamic JavaScript pages in Go. - -### Command Line & Shell -- [Bash / CLI Crash Course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line) — Beginner-friendly guide to navigating files, running scripts, and managing processes in the terminal. diff --git a/docs/Home.md b/docs/Home.md index 6e37190..10c2b2a 100644 --- a/docs/Home.md +++ b/docs/Home.md @@ -4,15 +4,7 @@ Welcome to the **API Tools Developer Documentation**! These docs are intended for developers of `api-tools` rather than users. For usage instructions, see [README.md](../README.md). ---- - -If you are a new developer, we recommend checking out these pages in this order: - -1. [Getting-Started.md](Getting-Started.md) -1. [Project-Architecture.md](Project-Architecture.md) -1. [Project-Structure.md](Project-Structure.md) -1. [How-to-Contribute.md](How-to-Contribute.md) -1. [Guides.md](Guides.md) +If you are a new developer, check out the pages under "Getting Started" in the Sidebar! ## Need Help? diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md index 06b5c1a..3487469 100644 --- a/docs/How-to-Contribute.md +++ b/docs/How-to-Contribute.md @@ -1,27 +1,30 @@ # How to Contribute -Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more. Don't worry if you don't quite know what you're doing, we're here to help. We don't expect perfection and appreciate anything you can do to help! +Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more. +Don't worry if you don't quite know what you're doing, we're here to help. We don't expect perfection and appreciate anything you can do to help! ## Find something to work on Look at our current issues, or make your own on our [issues page](../issues). -Write a comment asking if you can work on a particular issue, or assign it to yourself. You can work collaboratively, or individually on an issue. +For beginner issues, please write the code yourself rather than using AI to generate it. These issues are designed to help new members learn the codebase and build familiarity with the project. We allow AI in Nebula Labs, but review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy) for more information on acceptable uses of AI. -Be sure to **discuss the issue frequently** especially if you don't quite understand. You can do this on GitHub, Discord, or in person. +Once you find an issue, write a comment asking if you can work on it. + +If you're unsure about something, **don't hesitate to ask for help**. We're happy to answer questions and help you work through issues. You can reach out on GitHub, Discord, or in person. ## Create a branch Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork as they do not have permission to make a branch. -Our branch naming convention is `-` for example, `738-new-developer-docs`. GitHub can automatically create a branch for you following this convention if you select "Create a branch" on the right sidebar under "Development" - ## Make Your Changes It's time to code! Don't forget to format and check your code. We have helper scripts with **Make** and **build.bat**: +Check your code with + ```bash make check ``` @@ -44,16 +47,14 @@ or .\build.bat test ``` -Discuss your progress frequently, and push your commits to GitHub! If you get stuck anywhere we're here to help! Avoid using AI to write code for your first contribution. - -We'd love to see you make a draft Pull Request, so we can see your progress and help you out! +We would appreciate if you make a draft Pull Request as your working on it, so we can see your progress and help you out! ## Making a Pull Request (PR) Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft. -Maintainers will review your PR and provide suggestions or approval. Don't take suggestions personally, we just want to help! +Maintainers will review your PR and may provide suggestions or request changes before approval. ## Next Step -See [Guides.md](Guides.md) +You have everything you need to start, but feel free to check out other pages first. diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md index 0fab8e3..5eb73cd 100644 --- a/docs/Project-Architecture.md +++ b/docs/Project-Architecture.md @@ -1,100 +1,74 @@ -# Pipeline Architecture & Core Concepts +# Project-Architecture -`api-tools` is built around a unidirectional, staged data pipeline. Each stage has a single, well-defined responsibility. Understanding this separation of concerns is essential before modifying or adding code. +`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded to the Nebula API. ---- +The data comes from various UTD data sources including courses, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. -## High-Level Data Flow +Tools are written in Go. If you're new to Go (or just rusty), two good places to start are [A Tour of Go](https://go.dev/tour/list), which is interactive, runs in your browser, and needs no setup, +and [Effective Go](https://go.dev/doc/effective_go), which covers how to write Go the idiomatic way and is worth skimming before diving into `scrapers/`, `parser/`, or `uploader/`. -The lifecycle of data moving through `api-tools` follows a four-stage progression: +## Project Pipeline + +Data moves through `api-tools` in three stages: ```mermaid flowchart LR - A[UTD Data Sources\nWebsites, APIs, PDFs] -->|1. Scraper| B[(Raw Data Files\ndata/)] - B -->|2. Parser| C{Validator\nvalidator.go} - D[(Static Data\nstatic-data/)] -->|2. Parser| C - C -->|Validated Models| E[3. Uploader] - E -->|4. MongoDB Insert/Merge| F[(Nebula API Database)] - - subgraph Automation - G[Runners\nrunners/*.sh] -.->|Orchestrates| A - G -.->|Orchestrates| B - G -.->|Orchestrates| E + A[UTD Data Sources
Websites, APIs] --> B + + subgraph STAGE["Three Stages"] + direction LR + B[Scraper] --> C[Parser] --> D[Uploader] end + + D --> E[(Nebula API Database)] ``` ---- - -## The Four Core Components - -### 1. Scrapers (`scrapers/`) — *Data Acquisition* -- **What they do**: Connect to external UTD web servers, APIs, and portals to download raw information. -- **How they work**: They use HTTP requests (via standard Go `net/http`) or headless browser automation with [ChromeDP](https://github.com/chromedp/chromedp) to navigate pages, fill forms, trigger clicks, and capture raw HTML, JSON, or PDF files. -- **Rule of Thumb**: **Scrapers do NOT validate or parse data.** Their sole job is to faithfully capture raw data from the source and save it to disk in the `data/` directory (e.g., `data/24f/cp_cs/cs1337.001.24f.html`). -- **External Resources**: - - [MDN Web Docs: Introduction to the DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction) - - [ChromeDP Examples](https://github.com/chromedp/examples) - ---- - -### 2. Parsers (`parser/`) — *Data Transformation & Validation* -- **What they do**: Read raw files produced by scrapers alongside static reference files, extract meaningful fields, normalize inconsistent formatting, and assemble validated Go data structures. -- **How they work**: - - They parse HTML using Go HTML tokenizers and CSS query selectors (via `golang.org/x/net/html`). - - They cross-reference scraped records with static datasets (such as grade distribution CSVs in `static-data/grades` or budget PDFs in `static-data/budgets`). - - They run the parsed structs through validation checks in [`parser/validator.go`](file:///var/home/justin/Documents/Projects/api-tools/parser/validator.go) to ensure all required fields conform to the Nebula API schema. -- **Core Principle: Immutability**: Input files in `data/` are strictly **immutable**. Parsers must treat input files as read-only and never modify raw scraped data. -- **External Resources**: - - [Go `html` package documentation](https://pkg.go.dev/golang.org/x/net/html) - - [Go Regular Expressions (`regexp`) Guide](https://pkg.go.dev/regexp) - ---- - -### 3. Uploaders (`uploader/`) — *Database Synchronization* -- **What they do**: Take validated data models and push them to the [Nebula API](https://github.com/utdnebula/nebula-api) database (MongoDB). -- **How they work**: - - Connect to MongoDB using the official Go Mongo driver (`go.mongodb.org/mongo-driver`). - - Support both **merge** operations (updating existing records with new fields) and **replace** operations (overwriting outdated datasets). - - Compute static aggregation metrics where required before saving. -- **Rule of Thumb**: Uploaders assume data passed to them has already been parsed and validated. -- **External Resources**: - - [MongoDB Go Driver Tutorial](https://www.mongodb.com/docs/drivers/go/current/) - ---- - -### 4. Runners (`runners/`) — *Automated Orchestration* -- **What they do**: Shell scripts (`.sh`) that coordinate automated end-to-end execution of the pipeline on a scheduled cron cadence. -- **How they work**: - - Instead of manually typing CLI commands, runners chain together scraping, parsing, and uploading steps for specific domains. - - Examples in `runners/`: - - `daily.sh`: Daily scrape/parse/upload for volatile event data (Astra, Mazevo, Comet Calendar). - - `weekly.sh`: Weekly scrape/parse/upload for academic calendars, discount programs, and degree plans. - - `monthly.sh`: Monthly updates for campus map locations and budget reports. - - `setup.sh`: Docker entrypoint script that retrieves secrets from Google Cloud Secret Manager (in production) before launching a specified runner. -- **External Resources**: - - [Bash Scripting Tutorial for Beginners](https://linuxconfig.org/bash-scripting-tutorial-for-beginners) - ---- - -## Component Comparison - -| Component | Directory | Input | Output | Mutation / Side Effects | Key Goal | -|---|---|---|---|---|---| -| **Scraper** | `scrapers/` | UTD Websites & APIs | Raw files in `data/` (`.html`, `.json`, `.pdf`) | Writes raw data to disk | Reliable network extraction | -| **Parser** | `parser/` | `data/` + `static-data/` | Validated Go data models | **None** (inputs are read-only) | Structure & schema validation | -| **Uploader** | `uploader/` | Validated Go data models | MongoDB Collections | Inserts / updates database records | Database persistence | -| **Runner** | `runners/` | Scheduled Cron / Trigger | Executes CLI binary sequence | Runs multiple stages in sequence | End-to-end pipeline automation | - ---- - -## Why Separate These Concerns? - -1. **Testability & Offline Development**: Because scrapers save raw data to `data/`, parsers can be tested and developed offline without repeatedly pinging university servers or needing internet access. -2. **Resilience**: If a university website layout changes, only the scraper or selector logic needs updating; the uploader and downstream database logic remain completely unaffected. -3. **Reproducibility**: If an issue occurs during parsing or uploading, the raw input data is preserved on disk, allowing developers to inspect exact HTML responses and write reproducible test fixtures. - ---- +### Scrapers (`scrapers/`) + +Scrapers connect to websites, APIs, and portals to download raw information. They capture raw data in `.html`, `.json`, or other formats, and store it in the `data/` directory. For example, a scrapped course would be found at `data/24f/cp_cs/cs1337.001.24f.html` after running the scraper. + +For most scraping we use Go's built-in `net/http` library to directly make API requests and get our data. Sometimes we need a browser to recreate more complex actions. For automating browser actions, we use [ChromeDP](https://github.com/chromedp/chromedp) which uses Chrome DevTools. For extra scraping functionality not already covered by ChromeDP, we use [cdproto](https://github.com/chromedp/cdproto). + +### Parsers (`parser/`) + +Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then convert this data into a more machine-readable format. + +For HTML documents like courses, sections, professors, degree requirements, and discounts, we use [goquery](https://github.com/PuerkitoBio/goquery). Goquery lets us navigate our `.html` files programmatically with selectors. For more unstructured documents like calendars and budgets, we use [Google Gemini](https://pkg.go.dev/google.golang.org/genai). + +Everything gets structured and validated against the data models in [nebula-api/api](https://github.com/UTDNebula/nebula-api) (`api/schema`) via `parser/validator.go`. + +### Uploaders (`uploader/`) + +Uploaders take parsed data and upload it to the Nebula API MongoDB database. + +We use the official [MongoDB driver](https://pkg.go.dev/go.mongodb.org/mongo-driver) to connect to the database, turn Go structs into BSON documents, write them in bulk, and run aggregation queries. The shared schema definitions come from [nebula-api/api](https://github.com/UTDNebula/nebula-api), so what we store always matches what the API expects to find. + +## Cloud Automation + +Most data sources are updated automatically through shell scripts in `runners/`. These scripts run in containerized environments in Google Cloud. + +We use [Docker](https://www.docker.com/) to create a consistent environment to run `api-tools` in the cloud. We use [Google Cloud Build](https://cloud.google.com/build) to build our docker image, and [Google Cloud Scheduler](https://docs.cloud.google.com/scheduler/docs) to schedule our cloud pipeline runs. + +## Other Important Tools + +We use several tools while developing the API. + +Tests use Go's standard `testing` package for unit tests across the project. + +We use [Sentry](https://github.com/getsentry/sentry-go) for error tracking. + +We use [GitHub Actions](https://github.com/features/actions) to check Go code for PRs, deploy our Docker image, and deploy this Wiki! + +## Supporting Libraries + +We use several supporting libraries for more minor things. Here is a list of most of them, and their purpose + +- [fastjson](https://github.com/valyala/fastjson) - lets us read huge JSON payloads in scrapers without loading the whole thing into memory at once. It walks through the data piece by piece, which is how we handle things like the Astra room scheduling events. +- [dongri/phonenumber](https://github.com/dongri/phonenumber) - cleans up phone numbers from student discount programs. They show up in every format imaginable, so this gives us one consistent shape to store. +- [golang.org/x/text](https://pkg.go.dev/golang.org/x/text) - handles Unicode transformations and casing rules, which keeps event names and titles from getting mangled when they contain non-English characters. +- [google/go-cmp](https://github.com/google/go-cmp) - compares structs for our unit and regression tests, so when something breaks we can see exactly which field changed instead of just "not equal." +- [godotenv](https://github.com/joho/godotenv) - reads our `.env` files and loads things like URIs and credentials when the program starts, so we don't have to hardcode secrets or pass them on the command line. ## Next Step -See [Project-Structure.md](Project-Structure.md) +See [Project-Structure.md](Project-Structure.md) diff --git a/docs/Project-Structure.md b/docs/Project-Structure.md index 42ca8ea..6b72580 100644 --- a/docs/Project-Structure.md +++ b/docs/Project-Structure.md @@ -2,61 +2,42 @@ This document provides an overview of the `api-tools` directory layout, explaining what each folder contains and guiding you on where to make changes when adding features or fixing bugs. -## Directory Overview +## Project Structure ```text api-tools/ -├── data/ # Files from scrapers and parsers -├── docs/ # Source of wiki pages -├── logs/ # Timestamped log files from runtime executions -├── parser/ # Parsing, data normalization, and schema validation -│ └── testdata/ # HTML input fixtures and expected JSON, used for tests -├── runners/ # Shell automation scripts ran on Google Cloud (daily, weekly, monthly) -├── scrapers/ # Web scrapers for UTD websites, portals, and APIs -├── static-data/ # Static data that isn't scraped -├── uploader/ # Database ingestion modules for Nebula API / MongoDB -├── utils/ # Shared utilities (ChromeDP helpers, logger, regexes) -├── .env.template # Template for required environment variables -├── Dockerfile # Multi-stage container definition -├── Makefile # Task runner for Linux and macOS (build, check, test) -├── build.bat # Task runner for Windows (build, check, test) -├── cloudbuild.yaml # Google Cloud Build deployment configuration -├── go.mod / go.sum # Go dependency manifests -└── main.go # Application entrypoint and CLI flag definitions +├── data/ # Files from scrapers and parsers +├── docs/ # Source of wiki pages +├── logs/ # Timestamped log files from runtime executions +├── parser/ # Parsing, data normalization, and schema validation +│ ├── parser.go # Central parsing orchestrator +│ ├── Parser.go # Domain-specific parsers (courses, sections, maps, ASTRA, etc.) +│ ├── gradeLoader.go # Loads historical grade distributions +│ ├── requisiteParser.go # Parses prerequisite/corequisite strings +│ ├── validator.go # Validates parsed structs before upload +│ └── testdata/ # HTML fixtures and expected JSON for regression tests +├── runners/ # Shell automation scripts run on Google Cloud +├── scrapers/ # Web scrapers for UTD websites, portals, and APIs +├── static-data/ # Static data that isn't scraped +│ ├── grades/ # Historical grade-distribution CSVs by semester +│ └── budgets/ # Historical budget PDFs +├── uploader/ # Database ingestion modules for Nebula API / MongoDB +│ ├── database.go # Establishes MongoDB connections +│ ├── uploader.go # Coursebook uploads and merge/replace strategies +│ └── Uploader.go # Domain-specific uploaders +├── utils/ # Shared utilities +│ ├── methods.go # ChromeDP, environment variables, headless mode, tokens +│ ├── logger.go # Runtime logging to stdout and timestamped files +│ └── regexes.go # Shared parsing and extraction regexes +├── .env.template # Template for required environment variables +├── Dockerfile # Multi-stage container definition +├── Makefile # Build, check, and test tasks for Linux/macOS +├── build.bat # Build, check, and test tasks for Windows +├── cloudbuild.yaml # Google Cloud Build deployment configuration +├── go.mod / go.sum # Go dependency manifests +└── main.go # Application entrypoint and CLI flag definitions ``` -## Directory Breakdown - -### `parser/` -Responsible for interpreting raw files, sanitizing fields, resolving cross-references, and ensuring schema compliance: -- `Parser.go`: Dedicated parser files corresponding to each scraper (e.g., `courseParser.go`, `sectionParser.go`, `mapParser.go`, `astraParser.go`). -- `parser.go`: The central orchestrator that coordinates parsing of scraped coursebook files with static grade and profile data. -- `gradeLoader.go`: Reads historical grade distribution CSVs from `static-data/grades`. -- `requisiteParser.go`: Parses prerequisite and corequisite strings into structured dependency trees. -- `validator.go`: Validates that parsed Go structs contain all non-empty mandatory fields before upload. -- `testdata/`: Test suites containing realistic sample input files (`input.html`) and expected JSON outputs (`course.json`, `section.json`, `professors.json`) for automated regression testing. - -### `uploader/` -Handles pushing structured, validated data into MongoDB: -- `database.go`: Establishes MongoDB client connections using `MONGODB_URI`. -- `uploader.go`: Core coursebook data uploader with merge/replace strategies. -- `Uploader.go`: Specialized uploaders for events, maps, discounts, degrees, and academic calendars. - -### `static-data/` -Persistent reference datasets that cannot be scraped on-demand or serve as historical baselines: -- `static-data/grades/`: CSV files organized by semester (e.g., `22F.csv` for Fall 2022, `23S.csv` for Spring 2023, `23U.csv` for Summer 2023). -- `static-data/budgets/`: Historical PDF budget files for fiscal years where web links are no longer available. - -### `utils/` -Shared helper functions used across scrapers, parsers, and uploaders: -- `methods.go`: ChromeDP initialization, headless settings, environment variable readers (`GetEnv`), and token refresh helpers. -- `logger.go`: Dual logger implementation (`SplitWriter`) that writes output simultaneously to `stdout` and timestamped files in `logs/`. -- `regexes.go`: Compiled regular expressions for text parsing and course extraction. - -### Root Configuration Files -- `main.go`: The entrypoint of the CLI application. Parses command line flags (like `-scrape`, `-parse`, `-upload`, `-verbose`, `-headless`) and routes execution to the appropriate package functions. -- `Makefile` / `build.bat`: Scripts to automate setup, formatting checks (`staticcheck`, `gofmt`, `goimports`, `go vet`), compiling the binary, and running unit tests. -- `Dockerfile`: Defines the multi-stage Docker build for local testing and production deployment. - ## Next Step -See [How-to-Contribute.md](How-to-Contribute.md) + +See [How-to-Contribute.md](How-to-Contribute.md) diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md index ec6618e..3c53059 100644 --- a/docs/Troubleshooting.md +++ b/docs/Troubleshooting.md @@ -1,96 +1,27 @@ -# Troubleshooting & FAQ +# Troubleshooting -This guide provides fast, actionable solutions to common issues encountered when running or developing `api-tools`. +This guide gives solutions to common problems developers or users of this project may run into. ---- +## Enabling Verbose Debug Logging -## 1. Chromium & ChromeDP Issues - -### Issue: "exec: 'chromium': executable file not found in $PATH" or Chrome fails to launch -- **Cause**: ChromeDP cannot find a Chromium or Google Chrome binary on your operating system. -- **Solutions**: - - **Linux**: Install Chromium via your package manager: - ```bash - # Ubuntu / Debian - sudo apt update && sudo apt install -y chromium-browser - # Fedora / RHEL - sudo dnf install chromium - # Arch Linux - sudo pacman -S chromium - ``` - - **macOS / Windows**: Ensure [Google Chrome](https://www.google.com/chrome/) is installed in the standard applications directory. - - **Environment Variables**: If Chrome is installed in a non-standard location, specify its path explicitly before running `api-tools`: - ```bash - export CHROMIUM_BIN="/path/to/custom/chrome" - export GOOGLE_CHROME_BIN="/path/to/custom/chrome" - ``` - -### Issue: Chrome window opens during scraping and interferes with work -- **Solution**: Pass the `-headless` flag to run ChromeDP invisibly in background headless mode: - ```bash - ./api-tools -scrape -coursebook -term 24F -headless - ``` - ---- - -## 2. Environment Variables & MongoDB Connection Issues - -### Issue: Panic: ` is missing from .env!` -- **Cause**: An authenticated scraper or uploader attempted to read a required environment variable from `.env` via `utils.GetEnv`, but the key was either missing or empty. -- **Solutions**: - 1. Ensure you have copied `.env.template` to `.env`: - ```bash - cp .env.template .env - ``` - 2. For Coursebook, Astra, or Mazevo scrapers, populate the required credentials: - - `LOGIN_NETID` / `LOGIN_PASSWORD` (UTD NetID login) - - `LOGIN_ASTRA_USERNAME` / `LOGIN_ASTRA_PASSWORD` - - `MAZEVO_API_KEY` - 3. If you only want to parse existing local data or run tests, you do **not** need these credentials; avoid calling authenticated scraping flags. - -### Issue: MongoDB connection timed out / "server selection error" -- **Cause**: The uploader could not establish a connection to the MongoDB instance specified in `MONGODB_URI`. -- **Solutions**: - - **Verify Connection String**: Check that `MONGODB_URI` in `.env` is formatted correctly: - ```env - MONGODB_URI="mongodb+srv://:@cluster0.abcde.mongodb.net/?retryWrites=true&w=majority" - ``` - - **Check Network & IP Whitelist**: If using MongoDB Atlas, ensure your current IP address is whitelisted under Atlas Network Access. - - **Local Development**: If testing locally with a local MongoDB container or service, verify MongoDB is running: - ```bash - # Test local MongoDB connection - mongosh "mongodb://localhost:27017" - ``` - ---- - -## 3. Locating & Inspecting Log Files - -### How Logging Works -When `api-tools` executes, `utils.NewSplitWriter` writes all log output to two destinations simultaneously: -1. **Console (`stdout`)**: Live output in your terminal window. -2. **Log File (`logs/*.log`)**: A persistent log file saved to the `logs/` directory named with the execution timestamp: - ``` - logs/8-31-2026T12-30-15.log - ``` - -### Enabling Verbose Debug Logging If a scraper or parser is behaving unexpectedly, re-run the command with the `-verbose` flag: +For example: + ```bash ./api-tools -verbose -scrape -coursebook -term 24F ``` -Verbose mode enables: -- Microsecond timestamp precision (`log.Lmicroseconds`). -- Source code filename and line number for every log statement (`log.Lshortfile`). -- Additional internal debug logs (`utils.Lverbose`). +This will give more logs, and can help you better understand what's going on. + +## "exec: 'chromium': executable file not found in $PATH" or Chrome fails to launch + +ChromeDP cannot find a Chromium or Google Chrome binary on your operating system. You'll need to install it. + +## KEY is missing from .env ---- +Ensure you have filled environment variables in .env. If you need help, ask for help! ## Still Stuck? -Reach out to the team on [Discord](https://discord.utdnebula.com) with: -1. The exact command you ran. -2. The relevant lines from your latest log file in `logs/`. -3. Your operating system and Go version (`go version`). +Ask for help on [Discord](https://discord.utdnebula.com)! diff --git a/docs/_Sidebar.md b/docs/_Sidebar.md new file mode 100644 index 0000000..0fb1541 --- /dev/null +++ b/docs/_Sidebar.md @@ -0,0 +1,10 @@ +## Getting Started + +- [Getting Started](Getting-Started.md) +- [Project Architecture](Project-Architecture.md) +- [Project Structure](Project-Structure.md) +- [How to Contribute](How-to-Contribute.md) + +## References + +- [Troubleshooting](Troubleshooting.md) \ No newline at end of file