diff --git a/.github/workflows/wiki.yml b/.github/workflows/wiki.yml new file mode 100644 index 00000000..38d21aff --- /dev/null +++ b/.github/workflows/wiki.yml @@ -0,0 +1,27 @@ +name: Publish Wiki + +on: + push: + branches: [develop] + paths: + - 'docs/**' + workflow_dispatch: + +concurrency: + group: publish-wiki + cancel-in-progress: true + +permissions: + contents: write + +jobs: + publish-wiki: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Sync docs to GitHub Wiki + uses: Andrew-Chen-Wang/github-wiki-action@v5 + with: + path: docs/ \ No newline at end of file diff --git a/README.md b/README.md index d78eae06..27f1fa9d 100644 --- a/README.md +++ b/README.md @@ -1,116 +1,145 @@ # API Tools -_A CLI to scrape some really useful UTD data, parse it, and upload it to the [Nebula API](https://github.com/utdnebula/nebula-api) for community use._ +_A CLI toolchain to scrape, parse, validate, and upload UTD university data to the [Nebula API](https://github.com/utdnebula/nebula-api) database._ Project maintained by [Nebula Labs](https://about.utdnebula.com). -### Design +> **Developer Wiki**: If you are interested in developing this project, head over to the [**Developer Wiki**](docs/Home.md)! -#### - The `scrapers` directory contains the scrapers for various UTD data sources. This is where the data pipeline begins. - - The scrapers are concerned solely with data collection, not necessarily validation or processing of said data. Those responsibilities are left to the parsing stage. -#### - The `parser` directory contains the files and methods that parse the scraped data. This is the 'middle man' of the data pipeline. - - The parsing stage is responsible for 'making sense' of the scraped data; this consists of reading, validating, and merging/intermixing of various data sources. - - The input data is considered **immutable** by the parsing stage. This means the parsers should never modify the data being fed into them. -#### - The `uploader` directory contains the uploader that sends the parsed data to the Nebula API MongoDB database. This is the final stage of the data pipeline. - - The uploader(s) are concerned solely with pushing parsed data to the database. Data, at this point, is assumed to be valid and ready for use. -#### - The `static-data/grades` directory contains .csv files of UTD grade data. - - Files are named by year and semester, with a suffix of `S`, `U`, or `F` denoting Spring, Summer, and Fall semesters, respectively. - - This means that, for example, `22F.csv` corresponds to the 2022 Fall semester, whereas `18U.csv` corresponds with the 2018 Summer semester. - - This grade data is collected independently from the scrapers, and is used during the parsing process. -#### - The `static-data/budgets` directory contains .pdf files of UTD budget data. - - Files are named by fiscal year. - - This budget data is used as a backup of scraped data as some years have been removed from the website. +--- -### Contributing +## Quick Navigation -Please visit our [Discord](https://discord.utdnebula.com) and talk to us if you'd like to contribute! +- [Developer Wiki](docs/Home.md) +- [Getting Started Guide](/docs/Getting-Started.md) +- [Pipeline Architecture (Scraper -> Parser -> Uploader)](/docs/Project-Architecture.md) +- [Project Structure & Codebase Map](/docs/Project-Structure.md) +- [Docker & Container Guide](/docs/Docker-Guide.md) +- [Contributing Guide](/docs/How-to-Contribute.md) +- [Troubleshooting & FAQ](/docs/Troubleshooting.md) +- [Discord Community](https://discord.utdnebula.com) -### Prerequisites +--- -- Golang 1.24 (or higher) +## Prerequisites & Building -### Build +- **Go 1.24+** +- **Make** (Linux/macOS) -To build the project, simply clone the repository and then either: - - Run `make` in the root (top level) directory (for systems with `make` installed, i.e. most Linux distros, MacOS) - - Run `build.bat` on Windows systems (unless you want to deal with getting `make` to work on Windows :P) +To build the executable binary: -The build process will output an executable file named `api-tools`; this executable is the CLI and can be ran in your terminal! +```bash +# Linux / macOS +make setup # Install linter tools (staticcheck, goimports) +make check # Verify formatting, vetting, and linting +make build # Compiles ./api-tools binary +make test # Run automated test suite -Additionally, you can run build (on Windows) and make (on MacOS/Linux) with the following arguments: - - `setup`: Installs required dependencies for the tools. - - `check`: Verifies prequisites and ensures the executable can be built. - - `test`: Test run to see if the executable works after building - - `build`: Builds the executble and makes it ready for use. +# Windows +build.bat setup +build.bat check +build.bat build # Compiles api-tools.exe +build.bat test +``` -### Usage +--- -The `api-tools` command line interface supports three main modes: scraping, parsing and uploading data to the Nebula API. +## CLI Usage Reference -#### Environment Variables -Before being able to use the tool, configure the `.env` file by following these steps: - 1. Find the `.env.template` file and rename it to `.env` - 2. Specify the required credentials for your use case as a string ("") following the variable name. - Example: LOGIN_NETID="ABC123456" +The `api-tools` command line interface supports three main modes: **scraping**, **parsing**, and **uploading**. -#### Basic Usage +### Global Flags -Run the tool by changing directory using `cd` to the `api-tools` directory and running the executable with the appropriate flags in the command line. To see all available options with the tool, run: `./api-tools`. To enable logging for debugging, use the verbose flag: `./api-tools -verbose`. Find available flags for each mode in the following tables. +| Flag | Default | Description | +| ------------ | -------- | ------------------------------------------------------------------------ | +| `-i ` | `./data` | Input directory to read data from | +| `-o ` | `./data` | Output directory to write scraped/parsed data to | +| `-l ` | `./logs` | Directory to write timestamped log files to | +| `--verbose` | `false` | Enables verbose debug logging with file names and microsecond timestamps | +| `--headless` | `false` | Runs ChromeDP browser automation in background headless mode | --- -### Scraping Mode - -| Command | Description | -|---------|-------------| -| `./api-tools -scrape -academicCalendars` | Scrapes academic calendar PDFs. | -| `./api-tools -scrape -astra` | Scrapes Astra data. | -| `./api-tools -scrape -cometCalendar` | Scrapes Comet Calendar data. | -| `./api-tools -scrape -coursebook -term 24F` | Scrapes coursebook data for Fall 2024.
• Use `-resume` to continue from last prefix.
• Use `-startprefix [prefix]` to begin at a specific course prefix. | -| `./api-tools -scrape -discounts` | Scrapes discount programs. | -| `./api-tools -scrape -degrees` | Scrapes degrees data | -| `./api-tools -scrape -map` | Scrapes UTD Map data. | -| `./api-tools -scrape -mazevo` | Scrapes Mazevo data. | -| `./api-tools -scrape -profiles` | Scrapes UTD professor profiles. | -| `./api-tools -scrape -headless` | Runs ChromeDP in headless mode. | -| `./api-tools -o [directory]` | Sets output directory (default: `./data`). | - -### Parsing Mode: - -| Command | Description | -|---------|-------------| -| `./api-tools -parse -academicCalendars` | Parses academic calendar PDFs. | -| `./api-tools -parse -astra` | Parses Astra data. | -| `./api-tools -parse -cometCalendar` | Parses Comet Calendar data. | -| `./api-tools -parse -csv [directory]` | Outputs grade data CSVs (default: `./static-data/grades`). | -| `./api-tools -parse -discounts` | Parses discount programs HTML. | -| `./api-tools -parse -degrees` | Parses degrees from HTML. | -| `./api-tools -parse -map` | Parses UTD Map data. | -| `./api-tools -parse -mazevo` | Parses Mazevo data. | -| `./api-tools -parse -skipv` | Skips post-parse validation (**use with caution**). | - - -### Upload Mode: -| Command | Description | -|---------|-------------| -| `./api-tools -upload -academicCalendars` | Uploads academic calendars. | -| `./api-tools -upload -discounts` | Uploads discount programs. | -| `./api-tools -upload -events` | Uploads Astra, Mazevo, and Comet Calendar data. | -| `./api-tools -upload -map` | Uploads UTD Map data. | -| `./api-tools -upload -replace` | Replaces old data instead of merging. | -| `./api-tools -upload -static` | Uploads only static aggregations. | - -Additionally, you can use the `-i [directory]` flag to specify where to read data from (default: `./data`) and the `-l [directory]` flag to specify where logs must be dumped (default: `./logs`). - -#### Docker - -Docker is used for automated running on Google Cloud Platform. More info [here](https://nebula-labs.atlassian.net/wiki/x/AYBjFw). - -To build the container for local testing first make sure all scripts in the `runners` folder have LF line endings then run: -``` +### Scraping Mode (`--scrape`) + +Scrapes raw data from UTD sources and saves it to `./data` (or the directory specified by `-o`). + +| Command | Description | +| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `./api-tools --scrape --coursebook --term 24F` | Scrapes CourseBook data for Fall 2024.
• `--resume`: Resume from last completed course prefix.
• `--startprefix `: Begin at a specific course prefix (e.g. `cp_cs`). | +| `./api-tools --scrape --evals --term 24F` | Scrapes CourseBook course evaluation data for the specified term. | +| `./api-tools --scrape --profiles` | Scrapes faculty and professor directory profiles. | +| `./api-tools --scrape --astra` | Scrapes Astra schedule and room reservation data. | +| `./api-tools --scrape --mazevo` | Scrapes Mazevo event reservation data via API. | +| `./api-tools --scrape --cometCalendar` | Scrapes public UTD Comet Calendar events. | +| `./api-tools --scrape --map` | Scrapes UTD campus map building and parking coordinates. | +| `./api-tools --scrape --degrees` | Scrapes academic degrees and catalog majors. | +| `./api-tools --scrape --discounts` | Scrapes student discount program listings. | +| `./api-tools --scrape --academicCalendars` | Scrapes academic calendar PDF documents. | +| `./api-tools --scrape --budgets` | Scrapes university budget reports. | + +--- + +### Parsing Mode (`-parse`) + +Reads raw files from `./data` (or `-i`), normalizes records against `static-data/`, validates data models, and outputs structured schemas. + +| Command | Description | +| --------------------------------------- | ------------------------------------------------------------------------------------------- | +| `./api-tools -parse` | Runs full coursebook parsing pipeline across all scraped prefixes. | +| `./api-tools -parse -gradesDir ` | Specifies custom path for grade distribution CSVs (default: `./static-data/grades`). | +| `./api-tools -parse -astra` | Parses scraped Astra scheduling data. | +| `./api-tools -parse -mazevo` | Parses scraped Mazevo event data. | +| `./api-tools -parse -cometCalendar` | Parses scraped Comet Calendar events. | +| `./api-tools -parse -map` | Parses campus map data. | +| `./api-tools -parse -degrees` | Parses degree plans from HTML. | +| `./api-tools -parse -discounts` | Parses student discount listings from HTML. | +| `./api-tools -parse -academicCalendars` | Parses academic calendar dates from PDFs. | +| `./api-tools -parse -budgets` | Parses budget reports from PDFs (use `-useBackupBudgets` to include `static-data/budgets`). | +| `./api-tools -parse -skipv` | Skips post-parse schema validation (**use with caution**). | + +--- + +### Upload Mode (`-upload`) + +Pushes validated data to the MongoDB database specified in `MONGODB_URI`. + +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------- | +| `./api-tools -upload` | Uploads parsed coursebook data (merges into existing records by default). | +| `./api-tools -upload -replace` | Overwrites/replaces existing records instead of merging. | +| `./api-tools -upload -static` | Uploads static aggregations only. | +| `./api-tools -upload -events` | Uploads Astra, Mazevo, and Comet Calendar event datasets. | +| `./api-tools -upload -map` | Uploads campus map locations. | +| `./api-tools -upload -degrees` | Uploads degree plans. | +| `./api-tools -upload -discounts` | Uploads discount programs. | +| `./api-tools -upload -academicCalendars` | Uploads academic calendar dates. | +| `./api-tools -upload -budgets` | Uploads budget data. | + +--- + +## Docker & Automated Runners + +Automated scheduled scraping is executed via Docker containers and cloud cron jobs: + +```bash +# Build the container for local testing docker build --target local -t my-runner:local . + +# Run daily event tasks docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=daily.sh my-runner:local + +# Run weekly catalog tasks +docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=weekly.sh my-runner:local + +# Run monthly map & budget tasks +docker run --rm -e ENVIRONMENT=local -e RUNNER_SCRIPT_NAME=monthly.sh my-runner:local ``` -## Questions? -Reach out to the team on [Discord](https://discord.utdnebula.com) and with any questions you may have! +For more details on Docker execution and cloud configuration, see the [Docker Guide](/docs/Docker-Guide.md). + +--- + +## Community & Support + +Have questions, suggestions, or want to contribute? Reach out on our [Discord](https://discord.utdnebula.com)! diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md new file mode 100644 index 00000000..65c3ccb8 --- /dev/null +++ b/docs/Getting-Started.md @@ -0,0 +1,134 @@ +# Getting Started with API Tools + +This guide walks you through setting up your local development environment, configuring environment variables, compiling the CLI binary, and running tests. + +## Prerequisites + +Ensure you have the following installed: + +- Git - For version control + - If you've never used git, need a refresher, or need help setting it up, check out [Nebula's Git Workshop](https://github.com/UTDNebula/git-workshop). +- Go - For building and running + - You can install from the [Go website](https://go.dev/dl/), or from [Homebrew](https://brew.sh/) or another package manager for automatic updates +- Make _(Linux/macOS)_ - For build automation + - Pre-installed on macOS (via `xcode-select --install`) and most Linux distributions (`build-essential`) +- Google Chrome or Chromium - For headless browser scraping +- Docker _(Optional)_ - For running containerized runners locally + +**Make** is a build automation tool. Feel free to check out [`Makefile`](../Makefile) to see exactly what's being run. + +If you're using Windows, instead of using `make`, you can also use our [`build.bat`](../build.bat) file. When you see any command starting with `make`, you can instead use `.\build.bat`. +For example instead of `make setup`, you can run `.\build.bat setup`. + +## Local Setup + +### Clone the Repository + +Clone the repository with `git clone` and `cd` into the project directory or open it in your code editor + +**HTTPS:** + +```bash +git clone https://github.com/UTDNebula/api-tools.git +cd api-tools +``` + +or + +**SSH:** + +```bash +git clone git@github.com:UTDNebula/api-tools.git +cd api-tools +``` + +Now open the cloned project in your code editor! + +### Install Development Tooling + +Install the Go static analysis and formatting tools (`staticcheck` and `goimports`): + +```bash +make setup +``` + +or + +```cmd +.\build.bat setup +``` + +If you installed **Go** with [**Homebrew**](https://brew.sh/), you need to add Go tools to your path for tools like `staticcheck` and others to work. + +- **For zsh** (the default shell on MacOS) + + ```bash + echo 'export PATH=${PATH}:`go env GOPATH`/bin' >> ~/.zshrc && source ~/.zshrc + ``` + +- **For bash** (the default shell on most Linux distributions) + + ```bash + echo 'export PATH=${PATH}:`go env GOPATH`/bin' >> ~/.bashrc && source ~/.bashrc + ``` + +- **For fish** + + ```bash + echo 'fish_add_path (go env GOPATH)/bin' >> ~/.config/fish/config.fish + ``` + +### Configure Environment Variables + +Make a file called `.env` at the root of the project, and copy the contents of `.env.template` into it. Some tools in `api-tools` require certain environment variables, which you can fill in `.env` +If you're not sure what to put, please try contacting one of our members through discord. + +### Run Code Verification & Formatting + +Check your code with: + +```bash +make check +``` + +or + +```cmd +.\build.bat check +``` + +You'll want to run this frequently while developing + +### Build the CLI Executable + +Compile the Go source code into a runnable binary (`./api-tools` on Linux/macOS, `api-tools.exe` on Windows): + +```bash +make build +``` + +or + +```cmd +.\build.bat build +``` + +### Verify with Automated Tests + +Run the test suite to confirm your environment is ready: + +```bash +make test +``` + +or + +```cmd +.\build.bat test +``` + +If all tests pass, congratulations! Your local development environment is ready. + +## Next Step + +Now that your environment is running, it's time to dive deeper. Check out [Project-Architecture.md](Project-Architecture.md) diff --git a/docs/Home.md b/docs/Home.md new file mode 100644 index 00000000..10c2b2a3 --- /dev/null +++ b/docs/Home.md @@ -0,0 +1,17 @@ +# API Tools Developer Documentation + +Welcome to the **API Tools Developer Documentation**! + +These docs are intended for developers of `api-tools` rather than users. For usage instructions, see [README.md](../README.md). + +If you are a new developer, check out the pages under "Getting Started" in the Sidebar! + +## Need Help? + +Join [our community on Discord](https://discord.utdnebula.com) and ask questions in the `#api-help` or development channels. We are always happy to help! + +## Found something you want to change on these docs? + +Don't edit these docs manually on Github. Instead, suggest a change by [creating an issue](../issues), or [opening a pull request](../pulls) with your changes. + +These docs are automatically generated from the `docs/` folder of this repository. diff --git a/docs/How-to-Contribute.md b/docs/How-to-Contribute.md new file mode 100644 index 00000000..34874690 --- /dev/null +++ b/docs/How-to-Contribute.md @@ -0,0 +1,60 @@ +# How to Contribute + +Thank you for your interest in contributing to `api-tools`! This guide covers our development workflow, coding standards, and more. +Don't worry if you don't quite know what you're doing, we're here to help. We don't expect perfection and appreciate anything you can do to help! + +## Find something to work on + +Look at our current issues, or make your own on our [issues page](../issues). + +For beginner issues, please write the code yourself rather than using AI to generate it. These issues are designed to help new members learn the codebase and build familiarity with the project. We allow AI in Nebula Labs, but review our [AI Policy](https://nebula-labs.atlassian.net/wiki/spaces/NLG/pages/1135607810/AI+Policy) for more information on acceptable uses of AI. + +Once you find an issue, write a comment asking if you can work on it. + +If you're unsure about something, **don't hesitate to ask for help**. We're happy to answer questions and help you work through issues. You can reach out on GitHub, Discord, or in person. + +## Create a branch + +Nebula recruits and members should make their changes on a branch, external contributors should work off of a fork as they do not have permission to make a branch. + +## Make Your Changes + +It's time to code! + +Don't forget to format and check your code. We have helper scripts with **Make** and **build.bat**: + +Check your code with + +```bash +make check +``` + +or + +```cmd +.\build.bat check +``` + +Run tests with + +```bash +make test +``` + +or + +```cmd +.\build.bat test +``` + +We would appreciate if you make a draft Pull Request as your working on it, so we can see your progress and help you out! + +## Making a Pull Request (PR) + +Open [our Pull Request Page](../pulls), and create a PR. If you're not finished, you can make your PR a draft. + +Maintainers will review your PR and may provide suggestions or request changes before approval. + +## Next Step + +You have everything you need to start, but feel free to check out other pages first. diff --git a/docs/Project-Architecture.md b/docs/Project-Architecture.md new file mode 100644 index 00000000..5eb73cdf --- /dev/null +++ b/docs/Project-Architecture.md @@ -0,0 +1,74 @@ +# Project-Architecture + +`api-tools` is a collection of tools used by [nebula-api](https://github.com/UTDNebula/nebula-api). Data is collected, processed, validated, and uploaded to the Nebula API. + +The data comes from various UTD data sources including courses, maps, events, and more. Many of these tools are self-contained and can be run directly from the command line. See the [README.md](/README.md) for instructions on running them. + +Tools are written in Go. If you're new to Go (or just rusty), two good places to start are [A Tour of Go](https://go.dev/tour/list), which is interactive, runs in your browser, and needs no setup, +and [Effective Go](https://go.dev/doc/effective_go), which covers how to write Go the idiomatic way and is worth skimming before diving into `scrapers/`, `parser/`, or `uploader/`. + +## Project Pipeline + +Data moves through `api-tools` in three stages: + +```mermaid +flowchart LR + A[UTD Data Sources
Websites, APIs] --> B + + subgraph STAGE["Three Stages"] + direction LR + B[Scraper] --> C[Parser] --> D[Uploader] + end + + D --> E[(Nebula API Database)] +``` + +### Scrapers (`scrapers/`) + +Scrapers connect to websites, APIs, and portals to download raw information. They capture raw data in `.html`, `.json`, or other formats, and store it in the `data/` directory. For example, a scrapped course would be found at `data/24f/cp_cs/cs1337.001.24f.html` after running the scraper. + +For most scraping we use Go's built-in `net/http` library to directly make API requests and get our data. Sometimes we need a browser to recreate more complex actions. For automating browser actions, we use [ChromeDP](https://github.com/chromedp/chromedp) which uses Chrome DevTools. For extra scraping functionality not already covered by ChromeDP, we use [cdproto](https://github.com/chromedp/cdproto). + +### Parsers (`parser/`) + +Parsers read raw files produced by scrapers, and non scraped data stored in (`static-data/`). They then convert this data into a more machine-readable format. + +For HTML documents like courses, sections, professors, degree requirements, and discounts, we use [goquery](https://github.com/PuerkitoBio/goquery). Goquery lets us navigate our `.html` files programmatically with selectors. For more unstructured documents like calendars and budgets, we use [Google Gemini](https://pkg.go.dev/google.golang.org/genai). + +Everything gets structured and validated against the data models in [nebula-api/api](https://github.com/UTDNebula/nebula-api) (`api/schema`) via `parser/validator.go`. + +### Uploaders (`uploader/`) + +Uploaders take parsed data and upload it to the Nebula API MongoDB database. + +We use the official [MongoDB driver](https://pkg.go.dev/go.mongodb.org/mongo-driver) to connect to the database, turn Go structs into BSON documents, write them in bulk, and run aggregation queries. The shared schema definitions come from [nebula-api/api](https://github.com/UTDNebula/nebula-api), so what we store always matches what the API expects to find. + +## Cloud Automation + +Most data sources are updated automatically through shell scripts in `runners/`. These scripts run in containerized environments in Google Cloud. + +We use [Docker](https://www.docker.com/) to create a consistent environment to run `api-tools` in the cloud. We use [Google Cloud Build](https://cloud.google.com/build) to build our docker image, and [Google Cloud Scheduler](https://docs.cloud.google.com/scheduler/docs) to schedule our cloud pipeline runs. + +## Other Important Tools + +We use several tools while developing the API. + +Tests use Go's standard `testing` package for unit tests across the project. + +We use [Sentry](https://github.com/getsentry/sentry-go) for error tracking. + +We use [GitHub Actions](https://github.com/features/actions) to check Go code for PRs, deploy our Docker image, and deploy this Wiki! + +## Supporting Libraries + +We use several supporting libraries for more minor things. Here is a list of most of them, and their purpose + +- [fastjson](https://github.com/valyala/fastjson) - lets us read huge JSON payloads in scrapers without loading the whole thing into memory at once. It walks through the data piece by piece, which is how we handle things like the Astra room scheduling events. +- [dongri/phonenumber](https://github.com/dongri/phonenumber) - cleans up phone numbers from student discount programs. They show up in every format imaginable, so this gives us one consistent shape to store. +- [golang.org/x/text](https://pkg.go.dev/golang.org/x/text) - handles Unicode transformations and casing rules, which keeps event names and titles from getting mangled when they contain non-English characters. +- [google/go-cmp](https://github.com/google/go-cmp) - compares structs for our unit and regression tests, so when something breaks we can see exactly which field changed instead of just "not equal." +- [godotenv](https://github.com/joho/godotenv) - reads our `.env` files and loads things like URIs and credentials when the program starts, so we don't have to hardcode secrets or pass them on the command line. + +## Next Step + +See [Project-Structure.md](Project-Structure.md) diff --git a/docs/Project-Structure.md b/docs/Project-Structure.md new file mode 100644 index 00000000..6b725802 --- /dev/null +++ b/docs/Project-Structure.md @@ -0,0 +1,43 @@ +# Project Structure & Codebase Map + +This document provides an overview of the `api-tools` directory layout, explaining what each folder contains and guiding you on where to make changes when adding features or fixing bugs. + +## Project Structure + +```text +api-tools/ +├── data/ # Files from scrapers and parsers +├── docs/ # Source of wiki pages +├── logs/ # Timestamped log files from runtime executions +├── parser/ # Parsing, data normalization, and schema validation +│ ├── parser.go # Central parsing orchestrator +│ ├── Parser.go # Domain-specific parsers (courses, sections, maps, ASTRA, etc.) +│ ├── gradeLoader.go # Loads historical grade distributions +│ ├── requisiteParser.go # Parses prerequisite/corequisite strings +│ ├── validator.go # Validates parsed structs before upload +│ └── testdata/ # HTML fixtures and expected JSON for regression tests +├── runners/ # Shell automation scripts run on Google Cloud +├── scrapers/ # Web scrapers for UTD websites, portals, and APIs +├── static-data/ # Static data that isn't scraped +│ ├── grades/ # Historical grade-distribution CSVs by semester +│ └── budgets/ # Historical budget PDFs +├── uploader/ # Database ingestion modules for Nebula API / MongoDB +│ ├── database.go # Establishes MongoDB connections +│ ├── uploader.go # Coursebook uploads and merge/replace strategies +│ └── Uploader.go # Domain-specific uploaders +├── utils/ # Shared utilities +│ ├── methods.go # ChromeDP, environment variables, headless mode, tokens +│ ├── logger.go # Runtime logging to stdout and timestamped files +│ └── regexes.go # Shared parsing and extraction regexes +├── .env.template # Template for required environment variables +├── Dockerfile # Multi-stage container definition +├── Makefile # Build, check, and test tasks for Linux/macOS +├── build.bat # Build, check, and test tasks for Windows +├── cloudbuild.yaml # Google Cloud Build deployment configuration +├── go.mod / go.sum # Go dependency manifests +└── main.go # Application entrypoint and CLI flag definitions +``` + +## Next Step + +See [How-to-Contribute.md](How-to-Contribute.md) diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md new file mode 100644 index 00000000..3c53059f --- /dev/null +++ b/docs/Troubleshooting.md @@ -0,0 +1,27 @@ +# Troubleshooting + +This guide gives solutions to common problems developers or users of this project may run into. + +## Enabling Verbose Debug Logging + +If a scraper or parser is behaving unexpectedly, re-run the command with the `-verbose` flag: + +For example: + +```bash +./api-tools -verbose -scrape -coursebook -term 24F +``` + +This will give more logs, and can help you better understand what's going on. + +## "exec: 'chromium': executable file not found in $PATH" or Chrome fails to launch + +ChromeDP cannot find a Chromium or Google Chrome binary on your operating system. You'll need to install it. + +## KEY is missing from .env + +Ensure you have filled environment variables in .env. If you need help, ask for help! + +## Still Stuck? + +Ask for help on [Discord](https://discord.utdnebula.com)! diff --git a/docs/_Sidebar.md b/docs/_Sidebar.md new file mode 100644 index 00000000..0fb15416 --- /dev/null +++ b/docs/_Sidebar.md @@ -0,0 +1,10 @@ +## Getting Started + +- [Getting Started](Getting-Started.md) +- [Project Architecture](Project-Architecture.md) +- [Project Structure](Project-Structure.md) +- [How to Contribute](How-to-Contribute.md) + +## References + +- [Troubleshooting](Troubleshooting.md) \ No newline at end of file