Overview

Every scraper in the scrapers folder calls panic(err) when something goes wrong. This crashes the whole process and stops the scrape. Some scrapers run for a long time. For example, astra.go scrapes a full year of dates, and coursebook.go walks through every course prefix. If one HTTP request fails in the middle of a run, all the progress so far is lost.

Change the scrapers so that instead of calling panic(err), they return the error to the caller. Start with the smaller scraper files, for example scrapers/discounts.go and scrapers/map.go, then move to the larger ones. Do not change what the scrapers do, only how they report failures.

Some scrapers already use log.Fatal and log.Panic in a few places. Treat those the same way and return errors instead.

Resources

• https://pkg.go.dev/errors
• https://pkg.go.dev/fmt#Errorf
• https://go.dev/blog/error-handling-and-go