in api-tools in main.go: runs all the scrapers, we’ll be looking at scrapeCoursebook
initChromeDp first
RunResponse runs a set of actions on the context
Run would not be able to handle page navigation/changes, like what you trigger by clicking a button
use RunResponse to log in with your netID and password to scrape without being rate limited
ClearBrowserCookies (using ActionFunc which is like a custom action) to get a new token at the start of each new prefix which resets your rate limit
get headers from with cookies to use in later requests
make a new request to coursebooks’s behind-the-scenes url for each course (this reverse engineering makes scraping faster and easier, don’t have to click thru so many dropdowns/buttons)
and then for each section in each course
also refresh the token every 30 seconds, just in case