2024-09-05 Meeting notes

Go resources

• Get the API team role in channels & roles

• Resources are in the API announcements channel

  • Go

  • Gin (routing library)

  • Go MongoDB Driver

  • ChromeDP (for web scraping)

  • Video resources (more lengthy)

Nebula API overview

• server.go starts up

  • uses CORS and middleware (logging)

  • route http requests to controllers based on the path

  • run

• Request example: /course routes to course.go which routes based on the rest of the path. Course search is one of these:

  • validate query search and parameters, send error otherwise

  • talk to MongoDB to get results

  • format results and return to user, all within 10 seconds or send error

Questions

• Go vs. NodeJS

  • Original version of API actually used Node

  • Main 3 Go advantages: speed, easy to learn, and growing in popularity

  • Why not?

Nebula API tools overview

• main.go

  • command line interface that controls tools based on flags and params

  • run a scraper, the parser, or the uploader

• Professor profile scraper overview

  • go get all the links to all the pages of profiles

  • get all the profile links on each

  • go to each profile and pull all the info out

• The scraper, parser, and uploader are all separated so you can modify each part individually

• Parser overview

  • load data from scraped files, run parse on each

  • pull data out of html

    • parse natural language requisites

  • run validation

Last notes

• Keep looking at the Go resources

• Next time will be going through the resources

• Reach out to Josh !!