2025-01-30 Meeting notes

API Tools

• Getting data: more complex than the actual process

• UTD websites → Scraping → Parsing → Uploading

  • Scraping just pulls and stores all data in a readable format

    • Dumps in local storage

  • Parsing formats data correctly

  • Uploading to database (MongoDB)

• All steps are separate and can be tweaked and run without rerunning the other steps

Actual API

• Hosted on Google Cloud

• Reads data from MongoDB and sends it to users

Overall

• The main focus is on the API tools, not the actual API

  • Distributing the data is easy, but getting is hard

• Main users are the Nebula projects, which guide the API

Succession

• Josh is graduating at the end of the semester

  • All other election spots are open as well

• If you’re at all interest now or in the future, reach out

What does the team look like?

• Kanban board: categorizes issues into todo, in progress, and done

  • Also categorizes into L1, L2, L3: which means difficulty or time the issues will take

• Find a task you’re interested in then just make a comment to be assigned to it

Actual API - in depth

• A Docker container runs the Gin router

  • Communicates over HTTP at api.utdnebula.com

  • Creates routes and routes requests from routes to controllers

  • Controller examples: returning all courses, searching for courses

API Tools - in depth

• CLI tool which runs with paramaters

  • tell it to scrape/parse/upload data from a specific source, or all of it!

• Scraping is done with ChromeDP

• Scraping is not done automatically yet, it’s run once a semester

  • This is a change to make this semester

For new people

• Start learning Go,

• Check out the rest of the resources Josh posted in the Discord