2024-08-29 Meeting notes

Tasks

• On GitHub projects

• L1-L3 tasks, higher number is more difficult

  • claim in GitHub projects board via a comment, josh will then assign it

• help wanted: need input and conversation

• on hold: don't work on this

• feature request/bug report

Josh’s role

• keep the team running

• help y'all out

• not possible if y’all don’t ask questions!

Questions

• Working on multiple projects

  • Possible but don’t spread yourself too thin

• How to alter the scope of an issue?

  • Try to stay within scope but communicate if you want to expand the scope wither with Josh or on the GitHub issue

  • You can always make a new issue too

• Goal of API this semester

  • This is a long running project, it will never be “done“

  • Cloud storage for other projects

  • Improving tools

    • Currently manual, could be automated

    • Data validation

  • New data sources need scrapers, parses, etc.

Overview of how the API gets data

• Written in Golang

  • Josh will run Golang pseudo-workshops

• Scraper

  • scrapes data and dumps it in a raw format

    • separating this from parsing allows adding and changing the data saved by only updating the parser

  • Scrapers can be run individually while all parsers are run together

  • sources

    • coursebook

      • course data and specific section data

      • professor data

    • events

      • SOC website org events

    • organizations

      • SOC website org listings

    • profiles

      • professor profiles

• Parser

  • take raw scraped data and extract meaningful values

  • much faster than the scraper as it requires no web requests and is all local

  • validation for wrong school data

• Uploader

  • separate from parsing to not have to rerun parsing to change the data format in the db

  • Uploader is the newest part of this pipeline

  • Need to integrate new data with old while maintaining linked professors and courses and so on

  • The current solution is to parse all data back to 2017 at once and replace the whole database

  • Room for improvement here

Overview of how the API runs

• server.go starts up and listens for http requests on various routes

• controllers read from MongoDB and adjust it to the requested format using a response format

Questions

• When will we be going through the codebase

  • Josh will put resources in the Discord on learning Go

    • Including: A Tour of Go

    • Go through the resources and just get a feel for the language

    • Look around the repositories

  • Next meeting

    • Go through the code

    • Go over some of the libraries the API uses

• Why Go?

  • Why not?

  • But also Go is great for APIs and small command line tools

  • Takes inspiration from C, a pseudo-low-level language

  • Easy to learn and get started with

• Code quality

  • VSCode automatically enforces a standardized Go code format

  • The build files run various checks on code, make sure your code builds

  • Josh does code reviews

Final notes

• Get your interest form submitted

• Go over the resources in the Discord