Schedule

Materials Resources Assignments
Apr 3
  1. Course introduction and overview
  2. Notebook: Course set-up
  3. Optional background materials
  1. Levesque 2013
  2. Manning 2015
  3. Philosophy of understanding (§2.6 of the "Foundation Models" report)
  4. Artificial Intelligence: Last Week Tonight with John Oliver
Domain adaptation for supervised sentiment
Apr 5
  1. Overview of Assign/bakeoff 1
  2. Contextual word representations
  3. Diffusion objectives for text [Lisa]
  4. Fantastic language models and how to build them [Sidd]
  1. SST: Socher et al. 2013
  2. DynaSent: Potts, Wu et al. 2020
  3. Smith 2020
  4. The Pile: Gao et al. 2020
  5. Transformer: Vaswani et al. 2017
  6. The Annotated Transformer
  7. Relative positional encoding: Shaw et al. 2018
  8. GPT: Radford et al. 2018
  9. BERT: Devlin et al. 2018
  10. RoBERTa: Liu et al. 2019
  11. ELECTRA: Clark et al. 2019
  12. T5: Raffel et al. 2019
  13. BART: Lewis et al. 2020
  14. DistilBERT: Sanh et al. 2019
  15. Diffusion-LM: Li et al. 2022
  1. Assign/bakeoff 1: due Apr 17, 3:00 pm Pacific
  2. Quiz 0: due Apr 17, 3:00 pm Pacific
  3. Quiz 1: due Apr 17, 3:00 pm Pacific
Apr 10
Apr 12
Retrieval augmented in-context learning
Apr 17
  1. Overview of Assign/bakeoff 2
  2. Information retrieval
  3. In-context learning
  4. Prompters before prompts and promptees [Mina]
  1. Khattab et al. 2021
  2. ColBERT: Khattab and Zaharia 2020
  3. DPR: Karpukhin et al. 2020
  4. SPLADE: Formal et al. 2021
  5. RAG: Lewis et al. 2020
  6. GPT-3: Brown et al. 2020
  7. RLHF for LLMs: Ouyang et al. 2022
  8. CoT: Wei et al. 2022
  9. Retrive-then-read: Lazaridou et al. 2022
  10. DSP: Khattab et al. 2022
  1. Assign/bakeoff 2: due Apr 26, 3:00 pm Pacific
  2. Quiz 2: due Apr 26, 3:00 pm Pacific
Apr 19
Apr 24
Advanced behavioral evaluation
Apr 26
  1. Overview of Assign/bakeoff 3
  2. Advanced behavioral evaluation of NLU models
  1. Jia and Liang 2017
  2. Glockner et al. 2018
  3. Liu et al. 2019
  4. Naik et al. 2019
  5. ANLI: Nie et al. 2020
  6. Dynabench: Kiela et al. 2021
  7. COGS: Kim and Linzen 2020
  8. ReCOGS: Wu et al. 2023
  1. Assign/bakeoff 3 due May 8, 3:00 pm Pacific
  2. Quiz 3: due May 8, 3:00 pm Pacific
May 1
May 3
Analysis methods
May 8
  1. Lit review overview
  2. Siyan: What I did for 224u final project
  3. Course review in the form of a Jeopardy! game
  4. Analysis methods for NLU
  1. LIME: Ribeiro et al. 2016
  2. Probing: Tenney et al. 2018
  3. IG: Sundararajan et al. 2017
  4. Causal abstraction: Geiger et al. 2022
  5. IIT: Geiger, Wu, et al. 2022
  6. DAS: Geiger, Wu, et al. 2023
  7. Circuits: Cammarata et al. 2020
  1. Lit review: due May 17, 3:00 pm Pacific
  2. Quiz 4: due May 29, 3:00 pm Pacific
May 10
May 15
NLP methods
May 17
  1. Experiment protocol overview
  2. NLP methods and metrics
  3. Real-world NLP assessments [Kawin]
  1. Resnik and Lin 2010
  2. Smith 2011, Appendix B
  3. Dynascores: Ma et al. 2021
  4. Santhanam et al. 2022
  1. Experimental protocol: due May 29, 3:00 pm Pacific
May 22
May 24
Your projects
May 29
  1. Memorial Day (no class)
May 31
  1. Presenting your research
  1. Jason Eisner's Advice for Research Students
  2. Stuart Shieber on reporting research results
  3. David Goss on math style
  4. Novelist Cormac McCarthy’s tips on how to write a great science paper
  5. Geoff Pullum's Five Golden Rules (well, actually six) for giving academic presentations
  6. Patrick Blackburn: How to give a good talk
  7. Datasheets: Gebru et al 2018
  8. Model Cards: Mitchell et al. 2019
  1. Final paper: due June 10, 11:30 am Pacific (end of our scheduled exam time, which we will not use)
Jun 5
Jun 7