CS 224S / LINGUIST 281 Speech Recognition and Synthesis Winter 2007

COURSE INFORMATION
Instructor	Dan Jurafsky, jurafsky@stanford.edu Office: Margaret Jacks Hall (bld 460) 113 Office Hours: Mon 10:00-11:00am
TA	Yun-Hsuan Sung cs224s-ta@cs.stanford.edu Office: Margaret Jacks Hall (bld 460) 022 Office Hours: Mon 12:40-1:30pm and Wed 11:10-12:00am
Newsgroup	We have setup a class newsgroup, su.class.cs224s. Please post questions there and check to see if your question has been asked before posting it.
Time	Tuesdays and Thursdays, 10:00-11:15am (NOTE SLIGHT CHANGE IN TIME TO START AT 10:00am)
Location	200-217
Textbook	The new edition of Jurafsky and Martin. 2007. Speech and Language Processing. Selected chapters from Taylor, Paul. 2007. Text-to-Speech Synthesis Selected chapters from Huang, Acero, Hon. 2001. Spoken Language Processing. Prentice-Hall. Some papers from the literature.
Description	This course is an introduction to automatic speech recognition, speech understanding and speech synthesis/text-to-speech from the computer science and linguistics (rather than EE) perspective. Focus on understanding of key algorithms including noisy channel model, Hidden Markov Models (HMMs), GMM (Gaussian Mixture Model) acoustic models, A* and Viterbi decoding, N-gram language modeling, unit selection synthesis, dialogue architectures, and the role of linguistic knowledge (esp. phonetics, intonation, pronunciation variation, disfluencies, emotion). Prerequisites: programming experience and familiarity with probability. Recommended: cs221 or cs229.
Required Work	Homeworks: 6 homeworks (Homework Collaboration Policy). Homework is due at 9:59am on the day it is due (i.e. before class starts). LATE HOMEWORK WILL NOT BE ACCEPTED. But I will drop your lowest homework grade. Readings: To be read before the class period in which they will be discussed. THERE IS A LOT OF READING IN THIS COURSE!!! We are covering what are really three entire fields (speech recognition, speech synthesis, dialogue systems) in one 10-week quarter, and not everything can be covered in each lecture, so you need to do all the reading. Final Project: Any project in speech recognition, speech synthesis, speech understanding, dialogue design, speech user interface design, etc etc. Either individual or joint projects are fine. The final project will be presented as a poster at the poster session on Thursday March 15, and is due on Monday March 19 at noon PST by email. Details here. Some publishable sample projects involving collaboration with students or postdocs in my lab here. Determination of final grade: 45%: final project 45%: best 5 homeworks out of 6 10%: class participation

	SCHEDULE
Wk	Date	HW	Lec	Topic and Readings
1	Jan 9		Lec 1 (ppt) Lec 1 (6up PDF)	Course Overview and History, Articulatory Phonetics and ARPAbet transcription J+M New Chapter 7: Phonetics pages 190-195 from Chapter 7 "Phonetics and Phonology" from Taylor, Paul. 2007. Text-to-Speech Synthesis
1	Jan 11		Lec 2 (ppt) Lec 2 (6up PDF)	Phonetics: Acoustic Phonetics J+M New Chapter 7: Phonetics
2	Jan 16	HW 1 due	Lec 3 (ppt) Lec 3 (6up PDF)	TTS: Background (machine learning, classification, NLP) and Text Normalization Note that there is a lot of reading for today. This is more than we will normally have. J+M New Chapter 8 Speech Synthesis, pages 1-10 SKIM J+M New Chapter 5 Word Classes and Part-of-Speech Tagging, pages 1-35 Chapter 4 "Text Segmentation and Organisation" from from Taylor, Paul. 2007. Text-to-Speech Synthesis Chapter 5 "Text Decoding" from Taylor, Paul. 2007. Text-to-Speech Synthesis for those who haven't seen decision trees before read an introduction to decision trees like pages X-Y of Russell and Norvig or this one
2	Jan 18		Lec 4 (ppt) Lec 4 (6up PDF)	Grapheme-to-phoneme and the Festival software (plus Text Normalization spillover) J+M New Chapter 8, pages 10-15 Read sections 1, 2, 3, 4, 5, and 6.1 and 6.1.1 from Alan Black's lecture notes on TTS and Festival. You should be looking at the Festival manual. You don't have to read the whole thing through, but you should skim it so you know where things are in the manual. For those who don't know Scheme, (used as Festival's scripting language): Introduction to Scheme for C Programmers, from Cal Tech.
3	Jan 23	HW 2 due	Lec 5 (ppt) Lec 5 (6up PDF)	TTS: Prosody (Intonation, Boundaries, and Duration) J+M New Chapter 8, pages 15-25 Chapter 6 "Prosody Prediction from Text" from Taylor, Paul. 2007. Text-to-Speech Synthesis The rest of Section 6 of Alan Black's lecture notes
3	Jan 25		Lec 6 (ppt) Lec 6 (6up PDF)	TTS: Waveform Synthesis (Diphone and Unit Selection Synthesis) A lot of reading again today! Pace yourself! J+M New Chapter 8, pages 25-35 Chapter 14 "Synthesis by Concatenation and Signal Processing Modification" from Taylor, Paul. 2007. Text-to-Speech Synthesis Chapter 16 "Unit Selection Synthesis" from Taylor, Paul. 2007. Text-to-Speech Synthesis Section 7 from Alan Black's lecture notes on TTS and Festival.
4	Jan 30	HW 3 due	Lec 7 (ppt) Lec 7 (6up PDF)	ASR: Noisy Channel Model, Bayes, HMMs, Forward, Viterbi, Start of Baum-Welch J+M New Chapter 6: Hidden Markov Models, pages 1-19 J+M New Chapter 9: Automatic Speech Recognition, pages 1-12
4	Feb 1		Lec 8 (ppt) Lec 8 (6up PDF)	ASR: HMMs continued, and application to ASR J+M New Chapter 9: Automatic Speech Recognition pages 25-39 Getting a speech recognizer for your project: where are HTK, Sonic, and Sphinx? HTK Sonic Sphinx
5	Feb 6	1-pgraph project proposal	Lec 9 (ppt) Lec 9 (pdf)	ASR: Feature Extraction Chapter 10 "Signals and Filters" from from Taylor, Paul. 2007. Text-to-Speech Synthesis Chapter 12 "Analysis of Speech Signals" from from Taylor, Paul. 2007. Text-to-Speech Synthesis
5	Feb 8		Lec 10 (ppt) Lec 10 (6up PDF)	ASR: Acoustic Modeling J+M New Chapter 9: Automatic Speech Recognition pages 13-25, 52-58
6	Feb 13	HW 4 due	Lec 11 (ppt) Lec 11 (6up PDF)	N-grams and Language Modeling J+M New Chapter 4: N-grams Optional additional reading: Manning+Schuetze Chapter 6
6	Feb 15		Lec 12 (ppt) Lec 12 (6up PDF)	ASR: Search (Lattices, N-best lists, A*, etc) J+M New Chapter 9: Automatic Speech Recognition 42-52
7	Feb 20	HW 5 due	Lec 13 (ppt) Lec 13 (6up PDF)	Conversational Agents: Simple dialogue systems (frame-based systems, VXML, evaluation) J+M New Chapter 23
7	Feb 22		Lec 14 (ppt) Lec 14 (6up PDF)	Conversational Agents: Grounding, Confirmation, Dialogue Acts J+M New Chapter 23
8	Feb 27	HW 6 due	Lec 15 (ppt) Lec 15 (6up PDF)	Conversational Agents III: Markov Decision Processes (MDPs), etc
8	Mar 1		Lec 16 (ppt) Lec 16 (6up PDF)	ASR: Dealing with Variation (Adaptation, MLLR, Pronunciation modeling)
9	Mar 6		Lec 17 (ppt) Lec 17 (6up PDF)	Disfluencies and Metadata: Boundaries, Fillers, Edit Terms Yang Liu, Elizabeth Shriberg, Andreas Stolcke, Barbara Peskin, Jeremy Ang, Dustin Hillard, Mari Ostendorf, Marcus Tomalin, Phil Woodland, and Mary Harper. 2005. Structural Metatada Research in the EARS Program,. ICASSP 2005. Yang Liu, Elizabeth Shriberg, Andreas Stolcke, Dustin Hillard, Mari Ostendorf, Barbara Peskin, and Mary Harper. 2004. The ICSI-SRI-UW Metadata Extraction System, ICSLP 2004. Snover, Matthew, Bonnie Dorr and Richard Schwartz. 2004. A Lexically-Driven Algorithm for Disfluency Detection. Short Papers Proceedings of HLT-NAACL 2004. Boston: ACL. 157--160.
9	Mar 8		Lec 18 (ppt) Lec 18 (6up PDF)	Emotional speech
10	Mar 13			Guest Lecture: Luciana Ferrer on Speaker Recognition D. A. Reynolds and R. C. Rose. 1995. Robust text-independent speaker identification using gaussian mixture speaker models. IEEE Trans. Speech, Audio Processing, 3(1):72-83, January 1995. Luciana Ferrer, Elizabeth Shriberg, Sachin S. Kajarekar, Andreas Stolcke, Kemal Sönmez, Anand Venkataraman, and Harry Bratt. 2006. The contribution of cepstral and stylistic features to SRI'S 2005 NIST Speaker Recognition evaluation system. To appear in Proceedings of ICASSP 2006.
10	Mar 15			Final Project Presentations