Speech Research in the Jurafsky Lab

The lab studies a number of areas in speech recognition, understanding, synthesis, and dialogue. Our high-level focus is on the use of linguistic knowledge (phonetic, phonological, prosodic, syntactic, semantic, pragmatic) in machine speech processing. We are also interested in human speech processing.

Prosody and Disfluencies in Speech Synthesis and Recognition

A major focus this year is our work on the automatic prediction of prosodic phenomena like pitch accents from text, and detection of pitch accents from speech, as well as the automatic detection of disfluencies. Collaborators here include Jason Brenier on prosody and Costas Boulis, Yuan Zhao, Surabhi Gupta, and Yun-Hsuan Sung (together with recent postdoc Jiahong Yuan). Distant collaborators include Simon King, Rob Clark, Volker Strom, and Mark Steedman at the University of Edinburgh. We are also starting a new collaboration with David Beaver, Sasha Calhoun, and Bob Ladd. Recent papers include:

Yuan, Jiahong, Jason M. Brenier, and Dan Jurafsky. 2005. Pitch Accent Prediction: Effects of Genre and Speaker . In Proceedings of EUROSPEECH-05.

Jason M. Brenier, Daniel Cer and Daniel Jurafsky. 2005. The Detection of Emphatic Words Using Acoustic and Lexical Features. In Proceedings of EUROSPEECH-05.

Yuan, Zhao and Dan Jurafsky. 2005. A preliminary study of Mandarin filled pauses. Proceedings of DiSS'05, Disfluency in Spontaneous Speech Workshop.

Speech Recognition of Dialect-Accented Speech

We are interested in understanding how to make speech recognition more robust to foreign accents. Recent projects include the JHU Summer Workshop 2004 project on robust speech recognition of Mandarin Chinese spoken by speakers with southern (Shanghainese) accents. Related to this project, Stanford linguistics student Rebecca Starr is working on sociolinguistic and phonological causes of variation in southern Mandarin. We have also worked on Spanish-accented English:

Yanli Zheng, Richard Sproat, Liang Gu, Izhak Shafran, Haolang Zhou, Yi Su, Dan Jurafsky, Rebecca Starr and Su-Youn Yoon. 2005. Accent Detection and Speech Recognition for Shanghai-Accented Mandarin. In Proceedings of EUROSPEECH-05.

Ikeno, Ayako, Bryan Pellom, Dan Cer, Ashley Thornton, Jason M. Brenier, Dan Jurafsky, Wayne Ward, and William Byrne. 2003. Issues in Recognition of Spanish-Accented Spontaneous English. In Proceedings of IEEE/ISCA Workshop on Spontaneous Speech Processing and Recognition, Tokyo, Japan.

Pronunciation Modeling

A key problem in automatic speech recognition, especially in recognizing human-to-human conversational speech, is predicting how words are likely to be pronounced in context. Recent pronunciation modeling efforts includes PMLA-2002 (the Pronunciation Modeling/Lexicon Adaptation Workshop), co-organized with Eric Fosler-Lussier and Bill Byrne. Early work focused on phonological rules. Which didn't work very well. Recent research includes our efforts to understand why improvements in ASR due to pronunciation modeling have proven so elusive:
Jurafsky, Dan, Wayne Ward, Zhang Jianping, Keith Herold, Yu Xiuyang, and Zhang Sen. 2001. What Kind of Pronunciation Variation is Hard for Triphones to Model? Proceedings of ICASSP-01, I.577-580, Salt Lake City, Utah.

Together with phonetician Alan Bell, we have also been working on studying various factors which affect pronunciation reduction; shortening or weakening of words in production. Among the factors we have investigated are word probability, neighboring disfluency, and word sense. A sample paper:

( Alan Bell, Daniel Jurafsky, Eric Fosler-Lussier, Cynthia Girand, Michelle Gregory, and Daniel Gildea. 2003. Effects of disfluencies, predictability, and utterance position on word form variation in English conversation. Journal of the Acoustical Society of America 113 (2), 1001-1024.

Language Modeling

One of the most important problems in ASR is predicting the next word the user is likely to say. We have worked on Latent Semantic Analysis for language modeling, and also did early work on the use of SCFGs.

Dialogue Modeling:

We also work on probabilistic models of dialog, starting with our team at the 1997 Johns Hopkins Workshop on Innovative Techniques in LVCSR. We are especially interested in the automatic detection of dialogue structure, such as automatic labeling of speech acts or dialogue acts: