Associated Literature:
Llosa, L. (2007). Validating a standards-based
classroom assessment of English
proficiency: A multitrait-multimethod
approach.Language Testing 2007 24 (4)
Llosa, L. (2005). Assessing English Learners' Language Proficiency: A Qualitative Investigation of Teachers' Interpretations of the California ELD Standards. The CATESOL Journal 17 (1)
November 15: Lorena Llosa
Can we trust teacher judgments?
Validating a standards-based classroom assessment of English proficiency
6:15 pm
Cubberley 114
Dr. Lorena Llosa is an Assistant Professor in the Department of Teaching and Learning at New York University. She received her Ph.D. in Applied Linguistics from the University of California, Los Angeles. Her work focuses on second and foreign language teaching and learning, language testing, program evaluation, and research methods. Prior to NYU, she worked as a research analyst for the Los Angeles Unified School District where she directed a large scale evaluation of a computer-based literacy program. She also served as a research analyst at the Center for the Study of Evaluation/CRESST at UCLA where she worked on the development of performance assessments in English and Spanish.
She is currently working on two funded research projects. With Professor Sarah Beck at NYU, she’s working on the development of a diagnostic assessment of high school students’ difficulties with academic writing. This project is funded by The Spencer Foundation and a number of NYU internal grants. The second project, funded by The Hewlett Foundation and led by Dr. George Bunch at UC Santa Cruz, examines current practices surrounding the testing and placement of language-minority students transitioning from high school to California community colleges and identifies innovative approaches that hold promise for improving students' success in pursuing academic pathways.
The work she will present at the LEEP colloquium focuses on validity issues related to the use of standards-based classroom assessments of English proficiency in large urban school districts. This work was funded by a Spencer Dissertation Grant and a UC Linguistic Minority Research Institute Grant and was awarded the AERA Division H Outstanding Dissertation Award in 2006.
The use of standards-based classroom assessments to test English learners' language proficiency is becoming increasingly prevalent. In a large urban school district in California, for example, a classroom assessment is used to make high-stakes decisions about English learners' progress from one level to the next, and is also used as one of the criteria for reclassification as Fluent English Proficient. Yet many researchers have questioned the validity of using classroom assessments based on teacher judgments for making high-stakes decisions about students (Brindley, 1998, 2001; Rea-Dickins & Gardner, 2000).
Using Bachman's (2005) validation framework, this study investigates validity issues related to the use of the ELD Classroom Assessment, a standards-based classroom assessment of English proficiency used in a large urban school district in California. The purpose of this study was to determine the extent to which scores assigned by teachers on the ELD Classroom Assessment can be interpreted as indicators of English proficiency as defined by the California ELD standards. Two different research approaches were employed to investigate the validity of the inferences drawn from the ELD Classroom Assessment: 1) examining the assessment in relation to another measure of the same ability-the California English Language Development Test (CELDT)-using confirmatory factor analysis of multitrait-multimethod data; and 2) examining the processes teachers engaged in while scoring the classroom assessment using verbal protocol analysis.
In the presentation, I will first introduce the validity argument. Then I will present findings of the confirmatory factor analysis of multitrait-multimethod data, which suggest that the ELD Classroom Assessment does measure the aspects of English proficiency it claims to measure. Then, I will present the findings of the verbal protocol analysis, which indicate that factors other than a student's English proficiency are also reflected in the scores assigned by the teachers. I will then provide an explanation for these seemingly contradictory findings and discuss them in terms of their implications for the use of standards-based classroom assessments within a high-stakes accountability system.