Over the last 10 years, under the leadership of Professor Elizabeth Bernhardt, the Stanford Language Center has worked to implement a systemic and systematic assessment program for all languages. This has centered on the Stanford language requirement but has made use of the national standards laid out in Standards for Foreign Language Learning in the 21st Century (1999) by the American Council on the Teaching of Foreign Languages (ACTFL). While Stanford’s online learning management system (LMS) is a key component of this effort, it is important to note that the program is built around proficiency standards, which drive the pedagogy, and not around the LMS. The LMS, previously CourseWork, and now Sakai, is used to facilitate the movement of some formative and summative assessment out of the classroom in order to provide more time for teachers to actually interact with their students.
Similarly, it is crucial to understand that there are clear goals for this program, the foremost being to improve student proficiency. If we cannot show progress, then there really is very little reason for doing any of it. This improvement is the goal of the program, and indeed the goal of the university. Consequently, it is important for the Language Center to be able to show that we are fulfilling Stanford’s language requirement and we are able to use the successes we have as a way to appeal for the resources needed to achieve this goal. In addition, we enhance our credibility both inside and outside the university by showing that students are improving. Finally, when clear proficiency standards are used, we are able to ensure a certain level of programmatic consistency within and across languages.
Among the positive results of this effort to build curriculum around proficiency standards are more highly trained staff, a professional conversation about the role and use of these standards, and an increased interest among programs in the proficiency notation. Most importantly, however, students have shown a 20% increase in proficiency in the first year, and a 24% increase in the second year, and have given a very positive reaction to the curriculum.
The program starts with placement testing, which consists of an online, text-based assessment during the summer and then an on-campus oral assessment during freshman orientation, both done outside of the learning management system. At the end of the academic year, first and second year students take a Simulated Oral Proficiency Interview (SOPI) online, which is administered through the LMS using special software. As the name suggests, this assessment simulates a live interview with a series of questions that require oral responses. In order to do this, the software has to be designed so that users do not have control over the media and timing of the test: items play and recording starts and stops automatically, with no opportunity to pause or go back. Banks of these items are created by the staff at the Language Center, based on ACTFL oral proficiency interview guidelines, which can then be used for placement tests, year-end exit testing, overseas studies testing, occasional or external testing and proficiency notation guidance. In the 2007-2008 academic year, nearly 1000 students participated in placement testing across all languages, while roughly 650 students took the online first and second year exit exam in nine languages.
Multiple prompts are developed so that multiple unique tests can be created. A “task force”, including administrators, coordinators and instructors, meets several times to create the final drafts. Artwork is contracted and instructions and a range of native-language prompts are recorded. These are all then transformed into paper test booklets or loaded into the SOPI software. Paper placement tests are delivered in classrooms or auditoria around campus, but the SOPI is administered in the Digital Language Lab. Results are accessed through the LMS, archived to CD-ROM, and assessed over the summer by internal OPI testers. The results go directly to the office of the President, to show that the Language Center is indeed fulfilling the university’s language requirement. This development effort is ongoing, in response to new languages and new technologies, like Sakai.
Curriculum is also based on these proficiency guidelines, and in order to evaluate students, coordinators have created computer-assisted online diagnostic assignments (OnDAs) of two types: oral diagnostic assessments (ODAs) and written diagnostic assessments (WDAs). ODAs are delivered four times per term for first and second year students and, while WDAs are still in the pilot stage, they are run twice each term in several languages. Prompts in these OnDAs are tailored according to the proficiency level that is being assessed. The oral proficiency assessments are done in the assignments tool by embedding an image or video and the audio recording item type and the written diagnostic assessments are done using the short answer / essay item type in a timed assignment. These items are all created by the coordinators for each language and are posted as assignments to separate “resource” sites. Instructors are members of this site and they copy the necessary assignments into their own sites. The OnDAs are evaluated on a credit/no credit basis and students are given individual feedback, although each program determines how this is implemented: some programs give feedback in the LMS, but some programs give it verbally in individual meetings. The assessment criteria are based on ACTFL guidelines and students are rated on how the task is performed or how severely communication is impeded. The Language Center has found that because OnDAs can be taken individually, not only can class time can be maximized and optimized for useful practice, pressure to perform is also generally reduced. However, it is important to note that both students and instructors need to be familiar with the technology, and comfortable using it.
After the first year, students begin to work much more on oral and written presentational language. While the pedagogy involved in writing is easily facilitated by tools in Sakai such as file upload, presentations have always been much more difficult to implement in pedagogically sound ways. The ideal method has always been to provide feedback in addition to making the recordings of presentations available to both the students and the instructors. In the past, this was often done with videotape, but digitizing video has streamlined the process, although the size of the files produced often makes it difficult to send them by email. Posting these files is possible, but often students are not comfortable with individuals other than instructors having access to their video. Sakai’s drop box has provided the solution to this dilemma, and a WebDAV interface allows files of any size to be uploaded. The classrooms in the Digital Language Lab have been outfitted with video capture carts with simple mini-DV cameras and shotgun microphones connected to MacBookPros which capture the video in Quicktime Pro. The encoding settings are adjusted so that the file is as small as is reasonably possible, and a link to the WebDAV for each site working in that room is placed on the desktop of the machine. Instructors are given a brief orientation but most are able to handle the process themselves. They are encouraged to use a mini-DV tape in addition to the capture system, just in case there are technical difficulties. This system is also used for graduate level ESL classes that focus on presentation and teaching assistant skills.
This program has worked well for several years now, and its success is a credit to the efforts of both the Language Center and Academic Computing. We look forward to the new challenges and possibilities that changing technologies will bring us and have high hopes for the benefits we will be able to offer to our students.
January 20, 2009
Based on ACTFL2007 presentation (.ppt) and the Stanford Language Center Annual Report