This is a pre-publication draft.  Do not cite or copy. You are not allowed to link to this site in any manner, nor pass this URL to anyone else without permission from me.

Last modified 2 August 1999 // © 1998 Carolyn Gale // carolyn@relax.com

Computing Completion Rate By Observation

    Computing the completion rate of a non-credit workshop is difficult if the stated objectives and criteria are not clearly defined from the outset of the course. This supports the need to use clear, detailed objectives (Bohlin 1993, Eastmond 1995, Knowles 1978), and could also have an effect on participant completion. Several participants stated that directions in latter parts of the workshop were not clear, while others stated that instructions for publishing materials to the workshop Web server did not always work.

    The completion rate of the workshop dropped dramatically after the first few assignments. Students tended to drop out in significant numbers around Assignment 3 (creating your course webpage using FrontPage). A possible explanation for this is that the first three assignments took so much time to complete that students ran out of time to work on remaining sections. Several comments support this:

1. Other comments: "Allaire Forums (Assignment 1.1) is a "trip" to learn. Confusing at best. I posted things in the wrong places, etc"
2. Mechanics difficulty: "I never got to send materials to the web server but we encountered difficulties with the FTP component (Assignment 2)"

    Additionally, Assignments 2 and 3 required publishing materials to the workshop web server. The use of FTP to transfer materials to the server caused numerous technology problems, as discussed below.

    After reviewing the qualitative data from the survey, three factors appeared to have the most predominant effect on completion: time frame, feedback on work completed, and problems using and accessing technology.

Time Frame

    "Too much content, too little time" seemed to be the prevalent attitude of those who responded to the survey. In Table 3, the vast majority of participants (70%) felt the time frame suggested by the instructors to be adequate at best. With respect to answers from the open-ended questions, time issues ranked highest in number of comments in the "Worst Feature", "What to Improve", and "Reason for Not Completing" sections. Participants felt that either the amount of content should be reduced to allow completion in the given amount of time, or there should be more time to finish the content:

1. What to improve: "It would be nice if the workshop were thinned out a bit and spread over a few more weeks."
2. Reason for not completing: "Too much content in too little time...felt it was difficult to "catch up" once I fell behind the schedule..."

    The benefits of "anytime, anywhere" asynchronous learning tended to be a double-edged sword. For example, if there was a family or work emergency, participants fell behind, and found it difficult to catch up:

1. Reason for not completing: "I did not have any release time to complete this course and did marathon weekends & evenings to complete what I did."
2. Other comments: "I would still like to finish the course but at my own pace as my work schedule permits."
3. What to improve: "I would make it truly asynchronous, not synchronous."

    This raises the issue of allowing students as much time as needed to finish the workshop, which would make it truly asynchronous. However, with no formal deadlines, participation would likely be lower that it currently is. Even with informal deadlines now being used, the completion rate still trails off after the third assignment.

Several comments in the issue of time frame raise the question of calculating the completion rate:

"Too much work, too little time. I continued to have difficulties publishing and just gave up and didn't even try the conferencing materials."

Feedback and Community of Practice

    There is a noted difference between help and feedback – facilitators were assigned to help with any technology problems students had, and students indicated this a useful feature. However, many students also liked having others comment on their work in the conferencing system, and even requested more feedback from the instructors: (comments from "What to improve about the workshop:")

1. "Ensure that each student gets an "instructor comment" (using private email) about each assignment. That provides feedback on all assignments and encourages completion of all assignments because students know "an instructor" will be looking for the assignment."
2. "Accountability. Although this puts a lot extra on the professor or TA, having an email message sent which says something like, "Haven't heard from you in a while. Are there questions that we can help you with?" In my case, I was simply busy; but I wonder how many students put off getting started because of the intimidation factor. In a traditional classroom, the student is sucked into the activity by peer pressure and physical momentum. At a distance, this dynamic is lost."
3. "I was delighted when someone left a message commenting about the assignment I handed in (which impresses that we are never too old to enjoy a complement or to accept constructive criticism for our work – I guess it relates to recognition)."

    This reinforces what others have stated about instructor intervention and fostering a sense of community (Eastmond 1995a, Harasim 1995, Hiltz 1997, Jiang 1998, Schrum in press, Wegerif 1998). In this case, technical support was delivered in a timely manner; this was lauded as one of the best features of the workshop. However, students expected more personal attention from the instructors, and even from peers. This could be related to the number of participants. A significant number of comments in the "What to improve about this workshop" section were made about the class size being too large:

1. "Decrease the number of participants so that we actually can have time to view and critique each other's work."
2. "Not have so many participants. I couldn't read everyone's material, so I eventually gave up and read none of it."
3. "I felt that the workshop was just too big. Even though we were divided into interest groups, I think that something is lost when you only talk to people that are like you. (In the engineering forum, there was very little conversation between the participants). It took too much time to try to jump into another group and come up to speed with the conversation."

It is useful to note that most survey participants (68%) did not feel part of a supportive community of practice. These comments may help explain why.

Technology Problems

    Technology problems were the most cited reason in the content and mechanics difficulty sections of the survey, and second most cited reason for not completing the workshop. Predominant problems were the use of FTP to publish web pages to the workshop server, the slowness of the system when working at home or during busy periods, and passwords that were initially incorrect. This reinforces Hiltz's (1997) observation that while technologies improve over the years, the difficulties in using them still remain.

Other Factors of Interest

    Other survey responses that related to the literature were: 1. Strong support for case study, "real world" examples, 2. Unclear/incorrect directions, 3. Incorrect matching of course prerequisites to learner's abilities (i.e. students didn't know about preparing course materials to put online until after they registered – those that didn't had difficulties with completion), 4. Conflicts with other schedules, and 5. Offering credit or certification for the course.

Recommendations for Course Designers

Based on the survey responses and supporting literature, recommendations for improving the workshop are as follows.

Short Term Improvements

1. Plan for more feedback. This could be accomplished by reducing the class size or increasing the number of instructors who have knowledge of the subject. If reducing the class size is not an option, perhaps split the class into smaller segments that run concurrently.
2. Move optional sections to another workshop, but keep roughly the same time frame (6-7 weeks). Eastmond (1995) recommends allowing more time in the beginning of the course to iron out technology problems. Since the population is mostly faculty in academic institutions, they need more time to juggle the workshop with job duties. For this population, it may have worked better to have shorter, more intense summer sessions that last for only 2-4 weeks, with longer sessions in the fall and spring.
3. Have clearly stated prerequisites for the course at the beginning. For example, participants did not know that they should have course materials ready to place online until after they registered. For some participants, not knowing of the need to bring online materials hurt their chances of completion:

Reason for not completing: "I did not have a course per se to model, therefore I had to design course materials from scratch. Between learning the tools, and designing a course, I didn't have sufficient time to complete your on-line course."

4. List and number the completion criteria for each assignment at the top of the assignment. Learners can then note their completion of each part, or adherence to scoring criteria for a complex problem.
5. If optional sections are moved to another workshop, rewrite the directions for the workshop with the required assignments. The target audience for this "beginner" workshop is not as familiar with technology, and need more "hand-holding" to get them through.
6. Shorten the HTML assignment in Part 2 to something simpler. The rationale for this is that many participants had difficulties or were resistant to learning HTML. Also, graphs of completion rate showed that the largest dropout rate occurred during this time. A possible solution is to have students post a HTML message about themselves in Forums that include the previous criteria. Take this extra time to allow for creating a course in FrontPage

Long Term Improvements

1. Offer certification for the workshop. Although 55% of participants would not be more likely to complete the workshop if credit was offered, an offer of credit or certification could attract participants who would want to take the course for credit.
1. What to improve: "Accountability - some sort of credit or certification would add greatly to one's incentives for participating and completing this workshop."
2. Improve quality and number of case studies being offered in Assignment 1.2 (reviewing and rating other online courses). This was rated highly in two of the survey questions: "Best Feature", and "What to Improve".
3. Have one username/password for materials and web conference access. Many participants had difficulties remembering two sets of usernames and passwords.
4. Integrate learning designs and content development into the workshop. This was the most requested feature for new workshop topics. Students are equally interested in content development and learning theory.

Reason for not completing: "The workshop really did not give me what I as looking for in how to develop on-line courses. I was looking for how to apply instructional design and learning theory principles to on-line courses."

Initial reasons for taking workshop: "I wanted to learn about instructional design principles and how to effectively incorporate them into a online web workshop"

Other topics for future workshops: "Pedagogy, discussion of the end user environment. Too many of these workshops focus on the technical aspects of creating these courses without really addressing "Creative ways to use on-line environments." Further, what about some basic cognitive psychology in the sense of knowing which types of materials and interactions to use with different types and groups of learners."

Conclusion

    This study may strengthen our understanding of ways to support online learners. While these improvements are specific to the case study involved, I speculate that they may be valuable to design of other online courses. Even if a person taking an online course has a supervisor that will decrease their work load, a computer with fast Internet access, and works well in an independent study setting, there is still no guarantee of course completion. Results from this study support nearly all of the observations and recommendations from the literature, and future studies can further explore selected factors.

Return to "Thesis Cover Page"