Mia K. Stern
Computer Science Department
University of Massachusetts
Amherst, MA 01003
Phone: (413)545-0582
Fax: (413)545-1249
email: stern@cs.umass.edu
URL: http://www.cs.umass.edu/~stern
Abstract: In this paper, we discuss the difficulties that we have encountered in developing an intelligent tutoring system for the World Wide Web (WWW). The tutoring system in question is MANIC (Multimedia Asynchronous Networked Individualized Courseware), which is designed to enable instructors to take existing multimedia-based courses and convert them to on-line courses. In a MANIC course, students can view slides and pictures and listen to audio and view video from the lectures. Additionally, we have added an intelligent tutor to the system, which has not been a trivial task. In this paper, we discuss the difficulties in this process, which include (1) adapting the existing course material to be on-line and (2) individualizing the course for each student. We will also discuss some of our proposed solutions to these problems.
Traditionally, intelligent tutoring systems (ITSs) have been built and deployed using static mediums, such as CD-ROMs. There are many benefits to this kind of delivery. For example, a lot of multimedia can be easily provided to the student without worrying about network delays or web sites being unavailable. But there are also disadvantages to this kind of medium. Any changes made to the educational material involves repressing and redistributing the CDs. This is both costly and time consuming. Thus a better solution must be found.
One possibility for a better solution is the World Wide Web (WWW). With the advent of the WWW, education has a new tool to utilize. It offers the ability to easily distribute educational material around the globe. Also, with a web implementation of educational material, changing the content is trivial, and students need not be made aware of such changes.
However, this technology presents new challenges to instructional designers. It is not trivial to transfer either an existing intelligent tutoring system or a lecture based course to be on-line. The first hurdle to overcome is the statelessness of the HTTP protocol (Berners Lee, et al. 1996), which makes it difficult for the tutor to identify each student. Many solutions for this problem exist, including cookies, hidden fields, and structured URLs. The solution we have chosen is cookies, as these were easy to implement and sufficiently powerful. A second hurdle is that web based tutors cannot "control'' the way students see material. Many ITSs do not want students to go back to earlier problems, but with the web, students can do this easily, since web browsers have navigational buttons and history lists.
In this paper, we discuss the difficulties of designing a web-based tutor, MANIC (Multimedia Asynchronous Networked Individualized Courseware). This system did not originate as a stand-alone ITS, so the problems associated with that kind of transfer are not discussed. However, we do discuss the problems of converting an already existing video-taped course to be an on-line intelligent tutoring system. We also discuss the difficulties of student modeling in the MANIC system.
MANIC is a tool for putting existing multimedia-based courses on line. Within a MANIC course, students can see the HTML slides and hear the associated audio from lecture based courses. We are currently in the process of adding video to MANIC courses. We have utilized common web browsers (such as Netscape Navigator) and free plug-ins (such as RealAudio and RealPlayer), which enables synchronized playback of audio/video with the viewing of the slides. Students can start the audio/video at the beginning of the lecture, and let it play until the lecture is over. They can also take a more active role, starting and stopping the audio/video as desired, and jumping to other parts of the course. This can be accomplished using the navigation buttons we have provided or the index, searchable by keyword and organized by topic. For more information on how students interact with MANIC, see (Stern, et al. 1997).
In addition to the multimedia, we have also developed an intelligent tutor as part of the MANIC software. This tutor is used to help guide students through the course material, to generate dynamic quizzes which are appropriate for the student's knowledge, and to "prefetch" parts of the course to the client's site before they are explicitly requested. The first goal is accomplished by using adaptive hypertext techniques including adaptive navigation support (Kaplan, et al. 1993) and adaptive presentation (Boyle and Encarnacion 1994). The tutor suggests to the student which topics he should view, as well as determines the content of the page displayed. The second goal is achieved by dynamically constructing quizzes for the student to take. The tutor selects, from a question database constructed by the instructor, questions which cover the correct topics and are at the right level of difficulty. The third goal is accomplished by predicting the next pages the student will see, and downloading these pages to the client before they are requested. If the predictions are correct, the delays seen by the user will be significantly reduced. See (Stern, Woolf, and Kurose 1997) for more details on the implementation of the on-line tutor in MANIC.
Before deciding whether a MANIC course should have an intelligent tutor, we must consider the target audience for the course. Recently, during the spring semester of 1997, MANIC was used concurrently with a lecture-based course on computer networking. In this setting, students used MANIC to listen to a lecture that they missed or review parts of previous lectures to prepare for tests or homework assignments. Additionally, MANIC was used as part of a 1-credit course during the fall of 1996. Students were required to listen to one hour of lecture per week and to then attend a discussion session. In both of these settings, MANIC was used to supplement a live course. However, this is not the only possible use of MANIC. We are also hoping to use a MANIC course as a standalone course, without an instructor giving lectures. Table 1 indicates which features of MANIC can be used in both situations.
| With a course | Standalone | |
| topic suggestion | maybe | yes |
| alternate versions of material | maybe | yes |
| dynamic quizzing | maybe | yes |
Table 1: Intelligent features in two situations
It is not clear, given the current use of MANIC, that adding intelligent features is cost effective. If the only intent is as a repository for course materials, then an intelligent tutor would not be extremely beneficial. In this case, students will primarily just want to see the slides and hear the audio from class, since the material must be viewed linearly as the course progresses.
In the two courses that have used MANIC, this is indeed how most students used the software. Only when reviewing material for homeworks and tests did students take a non-linear approach to the material. Therefore, using a tutor to, for example, suggest topics for a student to see, is not beneficial. Our assumption is that students will choose to view the topics linearly, as they are placed on-line. So when initially learning the material, topic suggestions are irrelevant.
However, even when MANIC is used in conjunction with a currently occurring course, some intelligent features could be extremely useful. The difficulty in this case is that the tutor cannot, and should not, attempt to guess the student's ability and knowledge. The student will obviously gain knowledge when not working with the tutor. But in order for the tutor to be able to help guide the student, it must derive a student model. This can be accomplished by asking the student what lectures he has seen either live or on video tape. By doing this, the tutor will be able to have an accurate picture of the student's state of knowledge.
Given this information, the tutor can suggest topics on which the student should focus. For example, if the student has missed a lecture, than the tutor can suggest the student review that material. Furthermore, the MANIC tutor could be used to help the student review material he has already seen. This could be extremely useful when studying for exams or preparing a homework assignment. Since the tutor provides more background information than can be gained in class, the student can use that information for review.
The tutor can also make decisions about which versions of the slides the student should see. These decisions are based on how much knowledge the student has, which is heavily influenced by how many topics he has viewed. If the tutor knows how much material the student has seen, regardless of the setting, then it can construct slides appropriate for him.
Furthermore, MANIC could also be used for quizzing the student, although not to a great extent. The quizzing mechanism in MANIC is designed to review material the student has not seen for a given period of time, as well as to test new material. If MANIC is being used with a course, then quizzes can be given on material on which the student has not been tested, but he claims he knows. And the tutor could choose to review material on which the student has been quizzed, which may include topics viewed either on-line or off-line.
If MANIC is used as the sole method of providing instruction, there is no reason to adhere to the linear nature of the existing course. Also, unlike in the previous case, it is likely that students will need to view all the material within the MANIC structure, since there is no other source from which to gain the knowledge.
Since students will not have the assistance of a full-time instructor, they must rely more on an on-line helper, in the form of the intelligent tutor. Because the student will be required to cover all of the material for the course, the tutor will be able to gain a more accurate assessment of his knowledge. Thus the tutor can provide more individualized instruction for the student, which includes suggesting topics the student should view and more accurately deciding on slide content.
Furthermore, since the interactions with the tutor will be the students' most significant contact for help, they will most likely take advantage of all of the features, especially the quizzes. When using MANIC to take a course, the student proceeds at his own pace. Thus having frequent quizzes to check his knowledge would be beneficial. In fact, many students who have used MANIC have said they would take advantage of on-line quizzes.
It may still turn out that students just sit down, start the audio/video, and let it play until the end of the "lecture." And this remains an option for students. However, students may also enjoy being able to have more input into their learning. In the remainder of this paper, we assume that we are converting the lectures to be used as a standalone course, thus making the intelligence a required feature of the system.
When converting already existing course material to be part of a tutoring system, the first thing that must be done is representing the domain. In this section, we discuss the difficulties we have had in translating the domain for on-line use.
Because we are converting existing lecture courses to be on-line, the domain begins completely linear in nature. While this is sufficient for simulating the in-class experience, it does not take full advantage of the power of the web. Web courses should allow students multiple ways of progressing through the material, since they may do that anyway (as we will discuss in section 5.1).
The first task in making a course non-linear is identifying the parts of the material that do not have to follow a linear ordering. This is not easy to do when converting a lecture-based course. However, with some courses, it may be possible that a linear traversal through the material is not necessary. Therefore, it is the responsibility of the instructor to identify places where branch points exist. It may then be necessary to edit some of the audio which refers to previous lectures by saying "in lecture 5, we saw that...." For a non-linear path through the course, this kind of statement makes no sense.
Furthermore, it is often necessary to add course material, i.e., supplemental topics to the existing material. This can be done for two reasons: (1) adding material at branch points so the course is non-linear and (2) adding remedial topics on background information. In either case, a domain expert will be required to supplement the already existing material. This supplemental material is necessary since instructors have limited time in lectures to cover all the topics they would like to cover. However, with an on-line version of the course, there is no time limit, and the additional topics should be added.
Also, because the material was originally presented in a lecture, there are occasionally reviews of previous material or indications about the due dates for assignments. These parts of the audio should not be part of the on-line course, since there are no longer "lectures" and due dates will not be the same. Therefore some of the audio must be edited, so that it is not completely lecture-like. The desire for the audio is that it describes the content of the slides without giving any indication that it originally came from a lecture course.
There are two main problems we must confront about the content of the course material. First, the content of the existing video-taped courses is directed towards an in-class audience which can ask questions of the instructor. This facility is not usually available on-line. Developing a completely intelligent tutor that can answer any question is beyond our current capabilities. However, we have been developing a list of frequently asked questions (FAQ), originating from questions asked by students using MANIC during the Spring 1997 semester, to include on-line. In this way, students who have asked questions in the past can help other students by having the answers published as part of the course material.
Another possibility for identifying confusing parts of the course is to have the on-line tutor record places where many students have difficulties, and then in the future, provide better instruction at those places. This better instruction, for example, could be to provide more detailed information on the topic, which may require the instructor to add some supplemental information. However, this identification requires the tutor to be able to assess data for all students as a population, which is not the current focus of our research.
The second problem concerning the course content is, in lecture-based classes, there are many students, each with a different level of ability and knowledge. It is not possible for the instructor to individualize the teaching for each student in the class. Therefore, the instructor may teach too slowly for some students and too fast for others. This is illustrated explicitly in the course with which we are working, since both graduate and undergraduate students were enrolled. The instructor of this course (James Kurose) has noted that there were times when graduate students are bored because something must be explained to undergraduates. In fact, some graduate students stopped attending classes and used MANIC to listen to the lectures. By doing this, they could skip the parts of the course they already knew.
This problem can be handled if in the on-line version, students could also skip the parts of the course that they already know. With our navigational buttons, this will only work for full slides. However, it may be the case that within a slide, the professor had a side bar to answer a question, and the student would like to skip this. Our interface does not have a "fast forward" capability to move forward in the audio. This can be done with the RealAudio control panel, but we encourage students not to use that (however, some do anyway, as we discuss in section 5.1). Therefore, there must be a better mechanism to bypass material already learned.
One way to do this is to provide multiple ways of teaching essentially the same material. Some ways would provide more details on background information, while others would assume that knowledge is known. For example, when discussing the TCP checksum, 1's complement is discussed. One version of the material should describe, in detail, what 1's complement is. Another version should just use the term without defining it. However, in the second case, there would be a hypertext link to the definition, in case the student did want to look at it after all.
One simple way to do provide alternate ways of presenting material is to have multiple versions of slides, for example, three: easy, medium, and hard. The easy version would contain more background information and more detail, while the hard version would assume this information is known by the student.
However, this approach has some significant drawbacks. First, a great deal of work is required to create these different versions of the slides. Either the instructor or another domain expert must determine the material that should be included on each version of the slide. So instead of simply constructing one slide, three must be made, each with slightly different information. Second, there is not much flexibility in the teaching, since all of the material is hard coded; there are three ways to individualize the instruction. This is too restrictive for an intelligent tutoring system.
An alternative approach to developing multiple versions of the course material is to have only one version of the slide hard coded, the difficult version. However, at certain points in the slide content, supplemental material could be added if necessary. Using the 1's complement example again, for a novice student, the slide would be constructed with all the relevant information on 1's complement added. However, for the advanced student, the slide would be constructed without this information. This would reduce the work required to put the slides on line, since only one version of each slide would need to be written. But the places for supplemental material would have to be identified, and the additional details would have to be written.
This alternative would also offer more flexibility in teaching. We would not be restricted to a constant number of versions of the material. Rather, we could add information only when necessary, further specializing the course for each student. However, it is still not as flexible as we would like.
One other approach to developing alternate ways of teaching is to abandon the slide format all together. Rather, there would be a database of content objects, which are every piece of text, image, animation, and video that could be shown to a student. The tutor, using the student model, would decide which objects to display to the student at a given time, determining what constitutes a "slide". The advantage of this method is the flexibility in what is actually displayed to the user. All of the user's interactions would be customized based on the student model. Thus two students would almost never see the same "slide". However, the disadvantage is the increased complexity in representing the domain knowledge and in the reasoning required to create material to show to the user.
To represent the domain with this option, we would need to create a "content object net" for each topic in the domain. This object net would relate the content objects together. Essentially this net would indicate an order of presentation of objects, since there must be some kind of structured presentation. Furthermore, each object would need a "degree of difficulty" that the student model could use when choosing objects to show to the user.
Regardless of the solution that is chosen, the question remaining is what course content should the student see? This decision must be made by the intelligent tutor (especially for the second and third solutions). In the next section, we discuss the difficulties and problems in designing a tutor to make these decisions.
Student modeling is a difficult subject, in standalone tutors as well as web-based tutors. However, in order to have an intelligent tutoring system, it must be able to maintain a student model. Therefore, a tutoring system must record student actions and make decisions based on those actions. In this section, we discuss how students "break" modeling efforts as well as what can be modeled in the MANIC system.
A difficulty of web-based student modeling is students do not have to use the software as intended. In MANIC, if a student wants to jump to another part of the material, we expect and want him to use the provided index. However, some students do not do this. Rather, they simply change the slide number in the URL. By doing so, the slide displayed may not be correct, since the URL also contains information about which buttons should be active (e.g. whether the "next" button is enabled). Only if the index or navigational buttons are used can the slide be accurately displayed. One solution to this problem is to simply not display the slide number in the URL, but rather track it at the server side. Another option is to abandon the idea of slide numbers completely, as we have already discussed (section 4.2).
Furthermore, even though we have provided navigational buttons, students do not have to use them. Web browsers have their own "back" buttons and history lists that students can use to traverse the material. If students do not use our buttons, the audio playback does not stop (it does when the MANIC buttons are used). Since the audio is still playing, the slide will change when a synchronization point is reached. However, the student may not want the slide to change. Thus using the web browser's navigation buttons can lead to unexpected behavior from the user's perspective.
Another problem faced in the MANIC system is the use of audio. We have provided a "stop" button to terminate the play back of the audio. When students press this button, the tutor knows how long the audio has been playing, which it adds to the student model. However, many students use the RealAudio controls to stop the audio. If this method is used, there is no way to inform the tutor when the audio stops.
We are currently working on a solution to this problem. This solution involves embedding the RealAudio controls directly in the HTML pages and using a Java applet to record the student's actions. By embedding the controls, students will not have the option of using either our given controls or the ones from RealAudio; there will be only one set of controls. Furthermore, a Java applet will allow us to record all student actions, as we do with the current implementation.
In MANIC, we model all actions the student takes, even if they use the history list or the browser's navigation buttons. We are able to do this since all of our web pages are common gateway interface (CGI) scripts, which must be reloaded each time. Therefore, the HTTP server is contacted whenever the student loads a page.
Thus the MANIC tutor knows which slides the student has seen, the audio he has heard (but only approximately, due to the problem described in section 5.1), the quizzes he has taken, and his performance on those quizzes. The question that must be faced now is what to do with this information.
The quizzes in MANIC are optional, since some people may use a MANIC course without being officially registered for the course. The intelligent features should still be available in this case, but clearly quizzes should not be required. However, quizzing provides the tutor with a lot of information about the student's knowledge, so it would be beneficial, from a user modeling standpoint, for students to take quizzes. Quizzes generated by the tutor allow it to have a degree of expectation for student's answers. The questions are directed to test knowledge about which the tutor is lacking sufficient information.
Since quizzes in MANIC are optional, we cannot base our student model on being able to collect these data. The main interactions with students are through their viewing slides and listening to audio. Thus, we must devise a student model that uses this information as its basis for judgment.
This is very tricky, though. It is not always possible to accurately judge a student's knowledge based on the slides he has seen and the audio he has heard. If a student views all the information on a topic, does he sufficiently know the material? What if he starts on a particular topic and skips all its prerequisites? Should the tutor suggest he go back to that initial material? Also, if a student does not listen to the associated audio, does he know the material as well as someone who has (we are assuming "no")?
We have added features to the MANIC student model to try to account for these difficulties. The first time the student uses the system, he is given a pretest which helps the tutor determine his prior knowledge. Then, if the student starts on a topic without seeing its prerequisites, but on the pretest he demonstrated he knows that information, the tutor would not suggest that he go back to cover that material.
Additionally, we have added hypertext links to the slides, some of which point to related material, others of which point to background knowledge (either prerequisite knowledge for the course or items that were presented earlier in the course). If students follow these review hypertext links, then they probably do not know the material sufficiently.
The tutor must also make judgments about students viewing the same material multiple times. If a student sees the same slide two or three times, should he see a harder version because he has already been exposed to the material, or an easier version? In MANIC, we are partly basing this decision on why the student elected to see certain material again. If it is because the tutor has suggested it as part of remediation or the student is following a hypertext link, then an easier version is presented. However, if the student is just reviewing material on his own (e.g. for an upcoming test), then giving a harder version seems appropriate.
Another piece of datum the tutor has is time spent on each slide. For example, a student who spends 30 seconds reading a slide probably has a better understanding of the content than someone who is paging through very quickly, and only spends 2 seconds on the slide. However, this data collection is flawed. There is no way to differentiate a student studying a slide for a few minutes from a student who was otherwise occupied for a few minutes, but who has really only spent 30 seconds reading the slide. Therefore, timing data are not relied upon heavily.
Clearly, if students do not take quizzes, then the student model is at best a guess of the students knowledge. For this reason, the tutor does not control what topics a student sees. Rather, it provides suggestions to the student, which he can either follow or ignore. This is similar to the mechanism used in ELM-ART (Brusilovsky, et al. 1996). We are currently investigating using reasoning under uncertainty to help the tutor make better decisions.
During the Fall of 1996, 15 students (9 of whom returned questionnaires) used the MANIC system as part of a one credit course on UNIX network programming and Java. The version of MANIC at that time did not include adaptive quizzing, topic suggestion, or varying of slide versions. Our main goal was to determine students' likes and dislikes about the software, and to use this information to change the system.
Eight out of nine students expressed very positive feedback about MANIC. These eight students indicated that they enjoyed the on-line course, with the ability to study at their convenience. Six students liked the linear nature of the course, but the other three would have preferred to have been able to choose what material to see next. Also, seven students said they would have taken advantage of on-line quizzes.
Students taking this course, however, had weekly meetings with instructors to discuss an assigned section of the material. Thus, the students simply listened to that part of the lecture to prepare for the discussion session. This gives credence to our assumption that when MANIC material is being used to supplement a live class, intelligent features may not be as useful.
In the spring of 1997, MANIC was used in conjunction with Computer Science 453/653: Computer Networks. During this semester, we discovered similar trends in user's behavior. For example, only 34 users viewed more than 1/4 of the material. And in almost 50% of the sessions, no audio was used.
As students had the option of either going to class, viewing lectures on video tape, or using MANIC, these statistics are not unusual. We are interested in learning how MANIC is used by students who do not have these other options, and in the future, we will be exploring these results.
Designing web based intelligent tutoring systems is not an easy task. Network latencies must be dealt with, especially if a lot of multimedia (sound, pictures, and animations) is part of the course material. Also, the HTTP protocol is stateless, which is unacceptable for doing student modeling. While solutions to the second problem exist, solutions to the first are still under investigation.
The MANIC system provides other complications for web-based teaching. First, converting from an already existing lecture course may not be very efficient in terms of creating the domain material. The audio from the class is clearly from lectures, whereas for the on-line version, it should be topic based. It may be better to record audio appropriate for the material rather than taking it from the video tapes.
Second, determining the actions students take is incredibly difficult, since students will not always use the software as intended. Similarly, accurately recording the time spent on the material is not possible, since there is no way to distinguish between idle time and intense studying time.
Third, creating an accurate student model based on the slide viewing pattern of students is almost impossible. A student may see some material, but not its prerequisite, and from this the tutor might conclude that the student does not know the prerequisite. But if he does, the tutor will have an inaccurate student model. Pretesting can help with this, which is why we are including a small pretest for students first using the system.
Thus one of the main research questions in MANIC is determining what kind of student model we can in fact maintain. We will then investigate how the student model can effectively be used to make the instruction individualized to each learner.
T. Berners Lee, R. Fielding, and H. Frystyk. Hypertext Transfer Protocol - HTTP/1.0. Network Working Group, RFC 1945, May 1996.
C. Boyle and A. Encarnacion. Metadoc: an Adaptive Hypertext Reading System. In In User Models and User Adapted Interaction, volume 4(1), pages 1-19, 1994.
P. Brusilovsky, E. Schwarz, and G. Weber. ELM-ART: An Intelligent Tutoring System on World Wide Web. In Intelligent Tutoring Systems, pages 261-269, 1996.
C. Kaplan, J. Fenwick, and J. Chen. Adaptive Hypertext Navigation Based on User Goals and Context. In User Models and User Adapted Interaction, volume 3(2), pages 193-220, 1993..
V. Padmanabhan and J. Mogul. Using Predictive Prefetching to Improve World Wide Web Latency". In ACM SIGComm, pages 22-36, 1996.
M. Stern, J. Steinberg, H.I. Lee, J. Padhye, and J. Kurose. MANIC: Multimedia Asynchronous Networked Individualized Courseware. In Educational Media and Hypermedia, 1997.
M. Stern, B.P. Woolf, and J. Kurose. Intelligence on the Web? In Artificial Intelligence in Education, 1997.