HYPERTEXT'98, Pittsburgh, USA, June 20-24, 1998
An Evaluation of Adapted Hypermedia Techniques Using Static User Modelling
University Road, Southampton, Hampshire, UK
Abstract: Adaptive hypermedia has the potential to break through traditional educational barriers by allowing the tailoring of applications to specific user needs and requirements, to do this effectively the application must hold a model of the user. In this paper we present an empirical evaluation which was carried out to investigate various techniques in adapted hypermedia in education by comparing; adapted presentation, adapted navigation with a non adapted control hypermedia application. This experiment was also used to verify a static model (derived from the SaD static and dynamic) user model which was the result of initial analysis from a research project undertaken at Southampton University. Initial results from the evaluation look promising, with a shift in attitude of subjects towards the acceptance of adaptation in hypermedia applications with a high percentage of users preferring the adapted presentation application.Keywords: Adaptive Hypermedia, User Modelling, Educational Hypermedia, Static User Modelling, Empirical Evaluation
1. Introduction
This paper will present the preliminary findings from an evaluation which was carried out to investigate various forms of adapted hypermedia. The user models used for this experiment were derived from an in-depth survey which was carried out to establish a simple user model for use in educational hypermedia applications. From the initial findings from this survey we suggested a SaD (static and dynamic) user model, section 2 will introduce the SaD model, section 3 and 4 will explain the design and preliminary results of the experiment and section 5 will present the direction which future research will take.2. SaD (Static and Dynamic) User Modelling
A literature review revealed that over time most authors have shifted away from one level standard static user models which included information such as sex, age etc. (Totterdell et al, 1987) to more detailed two level models; a static user model with a dynamic user model containing user interaction information (Pohl et al, 1995) (Murphy et al, 1997) (Gutkauf et al, 1997) and (Beck et al, 1997). Although it is agreed that user models are required for adaptive hypermedia, ideas presented by individual authors vary in the level of detail required.To gain a deeper understanding of user models a survey was carried out to construct a simple user model to be used in educational applications. Students ranging from computer science, arts and the science faculties from Southampton University took part in the study, all the students involved were using Microcosm (Davis et al, 1992) applications as part of their studies. They were required to answer questions on their previous computing and Microcosm experience, personality traits and background. The results were used to answer the following questions:
- What variables should be modelled and to what level of detail?
- How closely tailored to individual users does the model need to be?
Users were asked to evaluate themselves on their personality and computing experience. The analysis was carried out by correlating each variable with each of the other variables to establish if certain types of users interacted with or accepted a system in similar ways. It was evident from the analysis that some variables bared no relevance to the interaction process. For example, work, home environment, the number of people an individual may live with etc.
In general not all personality variables were found to be
relevant to the interaction process however, traits such as shy,
pessimistic and quiet did reveal some significance. Again the most
relevant 'computing experience' attributes were quite clear i.e. users
computing skill, windows, application experience proved to be extremely
relevant for users when interacting with the application. More
confident users required less help from the system. It was interesting
to find that most users irrespective of their skill level wanted some
level of information when errors were made. When analysing the results
it became evident that a division in the type of relevant user model
variables fell into two distinct categories which required the
construction of two user models; a static user model which contained
static user information and a dynamic user model which contained data
which could be used by the system to dynamically adapt to the
individual user. Hence, SaD (Static and Dynamic User Modelling), the
basic SaD model can be seen in Figure 1.
Figure 1 SaD (Static and Dynamic) User Model
A more detailed model can be found here.
3. Design of the Experiment
The experiment was devised to establish user's subjective preference when comparing different forms of adaptation (Brusilovsky,1997) and to verify the static user model. The experiment involved the implementation of three 'Archaeology Dating Techniques' applications, one which adapted the presentation of links (Application A), another which adapted the document content (adapted presentation) (Application B) and one as a control hypermedia application without any form of adaptation (Application C).3.1 Adapted Hypermedia Applications
The applications were developed using Microcosm, an open hypermedia environment simply because it provides authors with the flexibility and freedom to explore different adaptive approaches. However, it is important to note here that the results from this experiment could be applicable to any hypermedia system as we are only considering adaptive hypermedia methods ie presentation issues and highlighted links.Varying windows management experience between subjects presented a problem, therefore a software tool called SHEP (screen handler enabling process) was used to keep the interface consistent, whereby issues such as the overlapping or multiple opening of windows etc. could be addressed by manipulating settings in the SHEP files called Shepherds (Hall et al, 1997).
In all, four basic (static) stereotypical user models were devised based on system and content knowledge. Each of these models corresponded to a particular linkbase which held the links relevant to that particular user model group. Each subject was required to complete a pre-test questionnaire which was used to allocate that subject a user model. A matrix describing the codes used for each user model can be seen in Figure 2.
Figure 2 Codes Used for Each User Model
For Application A, four linkbases (database of links in Microcosm) were
used each corresponding to one of the four different user models so
that the application displayed different links for the four types of
users.
For Application B only two linkbases (CNSN and CNSE) were applied
i.e. only the subject's system skill level was taken into account in
the user model, because for this application we were only interested in
the users' subjective view of the adapted presentation. In this
application the entire information base and highlighted link structure
was available to all users whatever their content skill level. However,
more complicated and dense material which was included solely for the
experts, was shaded out so that users could choose whether to read it
or not, thus giving them a form of self adaptation.
Application C was used as a non-adapted control hypermedia
application, which the subjects could use to compare the methods of
adaptation. This application also utilised only one linkbase where only
one user model was allocated to all types of users, however, No form of
adaptation was used and all the subjects were presented with the same
links, documents and hypermedia structure.
4. Empirical Evaluation For Adaptive Hypermedia
The results of the experiment were obtained using various evaluation techniques such as questionnaires, log of session data and observation. The experiment was set-up so that users were given pre-test questionnaires and depending on their subject and system knowledge they were allocated one out of four stereotypical static user models. The subjects were then allocated specific user names which corresponded to their user model, these user names were used to log into each application in turn.The students were told about the main aims of the evaluation and what was required of them, these instructions were enforced by an instruction sheet which explained step by step exactly how to log onto the system. The instruction sheets also set out tasks for each application which the subjects were asked to complete in turn. Once each set of tasks had been completed they were asked to fill out a short questionnaire on that particular application type.
The evaluation was carried out over three separate sessions using a laboratory set up, the subjects came from Computer Science and Archaeology backgrounds. 7 females and 9 males with an age range from 18 to 35 took part in the study with experience ranging from 1st year undergraduates, PhD students to lecturers.
4.1 Preliminary Evaluation Results
4.1.1 Static User Model
The static user model did appear to match the correct user model to each subject. It was found that for total expert or total novice users in one particular area, i.e. in computing or content, it was straight forward to allocate correct user models as these groups work in a predictable way. However, for the average computer users and average content skill level groups (i.e. for users not on the extremes) it was more difficult to assess their knowledge unless they were asked to complete an in-depth questionnaire covering all the areas of the application. Therefore, a major problem revolved around users being unable to assess themselves on their computing skills. For this reason it is felt that the static user model requires more detail for users without computing background simply to determine what their real computer literacy levels are. Having said this it is important to point our that the pre-test questionnaire was designed to be very basic and force the users to select from very broad categories i.e. the users were given a choice of one out of two possible answers for most of the questions, for example expert/ novice.A correlation analysis carried out on the data proved to be very useful. It clearly showed the variables which were significant to the way the subjects worked. In this experiment personality variables only showed some positive correlation. Having said this it is important to point out that the results from the first survey from which SaD was proposed did find certain personality variables extremely relevant in how well a user accepted a system. There is also a general acceptance within the user modelling community (Beck, 1997) that they are relevant to how a user interacts with a system. Therefore the lack of significance from the current analysis can be put down to the small population sample who took part in the study as opposed to the larger sample which took part in the first survey.
It was interesting to find that the number of hours a subject watched television and the number of hours a subject used a computer during a week was very significant in how a user accepted and worked with a system. The results showed that in general a high correlation between the subjects who spent more than 15 hours a week watching television and subjects with a low level of expertise in computing however, subjects who spent less then 15 hours a week watching television were more likely to be expert computer users. Also, the more windows a user claimed to work with at one time the more experienced they were on most of the packages listed in the questionnaire.
From the analysis it could be observed that some of the
variables which were studied could be used to dynamically determine
users' expertise. The time spent per document is a good indicator as
the analysis showed, the CESE group spent less time on a document
compared with the other two groups. Total session completion time per
group was also significantly varied as the CESE expert group had a
faster completion time and the CNSE group had the slowest session
completion time. There was not much difference found in the number of
nodes the different groups visited, however, the CESN group did visit
more nodes within each application compared with the other two groups.
The time spent on relevant documents proved to be more significant than
the average time spent on each document, as the CNSE group spent more
time on relevant documents than the CESN group and the CESE group
overall spent the least time per relevant document.
During the analysis it was obvious that members of each user model
worked in a similar manner as the number of links they followed, the
time they spent on each document and the total session times etc were
alike. This emphasised the relevance of allocating users stereotypical
user models at the start of an interactive session.
4.1.2 Adaptive Techniques in Hypermedia
Adapted hypermedia was a new concept to most of the subjects who took part in the evaluation process, therefore a detailed explanation of adaptation and various methods was given in the pre and post-test questionnaire before asking the subject for his subjective preference on which method of adapted hypermedia they would feel most comfortable with. In the pre-test questionnaire most of the subjects chose the control hypermedia application, stating that they preferred to have access to all the material and all the links. However, after the evaluation session the results showed a significant shift in preference towards the acceptance of adapted hypermedia applications with a very low percentage liking the idea of the control hypermedia application but a very high percentage preferred the use of the adapted content application with the adapted links application coming a close second. These results can be seen in Figure 3 below.
Figure 3 The Change of Preference Towards Adaptive Hypermedia (full size)
Some subjects even proposed the use of a combination of adaptation
methods within one application. Again the reasons stated for this was
that users would still have access to all the material but could avoid
wasting time reading material which would be too complex or irrelevant
for their particular task and that the information would be there in
case they wanted to know more about a particular topic, other
statements given included;
These views were reflected in the answers given in the
questionnaire where 78% of the subjects felt that all the material
within an application should be made available to all types of users,
just like a conventional book and 63% agreed that shading out chunks of
irrelevant text reduced information overload.
Figure 4 clearly shows that most of subjects spent the highest amount
of time visiting nodes while using Application A. The subjects also
spent on average a significantly greater amount of time visiting each
node in Application C as opposed to Application B where the average
time spent visiting each node was significantly less than with the
other two applications.
The results from this graph are significant in that although the
subjects spent less time per node on average within Application B the
percentage of correct answers was higher for this application in
comparison to correct answers for the other two applications. This can
be seen in Figure 5 below.
Figure 4 Average Time Spent On Each Document Within Each Application (full size)
Figure 5 Correct Answers For Each Application (full size)
5. Conclusion and Future Work
The results from the experiment revealed that adaptive hypermedia was accepted by the subjects who had reservations about it at the start of the evaluation session. The subjects felt that adaptation was an important concept and that it could support the subject during the learning process when using educational applications. Most of the users felt comfortable with Application B but did agree that Application A played an important role in aiding navigation. There was a high emphasis placed on allowing users access to the entire information base within an educational hypermedia application.This experiment also revealed some of the most important variables required for user modelling which included computing experience and subject studied etc. However, it was found that the static model requires more computing background information so the user has a better chance of assessing themselves correctly and the system has a more detailed model of the user to provide the correct level of adaptation.
The next stage of the research will involve gathering results from the evaluation described here design an educational adaptive hypermedia application which will aid navigation, reduce information overload and at the same time allow access to the whole information base within the application. The same experiment will be used to implement the refined version of the SaD model so that the adaptation will take place dynamically.
It is important to note here that the users cognitive model is one
division of the user model and due to it's complexity this area was
deliberately left out from this work.
References
Beck, J., Stern, M. and Woolf, B. P., (1997), Using The Student Model To Control Problem Difficulty in User Modelling; Proceedings of the 6th International Conference eds. A. Jameson, C. Paris and C. Tasso, Springer Wien.Brusilovsky, P. (1997), Efficient techniques for adaptive hypermedia in Intelligent hypertext: Advanced techniques for the World Wide Web, Lecture Notes in Computer Science, vol.1326 eds. C. Nicholas, J. Mayfield, pp. 12-30. Springer-Verlag, Berlin.
Davis, H., Hall, W., Heath, I. and Hill, G., (1992), Towards an Integrated Information Environment with Open Hypermedia Systems in Proceedings of the ACM conference on Hypertext eds. D. Lucarella, J. Namard, M. Namard and P. Paolini, ACM Press.
Gutkaut, B., Thies, S. and Domik, G., (1997), A User-Adaptive Chart Editing System Based On User Modelling and Critiquing in User Modelling; Proceedings of the 6th International Conference eds. A. Jameson, C. Paris and C. Tasso, Springer Wien.
Hall, W., Weal, M., Heath, I., Wills, G.B., & Crowder, R.M., (1997), Flexible Interfaces in the Industrial Environment in International Conference Managing Enterprises- Stakeholders, Engineering, Logistics and Achievement (ME-SELA'97), pp453-460, Loughborough, UK.
Murphy, M. and McTear, M., (1997), Learner Modelling for Intelligent CALL in User Modelling; Proceedings of the 6th International Conference eds. A. Jameson, C. Paris and C. Tasso, Springer Wien.
Pohl, W., Kobsa, A. and Knutter, O., (1995), User Model Aquisition Heuristics Based on Dialogue Acts in International Workshop on the Design of Cooperative Systems, 471-486, France.
Totterdell, P. and Cooper, P., (1987), Design and Evaluation of the AID Adaptive Front End to Telecom Gold in BCS Workshop Series Proceedings of the 3rd conference of BCS HCI Specialist People and Computers eds. D. Diaper and R. Winder, Cambridge University Press.