The Chat Reference Experience at Carnegie Mellon University
http://www.contrib.andrew.cmu.edu/~matthewm/ALA_2001_chat.html
Matt Marsteller (matthewm@andrew.cmu.edu)
Paul Neuhaus (neuhaus@andrew.cmu.edu)

Abstract: Carnegie Mellon University began its chat reference service on October 1, 2000. The service utilizes commercial off-the-shelf software called LivePerson. Service evaluation features of the software provide interesting data to share with the greater library community. Summary statistics of the service, logs of actual chat sessions, and cumulative results of post chat surveys (filled out by the chat patrons on a voluntary basis) will be presented. Plans for future enhancements of the chat reference service that are independent of the software used will be discussed. A live demonstration of LivePerson will be offered if technically feasible.

Statement of the Problem: The University Libraries sought an additional means to provide online reference service to remote users. Prospective users included faculty and students working off-campus or outside the libraries as well as distance education students. Our email reference service received only sparse usage at that time.

In the summer of 2000, the Carnegie Mellon University Libraries decided to offer a digital reference service that permits an effectively synchronous exchange between librarians and users. There was mounting concern that the Libraries should utilize existing technology to make reference services more visible and accessible. The Libraries’ administrators directed the Library Information Technology department to identify a quality product capable of providing this service. The customer contact center software produced by LivePerson was chosen.

A number of factors contributed to the decision to utilize LivePerson:

Carnegie Mellon University Libraries initiated their chat reference service in October of 2000. The Libraries formed a "Chat Reference Task Force" that oversaw the implementation of the service, provided additional training to that given by a LivePerson representative, and kept the project moving. The pilot project called for staffing the service from 1:00 PM to 5:00 PM, Monday through Friday. Initially, only reference librarians staffed the service. Gradually students employed by Carnegie Mellon from the nearby University of Pittsburgh School of Information Science began to help with chat as their experience with reference work in general deepened.

The Libraries soon learned that various technical matters would play a role when offering chat reference services. For example, during training librarians discovered that LivePerson did not function properly when the chat operator (librarian) utilized Netscape in the CMU computing environment. Technical staff subsequently loaded Internet Explorer 5.5 on all computers used by chat operators. Fortunately, patrons can choose either Netscape or Internet Explorer. The other prime technical concern is speed. Older computers can slow the chat interaction considerably.

The Libraries began advertising the forthcoming service via the Libraries' home page in August. Later promotional efforts included campus-wide email announcements, liaison librarians informing their constituencies through various means, and the insertion of a flyer in the student newspaper (February 2001). Interestingly, the flyer produced no noticeable effect. The number of chat sessions continues to range between 60 and 80 per month when classes are in session. The task force plans to renew marketing efforts when fall classes begin.

 

Evaluating the Service

As with any service, there are both positive and negative characteristics. After providing chat reference for six months, the Chat Reference Task Force developed and distributed a survey to the librarians and graduate students who serve as operators. The "Pros" and "Cons" identified below are based upon their feedback.

Pros

  1. Provides immediate assistance for remote users.
  2. Good for distance education students.
  3. Since the patron is already online, s/he can implement the librarian’s instructions right away.
  4. Better than email for conducting a reference interview.
  5. Allows anonymity.
  6. LivePerson defaults to email when a librarian is not available.
  7. Raises awareness of the library among user community.

Cons

  1. Staffing the chat service without hiring additional personnel places greater demands upon already busy librarians.
  2. Schedules for librarians become less flexible.
  3. Librarians sometimes lack the necessary subject knowledge.
  4. The librarian receives no visual or auditory cues during the reference interview.
  5. Less interactive than in-person or phone reference.
  6. Typing takes time.
  7. Some patrons log off before librarian finishes answering the query.
  8. If communications are slow, users may just log off or leave before connecting.
  9. LivePerson – for CMU’s configuration, librarians need to use Internet Explorer.

In the administrator's area, LivePerson offers a number of features that facilitate an analysis of the service. The software keeps a log of all chat conversations indefinitely. Here is an example of one chat session - "Andy" is the chat operator's name.

Andy: Hello Dizzy, how may I help you today?
Dizzy: I was wondering if you had any idea where i could information on incan foods...namely how to make them.
Dizzy: find information...sorry
Andy: Where have you looked so far?
Dizzy: mainly on the internet, for about the last three days and i havent turned up much...nothing actually
Andy: Can you hold for a couple of minutes and I will check some things?
Dizzy: yes i can
Andy: Thank you for waiting. I'll be with you momentarily.
Dizzy: thanks
Andy: You may get the information you need from an encyclopedia. I will push you a page for online encyclopedias.
Dizzy: ok thank you
Andy: http://www.library.cmu.edu/Research/Genref/ency.html
Andy: I tried the Encyclopedia and saw a couple items listed under canning and also "vacuum packaging."
Dizzy: ok...thank you
Andy: I tried the Encyclopedia Britannica.
Dizzy: ok..thank you...that was all i needed.
Andy: If this does not have the information you need let me know and I can check elsewhere.
Dizzy: ok...
Andy: Thank you for waiting. I'll be with you momentarily.
Andy: Unexpectedly left call
STALLER: Call terminated unexpectedly
Andy: Party left call

The authors analyzed the logs for the first seven months to determine the types of queries received and the librarians' responses. The following charts describe the subject breakdown:


It is interesting to note that slightly less than 50% of the queries involved identifiable subjects and the "general" category is the largest subject group. Given the nature of the chat medium, this finding perhaps should not be too surprising. Chat appears to be more appropriate for answering simpler types of questions than for providing in-depth research assistance. Librarians may be willing to spend the time needed for such research but many patrons are not. In the "Other" category, "Technical Problems" predominate. Various explanations are possible for this large number. Some definitely reflect problems with the software from time to time. These problems decreased after LivePerson upgraded their software in November 2000. We believe some aborted calls are due to the user changing his/her mind and just leaving. Other libraries using LSSI and Human Click software reported similar occurrences on the DIG_REF listserv a few months ago. Testing and anecdotal feedback suggest slow connections cause some patrons to abandon the attempt. This could be a LivePerson problem, slowness of older computers, or heavy afternoon Internet traffic decreasing performance speeds. Lastly, "Chat About Chat" refers to librarians from other institutions testing the service as part of their decision-making process.

The second method of analysis examines how librarians responded to the queries.

"Referrals" include referrals to libraries off-campus (principally to the nearby University of Pittsburgh and the Carnegie Library of Pittsburgh), to appropriate libraries on-campus, to library departments (e.g., Circulation), and to subject specialist librarians. "Source Suggestions" refer to the times librarians direct users to particular resources such as the library catalog, databases, print sources, or specific Internet sites. "Ready Reference" are simple questions (e.g., library hours) for which the librarian provides a quick answer. "Librarian Does Search" questions are similar in nature to "Source Suggestion" queries. In these cases, however, the librarian searches the source and locates the information rather than directing the user to the source.

Another LivePerson attribute is the ability to survey users when they exit the service. Approximately 20% of patrons answered at least part of the survey.



Since this is not a scientific sample, it is impossible to draw any firm conclusions though a few observations can be made. It is impossible to know how many (if any) people reporting a "Partial" answer logged off before the librarian finished answering the question. Secondly, at least three librarians from other institutions testing the service completed the survey and indicated they would not return. Lastly, we are somewhat surprised by the percentage of graduate student users. Given respective numbers on campus, we expected the largest group of patrons to be undergraduates. Two explanations suggest themselves. The greater amount of time spent on research by graduate students may naturally lead to more questions or these students may be simply more inclined to complete a survey.

Two other survey questions asked patrons to report technical problems and to suggest chat service enhancements. Most individuals indicated an absence of technical difficulties; a few said the connection was not so good or slow. The comments regarding improving the service included some very complimentary remarks (e.g., Andy is very helpful - he is the Best!). A few patrons suggested the Libraries get more people to staff the service or find people who can answer the patrons' questions. In looking at the call logs for this latter response, it appears patrons were looking for subject assistance but the librarian staffing the service was unfamiliar with that particular subject. These instances usually resulted in a quick referral by the librarian but left users a bit disgruntled. With a variety of librarians staffing the service and handling all incoming questions, such occurrences are regrettable, but probably unavoidable.

The final graphs display the hours of the day and days of the week for chat sessions. Thursdays and Fridays are somewhat slower but the Friday statistics also reflect occasional off-days when classes were not held.


One unexpected result of instituting the chat service was a noticeable increase in the number of email reference questions. For the first time the Libraries' home page included an email reference link along with the chat service link under "Ask A Librarian." Previously the only connection appeared on the reference services page. In addition, LivePerson allows patrons to leave an email message during off-hours or after failed attempts to connect to the service.

Since instituting the chat service, we also have tracked the time email messages arrive. So far the number of email questions received in the evening is quite small. Many more email queries arrive during the morning hours. If the Libraries decide to expand chat service hours, this may indicate which direction to go. The timing of email questions may not be transferable to chat though approximately 40% of morning email queries originated in the chat service.

 

Conclusion

Overall we believe the chat reference service is a worthwhile endeavor. On the internal chat operator survey we asked for responses to "The chat service provides a much needed service and should definitely be continued." The results: Strongly Agree-2; Agree-8; Disagree-1; Strongly Disagree-0. Chat provides another reference option and is a good resource for remote users. Survey feedback suggests most users are satisfied with the service. There is some concern regarding the number of times technical difficulties hinder access. The Chat Reference Task Force will examine this issue more carefully over the summer. In consultation with all the librarians staffing chat reference, the Task Force also will explore various enhancement possibilities:

Whatever the outcome on these issues, the Libraries have moved from offering chat on a trial basis to making chat a regular feature of Carnegie Mellon's reference services.

Last updated June 2001.