Samuel A. Rebelsky
Department of Mathematics and Computer Science
Science 2427 (Y-2)
Grinnell, Iowa 50112
Abstract: As increasing numbers of course webs are built, it is increasingly important to evaluate the effects of those webs on student learning. Web usage may depend on student expertise, classroom setting, and available computing resources. In past studies [Rebelsky 1996c] [Rebelsky 1996a], I examined student reactions to a wide variety of web resources, from assignments to transcriptions of individual class sessions. In this present study, I examine student usage of course webs in classroom situations in which computers are readily available and students are comfortable using hypertext systems. For these students, there is significant, but not uniform, use of web-based course supplements during class sessions. More importantly, these students report benefits from their self-directed use of those online materials.
The World-Wide Web [Berners-Lee et al. 1994] continues to change the ways in which we teach and are expected to teach. Many instructors are putting part or all of their courses on the web, and students are expecting to find course resources on the web. In spite of this growth, there is little formal or informal evaluation of student usage of course webs or the impact of course webs. For example, of the 182 full papers presented at the 1997 World Conference on Educational Multimedia, Hypermedia, and Telecommunications [Muldner and Reeves 1997], only six provided any form of evaluation of student usage of course webs, and many of those included only rough notes on observations (however, a number of others evaluated student use of particular hypertext learning tools). This lack of evaluation is surprising, given that "most researchers agree that insufficient evidence exists to determine whether outcomes match the proclaimed promises of hypermedia" [Fizgerald and Semrau 1997].
As developers of course webs, we must ask ourselves how students use, react to, benefit from, and are otherwise affected by our course webs. The questions developers of course webs might ask themselves and their students include,
In previous studies [Rebelsky 1994] [Rebelsky 1996a], I reported on student usage of resource-intensive course webs that included outlines of each class, copies of anything written on the blackboards during class sessions, assignments, selected questions and answers, traditional course handouts, such as the course syllabus, and, in one course, transcriptions of each class session. Students reacted positively towards the online availability of a wide variety of resources, and indicated that they appreciated effort required to produce these webs. At the same time, they felt overwhelmed by the materials, with many deciding that they needed to print and read everything. In an in-depth analysis of the effects of web usage on computing anxiety [King et al. 1997], King, Henderson, and Putt reported that use of course webs does not appreciably affect computing anxiety.
The present study approaches the problem from a different perspective: if students have in-class access to computers and the web, how does that affect their perception and usage of the web. In particular, what pages do they use most frequently, when do they use them, and what do they say about their usage? The study is based on two courses: a second course in computer science (an ACM CS2 course), and an upper-level course in software design.
In the fall of 1997, I prepared extensive course webs for two courses at Grinnell College, Computer Science 152 and Computer Science 223. The design of these webs was based on previous research on student use and preferences [Rebelsky 1996a]. Each web contained (1) a front door with links to the remainder of the web and appropriate external resources; (2) standard handouts, such as syllabus, rules and regulations, and assignments; (3) moderate-sized outlines for each class, developed on or before the day of each class; (4) a short news document documenting updates to the web; (5) online quizzes; and (6) appropriate reference materials, including readings and external links. The class outlines had three parts: (1) a short summary of the topics to be covered; (2) a list of administrative information with links to appropriate pages; (3) my notes on the day's topics. A typical outline is four pages long. Students were told about the course web the first day of class, and were rarely, if ever, given paper handouts. The web is primarily text-based; most of the images are on pages automatically included by javadoc.
Grinnell College's Computer Science 152 is a standard second course in Computer Science, with emphasis on data structures and algorithm analysis. CS152 meets four days per week. We teach the course in Java so that we may emphasize both object-oriented and imperative design. Grinnell has recently adopted a two language/three paradigm strategy in keeping with the new standards for introductory sequences [Walker and Schneider 1996]. In the fall term, there were seven students enrolled in CS 152, three of whom had a previous course based on functional programming, and four of whom had other exposure to programming. All students were familiar with the World-Wide Web and were unfamiliar with Java. The text for the course, a preprint of Bailey's Java Structures [Bailey 1997], included an online software library and reference manuals for the library that were included with the course web. Students also used a summary of the Java language [Flanagan 1997] as a reference.
Grinnell College's Computer Science 223 is an introduction to issues in software design aimed at liberal arts students. CS223 meets three days per week. CS223 covers a wide variety of topics, from forming a programming team and building large projects to issues in user interface design. As such, it draws upon a wide variety of texts, from general treatises on software design [Winograd 1996] to collections of articles on programming [Bentley 1986] [Bentley 1988]. This makes students more inclined to rely on instructor-generated resources, such as the course web. Like CS152, CS223 uses Java as the main language so as to better illustrate object-oriented design. In the fall term, there were twenty four students enrolled in CS 223. Of these, nineteen were majors and five were nonmajors (with majors in Biology, Spanish, English/Physics, Theater, and Chemistry). Students were generally unfamiliar with Java, but were uniformly familiar with the World-Wide Web, and about half were able to write HTML.
Both courses were taught in the departments MathLAN classroom. This room includes sixteen HP 712/60 Unix workstations. In CS223, students were forced to pair up on workstations, but they are used to this from other classes. There were some in-class exercises given in both classes, as well as some electronic quizzes (in CS223, students could collaborate on a quiz). Students were permitted to use the computers as they deemed fit during class. The MathLAN classroom is adjacent to the MathLAN laboratory, where students often look at course pages before and after class.
During the term, I observed a number of students loading the course web pages during class, doing "research" on in-class questions, and using the computer as a notebook. It therefore seemed appropriate to gather data on how they were using the combination of computers and web.
Data on student web usage were gathered in two ways: students took a mid-term survey on their class usage, and web logs were analyzed using custom tools. The survey focused on students perceived use of the web, particularly of the class outlines. The logs permitted more general analysis.
Before mid-semester break, students were given an electronic survey on the class web and were told that the intent of the survey was to determine if and how the class web should be changed for the remainder of the term. In CS152, all seven students answered the survey. In CS223, twenty one of the twenty four students answered the survey. The survey asked students (1) how often they referred to the class outlines during, before, and after class sessions (1: never, 2: rarely, 3: once in a while; 4: often; 5: always); (2) whether the outlines were a help or a hindrance or both; (3) whether they printed the outlines; (4) whether they took notes on the computer; (5) a number of free-form questions about the course web; and (6) some questions not pertinent to the study at hand.
The access logs generated by our httpd daemon were analyzed with a custom
Perl script. Cumulative data was used for overall analyses.
For each class, this script selected pages according to
appropriate criteria. For accuracy, the pages counted were limited
to those that were
(1) part of the course web (i.e., contain the site root as part
of their path);
(2) accessed from inside the college (i.e., contain
part of the requesting machine);
(3) accessed from machines the instructor does not regularly use (the
instructor's machines are never used by students, and the instructor
does not view the web from student machines);
(4) valid requests (i.e., requests for which a page exists).
For counts of usage of the current day's outline "during class", the time of access was restricted to the fifteen minutes before and ten minutes after the class session (for a class held from 1:15 to 2:05, the times were 1:00 to 2:15).
Reported student usage of class trends was in keeping with past observations. Most students report making at least occasional use of the course outlines, with the majority (five of seven) in CS152 reporting that they "always" used each course outline at some point. In CS223, there seems to be less regular usage (only eight of twenty one chose the highest usage in one of the categories), although the reasons for that are not clear. Surprisingly, students seem to pick a particular time to use the outlines, and stick to that time. Some students appear to use the outlines before and during class as a guide to what will happen and others use them later for review.
Student comments described a wide variety of uses of the webs during and near class time. As one might hope, some used them to prepare themselves for the topics in a given class
They give you an idea of where the class is going. [CS223 student]Others used them as aids to recall, to answer questions they had during class, or to find other perspectives.
I read the outlines before class so I know what to expect and to be in a proper mindset. [CS223 student]
Sometimes they give enough information beyond what is getting written on the whiteboard. That way, I can piece it together for myself without having to slow down the rest of the lecture for a silly question. [CS223 student]
I really like being able to jump back to any point in the lecture if I find that there is something I didn't quite catch and need repeated. [CS223 student]
One danger of online resources like these is that they adversely affect student note-taking. Since note-taking has a clear role in learning [Carrier 1983], it is worrisome that some students use class notes as a replacement for their own. I attempt to fight this problem, in part, by using my outlines as only a rough guide for what I teach each day. I often develop new or modified examples, and rarely look at the outlines during class. Nonetheless, students reported that they perceived positive effects on their note-taking. For some, this is because they have not yet learned to take good notes:
I have only mediocre note-taking skills so the outlines that you provide are superior to any notes I would take myself. [CS152 student]It is, however, worrisome that these students do not yet understand the implicit benefits of notetaking.
It's nice to have a record of what happened in class (since I'm a poor note taker). [CS223 student]
More importantly, many reported that the in-class availability of outlines permitted them to concentrate more on understanding than on copying.
Since I do not have to take many notes I am able to concentrate more on understanding what you say instead of writing down. [CS152 student] Instead of taking the time to write down everything and missing stuff, I can watch and listen more closely. [CS152 student] I can spend more time listening to what you and other people say than take notes. [CS223 student]
Interestingly, some of those who were confident in their note-taking abilities reported that they used the outlines to supplement their note-taking.
I like to organize notes when I take them and if I know what general subjects you cover, I'm happier. [CS223 student]
One of the perceived benefits of hypertext is that it supports a variety of learning styles. Student answers suggested that this is, indeed, the case.
I have much better comprehension of things that I read instead of hear. Since you do not follow the outline exactly, reading it can be a way of getting things rephrased or put differently. [CS152 student] Seeing something on the screen, however, makes me much more likely to remember something. [CS223 student]
Surprisingly, most students reported that they did not regularly print the class outlines. In CS152, only one of seven students reported printing the class outlines, and that student is a non-native speaker who has indicated he makes greater use of the outlines than other students. In CS223, only three of eighteen students reported that they printed the class outlines. This is somewhat surprising, given my own past experience in which students printed and read "everything available" [Rebelsky 1996a] as well as the experience of others (e.g., [Windley, 1994]). It is not clear whether or not this is because of the in-class availability of the resources, but it seems more likely that as computers become more available, students feel less needed for printed copies.
Table 1 summarizes student reports of their usage of course outlines. The first part of the table summarizes their reported usage before, during, and after class. The second part of the table summarizes the computed maximum and minimum reported usage for each student. It was surprising to discover that no students reported a usage of 4 (often), with many choosing either 5 (always) or 3 (once in a while). It is not clear why this is. It may be that students felt that 3 was appropriate for all but the most web usage, it may be that students are either regular or occasional web users, or it may be that the terminology biased students. A revised version of the survey, to be given at the end of the term, will eliminate this terminology.
Analysis of web log files shows that reported in-class usage of outlines is fairly close to actual usage. In CS152, the outline of the current class was accessed during class 5.9 times on average. These accesses came from an average of 5.0 machines. This is in keeping with the reported numbers. However, in CS223, the current class outline was accessed during class 15.0 times on average, from an average of 12.5 machines. A small number of in-class (or, presumably, shortly pre- and post-class) access came from machines outside the classroom, but this number was less than one document per day.
Table 2 reports on overall student usage of
the course web, with data taken from the server log files. A short
analysis of the files accessed suggested that certain sets of documents
biased the results. In CS152, documentation that accompanied the book
was included in the counts, so a separate analysis without that
documentation was done. In CS223, some documents generated by
javadoc included images that were counted towards the
totals. In addition, a directory loop in the Examples system
resulted in their being multiple paths to the same document, which
greatly increased the number of different documents loaded. Again,
separate analyses were done to handle those differences.
Surprisingly, the number of accesses per document per student is relatively uniform across classes, with each document being accessed approximately twice by each student. While there are clear variation in access patterns (e.g., the root of the web accounts for nearly 1/5 of the accesses, the two classes have different types of pages, some students use the web more or less), it is comforting to see that students do, in fact, regularly use the course materials.
| w/o book
| w/o examples
As table 3 suggests, students look at a wide variety of materials, but most visits are confined to the course root (presumably as a starting point for explorations) and outlines (which, in the webs used in this study, contain most of the information).
As one might expect, the types of documents used varies depending on whether students are inside or outside of class. In particular, use of class outlines increases significantly during class sessions. Presumably, this is due to a number of factors. First, students are less likely to look at "extraneous" material (such as assignments or syllabus) during class sessions. In addition, students who use the web during class time are likely to follow along in the current outline, or sometimes refer back to previous outlines for further information.
For many of the students in these courses, the in-class use of course webs has had an impact on how they learn in class, or at least on how they perceive their in-class learning. In each class session, over half the students load web pages (in CS223, the sharing of workstations makes presumed usage even higher). Students who use the pages in class report that the web helps them learn better in class, and both poor and good note-takers report benefits for their note taking.
[Bailey 1997] Bailey, D.: Java Structures. McGraw-Hill (1997).
[Bentley 1986] Bentley, J.: Programming Pearls. Reading, Massachusetts: Addison-Wesley Publishing Company (1986).
[Bentley 1988] Bentley, J.: More Programming Pearls. Reading, Massachusetts: Addison-Wesley Publishing Company (1988).
[Carrier 1983] Carrier., C.A.: "Notetaking research: implications for practice"; Journal of Instructional Development 6, 30 (1983), 19-26.
[Fitzgerald and Semrau 1997] Fitzgerald, G. E. and Semrau, L. P.: "Hypermediaed Learning: Learning Styles, Path Analysis, and Knowledge Outcomes". Proc. EdMedia'97 Conference on Educational Multimedia/Hypermedia. AACE, Virginia (1997).
[Flanagan 1997] Flanagan, D.: Java in a Nutshell, A Desktop Quick Reference, Second Edition; O'Reilly and Associates (1997).
[Greenwood and Recker 1996] Greenwood, J. and Recker, M.M.: "Networked, Asynchronous Student Evaluations of Courses and Teaching: An Architecture and Field Studies"; Proc. EdMedia'96 Conference on Educational Hypermedia and Multimedia, AACE, Virginia (1996).
[King et al. 1997] King, J., Henderson, L., Punt, I.: "Measuring Affective Aspects of WWW and E-mail Use in Course Delivery"; Proc. EdMedia'97 Conference on Educational Multimedia/Hypermedia. AACE, Virginia (1997).
[McKeachie 1986] McKeachie, W.: Teaching Tips: A Guidebook for the Beginning College Teacher, Eighth Edition. D.C. Heath, 1986.
[Muldner and Reeves 1997] Muldner, T., and Reeves, T. (editors): Proc. EdMedia'97 Conference on Educational Multimedia/Hypermedia. AACE, Virginia (1997).
[Rebelsky 1994] Rebelsky, S. A.: "A web of resources for introductory computer science." Proc. 2nd International Conference on World Wide Web and Mosaic (Chicago, IL, 1994).
[Rebelsky 1996a] Rebelsky, S. A. "Evaluating and improving WWW-aided instruction." Journal for Universal Computer Science, 2 (12), December 1996.
[Rebelsky 1996b] Rebelsky, S. A. "How and Why to Teach Hypermedia in Introductory Computer Science Courses (short paper)." EdMedia'96 Conference on Educational Multimedia and Hypermedia, Boston, Massachusetts, June 17-23, 1996.
[Rebelsky 1996c] Rebelsky, S. A. "Improving WWW-Aided Instruction: A Report from Experience." EdMedia'96 Conference on Educational Multimedia and Hypermedia, Boston, Massachusetts, June 17-23, 1996.
[Rebelsky 1997] Rebelsky, S. A. "CourseWeaver: A Tool for Building Course-Based Webs." EdMedia'97 Conference on Educational Multimedia and Hypermedia, Calgary, Canada, June 14-19, 1997.
[Walker and Schneider 1996] Walker, H. and Schneider, G. M. "A Revised Model Curriculum for a Liberal Arts Degree in Computer Science." Communications of the ACM 39(12), (December, 1996).
[Windley 1994] Windley, P. J.. "Using WWW to augment classroom instruction."; Laboratory for Applied Logic, Brigham Young University. URL http://lal.cs.byu.edu/people/windley/using.www.to.teach.html, 1994.
[Winograd 1996] Winograd, T. Bringing Design to Software. Reading, Massachusetts: Addison-Wesley Publishing Company (1996).