CHAPTER THREE

DESIGN AND METHODOLOGY

This section will review the methodology employed in researching student attitudes towards and use of the WWW in an educational setting. Following a discussion of selected research methodologies, the design of the pilot study will be described. Following this will be a description of the design and methodology of the current study, including a review of the statistical procedures employed.

Combining qualitative and quantitative approaches, the pilot study and the current research project employed open-ended questions, interviews, two types of survey research, and content analysis of WWW sites visited by students. Some of the data was collected with the active participation of the subjects, while other data was collected using passive data collection techniques. Using multiple methodologies allowed for increased richness of data and a clearer picture of the phenomena under investigation.

Quantitative Methodology: Survey Analysis

Uses and gratifications researchers, e.g., Greenberg (1974), Rubin (1981a), and Lin (1993b), have historically used questionnaires and interviews to seek information from the subjects themselves about their media use and motives for such use. Becker (1979) noted that the most common method employed by uses and gratifications researchers, "is to rely on reports from the audience members" (p. 54). This methodology is based on the assumption that audiences can provide information about why they attend to a particular medium. With this approach it is important that the subjects provide accurate information about their personal motivations for approaching specific media content.

Sample survey as a methodology has a rich legacy in social scientific research. And as with any social scientific methodology, survey research has advantages and disadvantages. According to Warwick and Lininger (1975), survey research is an excellent choice when attempting to ascertain attitudes and beliefs. It is especially appropriate for descriptive studies of large populations. Survey research has the additional benefit of being parsimonious both in terms of the number and scope of questions that can be addressed and in the effort required to reach conclusions about variability within a population (Comstock & McCombs, 1981). A weakness of survey research is that independent variables cannot be manipulated in ways that are possible with laboratory experiments. While correlational findings are possible, statements about causality are extremely difficult to make. Another potential weakness of survey research is the bias that can be introduced by word choice, phrasing, or question order.

Some researchers have questioned the assumptions that respondents are capable of knowing the reasons why they approach a particular medium and that they are capable of identifying with reasons presented in a survey instrument (e.g., Messaris, 1977; Zillmann, 1985). However, others within the uses and gratifications tradition have expressed strong support for sample survey research and self-report methodology (e.g., Katz, Gurevitch, & Haas, 1973; Becker, 1979; Rosengren, 1985).

Quantitative Methodology: Passive Data Collection

Research based on the sample survey methodology is premised on the sometimes questionable assumption that audiences can and will provide accurate information about why they attend to a particular medium. A weakness of survey-only approach is that there is always the possibility that the user may not know his or her true motives, or may misrepresent them to the researcher (Carey & Kreiling, 1974; Messaris, 1977; Rosengren, 1985; Zillmann, 1985; and Babrow, 1988). Obviously it is important that the user provide accurate information about his or her personal motivations for approaching specific media content. Cautioning against "use of the respondent as analyst," Messaris (p. 320) cited Birdwhistell who reminded researchers that the informant is not necessarily objective in the evaluation of his own behavior. And McQuail (cited in Windahl, 1981) criticized uses and gratifications research which "relies to a high degree on subjective reports of mental states and is therefore regarded as too ‘mentalistic’" (p. 175).

Self-reported data, then, should be interpreted in light of data collected using other approaches. Such an approach was suggested by Palmgreen (1984) who wrote,

Much doubt about the construct validity of gratification measures would be resolved, however, if it could be shown that such indices relate in theoretically predictable ways to other variables, particularly various measures of media consumption. Such empirical evidence would at the same time support the general postulate that mass media consumption is motivated by the gratifications identified by researchers. (p. 23)

And according to Zillman and Bryant (1985), "Inferences from the assessment of actual consumption behavior that varies as a function of known message variables are generally considered more compelling, however, as they bypass the distortions that are commonly associated with self-perception and self-reports" (p. 9).

Unobtrusive research methods (Webb, Campbell, Schwartz, & Sechrest, 1966) are those that collect their data from what people leave behind. Because they do not require interaction between the researcher and the subjects of research, unobtrusive research methods are not as vulnerable to some of the biases that hinder other data collection methods. One such approach is found in content analysis.

Qualitative Methodology: Content Analysis

Content analysis of WWW sites visited by students during times of non-directed access at school provides data that can be helpful to better understand the use to which students are putting this new media technology. According to Stempel (1989), content analysis must be conducted taking into account the meaning of four key terms found in Berelson’s 1952 definition: "content analysis is a research technique for the objective, systematic, and quantitative description of the manifest content of communication" (cited in Stempel, p. 125). Objectivity is achieved by carefully defining categories and laying out a procedure that can be followed without regard for or influence of personal and subjective opinion. The process must be systematic implying a consistent and thorough application of the procedures and careful attention to the research question or hypotheses. Content analysis is also quantitative meaning that occurrences of specific content are noted, counted, and tabulated using mathematical formulae. And finally content analysis must examine the manifest content of the message and take it at its face value (pp. 125-126).

Design of the Pilot Study

A pilot study was conducted in spring and summer of 1998. Like the current study it used a combination of several approaches for data collection: open-ended questions, computer-administered surveys, interviews, and passive data collection. Participants in the study were determined by a purposive sample drawn from middle school and high school students from four schools in a public school district.

The pilot study was initiated by asking middle school and high school students to respond anonymously to an open-ended question asking them to list several things "that the World-Wide Web is good for." A total of 78 students provided 328 responses. These were content-analyzed by the researcher and categorized into the following leading "uses" of the WWW: info/research (n = 91), communication (n = 61), learn/education (n = 28), projects/homework (n = 18), fun/interesting things (n = 17), games (n = 16), entertainment/music (n = 14), shopping/buying (n = 9), news (n = 6), help (n = 6), sports (n = 4), software downloads (n = 4), and other (n = 54).

Table 3.1

Use Categories and Selected Student Responses to the Question, "What is the WWW good for?"

Category Selected Examples of Student Responses

Students were also asked to respond with several things "that the WWW is NOT good for." The reason for this question was to identify beliefs or attitudes that may lead to media avoidance behavior (McLeod & Becker, 1974; Becker, 1979). Students may choose not to use the WWW either because of a perceived negative attribute that it is believed to possess or because they believe that it does not possess any useful or positively valued attributes. Responses to the question of what the WWW is not good for yielded 78 responses. These were content-analyzed by the researcher and placed into categories. By far the most common response made reference to pornographic material (n = 22), followed by hacking/illegal use (n = 10), and bomb/gun/drug-making info (n = 5). Others expressed concern about offensive people ("horney" people, "perverts," and "weirdoes") (n = 4), privacy of personal info (n = 3), finding incorrect or invalid information (n = 3), addiction (n = 2), satanic info (n = 2), hate nets (n = 2), what "little kids" can find (n = 2), and speed of downloading (n = 2).

The "use" categories generated by responses to the open-ended question were used to construct a short survey instrument that was administered via computer at four schools in the same public school district. Computer laboratories in the media centers of two middle schools and two high schools were made to display the short survey form as the default home page whenever the WWW browser (Netscape Navigator or Netscape Communicator) was launched. Respondents to the short survey (n = 228) were asked their grade, gender, how much they used the WWW, and most importantly, the purpose for using the WWW at this time. The choices available included the leading "use" categories as determined by the open-ended survey. In addition, the student was given the option to "write in" another reason for using the WWW if the choices presented were not acceptable.

After the student completed the short survey, he or she was presented with a link allowing continuation on to a longer survey or the option to proceed to use the WWW. The students who continued on to the longer survey (n = 42, 18% of those who completed the short survey) were informed that the 37-item survey would take approximately 10 minutes to complete. The longer survey asked more detailed questions about the students’ perceptions of the WWW, typical usage patterns, attitudes towards the WWW, and demographic information. Thirteen items measured computer and other media use, 20 items addressed questions about gratifications sought from the WWW, and four questions collected demographic data. The last item on this survey instrument allowed the student to submit his or her name and the name of his or her school if interested in providing a face-to-face interview. Of the students who completed the long survey, 19 (45%) expressed willingness to provide an interview. These students were presented with a parental consent form that was to be signed by a parent or guardian and returned to school. Of those who were given the parental consent form, six students returned the form with parental consent provided. In early June these students were interviewed in the schools’ media centers for approximately 20 minutes, and the interviews were recorded. In addition to reviewing the questions asked by the long survey, the interview provided an opportunity to ask additional questions as directed by the students’ responses during the course of the interview.

In addition to survey research and interviews, the pilot study made use of passive data collection and content analysis of WWW sites visited by students. This content analysis of sites visited by students during the time that the survey was in effect allowed for a comparison of reported usage with actual usage. This information was compared to the students’ reported uses of the WWW to reveal not only how students use the WWW, but also how they perceive their use of the WWW.

At the beginning of the data collection phase the cache files on the computers in the schools’ open labs were cleared. At the end of the collection period the cache files were copied to a disk and the data prepared for analysis. A total of 47,477 URLs were collected from the 31 Macintosh™ and Windows™ personal computers on which the survey instrument had been installed. In order to facilitate content analysis of the sites visited, URLs ending with .gif and .jpg were first stripped from the list. Systematic sampling was used to reduce the remaining list of URLs to a smaller and more manageable number. To accomplish this task, a UNIX grep script was written which allowed for every nth URL to be sorted out starting with the URL determined by a random number generator. The subsequent 200 sites were then collected into a single WWW page and three educators/media specialists from a nearby school district were asked to analyze a portion of the sites. The sites were assigned a "uses" category and rated for education and entertainment value on a scale of 1-5. The educators were instructed to look at each site with consideration for the grade level of the students being studied. The unit of analysis for this step was the individual WWW page. However, coders were given the flexibility to follow a link forward or backward in order to get a clearer picture of the contents of the page.

In preparation for determining the educational value of the sites the coders were asked to visit and explore the EvalWEB tutorial developed by Schinker. This web-based resource "is a tutorial on evaluating web pages to determine their suitability for use as research sources for middle and high school research" (Schinker, 1997). A meeting was held with each of the coders to discuss criteria to be applied to the WWW sites and to answer questions about the coding process. Intercoder reliability was tested by first having each coder rate a sample of 20 sites and comparing results on the five-step scale for educational value. Alpha reliability for the educational value assigned by the three coders was .93. Once intercoder reliability was established at an adequate level, the 200 sites coded by the educational media specialists were analyzed for educational value and use category.

Survey Analysis Design for This Study

Several different sample survey methods are employed in the design of this study. The first phase of survey data collection was actually a part of the pilot study. As described above, open-ended questions addressing the possible uses of the WWW were administered to a small group of students in two public schools. Responses to these questions were then fashioned into WWW "use" statements that became part of the paper survey instrument. A principal components analysis of responses to the paper survey was conducted and the resulting "use" categories were then used to construct the computer-administered survey instrument.

The two primary survey instruments employed in the current study will be referred to as the "paper survey" and the "computer survey." The paper survey is a 75-item survey instrument that was administered to students in their classrooms at selected Colorado public middle schools and high schools. The paper survey contains sections designed to measure the students’: 1) affinity for the WWW, 2) assessment of the value of the WWW for various purposes, 3) skill level for computer and WWW use, 4) use of the WWW, 5) avoidance of the WWW, and 6) demographics (see Appendix C). Following this, the computer survey was administered to the students at the time and place of their access to the WWW—specifically the school’s media center or library. The computer survey is comprised of just four questions: grade, gender, how much the student uses the WWW, and the student’s purpose for using the WWW at this particular time (see Appendix D).

Development of the Survey Instruments

As Becker (1979) noted, it is important that the measures be presented in the vernacular of the targeted respondents in order to prevent premature rejection. With this in mind, and in keeping with the methodology employed by Greenberg (1974), the first step in the process was to identify, in their own words, student motives for using the WWW.

The Paper Survey Instrument

Use and avoidance statements. Based on categories generated from both the responses to the open-ended question and the fill-in-the-blank responses to the computer survey questionnaire in the pilot study, a list of 40 Likert-type "uses" statements and 10 "avoidance" statements were generated for the paper survey instrument. While most of the use statements could be categorized using traditional dimensions from the uses and gratifications literature, some of the items were unique to the WWW. The 40 statements are as follows:

Education/Learning

Surveillance

Communication

Convenience

Entertainment

Diversion/Pass Time

Acquisition

Other

The ten avoidance statements—statements that might explain why students may choose to not use the WWW or may choose to use some other media—are as follows:

For each statement a five-point Likert scale allowed the student to choose a response that most accurately reflected the student’s use of the WWW at school: strongly disagree, disagree, neutral, agree, strongly agree.

Skill and using computers and the WWW. Also included in the paper survey were eight items to determine the skill level of the respondent. One item asked for the student’s self-appraisal of his or her overall skill at using computers, three items addressed length of time and frequency for using the WWW, and four items were Likert-scale, agree/disagree statements about skill level that were taken from the Novak and Hoffman (1997) survey (alpha=.86) who reported to have taken them from the 7th GVU survey.

Affinity for the WWW. Because a person’s affinity for a medium may be correlated with how he or she uses it (Abelman, 1988; Rubin, 1979; see also Greenberg, 1974; Wenner & Dennehy, 1993), five Likert-type items designed to measure the affinity that the respondent feels for the WWW were included. These items, modified from those used by Abelman, are as follows: "Using the WWW is one of the most important things that I do," "When the computer lab is down—making the WWW unavailable—I am very disappointed," "Using the WWW is very important to me," "I could easily do without the WWW for several days," and, "I would feel lost without the WWW." A five-step scale ranging from "strongly disagree" to "strongly agree" was used and the polarity was reversed for the fourth statement. Lichtenstein and Rosenfeld (1983) hypothesized that "frequent users and fans of a particular medium should perceive the gratifications provided by a particular medium differently than people who are infrequent users and who are not fans of that medium" (p. 101). While their research did not support their hypothesis, studies that have researched children’s attitudes towards and use of the media (e.g., Greenberg, 1974; Rubin, 1979) suggest that this variable may be appropriately applied to this population.

Other questions. Additional questions asked about the locations from which the respondent accesses the WWW. The choices presented included home, school, a friend’s house, the public library, and, other. Respondents were also asked what they consider to be their "favorite" and "most useful" WWW sites. Three items asked the respondents to rate the WWW as a source of information, a source of entertainment, and a means of communication. And another item asked them to compare the WWW to the traditional media, print or audio/visual. This item was included because of research that suggests that children’s perceptions of print and TV media are related to learning from them (Salomon, 1984). Schramm, Lyle, and Parker (1961) observed, "The act of going purposefully to the media for useful information is something that is learned chiefly in school, and more often sends the child to print rather than to the audiovisual media" (p. 75, emphasis in the original).

Demographic questions. Finally, five items sought demographic information: age, grade in school, grade point average (GPA), gender, and ethnicity. According to Greenberg and Heeter (1983), "prior research…indicates that two critical differentiators of mass media behaviors among young people are sex and age. Age has typically been used as an index of developmental differences" (p. 306). The present study sought to explore whether WWW use is differentiated by age and grade of the students. A question asking the student’s grade point average was included at the request of the Colorado Department of Education. Descriptive statistics and correlation analysis using GPA as a variable revealed possibilities for future study. The ethnicity of the respondents was requested to provided information to indicate whether our sample was representative of the school districts that were selected.

The relationship of gender to adolescent media use has been examined in prior studies (e.g., Greenberg & Heeter, 1983; Kubey & Larson, 1990; Lubans, 1998). Others have reported on the inequity of male to female attitudes towards and use of computer technology (Canada & Brusca, 1993; Chen, 1984; Krendl, Broihier & Fleetwood, 1989). Whitley (1997) found small but significant differences in a meta-analysis of studies exploring gender differences in computer-related attitudes and behavior. Interestingly, Whitley observed that the largest differences were found among high school students.

Pre-testing the Survey Instrument

The paper survey instrument was tested with 27 seventh and eighth grade students in September of 1998 to identify items that may cause confusion or misunderstanding. Students voiced concern that repeating the phrase, "I use the WWW…" for each of the 40 items created a sense of fatigue as they read the same words repeatedly. It was suggested that these words could be replaced by an ellipsis and this change was made in the final version of the survey instrument. Another reason for pretesting was to measure the time required to complete the survey. The time required to complete the survey varied from as little as six minutes for students who did not use the WWW at school, to as long as 20 minutes for a few students. It was observed that approximately 80% of the student were able to complete the entire survey in 12 to 15 minutes. This allowed the researcher to give an accurate estimate of time required when approaching school districts for permission to use classroom time to administer the survey instrument.

Computer-based Survey Design

The second phase of data collection made use of a four-item computer-administered survey instrument that was presented to students at the place and time that they choose to use the WWW in the schools’ media centers. This survey was intentionally kept very short in order to prevent frustration by students and a perception of "time-off-task" that may have jeopardized the support of school administrators.

The Computer Survey Instrument

Items included in this second and shorter survey instrument included: grade, gender, average amount of time that the respondent uses the WWW on a weekly basis, and, the respondent’s purpose for using the WWW at this particular time. The first three questions were presented in a multiple choice format and were selected by clicking on the appropriate radio button. The use statements were presented to the respondents as a multiple-choice, pull-down menu from which the students could choose. The seven items so identified were described in terms that were designed to be easily understood by the students. The seven uses for the WWW as presented in the computer-administered survey were: "for research and learning," "to communicate with other people," "for access to material otherwise unavailable," "to find something fun or exciting," "for something to do when I’m bored," "for sports and game information," and, "for shopping and consumer information." As an option to the seven use statements presented, the student could select "other" and use a text-box to enter a use that better described his or her purpose for using the WWW at that particular time. The phrasing of the question, "What is your purpose for using the WWW at this time?," was designed to measure gratifications sought and the "behavioral intention" (Palmgreen & Rayburn, 1982) of the student.

This WWW-based fill-out form was installed as the default home page on computers with WWW access in each school’s media center and adjacent computer laboratories. When a student launched the WWW-browsing software: i.e., Netscape Navigator, Netscape Communicator, or Internet Explorer, the default survey page loaded and the student was presented with a request to respond to the four-item questionnaire before continuing (see Appendix D). This survey instrument was installed for a period of between one and three weeks at each school with the goal of receiving approximately 100 responses from each school. The responses to the survey were directed to the researcher’s email account via a common-gateway-interface (cgi) email program. This allowed the researcher to track the frequency of responses as they were received and allowed for responses to be screened for date and time received.

Using a WWW-based survey instrument to survey WWW users is not new. The Graphic, Visualization, & Usability Center’s WWW User Surveys have used this approach since January of 1994 when they pioneered the concept (GVU’s 8th WWW User Survey, 1997). Information about creating and using WWW-based survey forms is now available (see e.g., Turner & Turner, 1998). Articles have appeared that address the ethical implications of "collecting social science data in cyberspace" (Thomas, 1996), and studies that have made use of these techniques are now beginning to appear in the literature (Comley, 1996; Kehoe & Pitkow, 1996; Pitkow & Recker, 1994; and Rosenfeld, Booth-Kewley & Edwards, 1993). One study used WWW-based surveying, passive data collection from the server, and email surveying to assess K-12 programs that used media technology to teach children with disabilities (Gold & Ethier, 1997). Smith (1997) outlined her use of email and WWW-based surveying of the Web presence provider community and reviewed the methodological issues surrounding this new research tool compared and contrasted traditional "snail mail" with electronic mail and WWW-based fill-out forms. And Coomber (1997) described Internet-based survey research among an unusual population with great concern for privacy—drug dealers. Other researchers have noted similar advantages in the collection of data about sensitive or illegal activity when computer-administered surveys were used (e.g., Turner, Rogers, Lindberg, Pleck, & Sonenstein, 1998).

Concerns Related to Computer Survey Methodology

However, using the WWW to collect data on WWW use can present some methodological problems (Coomber, 1997; Smith, 1997; Reynolds, Walther, Gurack & Eadie, 1998). Smith noted, "perhaps the most critical problem with Internet-based research is the practical impossibility of probability sampling" (n.p.). As the GVU survey indicated, the method is non-probabilistic and allows for self-selection by the sample population, resulting in a sample that favors those more experienced and those who are more frequent users of the WWW. Because this phase of the study was designed to survey students who already use the WWW, this issue was deemed to be less critical than it would have been had the researcher targeted students at large. With that understood, there remains concern over the self-selection process. Only students who used the WWW at one of the computers in the school’s media center during the several-week period of time that the survey was in effect were sampled. Of these students, those who used the WWW more frequently were presented with more opportunities to respond to the survey and some may have responded repeatedly. Another issue noted by Smith, and one that is of special concern to this researcher, is that the response rate is unavailable. While each students who sat down at a computer in the schools’ media centers and launched the WWW browser was presented with the survey form, there is no data to indicate what percentage of the students completed the form and what percentage bypassed it.

Because a WWW-based survey instrument was available to anyone on the WWW, some concern over the security of the survey instrument and opportunity for abuse by WWW surfers other than students from the schools in question was warranted. While it was doubtful that someone would stumble across the survey instrument accidentally, that possibility did exist. More likely perhaps would be a student telling friends about the survey with the goal of "stuffing" the ballot box, so to speak, with invalid data. In an effort to minimize this possibility, careful note was made as to the day and time of each email entry from the forms. The email entries were carefully screened so that entries received outside of the normal school day schedule, for example Monday through Friday, approximately 8 a.m. to 4 p.m., could be discarded before analysis.

The response to the Web-based survey was anonymous and no attempt was made to link responses to the earlier paper-based survey instrument with the computer-generated survey data. Nor could any link be made with the passive data, the third phase of data collection, that was collected from the computers’ cache files. However, the assumption is that each of the survey methods employed reflected the general student body and the attitudes and behaviors of the students who uses the WWW at school.

Passive Data Collection Design for This Study

December (1996) and Newhagen and Rafaeli (1996) recognized the fact that the Internet provides excellent opportunities for data collection. As Rafaeli noted, any social scientist who has looked at an Internet server must be struck by the research possibilities present in the data that is passing through that computer (p. 6). In order to take advantage of this unique feature of the WWW, the design of this study calls for passive data collection to follow the survey research. Both Netscape Navigator and Microsoft’s Internet Explorer browsing software generate a cache or "global history" file that resides on the user’s hard drive and which retains a list of addresses (Uniform Resource Locators or URLs) of WWW sites last visited. This list of URLs listing WWW pages and graphics visited most recently is extensive and can be thousands of sites long.

At the beginning of the data collection period the cache files on the computers in the schools’ open labs were deleted. At the end of the collection period the cache files were retrieved and the data prepared for analysis. First, the number of occurrences of web sites from the five commonly accessed domains: edu, com, gov, net, and org, were noted. Next, the lists of URLs were processed using a UNIX grep script to delete URLs that were locators for graphic files. These URLs were identified as those ending in gif, jpg, or png. The justification for this step was that graphically rich WWW sites generate several listings in the cache file, one for the page itself and one for each graphic contained on that page. Since the unit of analysis is the complete WWW page, discarding URLs representing images contained on a page would more accurately reflect the frequency with which a page was accessed by students. Additional script lines removed URLs that linked to banner advertisements. Companies such as valueclick, adforce, linkexchange, and adfinity place banner advertisements on commercial WWW pages of their clients. These links, when followed, displayed banner ads alone and were of no value in determining the origin or content of the WWW pages actually viewed by students.

Once graphic files and links to banner ads were discarded, the remaining lists of URLs from each of the 10 schools were randomly reduced to 70 sites using the skip-interval method. These WWW pages were visited and reviewed by the researcher to determine if any sites have been removed or were otherwise unavailable. The first 50 accessible WWW pages from each list of 70 were then combined to create a list with a total of 500 URLs. These URLs were assembled as a list of hyperlinks on a single WWW page which was then made available to the coders for the purpose of content analysis.

Privacy issues associated with passive data collection (see e.g., Thomas, 1996) are a concern. However, the anonymity afforded by this approach insures that no individual student can be linked with any particular bit of information. Also, students and parents had already been informed by the various school districts that their use of the WWW was a privilege that would be subject to periodic monitoring. These two points taken together were sufficient to override concerns about the collection and use of computer data in this particular research project.

Content Analysis Design for This Study

In order to better understand the educational value and potential use of the WWW pages actually visited by students, URLs representing WWW pages visited by students during the weeks surrounding the computer-administered survey were collected and content analysis of a sample of these pages was performed by professionally trained educators and media specialists.

Two goals were addressed by this phase of the research: 1) to apply an objective and reliable standard to a sample of WWW pages in order to better understand the educational value of the pages that were visited by students, and 2) to compare students’ stated purposes for using the WWW with coders’ analysis of the most likely use that visited WWW pages might serve. Following Stempel’s (1989) procedure (pp. 127-133), this phase of data collection began with the selection of the unit of analysis. Following this is a discussion of the category construction and an analysis of the reliability of coding.

The Unit of Analysis

The unit of analysis is the WWW page. A WWW page is defined by the content that appears on screen using standard browsing software. This content may be made up of text, graphics, animations, audio, and/or video. A page may be smaller than the area displayed by a 15-inch computer monitor or may be many screens long. Coders were given freedom to browse forwards or backwards from the initial page in order to better understand the context in which the page was presented.

Category Construction

Determining the categories for the analysis of these WWW pages was aided by the adoption of procedures designed by an educational media specialist for the purpose of training middle and high school students how to evaluate WWW resources. In preparation for determining the educational value of the pages the coders were asked to visit and explore the EvalWEB tutorial developed by Schinker. This web-based resource "is a tutorial on evaluating web pages to determine their suitability for use as research sources for middle and high school research" (Schinker, 1997). Some of the categories created by Schinker and used for evaluation in this study include: address, content, author, revision date, and links. A meeting was held with each of the coders to discuss criteria to be applied to the WWW sites and to answer questions about the coding process.

Reliability Analysis

The content analysis was performed by technology specialists from another public school district. For each WWW page a coder assigned a "use" category and a rating for "suitability as a source for academic research" on a scale of 1-3: 1 = not suitable, 2 = questionable, 3 = suitable. The educators were instructed to look at each site with consideration given to the grade level of the student participants. Also, for each site a use category was assigned by a coder. The use categories were the same as those presented to the students in the computer-administered survey: "for research and learning," "to communicate with other people," "for access to material otherwise unavailable," "to find something fun or exciting," "for something to do when I’m bored," "for sports and game information," and, "for shopping and consumer information."

Intercoder reliability was tested by first having each coder rate a sample of 50 sites and comparing results on the three-step scale for educational suitability. Once intercoder reliability was established at an adequate level, the 500 sites coded by the educational media specialists were analyzed for suitability and use category. This information, when compared to the students’ reported uses of the WWW, provided a more revealing portrait of how students use the WWW in school. In addition, the comparison of reported use with actual use provided information about how students perceive their own use of the WWW.

Variables

Dependent Variables

The dependent variable defined in this study is the student’s identified purpose for using the WWW.

Independent Variables

Independent variables defined in this study include: affinity for the WWW, skill level at using the WWW, amount of time spent using the WWW, and the demographic variables age/grade, and gender.

Statistical Procedures

As an exploratory study that addresses for the first time the uses of the WWW by school students in a public school setting, the research makes extensive use of descriptive statistics. More advanced statistical procedures employed include: principal components analysis, reliability analysis, correlation analysis, and multiple regression analysis.

Table 3.2

Statistical Procedures and Variables for Each Phase of Research

  Statistics Paper Survey Computer Survey Content Analysis
  Descriptive
  • Age/Grade of users
  • GPA of users
  • Gender of users
  • Ethnicity of users
  • Affinity towards the WWW
  • Skill level of users
  • WWW as a source of information
  • WWW as a source of entertainment
  • WWW as a source of communication
  • WWW more like Print or AV
  • "Favorite" and "Most Useful" sites
  • Location where WWW is accessed
  • Uses of the WWW
  • Reasons for avoiding the WWW
  • Grade of users
  • Gender of users
  • Average amount of time spent using WWW
  • Use categories—frequency of occurrence
  • Crosstabulation of use categories with grade, gender, and amount of time spent using the WWW
  • Domain name—frequency of occurrence
  • Suitability for educational research—frequency as determined by coders
  • Use category—frequency as determined by coders
  Correlation
  • Affinity, skill level, use categories
   
  Principal Components Analysis
  • Use statements
   
  Reliability Analysis
  • Use categories and other scales
 
  • Intercoder agreement
  t test
  • Gender with use categories
   
  Multiple Regression
  • Age, GPA, gender, affinity, and skill level with use categories
   

Principal Components Analysis

Following the methodology employed by numerous uses and gratifications researchers (e.g., Greenberg, 1974; Becker, 1979; Rubin, 1981a; Finn & Gorr, 1988; Vincent & Basil, 1997) exploratory principal components analysis was employed to identify the factors that explain the leading uses of the WWW by students in an educational context.

As described earlier, the use statements to be factor analyzed were obtained by soliciting free responses from research subjects, including "other" use statements gathered during the pilot study. Data analysis was performed on an incomplete data set of 477 responses to the paper survey instrument using Statistical Package for the Social Sciences (SPSS) version 8.0 (SPSS, 1997).

Applying criteria recommended by Stevens (1996) resulted in an eight-factor solution which accounted for 56% of the total variance. Because the last factor was composed of a single item it was deleted resulting in seven "use" statements: 1) "for research and learning," 2) "to communicate with other people," 3) "for access to material otherwise unavailable," 4) "to find something fun or exciting," 5) "for something to do when I’m bored," 6) "for sports and game information," and, 7) "for shopping and consumer information." These seven use statements were then incorporated into the computer-administered survey instrument.

Reliability Analysis

When multiple items are used to generate a scale, reliability analysis is employed to determine the degree to which the various items contribute to the construct. Alpha reliability was computed for the various use factors, and for the items that make up the affinity and skill scales. The content analysis phase of the research also made use of reliability analysis. Intercoder reliability was addressed by computing the alpha reliability for the "suitability" rating generated by the two coders.

Correlation Analysis

Correlation analysis explores the strength of the relationship between variables. The Pearson product-moment correlation was used to determine whether associations exist between the various independent variables and student uses of the WWW.

Multiple Regression Analysis

Multiple regression analysis allows the researcher to predict the occurrence of the dependent variable(s) by examining a set of independent or predictor variables (Stevens, 1996). In this study the student’s stated purpose for using the WWW was examined with regard to the following predictor variables: affinity for the WWW, skill level at using the WWW, amount of time spent using the WWW, age/grade, and gender.

The Population

The population for this study was comprised of middle-school and high-school students at selected public schools in five districts in the state of Colorado. The districts were selected in consultation with the Colorado Department of Education to reflect a cross-section of schools in urban and rural settings that have Internet access. The school districts selected were: Adams-Arapaho 28J in Denver, Douglas County RE-1 in Castle Rock, Academy #20 in Colorado Springs, Pueblo #60 in Pueblo, and Garfield RE-2 in Rifle. School selection took into account the size and demographic and socio-economic makeup of each school in order to create a pool that most accurately reflected the state of Colorado. Of the schools selected, only the schools in Academy #20 use filtering software. All districts employ an Acceptable Use Policy (AUP) and this document must be signed by both student and parent or guardian before the student is allowed access to the Internet via computers at school.

The ethnic makeup of the five districts combined is as follows: American Indian, 1%; Asian, 3%; Black, 9%; Hispanic, 18%; and White, 69%. These numbers are consistent with the representation of ethnic groups in public schools throughout the state of Colorado: American Indian, 1%; Asian, 3%; Black, 5%; Hispanic, 19%; and White, 71%. The ethnicity of respondents to the paper survey is as follows: American Indian, 2%; Asian, 3%; Black, 9%; Hispanic, 16%; White, 69%; and other, 2%. As you can see, the ethnic makeup of the sample is consistent with the public school population in the state of Colorado and compares very favorably with the makeup of the combined percentages of the five districts that were selected for this study.

Colorado’s K-12 public school enrollment in 1997 was 688,438. In 1998 the ratio of students per instructional multimedia computer was 13, which was also the nationwide average. The number of students per computer in Colorado public school media centers was 90—substantially better than the nationwide average of 114. Fifty-nine percent of Colorado public school classroom had Internet access and 91% of schools had Internet access in 1998. Again, Colorado exceeded the national average of 44% and 85% respectively (Technology Counts ‘98). The range for expenditure per pupil at the five districts in 1996-1997 was $5,260 to $7,817, with the mean expenditure per pupil at $6,809.60, slightly below the $7,063 mean for the state but considerably more than the mean expenditure per pupil for students in all 50 states at $6,330 (in 1996 constant dollars) in 1993-1994 as reported by the US Department of Education (National Center for Educational Statistics, 1998a).

Aside from asking where students accessed the WWW, this study was not designed to address student use of a computer at home. It is interesting to note however that as of 1998 Colorado ranked third, behind only Alaska and Utah, with 52% of homes having a personal computer as reported by the US Department of Commerce’s National Telecommunications and Information Administration (NTIA) report, Falling Through the Net II (Falling Through the Net II: Accompanying Graphs and Charts, 1998). Using Census Bureau data collected and compiled in late 1997, the NTIA report found that in the three years since the first "Falling Through the Net" report, PC ownership nationwide has increased 52%, modem ownership has gone up 139%, and access to electronic mail has increased by 397%. However, much of the gain has only served to widen the gap between groups that have access and those who do not. Nationwide, the poor and blacks and Hispanics lag even further behind than they did three years ago. White households are more than twice as likely to have a computer as black or Hispanic households. There is also a disparity when one considers geographical boundaries with the West leading other regions of the country, and central cities and rural areas lagging behind urban areas (Falling through the Net II, 1998).

As indicated earlier, Colorado exceeds the national average in terms of PC ownership in the home. One media center director at an upscale middle school reported that an informal survey of students revealed that nearly 95% of students had access to a computer at home. Of Colorado households with computers, it is difficult to determine how many computers have modems and are connected to an online service provider. Even if Colorado is only typical of Western region states, it can be assumed that approximately 22% of homes have a computer with online service using 1997 data reported by the NTIA. Students who have access to a computer at home are more likely to be comfortable with computer technology and more at ease with computer software. It is also anticipated that students who access the WWW at home are likely to differ from students whose only access is at a public institutions, e.g., school or the local public library.

In addition to WWW access at school and from home, some students gain access to the WWW at their local public library. A report released by the American Library Association found that in 1998 73% of the nation’s public libraries offer basic Internet access to the public, up from 28% in 1996 (Bertot & McClure, 1998).

The Sample

The participants for this study were selected using two different approaches. For the first survey, which was administered on paper, a stratified convenience sample was employed. At one middle school and one high school in each district a class representing each grade (sixth, seventh, eighth, ninth, tenth, eleventh, and twelfth) was selected to take the paper survey. These classes were selected with consideration given to avoiding groups that had been preselected for either high or low achievement. The paper survey was administered during the months of October and November of 1998 and January and February of 1999.

The size of the sample was determined by consulting Kraemer and Thieman’s (1987) guide and was based on a desired significance level of 0.05 with 90% power and a critical effect size of 0.15. It was decided to administer the paper survey to one "classroom" of students from each grade level in the 10 schools that had been selected. With an average size of approximately 25 students per classroom, the total number of students taking the paper survey would be approximately 800.

The second survey was administered electronically at the computer. Students attending middle and high schools in these districts have access to the WWW using computers available in classrooms and in the schools’ media centers. However, not all students have parental permission to access the WWW. Only students who have been granted parental permission and who have signed and submitted the required forms to their local school administrators are permitted to access the computer-administered survey. Of these, participation in the second phase of the survey was voluntary and by self-selection. The survey was installed as the default home page for a period of time sufficient to gather approximately 100 responses from each school.

Those who agree to participate in this phase of data collection when confronted by the survey screen were able to response to the online survey instrument as often as they liked. While this skewed the computer-administered survey to heavier users, several advantages were realized. First, the respondents were given the opportunity to reflect immediately on his or her purpose for using the WWW thus eliminating the need for recall. As uses and gratifications researchers who have used the Experience Sampling Method have argued (see e.g., Kubey & Larson, 1990), the immediate reporting of cognitive or affective states reduces reporting error. Also, research indicates that adolescents tend to respond more honestly when the survey is administered by computer (Turner, et al., 1998). And finally, each use of the WWW, as defined by a session at the computer, was counted separately. Although heavy users accounted for a larger percentage of the total responses, they were likewise responsible for a proportionately larger percentage of the URLs that were captured by the global history file. The computer survey was administered between November of 1998 and February of 1999. The global history file of sites visited was collected at the conclusion of the computer survey.

Anonymity and confidentiality of participants was assured at every stage of data collection. The paper survey forms were administered in the classroom by either the researcher or a representative of the school. The computer survey were administered by computer and had no mechanism by which to link a response with a particular student. And the collection of URLs from the computers likewise included no mechanism to allow for identification with any particular student.

Summary

The methodology outlined here includes interviews, paper and computer-administered surveys, as well as content analysis of WWW sites. In order to increase the richness of the data, both active and passive data collection, utilizing various methodologies, are employed. Quantitative and qualitative approaches together enable a multifaceted view of the complex phenomenon of WWW use in a public school setting.

 

Chapter 1 | Chapter 2 | Chapter 3 | Chapter 4 | Chapter 5 | References | Appendix