Session 3

How can new materials be evaluated to make sure they are accomplishing what is intended of them?

Evaluation is often the weak link in curriculum development projects like developing web-based teaching modules. There are three things to remember about evaluation:

You have to do it.

There are lots of ways to do it.

What we need is careful research and evaluation to know what works, when, and why.

We need to research and reflect on how students learn, access, and use information and ideas on the web. We need this information to make decisions about ways we can use the web to help students to think geographically.

Steps to Developing an Evaluation

Step 1: Develop research and evaluation questions.

There are at least five types (categories?) of research questions:

Causal, e.g., do geography students learn better using Web-based materials as opposed to other methods of instruction?

Non-causal, what geographic knowledge is used in everyday life?

Non-causal policy questions, e.g., what role do writing tutorials play in helping students to think geographically?

Non-causal evaluation questions, e.g.,what are the characteristics of web sites that promote geographic learning?

Non-causal management questions, e.g., what is the cost effectiveness of placing geography courses on the web?

Step 2: Use a wide variety of data collection methods to answer each of your evaluation questions.

There are lots of ways to collect and analyze data. You can collect quantitative data, qualitative data or both. Here are some ideas about models of qualitative research.

Anecdotal, personal accounts journal entries, content analysis

Structured observations, tallies and statistical analysis, codes for behavior, recording of observations

Case study, multiple or single, qualitative and quantitative data, structured interviews, document analysis, video-taping, model building

Ethnography, participant observation, interviews

Cognitive studies, think-aloud procedures, content analysis of written, verbal protocals, use of standard problems to elicit responses, tests, expert vs novice studies

There are a number of evaluation tools which can be used.