Civics, as a topic of study, prompts the adoption of research techniques that can be used on any range of topics. However, the techniques chosen also have to take into account the fact that civics, as a curricular subject, and unlike mathematics or even science, is hard to define. It was this consideration that pushed the Civic Education Study steering committee into adopting research methods and perspectives more commonly found within the broad and amorphous area of “qualitative research.” In doing so, they organized a type of study that offers a model for future studies of international achievement.In this regard, the results of the TIMSS case study and video project point to the likelihood that future such studies will require more complex research approaches than previously used. Even though the presence of “authoritative intended” curricula for subjects like mathematics and science seemingly “serve as reasonable starting points” for research in these fields, a growing number of studies suggests that cultural considerations also need to be taken into account (see, for example, Lee, Graham & Stevenson, 1996; Tsuchida & Lewis, 1996; Stigler, Fernandez & Yoshida, 1996). As international studies of achievement grow more complex, and the methods used to collect and analyze data more refined, the impact of various cultural domains (school culture, regional cultural and/or national culture) will play an increasingly important role in the entire research process. To fully understand how achievement is contextualized in a given nation requires not only sets of complex data but also a range of analytical methods that draw out conflicting views, contested areas and shared beliefs.The last decade has seen the continued call for more culturally sensitive (appropriate) analysis and interpretation of international educational data, particularly achievement data. There has been a growing expectation that researchers who use such data sets will be either experts in or have access to expert advice in the nations selected for comparison. However, this trend alone will not affect the basic collection of the data. If there is to be a significant change in the type of data collected — a change that will allow far more sophisticated qualitative analysis to be conducted — then qualitative data collection and analysis cannot be simply relegated to the “descriptive” mode. Both the Civic Education Study and TIMSS have demonstrated that rich qualitative data, analyzed using methods derived from current qualitative educational research, can have dramatic effects on the orientation of the research process.The use to which data gathered by TIMSS have been put also offers insights for future projects. One alarming trend has been the tendency for media and policy groups to use TIMSS’s qualitative data as evidence of the “real” conditions in a given country. This is particularly true of the video study conducted by Stigler. Demonstration clips of the videos have been widely shown on the implicit assumption that educators and policy-makers can directly use this type of “data” even when it has not been “interpreted” by scholars. While the aim of creating straightforward data free of disciplinary jargon is a good one, qualitative data are perhaps more easily misinterpreted than quantitative data. The impact of a five-minute video clip or an evocative quote from an interview transcript cannot be denied, but this impact may be misleading. All data, qualitative or quantitative, are representations of reality. Experienced ethnographers know that five minutes after the camera or tape-recorder is turned off, drastically different events may occur or conflicting statements may be made. The presentation of qualitative data, then, must be carefully framed within an explicit outline of the methods used to collect and analyze it.The greatest danger lies in assuming that all that can be done with qualitative data is to provide the cultural “context” for quantitative analyses. Unfortunately, this did occur with TIMSS. In contrast, the Civic Education Study researchers used the hypothesis-generating function of the qualitative work to challenge original hypotheses and to dramatically re-cast the project. This continued use of reflective analysis changed the Civic Education Study from a more traditional “input-output” study to one that was indeed truly process oriented. Moreover, because the Civic Education case studies were conducted well in advance of the survey and testing aspects, the Civic Education Study researchers were able to capitalize on the qualitative data in a way that the TIMSS researchers could not.Comparative education scholars have long been aware of the different cultural values attached to school, learning and teaching in different nations or among different groups within a given nation. However, recent international achievement studies have rarely attempted to analyze systematically the cultural components of achievement. Over 30 years ago, Jones (1971) hailed the first International Mathematics Study (FIMS), but nevertheless stressed that “because comparative education is concerned with cross-national or cross-cultural variability, one of its tasks ought to be the advancing of hypotheses which can be tested in either established or novel ways” (p. 153). The ethnographic case studies of the Civic Education Study offer established ways to use qualitative data to generate and test hypotheses. These approaches have been used for at least two decades in some areas of qualitative studies, but they present, for the field of comparative education, novel ways to think about the problems of comparison.The politically contested nature of civic education prompted the Civic Education Study’s NPRs and steering committee to alter significantly the research process that has traditionally been applied to cross-national studies of achievement. Topics like democracy and citizenship evoke very different associations in different countries, and even between various groups within a given nation, a fact that fortunately clarifies rather than obscures the impact of culture on curriculum, teaching and expectations for competence. The experience of the civic education researchers suggests that the use of qualitative data for hypothesis generation and the use of a reflective research process that links qualitative and quantitative data have significant promise for the study of achievement in any subject. In addition, continued recourse to highly integrated studies that utilize a range of research methodologies promises to bring comparative studies of achievement to a new and higher level of usefulness.The Civic Education researchers incorporated a range of qualitative techniques, innovative uses of technology, and extensive literature reviews, and were highly sensitive to the impact of methodological decisions on the kinds of data collected and the interpretations arrived at. Despite the tremendous amount of material reviewed and summarized, as well as the incredible cultural, social and political diversity represented by the participating nations, the work of these researchers not only is providing crucial insights into the actualization of qualitative methods on a large scale but also is offering “comparativists” and “qualitative researchers” alike a wealth of information.
