Reviews of teaching methods - what are the main problems identified?

Pedagogy and Modern teaching methods

Bugalaeva Ayman, Kaniyeva Dilnaz

72 secondary General Education school-Lyceum

Reviews of teaching methods - what are the main problems identified?

Abstract. The purpose of this study is to identify and discuss issues related to the tension between contextuality and generalization that are periodically identified over time in research reviews of teaching methods. Three main problems were identified that have faced research over time: 1) the abundance of constraints, 2) the need for highly qualified teachers, and 3) the gap between research and practice. These three issues are said to reflect the contradictions in the original research. The implications of these findings are discussed in the article.

Keywords: internal and external validity, moderating factors, overview, research-practice gap, review, teaching methods.

Given the global emphasis on education as a pathway to national and individual success, it is not surprising that a wealth of research has been done on which teaching methods enable education to achieve its goals. While education concerns many areas such as education policy, educational organization, financial systems and school governance, there seems to be widespread agreement that teaching is ultimately a key factor in the success of educational systems (e.g. Barber & Mourshed, 2007; Hargreaves and Fullan, 2012; Hattie, 2003; OECD, 2016; Stigler and Hebert, 2009).

How learning should be best organized has been identified in a large number of studies involving various theoretical assumptions (Hattie, 2009). Consequently, reviews of the effectiveness or appropriateness of teaching methods are becoming more readily available. Creating such reviews is a logical way to combine findings and insights from different studies. Systematic research reviews can contribute in many ways to knowledge that can be used in research, practice, and policy decisions (see Gough, Thomas, & Oliver, 2012). However, baseline research results often show mixed and sometimes even conflicting results due to many factors (eg, Shute, 2008). Teaching methods and interventions operate in complex systems, and their effects depend on various factors in context, as well as on how and by whom they are implemented and applied (see Cartwright & Hardie, 2012; Pawson, 2006; Pawson, Greenhalgh, Harvey, & Walshe, 2005; Rycroft-Malone et al., 2012).

The generalization of the results and effects of numerous primary studies inevitably entails some degree of decontextualization. However, at a secondary level of research, researchers recognize, relate and / or problematize the meaning and influence of context in different ways.

In the current research, we develop knowledge about how the tension between contextuality and generalizability is resolved, and how it is revealed in research reviews of teaching methods. We are particularly interested in whether and how questions about what works for whom and under what circumstances are problematic (see Pawson, 2006). Thus, we explore those issues that are repeated in the learned methods and overtime in research reviews of teaching methods, taking into account the tension between context and generalization. Subsequently, the identified problems will be discussed in terms of possible implications for research at both primary and intermediate levels.

This research is part of a research project that aims to expand and improve our knowledge of teaching and research teaching (Hirsh & Nilholm, 2019; Roman, Sundberg, Hirsh, Nilholm, & Forsberg, 2018). To clarify the context in which the present research arose, a brief description of the starting points and assumptions underlying the entire research project follows (see Nilholm & Göransson, 2017).

With the increase in the number of primary studies, the number of research reviews, that is, the secondary level that summarizes and synthesizes primary-level research on a particular topic, has also increased. This, in turn, paves the way for high-level research that summarizes and synthesizes research reviews (Polanin, Maynard, & Dell, 2017). Various terms are used to describe the type of third-order research that uses research reviews as empirical evidence, such as review (Polanin et al., 2017), meta-meta-analysis (Hattie, 2009; Kazrin, Durac, & Agteros) ... , 1979), metasynthesis (Cobb, Lehmann, Newman-Gonchar, & Alwell, 2009), review of reviews (Maag, 2006), tertiary review (Torgerson, 2007), mega-analysis (Terhart, 2011) and general review (Grant et al. Booth, 2009). Thus, the terms used vary, but it is common that the tertiary level aims to synthesize evidence on a particular topic of interest by examining only the highest level of evidence, that is, evidence presented in systematic reviews or meta-analyzes. In our project, we decided to use the term "overview".

A characteristic feature of our review methodology is, among other things, the selection of research reviews to be included. The starting point is that it is helpful to map and analyze research that the research community itself considers important (see Nilholm & Göransson, 2017). Therefore, we only include recognized, highly effective, and most cited studies listed in the Web of Science (WoS) .1 The overall interest in the project is to explore trends in influential research reviews on teaching methods and identify common results and themes. to discuss issues ˗ using WoS as an indicator of impact.

Other major differences between our review methodology and the evidence-extraction focused methodologies (see above) are that our review type: a) includes different types of second-tier research methodologies, b) includes different theoretical approaches to research. the level of review (including reviews from a critical interpreter's point of view) and c) displays and analyzes some aspects of the research area, such as the topics studied, theoretical / conceptual starting points, methodologies used, and the results and conclusions presented. However, one particular article cannot cover all of these aspects, and the present study focuses on the findings and conclusions presented, with particular attention to the tension between contextuality and generalizability.

During the careful reading that the coding process entailed, it became apparent that the baseline reviews mainly discussed similar issues, reached similar conclusions and / or indicated similar implications for practice and / or research, regardless of the learning method being studied. This, in turn, has led to further analysis based on a shared interest in inductively and deeper exploration of the most frequent problems in order to identify recurring problems and group problem models into categories (see Saini & Shlonsky, 2012; Thomas and Harden, 2008). Before presenting the main findings of the current study, that is, the three overview findings, we briefly outline some general observations regarding the format of the baseline reviews, as well as their temporal and geographical distribution.

Revealing the gap between research and practice is neither new nor unique to the research field of teaching methods. We have shown that the gap is explained, on the one hand, by factors at the teacher and context level, and, on the other hand, by the fact that primary research is not sufficiently naturalistic, didactic and specific. A high degree of experimental purity, which can be achieved, for example, in a laboratory setting, can provide high internal confidence, but at the same time limit the external or environmental validity of the results (Bernstein, 2018; Khorsan & Crawford, 2014). The failure of primary research to confirm why a particular teaching method works or not in a particular context is highlighted in many reviews as the cause of the gap between research and practice. In addition, some reviews contain elements of self-criticism regarding the secondary level of research.

We would also like to suggest that the long-standing gap between research and practice is a problem that needs to be addressed in both first and second order research. In the introduction, we argued that review writing is a logical and reasonable way to integrate the results and conclusions of different studies, and that systematic research reviews can contribute in different ways to knowledge that has the potential to advance research and inform both practice and policy. According to Gough et al. (2012, p. 5) research reviews are vital for a variety of reasons: reviews allow us to establish not only what is known from research; but also that which is unknown. They can inform decisions about what further research can best be undertaken, thereby creating a favorable cycle. They enable researchers, policymakers and practitioners to answer key questions: "what do we know, how do we know it?" And "what else do we want to know and how can we find out?"

In our review findings, we have highlighted issues that are often challenged in reviews of high-impact studies of teaching methods over a four-decade period. The significant aspects of the results are neither surprising nor previously unknown. The strength of this study lies in how we were able to show patterns and consistency of conclusions on the issues studied over time and their relevance to the tension between context and generalization.

Trying to determine where the effect of the method itself ends and where the influence of the context begins is perhaps an impossible mission. What can be done in both primary and second-order exploration is explicitly acknowledge (to a greater extent), explore, and discuss contextual complexity. Like the other researchers mentioned above, we want to emphasize the importance of considering validity as a multidimensional concept that includes both internal, external and environmental aspects. Essentially, there are two questions that research on teaching methods must answer: does a particular way of teaching affect student learning and achievement, and what and how others can learn from the completed research. Both are equally important, but the intrinsic credibility of the research seems to be more valuable than the extrinsic and ecological ones. As Bernstein (2018) argues, promoting one to the fore at the expense of the other does not contribute to the development of an area of ​​knowledge:

If we cannot determine if what we are doing is working, we are in an evidence-free zone where we cling in the dark to find the most effective ways to teach our content. In addition, if we cannot generalize our work to other contexts, we do not create a field and do not allow teaching practice to move beyond our individual classrooms. (p. 123)

Thus, more detailed descriptions and problematization of context are needed so that both practitioners and reviewers can define credibility in a multidimensional way. Last but not least - and because many teaching methods are both comprehensive and complex - it is important to emphasize the need for well-articulated research questions indicating which aspects of the intervention or method are being studied and that there is a limit to the territory that the survey can cover.

REFERENCES:

1. Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. Crossref.

2. Barber, M., & Mourshed, M. (2007). How the world’s best-performing school systems come out on top. London: McKinsey & Co.

3. Bernstein, J. L. (2018). Unifying SoTL methodology: Internal and external validity. Teaching & Learning Inquiry, 6(2), 115–126. Crossref.

4. Cartwright, N., & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better. New York: Oxford University Press. Crossref.

5. Cartwright, N., & Hardie, J. (2017). Predicting what will happen when you intervene. Clinical Social Work Journal, 45(1), 270–279.

6. Cavanagh, S. (1997). Content analysis: Concepts, methods and applications. Nurse Researcher, 4(3), 5–16. Crossref.