Methodology tutorial - introduction

This is part of the methodology tutorial

Research Design for Educational Technologies - Introduction[edit | edit source]

Introduction - A first look at research[edit | edit source]

Research = ask a question and answer

Major stages of a research project

What do you need to know in order to get a project done ?

The concept of “science”[edit | edit source]

Why bother ??

• to understand how to write a research plan
• to understand your academic partners (e.g. your thesis advisors)
• to find out why they don’t like your initial research subject ...

Which elements define a given piece of research ?[edit | edit source]

1. Theory of science:
   • what is knowledge ? academic knowledge ?
   • how should you reason ? deduce ? induce ? model ?
2. The methodology :
   • should fit your research subject
   • ..... legitimated by some theory of science.
3. The research object
   • you need to define exactly what you want to study
4. The research goals
   • what’s the purpose of your study?
5. Your means
   • time, money,
   • knowledge,
   • data access

An equilibrium between methods, object, goals and means:[edit | edit source]

What does this figure tell ?

• In educational technology research, there rarely is a ready solution for your problem !
• There are suggestions (freedom to choose) as well as interdictions (things not to do)
• In other words: You will have to come up with your own research design and its justification !

What do we mean by academic empirical research ( science )?[edit | edit source]

1. A systematic activity
   • produced knowledge is a coherent whole
   • it (your results) should integrate with a system of knowledge
     (build upon literature and compare with literature)
2. centred on reality
   • e.g. nature, la society, people’s behavior, people’s attitudes
   • in other words: don’t just speculate, look at things
3. precise tools (hypothesis, theories, methods, reliable techniques etc.)
   • be aware of your "confirmation biais", test your conclusions against alternative explanations,
4. generalization
   • contribute to theories by using (and testing) their theoretical statements
   • reuse (and criticize) their instruments (frameworks, analysis grids, etc.)
   • suggest modifications (or even new theories)

in addition:

1. a belief in determinism,
   • phenomenons are the necessary consequence of conditions (causes).
   • In other words: randomness in explanation is only due to ignorance, complexity, etc.
2. relativism :
   • our knowledge is not perfect ,
   • .... in particular in social sciences where man is subject and object, observer and observed and where many variables influence a phenomenon

What’s an interesting piece of research ?[edit | edit source]

You will have to produce something that is (somewhat) new

• answer new questions
• answer old questions without good answers
• answer otherwise to questions addressed by the literature
• provide support to answers found in literature with a new argumentation
• apply a theory to a new types of cases (e.g. does it apply to Mauritius school system ?)

It produces something that provides “satisfaction”

• to a certain community
  (you don’t write your thesis for yourself !).
• to you !

The role of method and theory[edit | edit source]

Epistemological dimensions of research[edit | edit source]

Theories of science

• Sets from a philosophical perspective the conditions of scientific knowledge
• example: "you can’t prove a hypothesis" (only evidence, show that alternatives are wrong,...)

Methodologies (also called approaches)

• general recommendations on how you should design a research plan .
• draws from a theory of science and suggests a set of legitimate methods.
• example "you should draw hypothesis from theory and then test it with quantitative research"

Methods

• general recipes to study a given class of phenomenons
• examples: "survey research methodology", "participatory software design"

Reasoning Methods

• how to pass from data to theory and from theory to data ?
• .... (influenced by theories of science and doctrine of approaches)

Techniques

• practical tools to gather, manipulate, analyze data, manipulate concepts, etc.

The range of theories[edit | edit source]

• big theories
  • go after complex topics (can’t fully be tested)
  • .... evolution of children’s minds, learning, society, .....

• Theories with limited scope
  • concern more restricted domains
  • examples: usability guidelines for software, conditions under which multimedia animations are effective, conditions under which e-learning projects can be sustainably implemented, ...

• Formal models
  • based on formal systems, e.g. mathematics, logics, rule systems, formal learning designs
  • sometimes tested with empirical data (not always, e.g. micro-economics is not).

• Conceptual models
  • e.g. “systems analysis”, activity theory
  • .... conceptual tools that allow you to talk about a phenomenon, to look at them in a certain way

• Hypothesis
  • Are frequently part of a theory or a formal model
  • Clear propositions that can be tested
  • e.g. "to introduce technology in schools, you need to provide a pedagogical support structure"

Everything together: components of knowledge[edit | edit source]

The paradigm concept[edit | edit source]

• Origin: Kuhn and his studies on “normal science”

Major components of a paradigm:

1. a general and “asymptotic” research goal
   • ex: “understand how to teach (instructional design)”.
   • At this level you will find general ideas at what you should look at.
2. Intermediate level: partial theories .
   • Par ex: to teach sustainable knowledge, one must engage students in practise and gradually introduce authentic problems that must be solved by themselves
3. Operational level: Empirically tested theories .
   • For example: how to teach procedural programming, drive a car, solve a simple geometry problem.
4. Each paradigm favors certain methodologies and provides you with "toolkits"

Why follow a paradigm ?

your are much more productive if you can count on confirmed research methodology

different researchers can work together , or at least profit from each other’s results

what happens if you don’t ??

people will not understand you and therefore ignore you if you don’t use accepted methodology or problems

your results are not comparable

Note: In experimental psychology, "paradigm" refers to a proto-typical experiment that can and is replicated in various studies. This does not convey the same meaning, but then experiment, comparison and replication are fundamental pillars of this field. In other words, a researcher in experimental psychology does not need to reflect about very general ways of doing things. It's a "paradigmatic science" and therefore the word "paradigm" can be used for a different purpose ...

The approach[edit | edit source]

There are in fact 2 different definitions

(1) approach = general methodology

• a way to do it ...
• includes a set of useful and tested methods for studying a set of phenomena e.g. the you could use the quasi-experimental design to study school reforms
• an approach is often transdisciplinary:
  example: the quasi-experimental approach was developed in educational science but has been exported to public policy analysis and many other domains

(2) approach = paradigm:

• for example: “activity theory approach” to say
  • you believe in a marxist activity-based scheme of looking at social phenomena
  • you may adopt Engeström’s related educational theories
  • you favor qualitative methodology
  • you are interested in change

A word on interdisciplinarity[edit | edit source]

• combinations of approaches or paradigms

3 variants

1. multi-disciplinary :
   • juxtaposition on the same object of various research paradigms, each one keeping its own language
2. Interdisciplinary:
   • confrontation and exchange of methods and/or adoption of a mix from various fields for a new field
3. Trans-disciplinary:
   • usually a high abstraction level, e.g. systems theory

Difficulties

multi-disciplinary research is difficult to coordinate.

• needs wide knowledge and very good communication skills to talk to people using another "language"

Interdisciplinary research is easier

• ..... because only methods and concepts that fit are taken from other fields,
• however, concerned scientific communities may hate you for that
• takes more time than disciplinary research (e.g. doing a "complete" educational technology thesis that involves pedagogy, psychology, sociology and ICT development takes more than doing a thesis in just one of these areas).

Types of research[edit | edit source]

Classification according to theory level[edit | edit source]

1. Simple description:
   • forget it, it doesn’t have much academic value (unless it is led to prepare further research)
2. Classifications and categorizations : put order in concepts or data:
   • The intelligent case study (exploratory research)
   • Typologies (identify characteristics of classes of cases, e.g. uses of technology in schools, types of teachers according to their beliefs in pedagogy, use of ICT, use of new pedagogies, etc.)
   • Ideal-types (theory-based identification of classes of cases)
   • The systems model (shows interactions between elements)
   • ....
3. Research where theory plays important role:
   • Theory attempts generalization and demonstrates regularities .
   • Theory tries to understand or to explain or to predict .

Research should aspire to level 3 !

Scientific ends[edit | edit source]

The following summary table is modified from Marshall & Rossmann (1995: 41)

Järvinen (2004: 10) proposed a similar taxonomy, but it is arranged in a tree shows how different approaches relate.

This typology is interesting because it integrates engineering sciences (something social science methdology texts usually do not. Since part of educational technology is research on technologies, the Järvinen typology is useful.

Research approaches in educational technology[edit | edit source]

Educational technology is an interdisciplinary field divided into many technical sub-fields often adopting different ways of looking things. We also could distinguish a series of "levels of research" going from the conceptual to the technical:

1. Fundamental research: Many researchers in the field choose to adopt a more fundamental research stance focusing on small well-defined problems such as “under which conditions can multimedia animations be effective”.
2. Technology-supported instructional design applied to various domains of education. Major categories are distance teaching, blended teaching, computer-enhanced classroom teaching, industrial training. Other specializations may concern subject matters (e.g. science or language teaching) or approaches (direct instruction vs. project-oriented learning for example).
3. Research on the design and application of technologies: Researcher may specialize on subjects like the use of computer simulations in education or more technically, how to build authoring and learning environments for simulations.

To further complicate things, some researchers may combine a fundamental research perspective with a particular kind of instructional design and a particular kind of technology. Depending upon these options, research interests and research methodology will not be the same. From these possible combinations, we probably could identify two major strands of thought:

1. Educational technology as part of the learning sciences: Research is inspired by and contributes to modern learning theory. This strand includes research communities like computer-supported collaborative learning, intelligent tutoring systems, ubiquitous computing.
2. Educational technology as instructional technology: It is inspired by and contributes to instructional design theory and methodology. This strand includes research communities on e-learning, distance teaching, and multimedia design.

Research topics in educational technology

One way to find out what kind of research is popular and recognized is to study authoring guidelines for good academic journals in the field.

The Journal of the Learning Sciences is the official journal of The International Society for Learning Sciences. It defines its aims in the following way (April 2009):

The Journal of the Learning Sciences provides a multidisciplinary forum for the presentation of research on learning and education. The journal seeks to foster new ways of thinking about learning that will allow our understanding of cognition and social cognition to have impact in education. It publishes research articles that advance our understanding of learning in real-world situations and of promoting learning in such venues, including articles that report on the roles of technology can play in promoting deep and lasting learning. The Journal of the Learning Sciences promotes engaging and thoughtful participation in learning activities, and articles reporting on new methodologies that enable rigorous investigation of learning in real-world situations.

The Educational Technology Research and Development journal is the official journal of the Association for Educational Communications & Technology. It defines itself like this (April 2009):

Educational Technology Research and Development is the only scholarly journal in the field focusing entirely on research and development in educational technology.

The Research Section assigns highest priority in reviewing manuscripts to rigorous original quantitative, qualitative, or mixed methods studies on topics relating to applications of technology or instructional design in educational settings. Such contexts include K-12, higher education, and adult learning (e.g., in corporate training settings). Analytical papers that evaluate important research issues related to educational technology research and reviews of the literature on similar topics are also published. This section features well documented articles on the practical aspects of research as well as applied theory in educational practice and provides a comprehensive source of current research information in instructional technology.

The Development Section publishes research on planning, implementation, evaluation and management of a variety of instructional technologies and learning environments. Empirically-based formative evaluations and theoretically-based instructional design research papers are welcome, as are papers that report outcomes of innovative approaches in applying technology to instructional development. Papers for the Development section may involve a variety of research methods and should focus on one or more aspect of the instructional development process; when relevant and possible, papers should discuss the implications of instructional design decisions and provide evidence linking outcomes to those decisions.

The Journal of Interactive Learning Research is edited by Association for the Advancement of Computing in Education. It publishes papers related to the underlying theory, design, implementation, effectiveness, and impact of interactive learning environments in education and training. We shall reproduce a longer extract of its definition of scope (May 2009):

Many researchers fail to distinguish clearly between the goals of their research and the methods they employ. [Table 9 andTable 10] present a classification scheme intended to distinguish between the goals and the methods of research. Most research studies submitted to JILR should be able to be classified according to the six research goals represented in Figure 1. This scheme reflects the debate about research "paradigms" that has dominated social science research literature for decades. For example, Soltis (1992) claims there are currently "three major paradigms, or three different ways of investigating important aspects of education" (p. 620): 1) the positivist or quantitative paradigm, 2) the interpretivist or qualitative paradigm, and 3) the critical theory or neomarxist paradigm. The three categories presented by Soltis (1992) fail to capture the full breadth of research goals in the fields of inquiry relevant to JILR, and therefore the scheme in Figure 1 includes more categories. However, the goals of inquiry represented in Figure 1 are not intended to be a complete and final listing of research goals.

A methodology classification scheme is represented in Figure 2. There are numerous methods available to researchers in areas as diverse as cognitive psychology, instructional technology, and computer science (cf., Driscoll, 1995), but for the sake of simplicity, these five methodological groupings provide sufficient discrimination to represent the major approaches likely to be used in investigations reported in JILR. This journal will be especially open to submissions that involve alternative methods (e.g., qualitative and critical theory) which seem to be underrepresented in more traditional publications.

The Journal of Interactive Media in Education is a good “open access” (free) on-line journal. Its aims are defined as follows (May 2009):

• To foster a multidisciplinary and intellectually rigorous debate on the theoretical and practical aspects of interactive media in education.
• To clarify the cognitive, social and cultural issues raised by the use of interactive media in education.
• To radically improve teaching and learning through better interactive media.
• To publish leading international research on the theories, practices and experiences in the field.
• To link scholars and commercial practitioners
• Through its innovative use of interactive Net-based media, to be an action research project which explores the changing face of journals, and more broadly, scholarly practice in the age of digital publishing and communication.

JIME's name is open to multiple interpretations, given the variation in usage of each of the component terms. They are to be interpreted broadly:

• Interactive - refers both to interaction through the media with other people (e.g. teacher-student, student-student, researcher-teacher), and to interaction with the materials embedded in the media (e.g. control of a simulation or educational game).
• Media - refers to the range of modalities which can be used to support learning. Media may represent educational materials, as well as other people relevant to that learning task. Media embraces text, sound, still and animated graphics, video, model worlds and virtual reality, as applied to the delivery of learning materials and communication between relevant communities.
• Education - includes all levels and types: schools, colleges, universities, home learning, open and flexible learning, distance learning, personal development, skill learning, work-based learning, and lifelong learning.

There exist many more academic journals and each of these has a somewhat different view of what kind of research is interesting and acceptable.

Other ways to find out what what kind of research is popular and recognized is to look at conference proceedings and PhD thesis. A third possibility is to refer to meta studies on research topics and methods. A very good source is the free on-line book “Multidisciplinary Methods in Educational Technology Research and Development” by Justus J. Randolph. Randolph gives answers to the four following questions:

• What are the methodological factors that need to be taken into consideration when designing and conducting educational technology research?
• What types of research questions do educational technology researchers tend to ask?
• How do educational technology researchers tend to conduct research? – What approaches do they use? What variables do they examine? What types of measures do they use? How do they report their research?
• How can the state of educational technology research be improved?

From his text, we extracted some examples of typical research questions:

Knowledge-based (literature analysis) questions:

• What is known about best practices in user-centered design?
• Across studies, what are the academic effects of tools that help students visualize algorithms?
• What variables are known to influence the effectiveness of educational interventions?

Empirical research questions

• What are the effects of a new technological intervention on the long-term and short-term memory retention of vocabulary words?
• To what degree do students and teachers report that they are satisfied with a new intervention?
• In what ways do teachers and students report that a new intervention can be improved?

Such questions are very general and they just summarize a research subject. As you shall see later, you must learn how to break down a general question into sub-questions: According to Randolph (17). The following question:

• What is the essence of the experience of sense of community in online courses?

This question then could be further specified with these three sub-questions:

• What do teachers experience in terms of the phenomenon of sense of community in online learning?
• What do students experience in terms of the phenomenon of sense of community in online learning?
• What medium-related contexts influence stakeholder’s experience of community in online learning?

A simple typology of research[edit | edit source]

In further installments of the methodology tutorials we will focus on these broad categories of research.

Read on ...

• Methodology tutorial - theory-driven research designs
• Methodology tutorial - theory-finding research designs
• Methodology tutorial - design-oriented research designs

You may of course combine elements from several research types...

References[edit | edit source]

• Järvinnen, (2004), On research methods, Tampere: Opinpajan Kirja. [Methods book for ICT-related issues, good but difficult and dense reading]

• Marshall, C. & Rossman, G. B. (1995) , Designing Qualitative Research, second edition, Sage, London. [conseillé aux débutants]

See Research methodology resources for web links and a larger bibliography.