Methodology tutorial - design-oriented research designs

This is part of the methodology tutorial

Introduction[edit | edit source]

Further complimentary reading: see the design methodologies category for a list of design-related articles in this wiki.

Key elements of a design-oriented approach[edit | edit source]

Design sciences existed for a long time, e.g. in architecture and engineering, but only somewhat recently instructional design and educational technology researchers started to making an explicit claim for the necessity of design-oriented research. A good example is the so-called design-based research movement.

The global picture[edit | edit source]

Typical ingredients or steps of design research can be summarized with the following picture (Pertti Järvinen, 2004)

Doing design-oriented research means to investigate at least one of the dotted lines.

The technological rule (or what we call design rules) as theory "on how to things" can be the research's input, output, or both.

Technological rules (design rules)

Design rules specify how to do things and are dependant on other theories (and beliefs)

Bunge (quoted by Järvinen:99) defines "technological rule" as “an instruction is defined as a chunk of general knowledge, linking an intervention or artifact with a desired outcome or performance in a certain field of application”.

Types of outcomes (artifacts, interventions)

Design outcomes must use and create clear language. We may distinguish between the following types of output:

• Constructs (or concept) that form the language of a domain
• Models that are sets of propositions expressing relationships among constructs
• Methods that are a set of steps to perform a task (guidelines, algorithms)
• Instantiations that are realizations of an artifact in its environment

Types of research

• Build: Demonstrate feasibility of an artifact or intervention
• Evaluate: Development of criteria, and assessment of both artifact building and artifact usage

What does this mean ?

There exist 4*2 ways (outcomes * types) to lead interesting design research.

Usually, it is the not the artefact (e.g. a software program) you build that is interesting, but something behind (constructs, models, methods) or around it (usage).

Examples of design rules[edit | edit source]

In educational technology, we find a variety of design levels. On example that tries to promote more systematic thinking about design is the Developing design documents (3D) model (Boot et al. 2007: 917).

This "3D" model includes three major dimensions:

1. stratification: Functionally different instructional and technical structures
2. degree of elaboration: conceptual, specification or implementation
3. formality: formal vs. informal

Below, we shall present a few examples of design rules (output of research) that are popular in educational technology. See the instructional design method article for more examples.

As you can see, design (or the essence of it in terms of design rules) can be expressed in various ways, e.g. as:

• concept map
• as list
• as UML diagram
• as formal or non-formal design language (see educational modelling languages).

The MISA instructional design method[edit | edit source]

MISA is an instructional engineering method describing graphically the instructional design processes and their products which define a learning system completely. MISA supports 35 main tasks or processes and around 150 subtasks. The method has been totally represented within the MOT knowledge editor.

A course designer works on "4 models"

1. Knowledge and Skill Representation
   DC: Design of Content (know-that and know-how)
2. Application of Teaching Methods and Approaches
   DP: Design of Pedagogical specifications
3. Specification of Learning Materials
   DM: Design of Materials
4. Delivery Planning
   DD: Design of Delivery

The 4 components split over the 6 phases lead to the 35 main tasks:

Using such a method is worth the effort:

• if you plan do it right (e.g. buy the MOT editor)
• if you focus on a whole course instead of difficult problems
• if you plan to train yourself in instructional design

It is not clear to us what parts this model are based on research or whether it represent an engineering doctrine that aims to provide rules an educational designer should follow ...

Gagné’s 9 steps of instruction for learning[edit | edit source]

Gagné's Nine events of instruction represents a set of nine sequential rules specifying the contents of a "good" lesson (unit of learning). It is grounded in behaviorist-cognitivist theory of instruction.

1. Gain attention e.g. present a good problem, a new situation, use a multimedia advertisement.
2. Describe the goal : e.g. state what students will be able to accomplish and how they will be able to use the knowledge, give a demonstration if appropriate.
3. Stimulate recall of prior knowledge e.g. remind the student of prior knowledge relevant to the current lesson (facts, rules, procedures or skills). Show how knowledge is connected, provide the student with a framework that helps learning and remembering. Tests can be included.
4. Present the material to be learned e.g. text, graphics, simulations, figures, pictures, sound, etc. Chunk information (avoid memory overload, recall information).
5. Provide guidance for learning e.g. presentation of content is different from instructions on how to learn. Use of different channel (e.g. side-boxes)
6. Elicit performance "practice", let the learner do something with the newly acquired behavior, practice skills or apply knowledge. At least use MCQ’s.
7. Provide informative feedback , show correctness of the trainee’s response, analyze learner’s behavior, maybe present a good (step-by-step) solution of the problem
8. Assess performance test, if the lesson has been learned. Also give sometimes general progress information
9. Enhance retention and transfer : inform the learner about similar problem situations, provide additional practice. Put the learner in a transfer situation. Maybe let the learner review the lesson.

Design rules from computer science[edit | edit source]

You can look at various modeling and diagram types of the Unified modeling language in order to have glance at the most popular design language in computer science with which many design problems can be modelled.

UML is a kind of qualitative data analysis tool with which various kinds of designs can be expressed. E.g. the following picture from the IMS LD Best Practice specification shows a diagram for competency-based learning with two major alternatives, advising-then-anticipating and anticipating-then-advising.

Activity Diagram for Competency-based learning. Source: IMS

.

Read the UML activity diagram article to learn more about activity diagrams.

The design process[edit | edit source]

Modern design science is influenced by several fields, e.g. architecture or software engineering.

In educational technology research, one uses most often some kind of "agile" and iterative design method for developing software. In this wiki, we don't cover (at least at the time of writing) software design methodology very much, but some of the approaches presented below have their origin in CS.

Anyhow, most research subjects (not just software development!) can be tackled through a design science approach.

Alternatives[edit | edit source]

Pertti Järvinen (2004: 103) formulated several alternatives for the design process.

Basically, you must choose between a more top-down approach (also called the waterfall model) and a more (fully) participatory "agile" approach.

The participatory design model[edit | edit source]

Read participatory design for more information.

User-centered design:

• involves users as much as possible so that they can influence it
• integrates knowledge and expertise from other disciplines than just IT
• is highly iterative so that testing can insure that design meets users’ requirements

Participatory design does not just mean to ask users what they want and to otherwise investigate their needs, but it means that they actively will participate in all the design cycles.

Traditional Information System prototyping approaches (after Grønbæk, 1991).

Cooperative prototype approach (after Grønbæk, 1991)

A similar model from Preece, Rogers and Sharp (2002), figure from Hakansson

Typical user analysis techniques[edit | edit source]

See the modules Methodology tutorial - qualitative data acquisition methods and Methodology tutorial - quantitative data analysis for some more operational details.

According toSeries of HTML Slides Håkansson's slides, the most popular user analysis techniques are:

• Questionnaires
  • if user number is high
  • if you know precisely what to ask (e.g. to identify user profiles, to test hypothesis gained from in-depth studies, etc.
• Semi-structured Interviews
  • to explore new issues
  • to let participants develop argumentation (subjective causalities)
• Focus groups
  • group interview”, collecting multiple viewpoints
• Observations/Ethnography
  • To observe work as it happens in its natural setting (observe task related workflow, interactions)
  • to understanding context (other interactions, conditions)
• Scenarios (for task description)
  • An “informal narrative description”, e.g. write real stories that describe in detail how someone will use your software (do not try to present specifications here !)
• Cultural probes
  • Alternative approach to understanding users and their needs, developed by Gaver (1999) ?

Definition of requirements[edit | edit source]

Before constructing an artifact, you will have to define requirements. There exist different types, that we shortly list:

• Functional requirements
• Environmental requirements
• Physical, social, organizational, technical
• User requirements
• Usability requirements

Building prototypes[edit | edit source]

Prototypes can be anything !!

• Quote: "From paper-based storyboards to complex pieces of software: 3D paper models, cardboard mock-ups, hyperlinked screen shots, video simulations of a task, metal or plastic versions of the final product" (Håkansson).

Prototypes are of different nature according to the stage and the evolution of the design process:

• Useful aid when discussing ideas (e.g. you only need a story-board here)
• Useful for clarifying vague requirements (e.g. you only need some UI interface mockup)
• Useful for testing with users (e.g. you only need partial functionality of the implementation)

Design-based research[edit | edit source]

Design-based research, also know as "design experiment" is an approach that has been specifically developed by educational technologists. See design-based research for more details.

Design experiments should be carried in some organized fashion. Daniel K. Schneider now systematically has his Ph.D. students work with so-called conjecture maps (Sandoval: 2004) and that allow to drive and to organize the research project in terms of a kind of concept map. Here is an example that identifies the important research components of a collaborative writing framework. It "tells" that research is base on a set of theories that we try to embody in a design and that will to (student cognitive) processes and finally to outcomes.

Conjecture map example for the DUAL-T collaborative writing framework

Such a drawing can and should be refined over time, i.e. arrows should not just point from big box to big box but from little boxes to little boxes. The idea is that some of these relations then should be tested with some serious research methodology...

Evaluation[edit | edit source]

Evaluation criteria[edit | edit source]

Evaluation usually happens according to some "technological rule"

A good example are Merril’s first principles of instruction:

1. Does the courseware relate to real world problems?
2. Does the courseware activate prior knowledge or experience?
3. Does the courseware demonstrate what is to be learned ?
4. Can learners practice and apply acquired knowledge or skill?
5. Are learners encouraged to integrate (transfer) the new knowledge or skill into their everyday life?

Another example is the LORI model, available as an online form consisting of rubrics, rating scales and comment fields.

Evaluation methodology[edit | edit source]

Design evaluation methodology draws on all major social science approaches for data gathering and data analysis

e.g. Håkansson cites the following data gathering methods

• Heuristics
• Experiments
• Questionnaires
• Interviews
• Observations
• Think-aloud

Therefore: have a look at the data acquisition and analysis modules of this tutorial series

Nielson's (1993) usability methods[edit | edit source]

Usability studies are usually part of design research. E.g. in TECFA's Master of Science in Learning and Teaching Technologies we require from students that choose to do a "development master" thesis to provide at least at usability study at the end.

Source: http://fdlwww.kub.nl/~krahmer/evaluation-introduction.ppt

Examples of design-oriented master thesis[edit | edit source]

Design and development of a scaffolding environment for students projects[edit | edit source]

Synteta, P. (2001). EVA_pm: Design and Development of a scaffolding Environment For Students projects. TECFA, University of Geneva. PDF version

This study is centered on the design and development of a scaffolding environment for students' projects. It's a constructivist environment that aims to scaffold Project-Based Learning (PBL) strategies to improve their effectiveness and serve as a reflection tool to help students develop meta-cognitive skills. It leverages XML (eXtended Markup Language) and suggests a vocabulary (DTD) that describes students' projects (assessments, theses, etc.) as a prompting tool. The vocabulary is well accompanied with the proposal of an appropriate XML editor to diminish the cognitive load of editing and also with an online toolset ("commNcontrol" and "Virtual Book") to scaffold communication, visualization, peering, and progress tracking.

Objectives (quotations from the thesis)

• This study is also an intervention for improving PBL efficiency. It entails the development of a Scaffolding Learning Environment (SLE1) that is trying to learn from the lessons of the past and leverage from stresses on new technologies like XML and the World Wide Web making a lightweight and easily portable environment.
• Most of the research for improving PBL efficiency tries to remediate specific weaknesses of PBL, but doesn’t propose a complete system that supports a substantial student project through all it’s phases and for all contexts.
• Our key goal was to develop a constructivist environment and a method for scaffolding students’ projects (assignments) from their management up to the writing of their final report.

So, the objectives of this SLE are:

• to help students develop scientific inquiry and knowledge integration skills, to focus on important and investigate key issues;
• to support them directing investigations;
• to make students better manage the time and respect the time constraints;
• to overcome possible writer’s block, or even better to avoid it;
• to help students acquire knowledge on project design and research skills;
• to improve team management and collaboration (especially collaborative editing of student groups);
• to make students reflect on their work;
• to support the tutor’s role in a PBL approach;
• to facilitate monitoring and evaluation for the tutor;
• to help the tutor verify whether knowledge is being acquired;
• to motivate the peers, and eventually to distribute the results to bigger audiences.

Research questions

see above

Method

Field exploration

• A very important part of this research was to conceive a grammar that would model the work of an academic project. There are different sources of information that have been used to achieve this goal. (....)
• Survey of needs with a questionnaire:
  • In order to gather precious information from the key persons involved in projects, like professors and their assistants, a questionnaire was articulated in such a way that would provoke a productive discussion, leading to comments and suggestions that would improve this research. The idea was to give the questionnaire to a small sample of the unit and stop the survey when the same answers came up again.

The development method

• ... that has been adopted corresponds to participatory design and specifically to cooperative prototyping .
• Both "prototyping" and "user involvement" (or "user centered design") are concepts that have frequently been suggested to address central problems within system development in recent years. The problems faced in many projects reduce to the fact that the systems being developed do not meet the needs of users and their organizations. [...]

XML-based argumentation authoring tool[edit | edit source]

Benetos, Kalliopi (2006). Computer-Supported Argumentative Writer. An authoring tool with built-in scaffolding and self-regulation for novice writers of argumentative texts. M Sc MALTT (Master of Science in Learning and Teaching Technologies), TECFA, University of Geneva. Home page, PDF file.

This master thesis relates to work on computer-supported argumentation, i.e. the design of a cognitive tool. K. Benetos produced an online authoring tool using PHP, DOM, XML, XSLT and JavaScript, to help novices of argumentative text composition to better develop and structure their written texts. The tool is based on an underlying XML-based schema (developed with instructors) that is representative of argumentative texts and will offer a visualization and scaffolding of the composition process.

Working hypothesis

1. A computer-supported authoring tool based on a schema inherent to written argumentative texts can help improve the texts written by novices of written argumentation:

a. in the quantity of arguments produced

b. in the quality of arguments produced

i. scope

1. variety of arguments

a. epistemological point of view (Baker, Quignard, Lund, Séjourné, 2003)

b. function of the argument (support and/or negotiate) (Dolz, 1996)

ii. depth

1. inclusion of counter-arguments and conclusions (Brassart, 1996)

c. structural quality of arguments and text as a whole

i. use of connectives (Akiguet and Piolat, 1996)

ii. organization of arguments

iii. conclusions

2. Through the use of a computer-supported authoring tool that offers structural and cognitive aid, novices will learn to recognize the components of the schema inherent to argumentative writing.

3. Feedback resulting from self-evaluation and procedural progress in the form of an actualized visual representation can enhance motivation, self-regulation and improve text structure and linearization.

Methodology (excepts)

• Needs Analysis – based on literature on user-centered design
• Theoretical Framework – To allow investigation into a computer-supported tool based on a schema inherent to written argumentative texts, it was necessary to first design a framework for argumentative writing
• Adaptation Of Prototype - refinement approach
• Testing – Testing and interviews with users and stakeholders were conducted in each phase
• Reporting – Reporting at the end of each phase permitted the recording of testing results for use in the identification of successes and problems within the design and testing processes.

The project was conducted in three phases: Design Analysis - System Design, Functionality And Usability - and Testing With Students.

Testing with students consisted of 4 activities over a 1 and a half hour session:

1. Writing of an argumentative essay
2. Post-test
3. Interview on writing and post-test activities
4. Interview on the visualization (of the tool)

A personalizable portal to organize academic resources[edit | edit source]

Title: Le portail personnalisable comme interface d'optimisation de ressources académiques

Research questions

La question principale de ce travail est : pourquoi les bibliothèques universitaires ne proposent-elles pas de portails personnalisables, ces derniers constituant vraisemblablement une bonne solution pour l'optisation des ressources des bibliothèques à l'ère numérique.

Cette question peut s'articuler en 3 sous-questions :

• pourquoi pas plus de portails personnalisables ?
• pourquoi les gens ne personnalisent-ils pas ?
• la personnalisation correspond-elle à un besoin ?

Method, not clearly articulated, e.g.

• La méthodologie retenue est une revue de la littérature, afin d'expliciter les concepts, de dégager les grands modèles et d'examiner les dispositifs existants, accompagnée d'un monitoring sur l'implémentation partielle d'un prototype de dispositif de portail personnalisable.
• L'implémentation partielle n'ayant pu être menée à terme, le monitoring a été supprimé et j'ai élargi le champ de la revue de littérature. L'analyse est donc qualitative, les données quantitatives initialement prévues n'ayant pu être collectées.
• Le matériel recueilli lors du prototype MyBCU ainsi que l'expérience acquise en 2001-2002 comme webmaster [...] m'ont néanmoins permis de tenter de répondre aux questions initiales.

Links[edit | edit source]

• Maria Håkansson, Intelligent System Design, IT-university, Goteburg, Design Methodology slides, Series of HTML Slides, retrieved 13:13, 7 October 2008 (UTC).

References[edit | edit source]

• Boot, Eddy W.; Jon Nelson, Jeroen J.G. van Merriënboer, Andrew S. Gibbons (2007). Stratification, elaboration and formalisation of design documents: Effects on the production of instructional materials, British Journal of Educational Technology 38 (5), 917-933. doi:10.1111/j.1467-8535.2006.00679.x

• Järvinnen, (2004), On research methods, Tampere: Opinpajan Kirja. [Methods book for ICT-related issues, good but difficult and dense reading]
• Nielsen, J. (1993), Usability engineering, Boston: AP professional.

More references about design science, etc.:

• See Design science and Design methodology (and follow up other links)