Methodology tutorial - quantitative data acquisition methods

This is part of the methodology tutorial

Introduction[edit | edit source]

In educational technology (as well in most other social sciences) one works with a variety of quantitative data sources. See also the overview of "measurement techniques"

Researcher generated data, for example

• Tests, e.g. a measure of task performance
• Questionnaires
• Quantified qualitative observations of various kinds, e.g. texts written by students, transcribed interviews.

Real data, for example

• Grades
• "Official and semi-official Statistics", e.g. aggregate population data

Log files

• I.e. traces of user interactions with a system
• This includes data from tools specifically made for research or data that just "exist" (e.g. database entries for forum participations)

In this tutorial we will focus on survey design and therefore questionnaire design

Basic Principles of questionnaire design[edit | edit source]

Below we formulate a short recipe that you should follow.

1. Make a list of concepts (theoretical variables) in your research questions for which you need data.
2. For each of these concepts make sure that you identify its dimensions (or make sure that they are not multi-dimensional). If don't know what "dimensions" are, go back and read about the operationalization of general concepts
   • Consult the research literature, i.e. find similar research that used questionnaires.
   • Discuss with domain experts
   • Make a list of conceptual variables (either simple concepts or dimensions of complex concepts)
3. For each conceptual variable think about how you plan to measure it
   • First of all (!!) go through the literature and find out if and how other people went about it
   • It is much better to use a suitable published instrument than building your own. You can then compare your results and you will have much less explanations and justifications to produce !
4. Plan some measurement redundancy
   • Do not measure a conceptual variable with just one question or observation. Use at least four questions
5. Rather ask people how they behave instead of how they think they behave
   • E.g. don’t ask: "Do you use socio-constructivist pedagogies ?" but ask several questions about typical tasks assigned to students.
6. Do not ask people to confirm your research questions
   • E.g in a survey or test do not ask: "Did you manage to make your teaching more socio-constructivist with this new tech." (Again) ask/observe what the teacher really does and what his students had to do.
7. Test your questionnaire with a least two people. The chance is very high, that some of your questions are really badly designed ...

Finally, you also could triangulate surveys with data of different nature, e.g. combine survey data with objective data and quantified observational data (like log file analysis) or even qualitative data like interviews with a few selected individuals.

Question design[edit | edit source]

Basic question and response item design[edit | edit source]

Wording and contents of the questions

• Only ask questions that your target population understands
• Questions should avoid addressing 2 issues in 1 question!
  • Bad example: Did you like this system and didn't you have any technical problems with it ?
• Make the questions as short as possible. Else people won't read/remember) the whole question.
• Ask several questions that measure the same concept
  • Try be all means to find sets of published items (questions) in the literature that you can reuse

Response items

• Avoid open-ended answers (these will give a lot of coding work)
• Use scales that have at least a range of 5 response options
  • otherwise people will have a tendency to drift to the "middle" and you will have no variance.
  • e.g. avoid:

agree () neither/or () disagree ()

• Response options should ideally be consistent across items measuring a same concept
• If you feel that most people will check a "middle" value, use a large "paired" scale without a middle point

e.g. 1=totally disagree, 10=totally agree
1 2 3 4 5 6 7 8 9 10

Again: Use published scales as much as you can. This strategy will help you in various ways:

• You get better reliability (user's understanding of questions) since published items have been tested
• Scales construction will be easier and faster (you can skip doing Kronbach alpha's)
• It will make your results more comparable
• ....

Testing

• You must test your questionnaire with at least 2 people !
• From my experience as a methodology crash course teacher I can say that I never have seen an even moderately acceptable questionnaire made in one go by a beginner student. Do not overestimate your skills ! - Daniel K. Schneider

Example questionnaire: Social presence[edit | edit source]

Social presence is an important variable in (informed) distance education. We shall have a look at a study that tries to link social presence to learner satisfaction.

The GlobalEd questionnaire by Gunawerda & Zittle (1997), and which can be found in the Compendium of Presence Measures, was developed to evaluate a virtual conference. Participants (n=50) of the conference filled out the questionnaire. Internal consistency of the social presence scale was a=0.88. Social presence was found to be a strong predictor of user satisfaction.

The questionnaire used the following 14 questions to measure social presence (the total questionnaire included 61 items):

• Messages in GlobalEd were impersonal
• CMC is an excellent medium for social interaction
• I felt comfortable conversing through this text-based medium
• I felt comfortable introducing myself on GlobalEd
• The introduction enabled me to form a sense of online community
• I felt comfortable participating in GlobalEd discussions
• The moderators created a feeling of online community
• The moderators facilitated discussions in the GlobalEd conference
• Discussions using the medium of CMC tend to be more impersonal than face-to-face discussion
• CMC discussions are more impersonal than audio conference discussions
• CMC discussions are more impersonal than video teleconference discussions
• I felt comfortable interacting with other participants in the conference
• I felt that my point of view was acknowledged by other participants in GlobalEd
• I was able to form distinct individual impressions of some GlobalEd participants even though we communicated only via a text-based medium.

A 5-point rating scale was used for each question

Example questionnaire: socio-constructivist teachers[edit | edit source]

In a questionnaire designed by B. Class and that was based on Dolmans (2004), the problem was how to identify socio-constructivist elements in a distance teaching course for interpreter trainers.

Decomposition of “socio-constructivist design” in (1) active or constructive learning, (2) self-directed learning, (3) contextual learning and (4) collaborative learning, (5) teacher’s interpersonal behavior (according to Dolmans et al., 1993)

Note that headers regarding these dimensions (e.g. "Constructive/active learning" are not shown to the subjects. We do not want them to reflect about theory, but just to answer the questions ... So they are just shown below to help your understanding

General questionnaire design issues[edit | edit source]

The Introduction

(applies to both written questionnaires on paper or on-line surveys)

• You should write a short introduction that states the purpose of this questionnaire

Such an introduction:

• guarantees that you only will publish statistical data (no names !)
• specify how long it will take to fill it in

Ergonomics

• Do not include anything else than questions and response items (besides the introduction)
• Make sure that people understand where to "tick".

Coding information for the researcher

• Assign a code (e.g. number) to each question item (variable) and assign a number (code) to each response item
• This will help you when you transcribe data or analyze data
• use "small fonts" (this information is for you)

Example of a question set

Below is a question from a paper-based survey with several sub-questions about teachers' behavior. You can see that each question is numbered and the response items have codes. These are irrelevant for the user (and you really should use small fonts for these), but they are useful for data transcription and also to help you map variables in your statistics program to the questions and response items.

Experiments[edit | edit source]

This section should some day become a separate tutorial - DKS

Designing a true experiment needs advice from some expert. Typically, a beginner makes the mistake to differentiate 2 experimental conditions by more than one variable, i.e. you will really be able to understand what exact treatment led to observed effects !!

There are many kinds of experimental measures

• observations (e.g. Video, or recordings of computer input)
• tests (similar to surveys)
• tests (similar to examination questions)
• tests (performance in seconds)
• tests (similar to IQ tests)

Consider all the variables you want to measure

• Most often, the dependant variables (to explain) are measured with tests
• Usually the independent (explanatory) variables are defined by the experimental conditions (so you don’t need to measure anything, just remember to which experimental group the subject belonged)

See the literature, experimental research publications usually explain fairly well the "method"!!

• First of all, read articles about similar research !
• Consult test psychologists if you need to measure intellectual performance, personality traits, etc.
• Use typical school tests if you want to measure typical learning achievement

Sampling[edit | edit source]

The ground rules[edit | edit source]

The number of cases you have to take into account is rather an absolute number

• Sample size is hence not dependent on the size of the total "population" you study

The best sampling strategy is random selection of individuals, because:

• you have a likely chance to find representatives of each "kind" in your sample
• you avoid auto-selection (i.e. that only "interested" persons will answer your survey or participate in experiments

When you work with small samples, you may use a quota system:

• e.g. make sure that you have both "experts" and "novices" in a usability study of some software
• e.g. make sure that you (a) both interview teachers who are enthusiastic users and the contrary, (b) schools that are well equipped and the contrary in a study on classroom use of new technologies.

A first look at significance[edit | edit source]

Significance of results depend both on strength of correlations and size of samples

Therefore: the more cases you have got, the more likely your results will be interpretable ! Let's illustrate this principle with an example:

Correlation between teacher training and average of student grades

Let's assume that initially you have data from only 6 teachers (the red dots): Your data suggest a negative correlation: more training days lead to worse averages

Now let's see what happens if we only add 2 new teachers (the 2 green dots). The observed relation will switch from negative to positive, i.e. the data suggests a (weak) positive correlation.

This reversal effect shows that doing a statistical analysis on very small data sets is like gambling. If your data set had included 20 teachers or more, adding these 2 additional "green" individuals wouldn’t have changed the relationship. This is why (expensive) experimental research usually attempts to have at least 20 subjects in a group.

Typical sampling for experiments[edit | edit source]

Sampling for experiments is a simpler art. You should have

• preferably 20 subjects / experimental condition
• at least 10 / experimental condition (but expect most relations to be non-significant)

Example - The effect of multimedia on retention

The model includes three variables

• Explanatory (independent) variable X: Static diagrams vs. animation vs. interactive animation
• Dependant (to be explained) variable Y1: Short term recall
• Dependant (to be explained) variable Y2: Long-term recall

Both dependant variables (Y1 and Y2) can be measured by recall tests

For variable X we have three conditions. Therefore we need 3 * 20 = 60 subjects

• If you expect very strong relations (don’t for this type of research !) you can get away with 3 * 15

Note: we can not administer the three different conditions to each individual (because by moving from one experiment to another they will learn). You may consider building 3 * 3 = 9 different kinds of experimental materials however and have each individual do each experiment in a different condition. However, they may get tired or show other experimentation effects ... and producing good material is more expensive than finding subjects.

Typical samples for survey research[edit | edit source]

Try to get as many participants as you can if you use written or on-line surveys. Dealing with on-line data doesn't cost very much, but you certainly will get a sampling bias.

40 participants is a minimum, 100 is good and 200 is excellent for a MSc thesis.

Otherwise you can’t do any sort of interesting data analysis, because your significance levels will be too high (i.e. bad) when you analyze even moderately complex relationships.

Typical samples for aggregate data[edit | edit source]

e.g. schools, districts, countries etc.

Since these data reflect real "realities" you can do with less (however talk to an expert, a lot depends on the kinds of analysis you plan to do).

Software[edit | edit source]

Questionnaires can be administered over the Internet. The main advantage is cost reduction. On the other hand you may get a higher sampling biais.

There exist some free web survey hosting services, e.g. see The Impoverished Social Scientist's Guide to Free Statistical Software and Resources

Recommended free survey server-side software (that you can install)

• LimeSurvey (php/mysql-based)

Bibliography[edit | edit source]

Textbooks and manuals

• Fowler, Floyd, J. (2001) Survey Research Methods, Sage Publications, ISBN 0761921915
• DeVellis, R. F. (2003). Scale development: theory and applications. Applied social research methods series Volume 26. Thousand oaks, London, Delhi: Sage Publications.

Examples cited

• Dolmans, D.H.J.M.; H.A.P.Wolfhagen, A.J.J.A. Scherpbier & C.P.M. Van Der Vleuten, (2003). Development of an instrument to evaluate the effectiveness of teachers in guiding small groups, Higher Education 46: 431-446,

• Gunawerda, C.N., & Zittle, F.J. (1997). Social presence as a predictor of satisfaction within a computer-mediated conferencing environment. The American journal of distance education, 11(3), 8-26.

• Taylor, Charles and Maor Dorit, Constructivist On-Line Learning Environment Survey (COLLES) questionnaire: http://surveylearning.moodle.com/colles/ , retrieved 15:43, 11 August 2007 (MEST)