School of Education University of Tampere, Finland

1
School of Education University of Tampere,
Finland
Issues in Study Design
Petri Nokelainen petri.nokelainen_at_uta.fi
2
CONTENTSIssues in Study DesignWriting
Scientific ReportsTen Questions About YOUR
Research
3
Issues in Study Design
Scientific research
Theoretical
Empirical
AND OR
ANDOR
E2 Textual / nominal data
T2 Innovation
T1 Research body
E1 Numerical data
AND
4
Issues in Study Design

• Qual T1,T2,E2 T1,T2

Quan E1 !
5
Issues in Study Design
Research question
6
Issues in Study Design
(Nokelainen, 2008, p. 119)
7
Issues in Study Design
8
Issues in Study Design

• The left-hand side of the figure shows two main
  categories of data collection
• Probability sample (PS) and
• Non-probability sample (NPS).
• Both methods aim to produce a scientific,
  representative sample from the target population.

9
Issues in Study Design

• According to Jackson (2006), a representative
  sample is like the population.
• Thus, we can be confident that the results we
  find based on the sample also hold for the
  population.
• This is not a problem with PS, which is based on
  random, stratified or cluster sampling.
• In random sampling each member of the population
  has an equal likelihood of being selected into
  the sample.
• Stratified random sampling allows taking into
  account different subgroups in the population.
• If the population is too large for random
  sampling of any sort, cluster sampling is
  applied.

10
Issues in Study Design

• Problems arise with NPS as the individual members
  of the population do not have an equal likelihood
  of being selected to be a member of the sample.
• The most commonly applied NPS technique is
  convenience sampling (CS) in which participants
  are obtained wherever they can be found and
  wherever is convenient for the researcher (Hair,
  Anderson, Tatham Black, 1998).

11
Issues in Study Design

• Why, then, educational scientists use NPS,
  typically CS?
• Simply because it tends to be less expensive
  than RS and it is easier to generate samples
  using this technique (Jackson, 2006, p. 84).

12
Issues in Study Design

• However, on the lower left-hand part of the
  figure, it is shown that when researcher ensures
  that the CS is like the population on certain
  characteristics (location and dispersion
  descriptive statistics about, for example, age
  and job title), it becomes a quota sample (QS).
• A quota sample is better than a CS as it allows
  us to ensure that the results we find based on
  the sample also hold for the population.

13
Issues in Study Design

• The upper part of the figure contains two
  sections, namely parametric and
  non-parametric divided into eight sub-sections
  (DNIMMOCS OLD).
• Parametric approach is viable only if
• 1) Both the phenomenon modeled and the sample
  follow normal distribution.
• 2) Sample size is large enough (at least 30
  observations).
• 3) Continuous indicators are used.
• 4) Dependencies between the observed variables
  are linear.
• Otherwise non-parametric techniques should be
  applied.

14
Issues in Study Design

• First, study design (D) is made on the basis of
  the research question and major goal.
• According to de Vaus (2004, p. 9), research
  design is to ensure that the evidence obtained
  enables us to answer the initial question as
  unambiguously as possible.

15
Issues in Study Design

• In order to obtain relevant evidence, we need to
  specify the type of evidence needed to answer the
  research question.
• More specifically, we need to ask Given this
  research question, what type of evidence (data)
  is needed to answer the question in a convincing
  way?

16
Issues in Study Design

• Sometimes we proceed with the so-called
  qualitative designs, sometimes a quantitative
  orientation is more appropriate, and sometimes we
  work both qualitatively and quantitatively
  (mixed-methods research, for a thorough
  discussion, see Brannen, 2004).
• Methodological, conceptual etc. triangulation.
• Design research is quite new approach, see
  Bannan-Ritland (2003).

17
Issues in Study Design

• Experimental design (a.k.a. pretest post-test
  randomized experiment) is the most recommended
  approach, but only possible with a random sample
  (a.k.a probability sample) and random
  assignment (participants are randomly selected
  for the experimental and control groups).
• Research is conducted in a controlled environment
  (e.g., laboratory) with experiment and control
  groups (threat to external validity due to
  artificial environment).
• Using experimental design, both reliability and
  validity are maximized via random sampling and
  control in the given experiment (de Vaus, 2004).

18
Issues in Study Design
19
Issues in Study Design
Random assignment to groups
Pretest
Intervention
Post-test
Experimentalgroup
Measurement (X)
Treatment
Measurement (Y)
Control group
Measurement (X)
No treatment
Measurement (Y)
20
Issues in Study Design

• Quasi-experimental design (a.k.a. non-equivalent
  groups design) resembles experimental design but
  lacks random assignment (sometimes also random
  sampling) and controlled research environment.
• This type of design is sometimes the only way to
  do research in certain populations as it
  minimizes the threats to external validity
  (natural environments instead of artificial ones).

21
Issues in Study Design

• The most popular quantitative approach in
  educational research, correlational design
  (a.k.a. descriptive study or observational
  study), allows the use of non-probability sample
  (a.k.a convenience sample).
• Most correlational designs are missing control,
  and thus loose some of their scientific power
  (Jackson, 2006).
• Some research journals accept factorial analysis
  (main and interaction effects, e.g., MANOVA)
  based on quasi-experimental design.

22
Issues in Study Design

• Observational studies can further be classified
  into cross-sectional and longitudinal studies
  (see Caskie Willis, 2006).
• Longitudinal design includes series of
  measurements over time.
• Change over time, age effect.
• Cross-sectional study involves usually one
  measurement and is thus considerably cheaper and
  faster to conduct (although producing less
  controllable and less powerful results).
• If there are several measurements, individual
  participants answers are not connected over time
  (e.g., due to anonymity).
• Causal conclusions are usually out of scope of
  this research type (ibid.).

23
Issues in Study Design

• Longitudinal design
• One sample that remains the same throughout the
  study.
• Longitudinal study produces more convincing
  results as it allows the understanding of change
  in a construct over time and variability and
  predictors of such change over time (ibid.).
• However, it takes naturally more time to carry
  out and suffers from participant drop-out.

Sample
Pretest
Intervention
Post-test
Random sample
Measurement (X)
Treatment
Measurement (Y)
24
Issues in Study Design

• Cross-sectional design
• Measurement is conducted once (or several times)
  and the sample varies throughout the study.

Sample
Pretest
Intervention
Post-test
Convenience or random sample
Treatment
Measurement (Y)
Convenience orrandom sample
No treatment
Measurement (Y)
25
Issues in Study Design
RANDOM SAMPLING
RANDOM SELECTION
pretest-posttest randomized experiment
Pre
Post
TEST
I
RS
Pre
Post
-
CONTROL
Non-Equivalent Groups Design
Pre
Post
I
TEST
RS
Pre
Post
-
CONTROL
Correlational design
CS
Pre
Post
I
TEST
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
Issues in Study Design

• Why do, then, educational scholars use
  correlational designs over controlled
  experiments?
• The first answer is simple Correlational designs
  are far easier, faster and inexpensive to conduct
  than experimental designs.
• The second answer is more complex as we need to
  ask if the controlled experiment approach is at
  all viable method to study educational research
  questions.

35
Issues in Study Design

• In science and psychology, most areas of interest
  are quite easily quantifiable and replicable
  (like, for example, freezing point of chocolate
  or systolic blood pressure).
• However, in educational research we study, for
  example, topics like pedagogical aspects of
  digital learning material (Nokelainen, 2006) or
  compare pre-existing characteristics of interest
  (e.g., gender, age, educational level).
• In such situations researchers do apply
  correlational designs, but still aim to employ
  different types of data in the analysis with a
  complementary way (quasi-experimental study).

36
Issues in Study Design

• Case study design is applied in qualitative
  research.
• The aim is to collect information from one or
  more cases and stydy, describe and explain them
  through how and why questions.
• Cases are represented, for example, by
  individuals, their communication and experiences.
  (For thorough discussion, see Flyvbjerg, 2004.)

37
Issues in Study Design

• As a conclusion, Abelsons (1995) concept of
  statistics as principled argument becomes useful
  
• Data analysis should not be pointlessly formal,
  but instead ... it should make an interesting
  claim it should tell a story that an informed
  audience will care about and it should do so by
  intelligent interpretation of appropriate
  evidence from empirical measurements or
  observations (p. 2).

38
Issues in Study Design

• Second, optimal sample size (N) is divided into
  two sections in the figure
• Samples that operate in the optimal area (n ? 30
  250) for traditional parametric frequentistic
  techniques (Black, 1993 Tabachnick Fidell,
  1996), such as t-test or exploratory factor
  analysis, and the samples that fail to do so (n lt
  30 or n gt 250).

39
Estimation of sample size

• N
• Population size.
• n
• Estimated sample size.
• Sampling error (e)
• Difference between the true (unknown) value and
  observed values, if the survey were repeated
  (sample collected) numerous times.
• Confidence interval
• Spread of the observed values that would be seen
  if the survey were repeated numerous times.
• Confidence level
• How often the observed values would be within
  sampling error of the true value if the survey
  were repeated numerous times.

(Murphy Myors, 1998)
40
Issues in Study Design

• Traditional non-parametric techniques, such as
  Mann-Whitney U-test, are considered to operate
  robustly, also with small samples (-gt lack of
  power?).
• Bayesian approach, however, is free of such
  restrictions.

41
Issues in Study Design

• Third, independent observations (IO) are always
  expected, also in time series analysis.

42
Issues in Study Design

• Controlled experiment designs, when conducted
  properly, rule out IO violations quite
  effectively (Martin, 2004), but correlational
  designs usually lack such control (e.g., to rule
  out employees co-operation when they respond to
  the survey questions).
• On the other hand, some qualitative techniques,
  like focus group analysis (Macnaghten Myers,
  2004), are heavily based on non-independent
  observations as informants are asked to talk to
  each other as an important part of the data
  collection.

43
Issues in Study Design

• Fourth, parametric techniques assume continuous
  (c) measurement level (ML) of indicators (i.e.,
  so called quantitative variables).

44
Issues in Study Design
PHENOMENON
OBSERVATION
45
Issues in Study Design
PHENOMENON
OBSERVATION
46
Issues in Study Design
Measurements
Quantitative
Qualitative
Discrete
Continuous
Nominal
Ordinal
Ordinal
Interval
Ratio
47
Issues in Study Design

• Non-parametric analysis is based on ordering of
  values and thus discrete (d) or, when applicable,
  nominal (n) values are expected (i.e., so called
  qualitative variables).
• A respondents income level (euros) or age (years
  or months) is a representative example of the
  first indicator type.
• A Likert scale from 1 to 5 is an example of the
  second indicator type (ordered discrete values).
• Respondents gender is an example of the third
  indicator type (nominal discrete values).

48
Issues in Study Design

• It is important to note that the central limit
  theorem, discovered by Pierre-Simon Laplace (1749
  - 1827), assures an approximate normal
  distribution for practically all sums of
  independent random variables.
• For example, it allows the use of parametric
  t-test with binomial or ordinal indicators (as
  the sample of normally distributed group means
  are compared, not the indicator values
  themselves).
• Bayesian analysis is based on discrete values,
  and thus, continuous values must be disceticized
  (automatically or manually) before the analysis.

49
Issues in Study Design

• Fifth, parametric techniques are technically
  based on the assumption of the multivariate
  distribution (MD) that is normal (n) by nature.
• Non-parametric techniques expect any shaped
  similar distributions (s).
• This is a great news to anyone who has collected
  real-life educational science empirical data and
  checked both univariate and multivariate variable
  distributions as usually almost all variables
  violate quite heavily against the normal
  distribution assumption with small sample sizes
  (e.g., below n 100).

50
Issues in Study Design

• Some researchers try to force their indicators to
  follow multivariate normal distribution by
  applying various transformation techniques (e.g.,
  logarithmic, square), but with varying success.
• The motivation for transformations lies behind
  the fact that in order to enable parametric
  analysis (i.e., based on, e.g., normal
  distribution) the bivariate or multivariate
  statistical dependencies (S) must be linear (l).
• It is important to note that this assumption does
  not hold for the Bayesian techniques.

51
Issues in Study Design

• Non-parametric statistics
• Chi-square ?2
• Multiway Frequency Analysis ?2
• Spearman Rank Order Correlation rS
• Mann-Whitney U
• Wilcoxon Signed Rank
• Kruskal-Wallis H
• Friedman
• Bayesian dependency modeling (B-Course)
• Logit analysis, Logistic regression
• Bayesian classification modeling (B-Course)
• Categorical variable modeling (Mplus)

Parametric statistics Pearson Product Moment
Correlation rP Independent-samples
t Paired-samples t One-way between-groups ANOVA
F Two-way repeated-measures ANOVA F ANCOVA,
MANOVA Regression analysis R Exploratory factor
analysis Principal component analysis Cluster
analysis Discriminant analysis Classification
analysis Confirmatory factor analysis
52
Issues in Study Design

• Sixth, extreme values, outliers (O), affect the
  results and, thus, the conclusions, of some
  parametric techniques severely (e.g., regression
  and discriminant analysis) and should be
  recognized and removed (see, e.g., Tabachnick
  Fidell, 1996).
• Non-parametric analysis techniques are not
  affected by such values as their analysis is not
  based on multivariate normal assumption (i.e.,
  linear dependencies between variables).

53
Issues in Study Design

• Seventh, when calculating correlations (C),
  Pearson product moment correlation (rP) should be
  applied with continuous indicators, and Spearman
  rank-order correlation (rS) with ordinal
  indicators.
• Both techniques are valid to detect linear
  dependencies.

54
Issues in Study Design

• The last point is to discuss about the two types
  of statistical dependencies (S) among the
  variables under analysis, namely linear (l) and
  non-linear (nl).

55
Issues in Study Design

• It is natural to assume, that both parametric and
  non-parametric techniques designed to detect
  linear dependencies work best with samples that
  contain linear dependencies.
• However, there are non-linear techniques, such as
  Bayesian analysis and neural networks that also
  allow the investigation of both dependency types.

56
Issues in Study Design

• The figure contains a reference to the
  qualitative analysis techniques, referring here
  to the empirical textual evidence based approach
  (e.g., individual or focus group interviews,
  narrative stories).

57
Issues in Study Design

• Firstly, it is obvious that qualitative research
  operates with small samples (usually n lt 30).
• There is nothing suspicious working with small
  samples Bartlett, Pavlov, Piaget and Skinner did
  that too!

58
Issues in Study Design

• Secondly, probability samples could also be used
  by qualitative researchers (as stated in the
  figure), but not as the only way to produce
  scientifically important findings.
• Gobo (2004) illustrates this by listing important
  qualitative research studies based solely on
  non-probability samples
• Alvin Gouldner (1920-1980)
• Howard Becker (1928-)
• Ernest De Martino (1908-1965)
• David Sudnow (1938-2007)
• Aaron Cicourel (1928-).

59
Issues in Study Design

• Gobo (2004) defines a new concept of
  generalizability for qualitative research by
  arguing that the concept of generalizability is
  based on the idea of social representativeness,
  which allows the generalizability to become a
  function of the invariance (regularities) of the
  phenomenon.
• Thus, The ethnographer does not generalize one
  case or event but its main structural aspects
  that can be noticed in other cases or events of
  the same kind or class. (id., p. 453.)

60
Issues in Study Design

• Thirdly, both qualitative and Bayesian analysis
  techniques allow researcher to apply a priori
  input to the modeling process and update the
  model on the basis of increased level of
  knowledge.

61
Issues in Study DesignWriting Scientific
ReportsTen Questions About YOUR Research
http//www.uta.fi/aktkk/lectures/sw
62
Writing scientific reports
Original idea for the research
Database of scientific knowledge
Conclusions
63
Writing scientific reports

• Title
• Author(s) name(s) and affiliation(s)
• Abstract and keywords
• Introduction / Goals or aims of the study
  (periodicals research questions)
• Theoretical framework / literature review
  (periodicals research questions)
• Research questions (dissertation)
• Method
• 7.1 Sample, participants
• 7.2 Measures / instruments
• 7.3 Procedure
• 7.4 Statistical analyses
• Results
• Conclusion(s) and/or Summary
• Discussion
• Acknowledgements / credits
• References
• Appendix(es)

64
Writing scientific reports

• 1. Introduction
• School leadership is currently one of the most
  widely studied and published areas in social
  sciences. However, leadership as a social
  process, affecting both end products and
  personnel emotions, is seldom studied (Nokelainen
  Ruohotie, 2006). In this sense one interesting
  direction to look at is Emotional Intelligence
  (EI) research that has recently become one of the
  most important constructs in modern psychological
  research. EI refers to the competence to
  identify, express and understand emotions,
  assimilate emotions in thought, and regulate both
  positive and negative emotions in one and others
  (Matthews, Zeidner, Roberts, 2002, p. 123).

65
Writing scientific reports

• 2. Theoretical Framework
• The theory as formulated by Salovey and Mayer
  (1990 Mayer Salovey, 1997) framed EI within a
  model of intelligence. Golemans model formulates
  EI in terms of a theory of performance (1998b).
  Goleman argues (2001) that an EI-based theory of
  performance has direct applicability to the
  domain of work and organizational effectiveness,
  particularly predicting excellence in jobs of all
  kinds, from sales to leadership. Goleman,
  Boyatzis and McKee further state (2002, p. 38)
  that EI characteristics are not innate talents,
  but learned abilities.

66
Writing scientific reports

• 2. Theoretical Framework
• Theoretical framework is summarized in Figure 1.
  Figure represents self-regulation (Zimmerman,
  1998, 2000) as a system concept (Boekaerts
  Niemivirta, 2000) managing leadership behavior
  through interactive processes between motivation,
  volition, emotion, attention, metacognition and
  action control systems. As Markku Hannula (2006)
  points out, self-regulation should be seen to be
  much more than mere metacognition.

67
Writing scientific reports
Figure 1. Self-regulation as a system concept
managing leadership competence through
interactive processes between different control
systems. (Adapted from Zimmerman, 2000, p.
15-16.)
68
Writing scientific reports

• 2. Theoretical Framework
• Daniel Goleman popularized the term emotional
  intelligence and claimed that EI was as powerful
  and at times more powerful than IQ in predicting
  life success (1995, p. 34). He aimed to show in
  his studies that emotional and social factors are
  important (1995, 1998a), but his views on EI
  often went far beyond the evidence available
  (Brackett et al., 2004). A recent study showed
  that most popular EI and ability measures are
  only related at r lt .22, i.e. about five per cent
  of common variance (Brackett Mayer, 2003).

Brackett, M. A., Lopes, P., Ivcevic, Z., Pizarro,
D., Mayer, J. D., Salovey, P. (2004).
Integrating emotion and cognition The role of
emotional intelligence. In D. Dai, R. J.
Sternberg (Eds.), Motivation, emotion, and
cognition Integrating perspectives on
intellectual functioning (pp. 175-194). Mahawah,
NJ Lawrence Erlbaum Associates.
69
Writing scientific reports

• 3. Method
• 3.1 Sample
• The non-probability sample consists of 124
  Finnish teachers from four comprehensive (n 84)
  and two upper secondary (n 40) schools. All the
  schools were located in Helsinki, capital of
  Finland (about 560 000 inhabitants, 9.3 of total
  population 5 223 442). Each respondent was
  personally invited to complete a paper and pencil
  version of the questionnaire. Participants were
  asked to evaluate their attitude towards the
  statements measuring emotional leadership.

70
Writing scientific reports

• 3. Method
• 3.1 Sample
• The respondents age was classified into four
  categories (1) 21 to 30 years old (n 18,
  14.5) (2) 31 to 40 years old (n 25, 20.2)
  (3) 41 to 50 years old (n 34, 27.4) (4) over
  50 years old (n 39, 31.5). Seventy per cent of
  the respondents were females (n 87, 70.2), the
  rest were males (n 29, 23.4).

71
Writing scientific reports

• 3. Method
• 3.2 Instrument
• Emotional Leadership Questionnaire
  operationalises Goleman and his colleagues (2002)
  four domains of emotional intelligence
  characteristics with 51 items (1)
  self-awareness, (2) self-management, (3) social
  awareness and (4) relationship management. Table
  1 depicts four EL domains and the eighteen
  associated characteristics (see Appendix for item
  level details).

72
(No Transcript)
73
Writing scientific reports

• 4. Results
• Next, we examine with descriptive statistics how
  subordinates evaluated their superiors
  emotional leadership. Table 1 shows that the
  school principals had quite strong self-awareness
  (M 3.7 3.8, SD 0.8 1.0). This finding is
  natural, as especially self-confidence is an
  important characteristic of a good leader. On the
  other hand, we suspect that this result is partly
  a self-fulfilling prophecy as teachers expect to
  see those atypical Finnish mentality
  characteristics strongly present in their
  leaders.

74
Writing scientific reports

• 4. Results

75
Writing scientific reports
Figure 2. Comparison of disagreement (SD) between
the three age groups on the IRSSQ dimensions.
76
Writing scientific reports
Figure 2. Comparison of disagreement (SD) between
the three age groups on the IRSSQ dimensions.
77
Writing scientific reports
Figure 3. Bayesian network of Finnish school
principals Emotional Leadership competencies.
78
Writing scientific reports
Figure 3. Bayesian network of Finnish school
principals Emotional Leadership competencies.
79
Writing scientific reports
80
Writing scientific reports
81
Writing scientific reports
Box-plot
82
Writing scientific reports

• 5. Conclusions
• In this paper, we presented a 51 item self-rating
  Likert-scale Emotional Leadership Questionnaire
  (ELQ) that operationalises Golemans et al.
  (2002) four domains of emotional intelligence.
  Our goal in this paper was to study with an
  empirical sample the construct validity of the
  four-domain model (Goleman et al., 2002) of EL.
  The non-probability sample consisted of 124
  Finnish school teachers from six different
  capital area schools.

83
Writing scientific reports

• 6. Discussion
• We asked teachers to evaluate their superiors
  according to our fixed, person-related questions.
  In the next version of the ELQ, we will add an
  additional scale measuring the importance of each
  question in a five-point Likert scale. This
  allows us to compare personal level EL factors to
  other measures, for example, the Multiple
  Intelligences Profiling Questionnaire (MIPQ), an
  operationalization of Howard Gardners MI
  theory, (Tirri, K., Komulainen, Nokelainen,
  Tirri, H., 2002).

84
Writing scientific reports

• References (APA style, http//www.apa.org)
• Bar-On, R., Tranel, D., Denburg, N. L.,
  Bechara, A. (2003). Exploring the neurological
  substrate of emotional and social intelligence,
  Brain, 126(3), 1790-1800.
• Boekaerts, M., Niemivirta, M. (2000).
  Self-regulation in learning finding a balance
  between learning and ego-protective goals. In M.
  Boekaerts, P. R. Pintrich, M. Zeidner (Eds.),
  Handbook of Self-regulation (pp. 417-450). San
  Diego, CA Academic Press.
• Carmines, E. G., Zeller, R. A. (1979).
  Reliability and Validity. Beverly Hills, CA Sage
  Publications.
• EQ Symposium (2004). About Reuven BarOns
  Involvement in Emotional Intelligence. Retrieved
  April 13, 2007, from http//www.cgrowth.com/rb_bio
  lrg.html

85
Writing scientific reports

• Test your APA knowledge What is WRONG with the
  following reference list?
• Cohen, J. (1988) Statistical power analysis for
  the behavioral sciences. Second Edition.
  Hillsdale, NJ, Lawrence Erlbaum Associates.
• Hair, J. F., Anderson, R. E., Tatham R. L.
  Black, W. C. (1995). Multivariate data analysis.
  Fourth edition. Englewood Cliffs Prentice Hall.
• Howell, David (1997). Statistical Methods for
  Psychology. Belmont, CA Wadsworth Publishing
  Company.
• Nokelainen, P., Tirri, H. (2007). The Essential
  Benefits of Using Bayesian Modeling in
  Professional Growth Research. In S. Saari T.
  Varis (Eds.), Professional Growth, 413-423).
  Hämeenlinna, FI RCVE.
• Kerlinger, F. 1986. Foundations of Behavioral
  Research. Third Edition. New York CBS College
  Publishing.
• Tirri, K., and Nokelainen, P. (2008).
  Identification of multiple intelligences with the
  Multiple Intelligence Profiling Questionnaire
  III. Psychology Science Quarterly, 50(2),
  206-221.

86
Issues in Study DesignWriting Scientific
ReportsTen Questions About YOUR Research
87
Ten questions about YOUR research

• Critical Appraisal Skills Programme (CASP)
  http//www.phru.nhs.uk/casp/casp.htm
• Rigor
• Credibility
• Relevance
• (Abelson, 1995.)

88
Ten questions about YOUR research

• Screening
• Was there a clear statement of the aims of the
  study?
• Is a qualitative/quantitative methodology
  appropriate?
• Research Design
• Was the research design appropriate to address
  the aims of the research?
• Was the sampling technique/recruitment strategy
  appropriate to the aims of the research?

89
Ten questions about YOUR research

• Data collection
• Were the data collected in a way that addressed
  the research issue?
• Justification of the setting for data collection.
• Clarification how data were collected (e.g.,
  questionnaire, focus group, semi-structured
  interview,..).
• Justification of the chosen methods.
• Explicit report on methods (e.g., how the
  instruments were delivered, what were the
  instructions, how interviews were conducted, was
  there a topic guide, how data was stored, ..).
• Explicit report on modifications to the methods
  during the study.
• Discussion of the sample size (effect size,
  power)/saturation of data.

90
Ten questions about YOUR research

• Reflexivity
• Has the relationship between researcher and
  participants been adequately considered?
• Critical examination of researchers own role,
  potential bias and influence during
• formulation of research questions
• data collection, including sample recruitment and
  choice of location.
• How researcher responded to events during the
  study.

91
Ten questions about YOUR research

• Ethical issues
• Have ethical issues been taken into
  consideration?
• Detailed description how the research was
  explained to participants so that reader is
  able to assess whether ethical standards were
  maintained.
• Discussion of the issues raised by the study.

92
Ten questions about YOUR research

• Data analysis
• Was the data analysis sufficiently rigorous?
• In-depth description of the analysis process.
• Selection of the statistical techniques/use of
  thematic analysis (e.g., how the
  categories/themes were derived from the data?)
• Qualitative How the data presented was selected
  from the original sample to demonstrate the
  analysis process?
• Is a sufficient data presented to support the
  findings?
• To what extent contradictory data are taken into
  account?
• Whether the researcher critically examined their
  own role, potential bias and influence during
  analysis (qualitative selection of data for
  presentation.

93
Ten questions about YOUR research

• Findings
• Is there a clear statement of findings?
• Explicit findings.
• Adequate discussion of the evidence both for and
  against the researchers arguments.
• Discussion of the credibility of the findings
  (triangulation, respondent validation, more than
  one analyst,..).
• Discussion of the findings in relation to the
  original research questions.

94
Ten questions about YOUR research

• Value of the research
• How valuable is the research?
• Contribution to existing knowledge and
  understanding.
• Identification of new areas where research is
  necessary.
• Transferability (generalizability/representativene
  ss) of findings to other populations.
• Consideration of other ways how the research may
  be used.

95
References

• Abelson, R. P. (1995). Statistics as Principled
  Argument. Hillsdale, NJ Lawrence Erlbaum
  Associates.
• Anderson, J. (1995). Cognitive Psychology and Its
  Implications. Freeman New York.
• Bannan-Ritland, B. (2003). The Role of Design in
  Research The Integrative Learning Design
  Framework. Educational Researcher, 32(1), 21-24.
• Brannen, J. (2004). Working qualitatively and
  quantitatively. In C. Seale, G. Gobo, J. Gubrium,
  D. Silverman (Eds.), Qualitative Research
  Practice (pp. 312-326). London Sage.
• Cohen, J. (1988). Statistical power analysis for
  the behavioral sciences. Second edition.
  Hillsdale, NJ Lawrence Erlbaum Associates.
• Fisher, R. (1935). The design of experiments.
  Edinburgh Oliver Boyd.
• Flyvbjerg, B. (2004). Five misunderstandings
  about case-study research. In C. Seale, J. F.
  Gubrium, G. Gobo, D. Silverman (Eds.),
  Qualitative Research Practice (pp. 420-434).
  London Sage.

96
References

• Gigerenzer, G. (2000). Adaptive thinking. New
  York Oxford University Press.
• Gigerenzer, G., Krauss, S., Vitouch, O. (2004).
  The null ritual What you always wanted to know
  about significance testing but were afraid to
  ask. In D. Kaplan (Ed.), The SAGE handbook of
  quantitative methodology for the social sciences
  (pp. 391-408). Thousand Oaks Sage.
• Gobo, G. (2004). Sampling, representativeness and
  generalizability. In C. Seale, J. F. Gubrium, G.
  Gobo, D. Silverman (Eds.), Qualitative Research
  Practice (pp. 435-456). London Sage.
• Hair, J. F., Anderson, R. E., Tatham, R. L.,
  Black, W. C. (1998). Multivariate Data Analysis.
  Fifth edition. Englewood Cliffs, NJ Prentice
  Hall.

97
References

• Jackson, S. (2006). Research Methods and
  Statistics. A Critical Thinking Approach. Second
  edition. Belmont, CS Thomson.
• Lavine, M. L. (1999). What is Bayesian Statistics
  and Why Everything Else is Wrong. The Journal of
  Undergraduate Mathematics and Its Applications,
  20, 165-174.
• Luoma, M., Nokelainen, P., Ruohotie, P. (2003,
  April). Learning Strategies for Police
  Organization - Modeling Organizational Learning
  Prerequisites. Paper presented at the Annual
  Meeting of American Educational Research
  Association (AERA 2002). New Orleans, USA.
• Nokelainen, P. (2006). An Empirical Assessment of
  Pedagogical Usability Criteria for Digital
  Learning Material with Elementary School
  Students. Journal of Educational Technology
  Society, 9(2), 178-197.

98
References

• Nokelainen, P. (2008). Modeling of Professional
  Growth and Learning Bayesian Approach. Tampere
  Tampere University Press.
• Nokelainen, P., Ruohotie, P. (2005).
  Investigating the Construct Validity of the
  Leadership Competence and Characteristics Scale.
  In the Proceedings of International Research on
  Work and Learning 2005 Conference, Sydney,
  Australia.
• Nokelainen, P., Ruohotie, P. (2009). Non-linear
  Modeling of Growth Prerequisites in a Finnish
  Polytechnic Institution of Higher Education.
  Journal of Workplace Learning, 21(1), 36-57.
• Thompson, B. (1994). Guidelines for authors.
  Educational and Psychological Measurement, 54(4),
  837-847.
• de Vaus, D. A. (2004). Research Design in Social
  Research. Third edition. London Sage.