Seven possibly controversial but hopefully useful rules

1
Seven possibly controversial but hopefully useful
rules

• Paul De Boeck
• K.U.Leuven

2
Examples

• Political openness and economic openness
• Life satisfaction and cost of living
• Research performance and innovation intensity
• Socioeconomic status and intolerance

3
Multi-level governance structure
4
Link concept-operationalisation
Participation Competition Transparance Accountabil
ity Rule of law Interfaces
various aspects

• Political
• openness

combine subindicies
5
Implicit theory
X1 numerical obs X2 numerical obs X3
numerical obs X4 numerical obs

the thing to be measured
test scoreglobal index
6
Measurement
X1 numerical obs X2 numerical obs X3
numerical obs X4 numerical obs

the thing to be measured
assignment rulee.g., sum, first component score
7

• Heavy vs light on meaning

8
Alternative
X1 numerical obs X2 numerical obs X3
numerical obs X4 numerical obs

Can I explain?Which model can explain?
9

• The data are not meant as a measurement of
  something, but as to be explained.
• For example, responses to inventory items,how
  can they be explained?Do the correlations
  between items stem from overlap in the
  information used to respond?Which information is
  it?Why not extract the information
  directly?What is the origin of the information?
• When explained with a quantitative theory, then
  measurement is a by-product

10

• 1. Not everything is worth being measured or can
  be measured, often the data are more interesting
  than the concept

11
Assignment of numbers

• number finding counts, percentages
• number asking ratings
• number construction apply a rule on original
  numbers in order to obtain a derived number

12
Measurement
X1 numerical obs X2 numerical obs X3
numerical obs X4 numerical obs

the thing to be measured
assignment rulee.g., sum, first component score
13
Measurement

• A quantity
• Increasing or decreasing doesnt change the
  nature
• Addition from two sources is possible
• Splitting is possible, e.g. in halves

14
Questions

• Why are you interested in the link between the
  two concepts?Why do you want to measure?Because
  I want to test a theorydata for the theoryWhy
  arent you interested in the data?and try to
  explain the data?theory for the data
• Arent your numerical variables of sufficient
  interest to keep them as they are?

15
Examples

• Woodworth Personal Schedule 1917 to measure
  psychological adaptation
• Before, lists of questions were used and one
  would listen to the responses
• Hirsch index the maximum obtained by selecting a
  number of publications with each at least the
  same number of citations, e.g., 15 articles with
  15 or more citations

16

• A strong dimension does not mean the conceptual
  component is important.It shows there are large
  individual differences in the component.

17

• 2. Psychometric criteria such as reliability and
  validity are not theory-independent

18

• The underlying theory is the simple implicit
  theory
• Alternatives- canalization one behavior has
  developed into a the dominant one and excludes
  the other behaviors- behavior competition the
  strongest takes it all- negative feedback
  showing a behavior makes it less likely to occur
  next- drop-out only occasionally it is affected
  by

19
Dynamic theories

20
Dynamic theories

21
Dynamic theories

22
Reliability

• Repetition over
• Situations
• Behaviors
• Time

23
Questions

• Do you have the simple theory for your data that
  they are a direct and linear reflection of the
  concept?
• What is your theory of stability?Stability over?

24

• 3. Always reflect on which type of covariation is
  meant when speaking about the link between two
  concepts

25
The case of shame and guilt

• Covariation over situations guilt vs shame is
  one of two dimensions
• Covariation over personsguilt shame define a
  dimension together with fear and anger
• Covariation over culturesguilt and shame define
  their own common dimension

26
Negative emotions

• Fear and anger are positively correlated over
  persons
• Fear and anger cannot co-occur because they rely
  on opposite action tendencies (flight and fight)

27
Guilt

• Experienced norm violation
• Self-reproach
• Tendency to restitute
• Unidimensional in the sense of individual-differen
  ces, and they each contribute separately to the
  probability of feeling guilty

28
Questions

• Are you interested in individual differences?Are
  you ready to find traits?
• Components of?Meaning semanticIndividual
  differencesSituational differencesTime
  differencesProbability of occurence

29

• 4. Measurement, reliability, validity, hypothesis
  testing dont need to be sequential steps

30

• Hypothesis
• link between concept A and B
• Step 1 construct a measurement for A, B
• Step 2 test reliability measurements
• Step 3 test validity measurements
• Step 4 test hypothesis

31

measurement

32

measurement reliability

33

validity measurement

34

measurement
hypothesis testing

35

validity measurement reliability
hypothesis testing

36
Questions

• Do you want to construct a test?
• ?Meaning semanticIndividual
  differencesSituational differencesTime
  differencesProbability of occurence

37

• 5. Always do a PCA

38

• PCA tells you about the sources of differences
  between the row elements
• PCA tells you whether there is interaction and
  where it is

39

• PCA is a quite robust way to check
  multidimensionality
• PCA shows the main interactions in a repeated
  measures data matrix- unidimensional equal
  loadings- unidimensional unequal positive
  loadings- unidimensional bipolar-
  multidimensional

40
Questions

• Show me your PCA before we continue, especially
  when complex methods are going to be used, such
  as SEMs

41

• 6. One does not necessarily have to care about
  the scale of the data

42

• Common concernwhat is the scale level?are
  parametric statistics permissible?
• Scale level only matters when - numbers are
  taken for an index of something else, how does
  the index relate to the something else?
• Transformations are interesting when a simpler
  and better structure can be found

43
Representations of relations

• Example
• P(Xpi1)/(1-P(Xpi1)) ?p / ?i
• ?p and ?i are on a ratio scale,as far as they
  represent odds ratios

44
Questions

• Suppose you forget about the scale level and you
  find an interesting relationship
• Do you want to generalize over other number
  assignment procedures?
• How meaningful are the numerical variables as
  they are?

45

• 7. Dont construct indices of concepts, unless
  for descriptive summaries

46
Problems

• The global index depends on the components, and
  hence, on the definition.
• Often definitions are arbitrary or they are
  mainly semantic
• Perhaps the relationships of the index follow
  from the relationships of the components

47
Questions

• What is the definition?
• What do others say?
• Arent you interested in the components?