Title: RES 341 RESEARCH AND EVALUATION
1RES 341RESEARCH AND EVALUATION
- WORKSHOP 1
- By
- Dr. Serhat Eren
- University OF PHOENIX
- Spring 2002
2CHAPTER 1
- THE ROLE OF STATISTICAL THINKING IN BUSINESS
RESEARCH
3CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.1 CHAPTER OBJECTIVES
- What Managers Should Know About Research and
Statistics Statistical Thinking-A New Paradigm
for Management - Some Situations That Call for Research and
Statistical Thinking Types of Research - Steps in Conducting Business Research
- Key Components of Research and Statistical
Thinking Organization of This Course
4CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.3 WHAT MANAGERS SHOULD KNOW
ABOUT RESEARCH AND STATISTICS
- As a manager you will be making several decisions
each day at work. - What would help you to make the right decisions?
- Will it be your experience on the job, your sixth
sense or hunch, or will you just trust to good
luck? - For sure, all these will play a part after you
have thoroughly investigated or researched the
problem, analyzed some data, and generated some
alternative solutions to choose from.
5 CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.3 WHAT MANAGERS SHOULD KNOW
ABOUT RESEARCH AND STATISTICS
- Some of the main tools of research are
statistical techniques because in attempting to
understand a problem you will almost always
gather and analyze data. - Statistical techniques allow you to see the
information in the data. Thus, an understanding
of statistics is needed to do the research
necessary for you to be a good manager. - The use of statistical techniques has long played
an important role in quality control and quality
improvement in business and industry.
6CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.3 WHAT MANAGERS SHOULD KNOW
ABOUT RESEARCH AND STATISTICS
- Unfortunately, for too long quality has been
relegated to the "quality department" and not
integrated into the whole organization. - The purchasing department would purchase the raw
materials needed at the cheapest price without
regard to quality and then would throw the
material "over the wall" to the engineers. - One of the reasons that people prefer to have the
quality department handle quality issues is that
thinking about quality leads to thinking about
data analysis.
7CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.3 WHAT MANAGERS SHOULD KNOW
ABOUT RESEARCH AND STATISTICS
- Data analysis requires the use of statistical
techniques, which are often viewed as difficult. - Four issues must be addressed if statistics is to
become an integral part of management - Managers must understand why they need to possess
statistical knowledge. - Current and future managers (that's you) must
have this knowledge. - Steps must be taken to ensure that the knowledge
is used. - The payoff from the knowledge and its application
must be measured.
8CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH1.4 Statistical Thinking- A
New Paradigm and Statistics
- Look at the Figure 1.1 below. What do you see?
- Some of you will see two faces and some of you
will see a vase. You probably can see both of
these things, but which one did you see first? - There is no right or wrong answer here. What you
see first depends on your viewpoint, your lens,
and your glasses. This is your paradigm this is
how you see things.
9CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.6 TYPES OF RESEARCH
- Research is simply the process of finding
solutions to a problem after a thorough study and
analysis of the situation and data. - The term business research simply tells us that
the problem to be investigated is in the work
setting. Business research comprises a series of
steps with the goal of finding answers to the
issues that are of concern to the manager. - Business research is process of finding solutions
to a specific problem encountered in the work
setting.
10CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.6 TYPES OF RESEARCH
- Business research can be undertaken for two
different purposes. One is to solve a current
problem faced by the manager. - For example, a particular product may not be
selling well and the manager might want to find
the reasons for this in order to take corrective
action. Such research is called applied research.
- The results are applicable only to this
particular product in this particular company at
this particular time.
11CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.6 TYPES OF RESEARCH
- Applied research is research, done with the
intention of applying the results of the findings
to solve specific problems being experienced in
the organization. - Basic research is research done mainly to
increase the understanding of certain problems
that commonly occur in organizational settings.
12CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.7 STEPS IN CONDUCTING
BUSINESS RESEARCH
- As you gain some experience with doing business
research, you will find that although the
situations and problems are varied, the steps
necessary to successfully complete the research
can be generalized to apply in any situation. - Many authors have summarized these steps and you
may find some procedures with 5 steps, some with
6 or 8 steps.
13CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.7 STEPS IN CONDUCTING
BUSINESS RESEARCH
- These procedures simply differ in how the
necessary work is divided up into manageable and
logical steps. - Step 1. Problem Identification
- Step 2. Statement of Desired Goal or Outcome
- Step 3. Research Evidence and Hard Data
- Step 4. Outcomes
- Step 5. Identification of Possible Cause(s)
- Step 6. Proposed Solutions
14CHAPTER I THE ROLE OF STATISTICAL THINKING IN
BUSINESS RESEARCH 1.8 KEY COMPONENTS OF
STATISTICAL THINKING
- It is important to understand the components of
this new paradigm that we have called statistical
thinking. There are three key components - We must use data whenever possible to guide us.
- We must look for connections and relationships.
- We must understand why data values differ from
each other. - Statistics is a branch of mathematics dealing
with the analysis and interpretation of masses of
data.
15CHAPTER 11
16CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.1 CHAPTER OBJECTIVES
- The three-level approach to needs assessment
- Data collection methods used for needs assessment
- Four types of needs assessment, classified
according to the type of problem being addressed
and the conditions under which the problem occurs - Some guidelines that may be applied to almost any
needs assessment
17CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.2 LEVELS OF ANALYSIS
- Needs assessments have traditionally been
classified as focusing on three levels of
analysis organization, job or task, and
individual or person? - The assessors should start by analyzing the
organization to determine what results are not
occurring and should be, and what organizational
factors are contributing to that condition. The
assessors should then analyze work, jobs, or
tasks to determine what performance should be
occurring.
18CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.2 LEVELS OF ANALYSIS
- Finally, assessors should study individuals to
determine who needs learning to accomplish those
jobs or tasks. - Figure 11.1 shows how these corresponding levels
can be viewed as a U-shaped process of needs
assessment leading to evaluation.
19(No Transcript)
20CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.2 LEVELS OF ANALYSIS
- By starting at the organizational level, needs
assessment is most likely to lead to
well-designed interventions with a very good
chance of solving real performance problems. - Evaluation after the program is easy because the
criteria for each level were determined before
the interventions were designed.
21CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.2 LEVELS OF ANALYSIS
- Assessors are usually cautioned to be sure that a
needs assessment encompasses these three
traditional levels (organization, work or task,
and individual). - Failure to collect assessment data at all levels
is likely to lead to misdirected resources and
low-impact interventions that are difficult to
evaluate.
22CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.2 LEVELS OF ANALYSIS
- It should be pointed out, however, that not all
needs assessors are given the opportunity to
perform an all-encompassing three-level analysis.
- The levels of analysis included in an assessment
depend on such factors as the status of the
assessor in the organization, available time and
resources, management policy, and the assessor's
biases.
23CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.3 METHODS EMPLOYED
- Needs assessors have long been known for
employing a wide variety of methods? - Table 11.1 summarizes the methods used.
24(No Transcript)
25CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.3 METHODS EMPLOYED
- Some notable and encouraging trends evident in
needs assessment include the following - Multi-method strategies- Although surveys and
interviews are the most popular methods, many
others are used as well.
26CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.3 METHODS EMPLOYED
- Integration of qualitative methods-Interviews and
focus groups provide open-ended information that
enriches an assessment. True revelations tend to
occur more frequently with qualitative methods,
largely because quantitative data collection
instruments constrain the range of responses.
27CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.3 METHODS EMPLOYED
- Appropriate use of surveys- The message to
assessors is this Surveys are powerful tools,
but they are best used in conjunction with other
methods, and they are not a necessary part of
needs assessment, even with a large group.
28CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.4 FOCUS OF NEEDS ASSESSMENT
- Traditionally, needs assessment had been
deficiency oriented, designed to identify and
address existing deficiencies or gaps in
performance. A deficiency approach is, by
definition, focused in the present. - Opportunity-oriented needs assessment is future
oriented, identifying performance gaps likely to
occur in the future and proactively implementing
solutions to prevent them.
29CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.4 FOCUS OF NEEDS ASSESSMENT
- And if organizations are to handle today's
fast-changing business environments, needs
assessments must be more focused on the future. - R. Revans has suggested that organizations face
four types of change, varying along two
dimensions.
30CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.4 FOCUS OF NEEDS ASSESSMENT
- One dimension is the type of problem-known or
unknown. - The other dimension is the conditions under which
the problem occurs-also divided into known and
unknown. - The four-cell matrix that results, Figure 11.2,
can be used to describe the following four types
of needs assessments
31(No Transcript)
32CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.4 FOCUS OF NEEDS ASSESSMENT
- Corrective needs assessments are those that
analyze existing problems in existing
circumstances to identify performance problems. - Adaptive needs assessments occur in organizations
that find themselves performing under new
conditions, but facing the same job demands as in
the past.
33CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.4 FOCUS OF NEEDS ASSESSMENT
- Developmental needs assessments are those
designed to improve an organization's ability to
deal with additional problems in the existing
environment and conditions. - Strategic needs assessments require the
anticipation of unknown problems or deficiencies
that are likely to occur in the future under
changing, but unknown, conditions.
34CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.4 FOCUS OF NEEDS ASSESSMENT
- Traditional needs assessment and performance
analysis methodologies are strong tools for
corrective needs assessments, limited tools for
adaptive and developmental needs assessments, and
weak tools for strategic needs assessments. - Needs assessors have traditionally avoided
strategic needs assessments because they require
a great deal of "crystal ball" analysis.
35CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.4 FOCUS OF NEEDS ASSESSMENT
- However, needs assessors must develop the
capability to assist organizations with
non-corrective assessments. - Corrective assessments, by definition, place the
human resource development (HRD) person
conducting the assessment in a reactive mode. But
no organization can afford to wait for
performance problems to occur in order to figure
out how to prevent them.
36CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.4 FOCUS OF NEEDS ASSESSMENT
- Experience indicates that the basic assessment
methods appear to work in strategic assessments
with minimal adaptation. - Perhaps the barriers lie more in a lack of
strategic thinking in HRD, not in the lack of
methodologies.
37CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.1 Plan to use data
- Someone once said, "Planning is a substitute for
good luck." The data you collect in a needs
assessment can be useful long after you've used
it to determine training needs. - You should plan to make your needs-assessment
data an integral part of your whole training
project. The data can keep you from having to
"reinvent the wheel" at every stage of the
training, as new needs arise.
38CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.2 Establish criteria or goals
- What do you expect your needs assessment to find?
- List your expectations and make a note of
conditions prior to the assessment.
39CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.3 Research the topic, task, or focus
- Become familiar with the subject that is targeted
by your needs assessment. - Interview people in the field.
- Review books, journals, and reports on the topic.
40CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.4 Create a guidance group
- Get together a group of people whose function is
to make decisions and keep the needs assessment
on track. - The group should include some of the people who
requested the needs assessment, some subject
matter experts, and some stakeholders-people who
may be affected by the outcome.
41CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.5 Identify your data sources
- With the group's help, determine the best sources
for the data you'll need. - Even while collecting data, continue looking for
other sources of information. Ask the people who
provide you with information to recommend
additional resources.
42CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.5 Identify your data sources
- There are two kinds of data hard and soft.
- Hard data are factual. They can take the form of
production reports, defective-parts reports,
recall reports, and absentee reports. - Such data provide real numbers that you can
count, analyze, and translate into statistics.
43CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.5 Identify your data sources
- Soft data are pieces of information that are
obtained through such means as group discussions,
interviews, questionnaires, and literature
reviews. - Soft data are subjective-similar to opinions and
beliefs-but they can often be supported by hard
data.
44CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.6 Design an interview format
- A trial run can help you refine the content of
your interviews with the people on your list of
sources and stakeholders. - The more refined an interview is, the more
information you're likely to get. Carefully
planned interviews also take less time to conduct.
45CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.6 Design an interview format
- Construct a questionnaire-either on paper or on a
computer disk-to use as the basis for your
interviews. - For the trial run, ask several people in your
guidance group to answer the questionnaire.
46CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 5 PLANNING AHEAD
- 11.5.6 Design an interview format
- In both the trial and actual interviews, describe
for participants the goals of your needs
assessment. Tell them who authorized it. - You also should tell them why they were selected,
how you plan to use the information they provide,
how much you value their participation, what
kinds of feedback they'll receive, and how they
might benefit from the outcome of the assessment.
47CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 6 COLLECTING DATA
- 11.6.1 Interviews
- Interviewing is an excellent way to get
information. The best interview subjects tend to
be those who are professionals in the area you're
assessing and who may be directly affected by the
outcome of the assessment. - Let an interviewee know in advance how much time
you expect the interview to take. You should have
some idea from your trial interviews.
48CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 6 COLLECTING DATA
- 11.6.1 Interviews
- When you begin an interview, ask for permission
to record the session. Warn people that you may
ask probing questions. Invite them to ask
questions as well. - Feel free to stray from your present format.
49CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 6 COLLECTING DATA
- 11.6.1 Interviews
- For example, interviewees may elaborate
immediately on some points that you'd planned to
ask later. Just glance through the questions at
the end to ensure that all of them have been
addressed. - When you can't interview people in person,
consider sending them questionnaires to fill out
instead.
50CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 6 COLLECTING DATA
- 11.6.1 Interviews
- One study shows that disk" based surveys reap
better response rates than written surveys. - Afterwards, be sure to send interviewees
thank-you notes that emphasize how valuable their
input will be to the outcome of the needs
assessment.
51CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 6 COLLECTING DATA
- 11.6.2 Group discussion
- A group discussion can bring out all kinds of
data an interactive environment tends to elicit
information that individuals might not bring up
on their own. - It's important to select a diverse group of
people, to follow an agenda, and to keep the
number of members manageable.
52CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 6 COLLECTING DATA
- 11.6.2 Group discussion
- Group size can affect the amount and kinds of
information you get. If a group is too large,
some people may not join the discussion. If a
group is too small, it may not provide the
richness or variety of information you seek. - Don't let one or two people in the group
dominate. And don't let anyone stray from the
topic.
53CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 6 COLLECTING DATA
- 11.6.3 Hard data
- If your needs assessment requires hard data, make
sure the data are relevant to and accurately
reflect work conditions that are observed during
the assessment. - Also, it's important that the data address the
concerns of people who requested the needs
assessment.
54CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 7 ANALYZING THE DATA
- 11.7.1 Compile the data
- By now, you probably have a lot of information.
- What are you going to do with it? It's up to you
and the group to decide which data is relevant.
55CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11. 7 ANALYZING THE DATA
- 11.7.2 Provide statistics
- If any of the hard data include statistics, a
statistical analysis can help clarify the
results. And sometimes, it may be necessary to
provide statistical documentation in order to
obtain resources for projects. - The data used for statistics also may be used to
monitor any changes that are implemented.
56CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.7 ANALYZING THE DATA
- 11.7.3 Prepare a report
- Your report should flow. It should lead readers
step-by-step through the needs assessment to the
results. - The report should contain a statement of purpose
and goals, dates, methodologies, data sources,
analysis techniques, and results-as well as a
list of the needs you and the group identified.
57CHAPTER 11NEEDS ASSESSMENT ANALYSIS AND
EVALUATION 11.7 ANALYZING THE DATA
- 11.7.3 Prepare a report
- Include charts and graphs for clarification. Use
the data you have collected as documentation for
the needs you list. - The report provides a solid basis for making
changes and for measuring the effectiveness of
those changes. You may even want to go ahead and
make some recommendations at this point.
58CHAPTER 8
- THE BUSINESS RESEARCH PROPOSAL
59CHAPTER VIIITHE BUSINESS RESEARCH PROPOSAL
- 8.3 Ethics in Business Research
60CHAPTER 8THE BUSINESS RESEARCH PROPOSAL8.3
ETHICS IN BUSINESS RESEARCH
- 8.3.1 Ethical Issues in the Preliminary Stages of
Investigation - Once a problem is recognized and an investigation
is decided on, it is important to inform all
employees, particularly those who will be
interviewed for preliminary data gathering
through structured and unstructured interviews,
of the proposed study. - It is also necessary to assure them that their
responses will be kept confidential by the
interviewer(s) and that individual responses will
not be divulged to anyone in the organization.
61CHAPTER 8THE BUSINESS RESEARCH PROPOSAL8.3
ETHICS IN BUSINESS RESEARCH
- 8.3.2 Ethics in Handling Information Technology
- PC technology imposes certain obligations on the
part of its users. - First, it is important that the privacy of all
individuals is protected whether they are
consumers, suppliers, employees, or others. - Second, companies also need to ensure that
confidential information relating to individuals
is protected and does not find its way to
unscrupulous vendors. - Third, care should be taken to ensure that
incorrect information is not distributed across
the many different files of the company. - Fourth, those who collect data for the company
should be honest, trustworthy, and careful in
obtaining and recording the data.
62CHAPTER 8THE BUSINESS RESEARCH PROPOSAL8.3
ETHICS IN BUSINESS RESEARCH
- 8.3.3 Ethical Issues in Experimental Design
Research - The following practices are considered unethical
- Pressuring individuals to participate in
experiments through coercion, or applying social
pressure. - Giving menial tasks and asking demeaning
questions that diminish their self respect. - Deceiving subjects by deliberately misleading
them as to the true purpose of the research. - Exposing participants to physical or mental
stress. - Not allowing them to withdraw from the research
when they want to.
63CHAPTER 8THE BUSINESS RESEARCH PROPOSAL8.3
ETHICS IN BUSINESS RESEARCH
- 8.3.3 Ethical Issues in Experimental Design
Research - Using the research results to disadvantage the
participants, or for purposes that they would not
like. - Not explaining the procedures to be followed in
the experiment. - Exposing respondents to hazardous and unsafe
environments. - Not debriefing participants fully and accurately
after the experiment is over. - Not preserving the privacy and confidentiality of
the information given by the participants. - Withholding benefits from control groups.
64CHAPTER 8THE BUSINESS RESEARCH PROPOSAL8.3
ETHICS IN BUSINESS RESEARCH
- 8.3.4 Ethics in Data Collection
- 8.3.4.1 Ethics and the Researcher
- Treating the information given by the respondent
as strictly confidential and guarding his or her
privacy is one of the primary responsibilities of
the researcher. - Researchers should not misrepresent the nature of
the study to subjects, especially in lab
experiments. - Personal or seemingly intrusive information
should not be solicited, and if it is absolutely
necessary for the project, it should be tapped
with high sensitivity to the respondent, offering
specific reasons therefor.
65CHAPTER 8THE BUSINESS RESEARCH PROPOSAL8.3
ETHICS IN BUSINESS RESEARCH
- 8.3.4 Ethics in Data Collection
- 8.3.4.1 Ethics and the Researcher
- Whatever be the nature of the data collection
method, the self-esteem and self-respect of the
subjects should never be violated. - No one should be forced to respond to the survey,
and if someone does not want to avail of the
opportunity to participate, the individual's
desire should be respected. - Nonparticipant observers should be as
nonintrusive as possible.
66CHAPTER 8THE BUSINESS RESEARCH PROPOSAL8.3
ETHICS IN BUSINESS RESEARCH
- 8.3.4 Ethics in Data Collection
- 8.3.4.1 Ethics and the Researcher
- In lab studies, the subjects should be debriefed
with full disclosure of the reason for the
experiment after they have participated in the
study. - Subjects should never be exposed to situations
where they could be subject to physical or mental
harm. - There should be absolutely no misrepresentation
or distortion in reporting the data collected
during the study.
67CHAPTER 8THE BUSINESS RESEARCH PROPOSAL8.3
ETHICS IN BUSINESS RESEARCH
- 8.3.4 Ethics in Data Collection
- 8.3.4.2 Ethical Behaviors of Respondents
- The subject, once having exercised the choice to
participate in a study, should cooperate fully in
the tasks ahead, like responding to a surveyor
taking part in an experiment. - The respondent also has an obligation to be
truthful and honest in the responses.