Reliability Analysis - Stanford University



Statistical Tools for Research ---SPSS (2)

Topic: Quantitative Data Analysis (Intermediate)

Date: April 14 & 15, 2003

Time: 6:00 - 8:30pm

Venue: B0415 & B0416

Facilitators: Dr. Zhang Wei-yuan (CRIDAL, OUHK)

Ms. Elaine Kwok (CRIDAL, OUHK)

This is the second session of workshops on quantitative data analysis using Statistical Package for the Social Science (SPSS) for Windows. This workshop will describe some basic statistical concepts and introduce techniques in One-Way ANOVA (Analysis of Variance), Reliability Analysis, Non-Parametric Techniques, and Multiple Response and Multiple Dichotomy analysis.

Recommended reading:

Norusis, M. J. (2000). SPSS10.0: Guide to Data Analysis, New Jersey: Prentice Hall.

Ferguson, G. A. & Takane, Y. (1989). Statistical Analysis in Psychology and Education, 6th ed,. New York: McGraw-Hill Publishing Company.

Mertens, D. M. (1998). Research Methods in Education & Psychology: Integrating Diversity with Quantitative & Qualitative Approaches, California: Sage Publications.

Wiersma, W. (2000). Research Methods in Education: An Introduction Research, 7th edn, MA, USA: Allyn & Bacon.

Lesson 5: To Run One-Way ANOVA (Analysis of Variance)

Comparing more than two population means

Example: If you use four different methods for teaching English, you want to compare average test scores for all four groups.

Independent variable and dependent variable

■ Independent variable: a variable that affects (or is assumed to affect) the dependent variable under study and is included in the research design so that its effect can be determined.

■ Dependent variable: a variable being affected or assumed to be affected by the independent variable.

Example 1: The effect of four teaching methods on reading scores on students.

■ Independent variable: teaching methods

■ Dependent variable: reading scores

Example 2: People’s average number of working hours are affected by their educational levels

■ Independent variable: educational levels (less than high school; high school; junior college; bachelor; and graduate).

■ Dependent variable: the average number of hours worked in a week

To obtain a one-way analysis of variance (ANOVA):

• You must indicate the variable whose mean you want to compare, and move it into “Dependent List”

• Select the variable whose values define the groups and move it into “Factor” box

• Click OK

Exercise:

>Open file “gssft”

>Click Analyze - Compare Means - One-way ANOVA

>Select the variable “hrs1” and move it into “Dependent List”

>Select the variable “degree” and move it into “Factor box”.

>Click “OK”

Bonferroni Multiple Comparison Test

Many multiple comparison procedures are available. One of the simplest is the Bonferroni procedure.

Select the variable “hrs1” and move it into “Dependent List”

>Select the variable “degree” and move it into “Factor box”

>click “Post Hoc” and tick Bonferroni

>Set significance level at 0.05 or 0.01

>click “Continue” and then “OK”

The difference in hours worked between the two groups is shown in the column labeled Mean Difference. Pairs of means that are significantly different form each other marked with an asterisk.

Results:

People with graduate degree work significantly longer than people with less than a high school education;

People with graduate degree work significantly longer than people with just a high school education.

Exercise

1. Repeating the sample above.

2. Use the “Gss.sav”data file:

• Is there a relationship between highest degree earned and number of hours of television viewed a day (variable “degree” & “tvhours”)?

• Dependent variable: the average number of hours of TV viewed a day

• Independent variable: educational levels (less than high school; high school; junior college; bachelor; & graduate).

Further Reading:

Norusis, M. J (2000) SPSS10.0: Guide to Data Analysis, New Jersey: Prentice Hall.

259 – 277.

Lesson 6: Reliability Analysis

• Reliability means consistency. It is the degree to which an instrument will give similar results for the same individuals at different times.

• Reliability can take on values of 0 to 1.0, inclusive.

Methods for checking Reliability:

Test-retest reliability

The calculation of test-retest reliability is straightforward. The same test is administrated on two occasions to the same individuals under the same conditions. This yields two scores for each person and the correlation between these two sets of scores is the test-retest reliability coefficient. If the test is reliable, there will be a high positive association between the scores.

• Exercise: The scores of 20 students in language proficiency test and retest

• Inputting the following data

|Student |Test |Retest |

|1 |94 |96 |

|2 |92 |87 |

|3 |88 |91 |

|4 |87 |86 |

|5 |87 |89 |

|6 |86 |86 |

|7 |85 |89 |

|8 |95 |91 |

|9 |85 |84 |

|10 |83 |86 |

|11 |82 |84 |

|12 |81 |77 |

|13 |78 |81 |

|14 |76 |71 |

|15 |72 |76 |

|16 |68 |72 |

|17 |66 |66 |

|18 |65 |72 |

|19 |63 |59 |

|20 |58 |55 |

To conduct a reliability analysis

Analyze-Correlate– Bivariate – click “Pearson” and “Flag”

Move “Test” and “Retest” to “Variables”-click “OK”

Pearson Correlation Result: r = 0.947

Split half

Only need one administration. The test items are divided into two halves, with the items of the two halves matched on content and difficulty.

Exercise: Interest Inventory (RIASEC)

Five-point scale:

very much like me 5

somewhat like me 4

neither like nor unlike me 3

somewhat unlike me 2

very much unlike me 1

Social type

1. Easy to talk with all kinds of people

2. Good at explaining things to others

3. Enjoying working as a neighbourhood organiser

4. Teach children easily

5. Teach adults easily

6. Help people who are upset or troubled

7. Good understanding of social relationships

8. Good at teaching others

9. Making people fell at ease

10. Better at working with people than things or ideas

Inputting the following data

| |It1 |It2 |It3 |It4 |It5 |

|I want to develop my mathematical (science) skills and study this|5 |4 |3 |2 |1 |

|subject more. | | | | | |

|Mathematics (science) is not a very interesting subject. |5 |4 |3 |2 |1 |

|Mathematics (science) is a very worthwhile and necessary subject.|5 |4 |3 |2 |1 |

|Mathematics (Science) makes me feel nervous and uncomfortable. |5 |4 |3 |2 |1 |

|I have usually enjoyed studying mathematics (science) in school. |5 |4 |3 |2 |1 |

|I don’t want to take any more mathematics (science) than I |5 |4 |3 |2 |1 |

|absolutely have to. | | | | | |

|Other subjects are more important to people than mathematics |5 |4 |3 |2 |1 |

|(science). | | | | | |

|I am very calm and unafraid when studying mathematics (Science). |5 |4 |3 |2 |1 |

|I have seldom liked studying mathematics (Science). |5 |4 |3 |2 |1 |

|I am interested in acquiring further knowledge of mathematics |5 |4 |3 |2 |1 |

|(science). | | | | | |

Inputting the following data

| |S1 |

|Ease of passing |12 |

|Course instructor |7 |

|Time of day |5 |

• Inputting the following data

| |Factors (factor) |Frequency (freq) |

|1 |1 (Interest in topic) |26 |

|2 |2 (Ease of passing) |12 |

|3 |3 (Course instructor) |7 |

|4 |4 (Time of day) |5 |

>Go to tool bar “Data” - Weight Cases - move “Freq” into “Frequency Variable” box-click “OK”

>Analyze – Nonparametric Tests – Chi-square – move “Factor” into the “Test Variable List” box-click “OK”

• Results: P = 0.000 < 0.05

• Conclusion: some of the factors are more important than others in course selection.

Chi-square test for relatedness or independence (2 x 2 table)

Exercise:

Purpose: To determine whether or not the gender is an important factor in students’ course selection

Sample: 74 male students and 72 female students

Preference for course selecting

| |Male |Female |

|Science |46 |26 |

|Social science |28 |45 |

• Inputting the following data

Labels

• Gender: 1=male 2=female

• Preference of courses (pref): 1=science 2=social science

| |Gender |Preference of courses (pref) |Number of frequency (freq) |

|1 |1 |1 |46 |

|2 |1 |2 |28 |

|3 |2 |1 |26 |

|4 |2 |2 |45 |

>Go to tool bar “Data” - Weight Cases - move “Frequency” into “Frequency Variable” box-click “OK”

>Analyze – Descriptives statistics – Crosstabs – move “pref” into “Row(s)”– move “gender” into “column(s)”

Go to “Analyze”- “Multiple Response”- “Define Sets” – “Define Multiple Response Sets”

> Move the variables from “Set Definition” (i.e. crit1 to crit 6) into “Variables in Set” box.

> In the “Variables Are Coded as” tick “Categories”

• Under the “Range: …. through …..” box - put 1 as the lowest code and 14 as the highest code

• Under the “Name”-type a suitable variable name ($crits)

• Under the “Label”-type a description of this variable (e.g. Factor considered in choosing jobs)

>Click “Add”

>Click “Close”

> Go to “Analyze”- “Multiple Response” – “Frequencies”

>Move “Selection criteria ($crits)” from “Mult Response Sets” into “Table(s) for”

>Click “OK”

Result:

• Percentage of responses refers to the proportion of a given response in relation to the count: count/total responses.

• Percentage of cases refers to the proportion of a given response in relation to the number of valid cases: count/total valid cases.

Multiple Dichotomy Analysis

Multiple dichotomy analysis is very similar to the multiple response analysis.

Exercise:

Question: please tick the important reasons why you study at the Open University.

___ Job change

___ Professional development

___ Earning university degree

___ Personal interest

__ Career advancement

• Data

• Each item would be given a variable (labels)

• If the item is ticked, give 1

• If the item is not ticked, give 0

• No items were ticked from one case, tick 9 (no answer)

• The data from the first participants may look as follows:

• Participant 1 0 1 1 0 0 (This participant ticked 2 & 3)

• Participant 2 1 1 1 1 1 (This participant ticked all five)

• Participant 3 9 9 9 9 9 (This participant didn’t tick any items)

• Participant 4 1 1 0 0 0 (This participant ticked the first two)

• Inputting the following data

| |Job |Profe |Degree |Interest |career |

|Part1 |0 |1 |1 |0 |0 |

|Part2 |1 |1 |1 |1 |1 |

|Part3 |9 |9 |9 |9 |9 |

|Part4 |1 |1 |0 |0 |0 |

To run a multiple dichotomy analysis:

> Go to “Analyze”- “Multiple Response”- “Define Sets” – “Define Multiple Response Sets”

> Move the variables from “Set Definition” (i.e. job, profe, degree, interest, career) into “Variables in Set” box.

• In the “Variables Are Coded as” tick “Dichotomies Counted value” and type the value that you assigned to those items which were ticked by respondents (i.e. 1)

• Under the “Name”-type a suitable variable name (Reasons)

• Under the “Label”-type a description of this variable (The reasons why choosing OU)

>Click “Add”

>Click “Close”

> Go to “Analyze”- “Multiple Response” – “Frequencies”

>Move “$reasons” from “Mult Response Sets” into “Table(s) for”

>Click “OK”

Result:

• Percentage of responses refers to the proportion of a given response in relation to the count: count/total responses.

• Percentage of cases refers to the proportion of a given response in relation to the number of valid cases: count/total valid cases.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download