Randomized Experimental Design - PiratePanel



Randomized Experimental DesignWhen defining the essence of an experiment, Campbell and Stanley stressed the random assignment of subjects to experimental treatments. I stress the manipulation of the treatment variable. In this document I shall discuss designs where the treatment variable is experimentally manipulated and the subjects are randomly assigned to experimental conditions. In a later document I shall discuss the so-called quasi-experimental designs, where a treatment is manipulated but subjects are not randomly assigned to experimental conditions.Random Selection and Random AssignmentLet me remind you of the difference between random selection and random assignment. Random selection is accomplished when you randomly select subjects into your research sample from a specified population. Random assignment refers to the process of deciding which subjects get which treatments. If those decisions are made randomly, then random assignment has been accomplishedRandom SelectionSampling N data points from a population is random if every possible different sample of size N was equally likely to be selected.We want our samples to be representative of the population to which we are making inferences. With moderately large random samples, we can be moderately confident that our sample is representative.The inferential statistics we shall use assume that random sampling is employed.In fact, we rarely if ever achieve truly random sampling, but we try to get as close to it as is reasonable.You may have seen a definition of random sampling that looked something like this: "Each and every element of the population has an equal chance of being selected."). A sampling procedure may this definition but not mine. For example, sampling from a population of 4 objects (A,B,C,& D) without replacement, N = 2, contrast sampling procedure X with Y:Sampling ProcedureSampleXYAB1/21/6AC01/6AD01/6BC01/6BD01/6CD1/21/6Note. The tabled values are probabilities.If each time a single score is sampled, all scores in the population are equally likely to be selected, then a random sample will be obtained.Random AssignmentWe have already discussed the Randomized Pretest-Posttest Control Group Design and the Randomized Posttest Only Control Group Design.1068705118110R??O??X??OR??O?????O00R??O??X??OR??O?????O3443605118110R??X??OR?????O00R??X??OR?????OIn both of these designs we wish to evaluate the effectiveness of the treatment by comparing the two groups, one which received the treatment and one which did not. If we can consider our two groups to have been equivalent prior to the administration of the treatment, then their differing after the treatment is strong evidence that the treatment caused that difference. Random assignment to groups allows to consider the two groups as equivalent prior to the treatment. Technically it is the two populations from which our samples were drawn that we can consider to be equivalent -- we can expect that our two samples will differ at least a bit on this and that due to sampling error.Noise Reducing DesignsSome designs, and their associated statistical analysis, are known to enhance power. Power, you should recall, is the conditional probability of detecting an effect of the experimental treatment, given that it really does have an effect.Randomized Blocks DesignEarlier I mentioned the power-enhancing quality of the randomized blocks design. Let me give you more detail now. Suppose you were going to evaluate the effectiveness of three different methods of teaching young children the alphabet. To enhance power, you wish to use a randomized blocks design. You administer to every potential subject a test of readiness to learn the alphabet, and then you match (block) subjects on that variable. Next you randomly assign them (within each block) to groups. In your statistical analysis, the effect of the matching/blocking variable is taken out of what would otherwise be “error variance” in your statistical model. Such error variance is generally the denominator of the ratio that you use as the test statistic for a test of statistical significance, and the numerator of that ratio is generally a measure of the apparent magnitude of the treatment effect. Lets look at that ratio., for example, , or .Look at this pie chart, in which I have partitioned the total variance in the DV into variance due to the treatment, due to the blocking variable, and due to everything else (error). If we had just ignored the blocking variable, rather than controlling it by using the randomized blocks design, the variance identified as due to blocks would be included in the error variance. Look back at the test statistic ratio. Since error variance is in the denominator, removing some of it makes the absolute value of the test statistic greater, giving you more power (a greater probability of obtaining a significant result).Some of you have already studied “repeated measures” or “within subjects” designs, where each subject is tested under each treatment condition. This is really just a special case of the randomized blocks design, where subjects are blocked up on all subject variables. If you have not studied such designs or need a refresher on important topics such as order effects and counterbalancing, please read the document An Introduction to WithinSubjects Analysis of Variance.91440012001500Analysis of Covariance. Another way to reduce noise and increase power is to have available for every subject data on one or more covariate. Each covariate should be an extraneous variable which is well correlated with the dependent variable. We can then use an ANCOV (analysis of covariance) to remove from the error term that variance due to the covariate (just like the randomized blocks analysis does), but we don’t need to do the blocking and random assignment within blocks. This analysis is most straightforward when we are using it along with random assignment of subjects to groups, rather than trying to use ANCOV to “unconfound” a static-group design (more on this later in the semester).Factorial ANOVAIf the extraneous variable you wish to control is a categorical variable, one method to remove its effect from the error variance is just to designate the extraneous variable as being a classification (“independent”) variable in a factorial ANOVA. Please read my document An Introduction to Factorial Analysis of Variance. A 2 x 2 factorial design would be represented like this, using the design notation we have adopted:1899920218440R??X11??OR??X12??OR??X21??OR??X22??O00R??X11??OR??X12??OR??X21??OR??X22??OOther Randomized Experimental DesignsThere are many, I’ll illustrate a couple here.451231022860R??O??X??OR??O?????OR?????X??OR????????O00R??O??X??OR??O?????OR?????X??OR????????OSolomon Four Group DesignThis design is a combination of the pretest-posttest control group design and the posttest only control group design. It is as good at controlling threats to internal and external validity as the posttest only control group design, but superior to that design with respect to statistical conclusion validity. However, it gains that advantage at the cost of considerably greater expense and effort in the collection and analysis of the data.To test the hypothesis of no testing by treatment interaction, one could arrange all four groups’ posttest scores into a 2 x 2 factorial ANOVA design, (pretested or not) X (given the experimental treatment or not). A significant interaction would indicate that the simple main effect of the treatment among those that were pretested was not the same as the simple main effect of the treatment among those that were not pretested. The main effect of pretesting and the main effect of the treatment could also be assessed with such an ANOVA. If there existed a significant testing by treatment interaction, one could test the simple main effect of the treatment for the pretested subjects and for the not pretested subjects. One might also want to analyze the data from the pretest-posttest control group part of the design with those techniques which are appropriate for such a design (such as ANCOV on the posttest scores using pretest scores as a covariate).Here are some examples of patterns of results from the Solomon four group design. Each cell contains a posttest mean. Where a pretest was given, I have indicated the pretest mean in parentheses.Treatment effect but no testing or Testing x Treatment interactionTreatmentPre-testedControlXmarginalno103020yes10 (10)30 (10)20marginal103020Treatment and testing effects but no Testing x Treatment interactionTreatmentPre-testednoneXmarginalno103020yes20 (10)40 (10)30marginal153525Treatment and testing effects and a Testing x Treatment interactionTreatmentPre-testednoneXmarginalno103020yes20 (10)60 (10)40marginal154530Randomized Switching-Replications Design3918585113665R??O??X??O?????OR??O?????O??X??O00R??O??X??O?????OR??O?????O??X??OThis design represents an attempt to control some of the social threats to internal validity -- that is, threats that arise when members of one experimental group know that they did not get the special treatment but that the other group did. What we do is give both groups the special treatment, but one group gets the special treatment first and the other group gets it later. Of course, there still may be social effects with respect to who gets the special treatment first.Here are three patterns of results for this design. Group 1 is that which gets the special treatment first.Temporary Treatment EffectGroupPretestPosttest 1Posttest 211501301502150150130Imagine that the dependent variable is a measure of the amount of anxiety a patient feels and the treatment is administration of a drug designed to lower anxiety. The results above show that the treatment has a temporary 20 point effect.Persistent Treatment EffectGroupPretestPosttest 1Posttest 211501301302150150130Imagine that the treatment is a particular type of psychotherapy. The results above indicate that the treatment has a persistent effect of lowering anxiety by 20 points. Of course, if we took a fourth measurement later in time we might find that the effect does not persist indefinitely.Continuing Treatment EffectGroupPretestPosttest 1Posttest 211501301202150150130Here we see that anxiety continues to decline beyond the first post-treatment observation. For example, our patients may have been given some cognitive psychotherapy that teaches them ways to avoid irrational, anxiety-provoking thoughts. Although improvement was shown already at the time of the first post-treatment observation, the patients continue to improve beyond that, as they become better and better at employing the cognitive techniques.Copyright 2016, Karl L. Wuensch - All rights reserved.Fair Use of this Document ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download