MVN responses and missing data - University of Bristol
Practical 22: Multivariate Normal response models and missing data.
We have so far concentrated on problems where we have one distinct ‘response’ variable that we are interested in and any other variables in our dataset are treated as predictor variables for our response. Whether a variable is chosen as a predictor or a response in an analysis generally depends on the research questions that we have formed. Sometimes factors such as collection time will govern our choice, for example in the tutorial dataset it would not make much sense to treat the exams scores at 16 as predictors for the London reading test scores which were taken five years earlier.
We may find however that our research questions result in us identifying several variables as responses. We could then fit models to each response variable in turn using the techniques we have so far discussed. This will result in separate analyses for each response variable, but we may also be interested in correlations between the responses. To investigate these correlations we would have to fit the responses as a multivariate response vector.
Another area where multivariate modelling is useful is the treatment of missing data. Generally when we fit a univariate model with missing responses, the missing data have to be discarded unless we make some additional assumptions and impute values for them. In multivariate modelling we will often have incomplete response vectors but we can still use such data by imputing the missing responses using the correlations that have been found from the complete records (See later).
In this chapter we will firstly consider a dataset with two responses and complete records for every individual. This dataset is a subset of a larger dataset, which also includes individuals who have one or other response missing. We will then analyse the complete dataset. We finally look at a dataset with more variables and show how we can use a multivariate multilevel model to perform multiple imputation (Rubin 1987). In this chapter we will consider continuous responses.
GCSE Science data with complete records only
We will firstly consider a dataset of pupil’s marks from the General Certificate of Secondary Education (GCSE) exams taken in 1989. The examination consisted of two parts, the first being a traditional written question paper (marked out of 160 but rescaled to create a score out of 100) and the second being coursework assignments (marked out of 108 but again rescaled to create a score out of 100). The dataset consists of data on 1905 students from 73 schools in England although we only have complete records on 1523 students. First we will open up the worksheet and look at the variable names:
The variables will then appear as follows:
[pic]
Here we see the two response variables, ‘written’ and ‘csework’, identifiers for the ‘student’ and the ‘school’ for each observation and one gender-based predictor variable ‘female’. As with analysis of univariate responses we can start by considering simple summary statistics and single level models.
We will first look at some summary statistics for the two responses:
The window should now look as follows:
[pic]
If we now hit the Calculate button we will get the following estimates:
->CORRelate 2 "written" "csework"
1523 observations
Means Correlations
written csework written csework
46.937 73.429 written 1.0000
S.D.'s csework 0.4749 1.0000
written csework
13.492 16.436
So here we see that the average coursework marks are higher than the written marks, coursework marks are more variable than written marks and there is a fairly high positive correlation (0.475) between the pairs of marks.
Fitting single level multivariate models
To fit multivariate models in MLwiN the software needs to create the correct format for the data. In earlier versions of MLwiN this was done via a special Multivariate window but now can be done directly via the Equations window. Perhaps the simplest multivariate model we could fit would simply replicate the summary statistics above and we will now show how to construct such a model.
The Responses window should now look as follows:
[pic]
The Y variable window will now appear and indicates that one level has been set up with identifiers in a column labeled ‘resp_indicator’. As explained in the users guide, the IGLS algorithm in MLwiN fits multivariate models by the clever trick of considering them as univariate models with no random variation at level 1 (an additional level which is created underneath the individual level). This can be achieved by transforming our original data by replication and pre-multiplication by indicator vectors that identify which variable is associated with each response. Note that this is done automatically by the software if we set up more than one response variable.
For example if we have the two equations
Writteni = Cons*β1 + u1i
Cseworki = Cons*β2 + u2i
Then we can stack the two response columns into one response column as follows:
Respir = I(r=1)*Cons*β1 + I(r=2)*Cons*β2 + I(r=1)*u1i + I(r=2)*u2i
Here r is 1 for a written response and 2 for a coursework response and the function I(x) is equal to 1 if x is true and 0 otherwise.
To set up a single-level multivariate model we now need to specify the individual level identifiers and the intercept terms in the Equations window (as we would for a univariate model):
The Add Term window will then look as follows:
[pic]
As you can see there are two options for adding terms into multivariate model. Firstly, as we will use here, we can ‘Add Separate coefficients’ which will add one term to each response equation. Alternatively we can use the ‘Add Common coefficient’ option, which allows the user to specify which responses will share a common term. This is useful if we have several responses that we believe have the same relationship with a given predictor.
Click on the Add Separate coefficients button and we will see the following:
[pic]
We now need to specify the random part of the model
This will produce (after clicking on the + and Estimates buttons twice) the following IGLS estimates:
[pic]
Here if we compare our results with the earlier summary statistics we see that the means are represented in the model by the fixed effects. The variance matrix in the model at level 2 will give variances that are approximately equal to the square of the standard deviations quoted earlier. Note however that IGLS is a maximum likelihood method and so the variances here are based on a variance formula using a divisor of n whilst RIGLS and the standard deviations earlier use a divisor of n-1 (i.e. we get exactly the same results as before if we switch to RIGLS). The covariance here can also be used along with the variance to give approximately the same estimate for the correlation quoted earlier.
If we now want to fit this model using MCMC we need to do the following:
After the 5,000 iterations have run we get (after pressing the + button once again) the following estimates:
[pic]
Here we see that MCMC gives slightly larger variance estimates but this is mainly because the estimates are means rather than modes and the parameters have skew distributions. As with univariate models we can now use the DIC diagnostic for model comparison. The deviance formula for multivariate Normal models is:
[pic]
Selecting MCMC/DIC diagnostic from the Model menu we get the following:
->BDIC
Bayesian Deviance Information Criterion (DIC)
Dbar D(thetabar) pD DIC
24711.27 24706.24 5.02 24716.29
25099.52 25095.50 4.02 25103.54
Here as with other one level models the pD diagnostic corresponds almost exactly to the correct degrees of freedom. We can compare this model with a model that assumes no correlation between responses and hence separate models for each response and we see a large reduction of DIC.
Adding predictor variables
As with single response models we can now add predictor variables to our model as fixed effects. In this dataset we only have one additional predictor, the sex of the individual students. To add new variables into a multivariate model we need to use the Add Term window.
This will add the additional two fixed effects, one to each of the response equations. We can now run the model:
After running for 5,000 iterations we get the following estimates:
[pic]
Here we see that girls do 6.3 points better on average at coursework but 3.3 points worse on average at the written paper. If we compare our model with the last model via the DIC diagnostic we see a significant reduction in DIC, which is to be expected given the two gender effects are both significantly larger than their standard errors.
->BDIC
Bayesian Deviance Information Criterion (DIC)
Dbar D(thetabar) pD DIC
24566.19 24559.21 6.98 24573.16
24711.27 24706.24 5.02 24716.29
A multilevel multivariate model
We can now consider the effect of school attended on the exam score. As there are 73 schools we will fit the school effects as random terms and this results in a two level multivariate response model which is treated in MLwiN as a three level model (with the responses treated as an additional lower level). In the bivariate case the school attended can affect both responses and so we will have two school level effects for each school and these could be correlated. This will result in a 2x2 school level variance matrix. To fit this model we firstly need to return to the Multivariate window and define our additional level.
We now need to add the two sets of random effects in the Equations window and run the model using firstly IGLS and then MCMC.
After running for 5,000 iterations we get the following estimates:
[pic]
If we were to compare the multilevel model to the single level model via the DIC diagnostic we will see the following:
->BDIC
Bayesian Deviance Information Criterion (DIC)
Dbar D(thetabar) pD DIC
23524.14 23397.53 126.61 23650.76
24566.19 24559.20 6.98 24573.17
Here we see that the multilevel specification reduces the DIC by roughly 900 suggesting a much better model. The effective number of parameters (126.66) is slightly less than the 153 parameters in the model.
As with the univariate models in earlier chapters we have partitioned our unexplained variation into that which is due to the schools and that which is due to the students.
Here the school level explains 29.3% (52.197/(52.197+125.441)) of the remaining variation in the written scores and 30.2% (79.696/(79.696+184.120)) of the remaining variation in the coursework scores. There is also a fairly strong positive correlation between responses at the school level (0.45) suggesting that schools with high average coursework scores also have high average written scores.
We can investigate this further by looking at the school level residuals for this model.
(It should be noted that MLwiN may give some error messages here. If so simply click on the OK button. These messages refer to the deletion residuals, which are currently not calculated correctly when MCMC is used. If the tick for deletion residuals is removed on the Residuals window the error messages will not appear.)
The school residuals have been stored in columns c300 and c301. We can now look at ‘caterpillar’ plots of the residuals as we have seen in earlier chapters
Two ‘caterpillar’ plots (minus the highlighting) will appear as shown below:
[pic]
In the graphs we have highlighted (in various colours) the highest and lowest performing school for each response, and two other schools that are interesting as we will see in the next plot. As we have two residuals we can also look at pair-wise comparisons, which construct a 2 dimensional representation of the above caterpillar plots.
The following graph will then appear
[pic]
Here we see the school effects for both the written and coursework scores and we see why the two additional schools have been highlighted. School 22710 (highlighted yellow in the top left of the graph) has a below average written test effect but an above average coursework effect whilst school 67105 (highlighted grey in the bottom right of the graph) has an above average written score but a below average coursework. The reason these schools are interesting is that the written test score was externally marked whilst the coursework score was internally assessed but externally moderated. Thus it would be interesting to investigate these schools more closely to confirm whether the difference in scores is due to actual disparities between pupils practical and examination abilities or due to differences between external and internal assessment.
GCSE Science data with missing records
The analyses we have performed so far would be appropriate if we had complete records for all the data collected, but in our dataset we have other partial records that we have so far ignored. Ignoring missing data is dangerous because this can introduce biases into all the estimates in the model. We will now consider how to deal with missing data in a more sensible way. There are many techniques that deal with missing data and these can be split into two families: First imputation-based techniques that attempt to generate complete datasets before fitting models in the usual way to the imputed data. Secondly model-based techniques that include the missing data as a part of the model, and hence fit more complex models that account for the missing data.
Imputation-based techniques are usually simpler but may perform worse if they fail to account for the uncertainty in the missing data whilst model-based techniques may become impractical for more complex problems. In this example we will consider a model based MCMC algorithm for multivariate normal models with missing responses. Then for our second example we will consider an imputation technique called ‘multiple imputation’ (Rubin 1987) which gets around some of the lack of uncertainty in the missing data by generating several imputed datasets.
Firstly we will load up the complete data for the Science GCSE example:
The Names window will then appear as shown below. It should be noted here that the missing data has been coded globally (you can use the Options/Numbers window to do this with your own data) as missing rather than –1 as in the example in the Users Guide. When using MCMC missing data MUST be coded globally as missing rather than using the Missing option on the Multivariate window (which will be removed in a later version).
[pic]
We can now repeat the earlier analyses using this complete dataset. We will ignore the simpler model and consider firstly the single level model with gender effects that we fitted earlier. We can set up this model in an identical way as described earlier so if you are unsure of how to do this refer back to the earlier sections. In MLwiN the IGLS and MCMC methods fit this model using different approaches which are both equivalent to assuming a ‘missing at random’ or MAR assumption (Rubin 1976). The IGLS method, due to the clever trick of treating the multivariate problem as a special case of a univariate problem, can simply remove the missing rows in the longer data vector, which contains one row for each response. (See the Users Guide for more details).
The MCMC method considers the missing data as additional parameters in the model and assumes an independent uniform prior for each missing response. Then the missing records have normal posterior distributions (multivariate normal for individuals with more than one missing response) and the Gibbs sampling algorithm is extended to include an additional step that generates values for the missing data values at each iteration in the sampling.
To run the two approaches on our dataset we need to do the following:
When the 5,000 iterations for the MCMC method have finished the Equations window will look as follows:
[pic]
Notice that we now have 3428 responses observed and 382 responses missing out of our total of 3810 responses. The results we see here are fairly similar to those for the complete records only, although the female difference effects have changed from 6.25 to 5.89 on coursework and –3.32 to –3.43 on written examination. If we look at the DIC diagnostic for this model we get the following:
->BDIC
Bayesian Deviance Information Criterion (DIC)
Dbar D(thetabar) pD DIC
30681.18 30292.23 388.95 31070.13
We cannot compare this diagnostic with other previous models as we now effectively have a new dataset. However what is interesting is that the effective number of parameters (pD) is approximately 389, which equals the number of missing responses (382) plus the number of parameters (7). So we can see clearly that the MCMC algorithm treats missing data as additional parameters in the model. Note that an alternative here would be to reformulate the deviance function to marginalize out the missing data and this would give different pD and DIC estimates.
We can now look again at the multilevel model considered earlier with the complete cases:
After running this model we get the following results, which are again similar to the estimates for the complete case data:
[pic]
The DIC diagnostic again shows a great improvement in fitting a multilevel model:
->BDIC
Bayesian Deviance Information Criterion (DIC)
Dbar D(thetabar) pD DIC
29389.19 28878.14 511.05 29900.25
30681.18 30292.23 388.95 31070.13
The MCMC method is generating values at each iteration for the missing data and we may be interested in the values generated. Currently MLwiN does not allow the user to store the chains of these missing values but summary statistics are available.
The window should then look as follows:
[pic]
Now click on the Calculate button and then the Done button to store the results and close the window.
The window should look as follows:
[pic]
Here we see that the first individual had a missing coursework response, which on average in the chain has estimate 51.2 with a standard deviation of 12.6. The second person has a missing written exam score which has predicted value 36.8 with a standard deviation of 10.4. Currently MLwiN only allows an MAR assumption for the response values although it is hoped that in future releases we will include options that allow informative prior distributions for missing data. Note that there are other modelling techniques for handling non MAR data we shall not discuss these here.
Imputation methods for missing data
As mentioned earlier there are two families of techniques that can be used for missing data; imputation and model based methods. The model-based methods as we saw in the last example make special provisions in the model for the missing data. The imputation-based methods however are techniques that are used prior to modelling. We can think in fact of imputation techniques as being two stage techniques where the first stage involves imputing a complete dataset and the second stage involves modelling using this complete dataset.
The imputation step can be deterministic, for example simple imputation techniques involve substituting missing data with a determined value. A simple example of this determined value is the mean for the variable. Alternatively a model can be created for the variables to be imputed, for example a regression model and this model can then be used to generate predicted values to replace the missing values. Of course the model will give a point estimate and a standard error for each missing observation and so we could use an alternative approach of generating a random draw from the predictive distribution (under the imputation model) of the missing observation. This will then produce a stochastic imputation procedure.
The disadvantage of imputation-based methods is that once the complete dataset is generated all uncertainty about the missing values is lost. Multiple imputation (Rubin 1987) aims to bridge the gap between standard imputation-based methods and model-based methods by using a stochastic imputation procedure several times. Then we can perform our modelling on each of these N imputed complete datasets and use for each parameter estimate an average of the N estimates produced by the datasets, along with a standard error that is constructed from a combination of the variance for each dataset and the variance between the datasets. So for a parameter ( we have an estimate
This approach would be equivalent to a model based approach if N was very large but even an N as small as 5 will give a reasonable approximation.
For more details on missing data and multiple imputation we recommend Schafer (1996). In this section we will simply show how to generate imputed datasets using multivariate multilevel models in MLwiN. The effectiveness of the imputation procedure will depend on the imputation model used. Typically single level multivariate normal models that take account of correlation between variables are used in multiple imputation. However it may be more appropriate to use a multilevel multivariate model as an imputation model (note that such models have been used independently in Schafer 1997).
Hungarian Science Exam dataset
This dataset was analysed in Goldstein(1995) and is part of the Second International Science Survey carried out in 1984. The data we are using is the component from Hungary and consists of marks from six tests taken by 2,439 students in 99 schools. Of the six papers, three are core papers in earth sciences, biology and physics which were taken by all students whilst the other three are optional papers, two in biology and one in physics of which no student took all three. Although these tests were marked out of different totals ranging from 4 to 10, Goldstein(1995) converted each test to have scores between 0 and 1. Here we will use instead scores between 0 and 10 as these are closer to the original marks.
The variables in the dataset are as follows:
[pic]
Here we see the two level identifiers, ‘school’ and ‘student’, the six test scores that are stored in columns c3-c8 and one additional predictor ‘female’ which indicates the sex of each student. As with the earlier example we could try and fit several different models to the dataset and compare the DIC diagnostic to find the ‘best’ model, here however we skip this stage and fit the model that Goldstein fits in his book.
To set up the model we first need to set up the model structure using the multivariate window.
This will set up the responses and the Equations window should look as follows:
[pic]
We now need to specify the level identifier variables and the predictor variables
We have now added the fixed effects into the model and we now need to add the random effects via the Equations window.
After the IGLS model converges you should get fixed estimates as shown below. Note that due the rescaling of the dataset these estimates are ten times the estimates in Goldstein (1995).
[pic]
We will now consider fitting this model using MCMC. Although this model is itself interesting we are using it purely as an imputation model and are more interested in the missing values imputed than the fixed effect and variance estimates. To use multiple imputation it is desirable to have independent samples from the posterior distributions of the missing data. Consequently we will run for a burn-in of 500 as usual and then store the values every thousand iterations. To get our first sample we will do the following:
After the estimation finishes (which may take a couple of minutes) we need to then do the following:
This will generate the first imputed dataset. It should be noted here that the values stored are the actual chain values after 1000 iterations and NOT the means of the sets of 1000 values. We can now repeat the procedure four more times by in turn changing the monitoring chain length to 2,000, 3,000, 4,000 and 5,000 iterations and clicking on the More button. Then each time we can change the Output column on the Missing Data screen to c32, c33, c34 and c35 respectively and generate an imputed dataset.
It should at this point be noted that we could have written a macro to generate these datasets that would use the MCMC and DAMI commands (see the Command manual). After generating the 5 datasets we can view the results in the data window:
The data will then look as follows:
[pic]
Here we see the data for the first three individuals in the dataset. As can be seen they each have one missing paper and the values for these papers have been generated from the respective posterior distribution. It should be noted at this point that in this model we have treating marks as continuous normally distributed variables. This of course means that the imputed marks will follow these normal distributions rather than the smaller set of values possible in the test. We are actively researching methods to impute ordered categorical data, although Schafer (1996) considers this subject in a non-multilevel context in great detail.
What you should have learnt from this practical
• How to fit multivariate response models using MCMC in MLwiN.
• How MLwiN deals with missing data.
• How to fit multilevel multivariate models in MLwiN.
• How to compare multivariate models using the DIC diagnostic.
• How to generate datasets for multiple imputation in MLwiN.
References
Goldstein, H. (1995). Multilevel Statistical Models (2nd edition). London: Edward Arnold.
Rubin, D.B. (1976). Inference and missing data. Biometrika, 63, 581-592.
Rubin, D.B. (1987). Multiple Imputation for Nonresponse in Surveys. New York: J. Wiley and Sons.
Schafer, J.L. (1996). Analysis of Incomplete Multivariate Data. London: Chapman and Hall.
Schafer, J.L. (1997). Imputation of missing covariates under a multivariate linear mixed model. Unpublished Technical Report.
-----------------------
Select Open Worksheet from the File menu.
Select ‘gcsecomp1.ws’ from the list and click on the Open button.
Select Average and Correlations from the Basic Statistics menu.
Select the Correlation button (as this gives averages as well).
Select both ‘written’ and ‘csework’ (you can use the mouse and the ‘Ctrl’ button).
Select Equations from the Model menu.
Click on the Responses button at the bottom of the window
Click on ‘written’ from the list that appears.
Next click on csework f click on ‘csework’ from the list that appears.
Click on the β0 (‘cons.written’) and select the ‘j(student_long)’ tick box (keeping fixed effect selected) and then click on the Done button.
Click on the β1 (‘cons.csework’) and select the ‘j(student_long)’ tick box (keeping fixed effect selected) and then click on the Done button.
Note that ‘student_long’ is a column the software has created by repeating the student identifiers twice, once for each response.
Click on the Start button.
Change estimation method to MCMC.
Click on the Start button.
Change Estimation method to IGLS.
Click on the Add Term button on the Equations window.
Select ‘female’ from the variable list.
Click on the Add Separate coefficients button.
Click on the Start button.
Change estimation method to MCMC.
Click on the Start button.
Change estimation method to IGLS
Select the Equations window from the Model menu.
Click on the response names to bring up the Y variable window.
Select ‘3-ijk’ as the number of levels.
Select ‘school’ from the level ‘3(k)’ pull down list.
Click on the Done button.
Select Equations from the Model menu.
Click on the β0 (‘cons_written’) and click on the ‘k(school_long)’ tickbox.
Click on the Done button.
Click on the β1 (‘cons_csework’) and click on the ‘k(school_long)’ tickbox.
Click on the Done button.
Click on the Start button.
Change estimation method to MCMC.
Click on the Start button.
Select Residuals from the Model menu.
Select ‘3:school_long’ from the level pull down list.
Change the SD multiplier from 1.0 to 1.4.
Click on the Calc button.
Select the Residuals button in the pair-wise box.
Click on the Apply button.
Select the Plot tab on the Residuals window.
Select ‘Residual +/- 1.4 sd x rank’.
Click on the Apply button.
Select Open Worksheet from the File menu.
Select ‘gcsemv1.ws’ from the list and click on the Open button.
Set up the single level model with gender effects as described earlier.
Click on the Start button.
Change estimation method to MCMC.
Click on the Start button.
Change estimation method to IGLS
Select Equations from the Model menu and click on the response names to bring up the Y variable window.
Select ‘3-ijk’ for the number of levels.
Select ‘school’ from the level ‘3(k)’ pull down list.
Click on the Done button.
Click on the β0 (‘cons_written’) and click on the ‘k(school_long)’ tickbox.
Click on the β1 (‘cons_csework’) and click on the ‘k(school_long)’ tickbox.
Click on the Start button.
Change estimation method to MCMC.
Click on the Start button.
Select MCMC/Missing data from the Model menu.
Select the tickbox to store SDs (in c301).
Select View or Edit Data from the Data Manipulation menu.
Click on the View button.
Select columns resp_indicator, resp, c300, and c301 which are lower down the list.
Click on the OK button and resize the window to see all four columns.
Select Open Worksheet from the File menu.
Select ‘hungary1.ws’ from the list and click on the Open button.
Select Equations from the Model menu.
Click on the Responses button.
On the list that appears click on ‘es_core’, ‘biol_core’, ‘biol_r3’, ‘biol_r4’, ‘phys_core’ and ‘phys_r2’.
Click on the Done button.
Select Equations from the Model menu.
Click on β0 (‘cons_es_core’) and select the ‘j(student_long)’ and ‘k(school_long)’ tick boxes (keeping fixed effect selected).
Click on the Done button.
Repeat this for the other constants, β1,β2,β3,β4 and β5.
Click on the Start button to run the model using IGLS.
Change estimation method to MCMC and Open the Estimation Control window.
Change the monitoring chain length to 1,000.
Click on the Start button.
Select MCMC/Missing Data from the Model menu.
Select the Calculate 1 imputation button and change the output column to c31.
Click on the Calculate button.
Select View or edit data from the Data Manipulation menu.
Click on the view button.
Select columns resp_indicator, resp, c31, c32, c33, c34, and c35.
Click on the OK button and resize the window.
[pic]
Click on the Done button on the Responses window.
On the Equations window click on the response names to bring up the Y variable window.
On the Y response window change the number of levels to ‘2-ij’.
Select ‘student’ for level 2(j).
Click on the Done button.
On the Equations window, click on the Add term button.
Choose ‘cons’ from the variable list on the Add Term window.
Click on the responses in the Equations window.
On the Y variable window that appears select ‘3-ijk’ as the number of levels.
Select ‘student’ from the level ‘2(j)’ pull down list.
Select ‘school’ from the level ‘3(k)’ pull down list.
Click on the Done button.
Click on the Add Term button.
Select ‘cons’ from the variable list and click on the Add separate coefficients button.
Click on the Add Term button.
Select ‘female’ from the variable list and click on the Add separate coefficients button.
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
Related searches
- university of scranton tuition and fees
- analyzing and interpreting data worksheet
- university of illinois track and field
- structured and unstructured data examples
- qualitative and quantitative data examples
- sas missing data imputation
- discrete and continuous data worksheets
- quantitative and qualitative data analysis
- structured and unstructured data example
- university of michigan sat scores and gpa
- race and ethnic data hud
- sun and moon data for one day