Introduction



SHS income project - final reportTable of ContentsTOC \o "1-1" \h \z \u2Introduction PAGEREF _Toc56588113 \h 13Conclusions PAGEREF _Toc56588114 \h 24Guidance PAGEREF _Toc56588115 \h 55Detailed results PAGEREF _Toc56588116 \h 6We are very grateful to our analyst colleagues in the Scottish Government and also to Prof Nick Bailey and Prof Glen Bramley for their contributions in quality assuring this work. IntroductionThis report is the result of a project that assessed in detail how accurately the Scottish Household Survey (SHS) measures household income. To this end, we compared SHS data to data from the Households Below Average Income (HBAI) dataset, the “gold standard” for estimating household income data in the UK.The comparison showed that SHS-based income estimates are accurate enough to determine whether a household is in poverty (before housing costs). We therefore plan to include a poverty flag in the SHS dataset, enabling SHS data users to produce poverty breakdowns of any SHS-based measure of interest.PurposeThe purpose of the project was to understand whether SHS income data is accurate enough to allow household income and poverty analysis in Scotland where this is not possible using the HBAI or administrative data sources. For example, the SHS is a great data source with a large sample size for a wide range of contexts such as travel patterns, social capital, satisfaction with services, house condition, digital inclusion and financial inclusion. Robust income data in the SHS would enable SHS data users to analyse these areas of interest by income decile, or poverty status.It should be noted that we are not proposing to produce SHS-based poverty statistics. The HBAI remains the best source of national (Scotland-level) poverty and household income statistics. For local area analysis, administrative data offers high quality poverty estimates. The added value of the SHS income data is to provide analysis of any SHS-based measure of interest by income level and poverty status.BackgroundThe Scottish Government published SHS-based local authority level poverty estimates in 2010 as experimental statistics. Following publication, there were concerns about how these related to other proxy measures, such as benefit claimants. A review conducted in 2011 concluded that the methodology was robust, even though the incomes of additional adults within a multi-adult household were imputed rather than captured directly.Further work was carried out to investigate the quality of the SHS data and the imputation procedures, including looking in particular at pensioners’ incomes, as there seemed to be a problem with under-reporting of pensioners’ incomes. A 2013 report concluded that there were many reasons to expect differences between poverty and benefit-based proxies at local level. It also found good agreement between the FRS and SHS; although there were several reasons why SHS income data may not be appropriate to use with the same confidence as the FRS data:Information is provided “off the top of the head” as part of an interview on many other topics. There is no requirement to refer to pay slips or bank statements to check the figures.Although overall income from benefits agrees well between the surveys, individual benefits may be less accurately classified in the SHS.Around a third of households in the SHS are unwilling or unable to provide complete income information and have some component of income imputed at the data processing stage of the survey.The 2013 report also suggested developing a suite of SHS questions looking at material deprivation to add to the understanding of poverty. That work resulted in the ‘families with limited resources’ measure first published in 2017. This measure combines low levels of income and being unable to afford a number of basic material necessities. However, similarly to the initial release of the SHS local poverty estimates, stakeholder response to this new measure included a degree of uncertainty, as some of the results were counterintuitive. It is not clear whether the unexpected results stem from the income, or the material deprivation aspect of the measure.In 2018, additional questions were added to the SHS to collect income data from the additional adults, reducing the amount of information that needs to be imputed. With this new income information now available, this project aims to conclusively describe how accurately the SHS is able to estimate income and poverty.ConclusionsWe produced detailed analysis to compare SHS and HBAI income sources and shared the results with a group of Scottish Government analysts and academic experts on measurement of household income. The group agreed with the conclusions set out below.We conclude that the SHS measures income accurately enough to identify households that are in poverty. However, there are some groups of households where income is underreported, namely those with a large proportion of their income coming from benefits, such as the state pension, disability benefits, and low income benefits.Key resultsThe SHS income variable compares reasonably well with HBAI income, and produces very good income estimates considering the wide remit of the survey.However, benefit payments are underreported in the SHS, which affects those households most for whom benefits are the main income source. That includes most pensioner households, as the state pension is a major income source for most of them. Other economically inactive and unemployed households are also affected by benefit underreporting.Average self-employed earnings are much higher in the SHS compared to the HBAI. Further work is required to unpick the reasons for this.The SHS income variable produces reliable before housing costs (BHC) poverty estimates for children and working-age adults, although care needs to be taken when looking specifically at economically inactive or self-employed households.Pensioners’ incomes in the SHS are not reported accurately enough to provide robust poverty estimates. Work is now underway to impute state pension and thereby improve pensioners’ incomes estimates. Until this has been done we recommend to exclude pensioners from any SHS-based income and poverty analysis.SHS-based income and poverty estimates differ from Official Statistics and can therefore not be compared directly; this needs to be highlighted and differences explained in any publications using this data.Guidance is required, explaining when to use which data source. Official Statistics (which are based on HBAI data) and administrative local poverty data, where available, are a better choice than any SHS-based poverty estimates. This guidance is provided in the Guidance section.It is possible to produce SHS-based regional poverty breakdowns where sample sizes allow, although these would be subject to fairly large measurement uncertainty. We instead recommend to use administrative data (see footnote) for regional poverty estimates.The members of the Quality Assurance group found that the income analysis undertaken was thorough, they considered the findings plausible, and agreed with the conclusions. They made a number of comments relating to how the SHS income data could be improved using imputation (state pension, council tax, disability benefits) and the suggestion to allow negative self-employed earnings responses in the SHS, consistent with HBAI.Detailed results can be found in the Detailed results section.RecommendationsWe recommend that the SHS continue to collect income data from all adults in the household.We further recommend the following changes to the SHS datasets, and there are early plans to implement these:Impute state pension and use imputed state pension to improve the existing net household income variableImpute council tax and water / sewerage charges and add as new derived variablesAdd new variable: equivalisation factorAdd new derived variable: equivalised net income after council taxAdd new derived variable: poverty flag (before housing costs)And finally we suggest the SHS team consider the following changes to the questionnaire:Consider allowing negative incomes for self-employed peopleReview how State Pension data is collected in the SHS, and whether changes to the questionnaire wording may improve the data quality for this key benefit Review self-employed data collection with a view to potentially harmonise questions with the Family Resources SurveyFurther workTwo areas were identified where further work is needed to understand discrepancies between HBAI and SHS. These are housing costs and self-employed earnings.The SHS includes a housing cost variable which could be used to produce after housing cost (AHC) income and poverty measures. There are some differences between surveys in what is considered a housing cost. The main difference affects home owners with a mortgage: While the HBAI considers the mortgage interest a housing cost but not capital repayments, the SHS includes both. Either method has merit, but each will lead to a quite different result for these households: Housing costs for home owners with a mortgage are higher in the SHS compared to the HBAI; therefore AHC poverty is higher for this group as well. Further work might involve a comparison of housing costs across surveys, investigating the impact of the different definitions on AHC poverty estimates, and finding ways to clearly communicate the differences between HBAI- and SHS-based housing costs.Self-employed earnings are much higher in the SHS compared to the HBAI. Checking with SHS data from previous years shows that this high level of self-employed earnings is consistent across years. The difference between surveys could be caused by questionnaire wording, survey time spent on self-employed earnings, or different definitions of self-employment. Further work could investigate the causes of the difference further.GuidanceThere are a range of data sources for poverty and child poverty reporting. We have listed currently available local child poverty data sources on the Scottish Government Child Poverty statistics webpage.When to use which data source for poverty reportingRequired analysisBest datasetMore informationUK comparisons of poverty levels and trendsHBAIHouseholds below average incomes publication (DWP), annual report and tablesScotland-level poverty estimatesHBAIPoverty and Income Inequality analysis, annual report, tables, additional analysisScotland-level poverty estimates broken down by household or personal characteristics, or urban/rural classHBAIPoverty and Income Inequality analysisScotland-level child poverty estimatesHBAIChild poverty analysisScotland-level child poverty estimates broken down by household or personal characteristics, or urban/rural classHBAIChild poverty analysisBelow Scotland-level (e.g.?council area level) poverty estimatesnot yet available, but could use administrative child poverty estimates (CiLIF) as proxyChildren in low income families publication (DWP), annual report, tables; small area data available on StatXPloreBelow Scotland-level (e.g.?council area level) child poverty estimatesCiLIFChildren in low income families publication (DWP)Below Scotland-level (e.g.?council area level) child poverty estimates broken down by family type, work status, child ageCiLIFChildren in low income families publication (DWP)SHS indicators broken down by poverty or child poverty status (before housing costs)SHSNone published yet; SHS data users will be able to produce these breakdowns once SHS dataset changes are implemented.SHS indicators broken down by poverty or child poverty status (after housing costs)not yet available, but BHC measure together with local housing market information could serve as proxyNoneExpected differencesAs with all statistical estimates based on different data sources, poverty estimates based on different surveys differ from each other. In this analysis, SHS income estimates tended to be higher than HBAI income estimates, and poverty estimates lower. However, both surveys identified the same characteristics associated with a high or low poverty risk. The exception are the groups highlighted earlier, retired and other economically inactive and unemployed households, as well as the self-employed. Until the data quality issues are resolved, any SHS analysis singling out these groups will not result in robust income and poverty estimates.Detailed resultsThis section summarises the analysis undertaken and the results. The full analysis can be made available on request.Income methodologyWe compared SHS 2018 income data (the new, broader definition, which includes income from other adults in the household) with HBAI 2018/19 income data. For the HBAI data, we used net income without the top-income adjustment. Except for this, the HBAI income definition we used is the same as that used for official household income and poverty statistics. This definition includes some components that are not available in the SHS, such as child income, and deductions (council tax, pension contributions, maintenance and child support payments, parental contributions to students living away from home, and student loan repayments). Of the deductions, council tax is the only information (indirectly) available in the SHS, and we deducted it from SHS household income. We calculated council tax using the council tax bands and imputing single-person discounts. Where the council tax band was missing (half of the households in Shetland in the 2018 dataset, but only small numbers in other council areas), we assigned council tax band B, the most common band.We also looked at aggregated administrative data on benefits to quantify the amount of benefit underreporting in the SHS (and the HBAI).Note that the income discussed here is net household income after benefits, direct taxes and national insurance contributions, and not taking housing costs into account (= before housing cost). We equivalised all incomes to account for household size, using the modified OECD scale, except when looking at aggregated incomes.For the analysis, 95% confidence intervals (C.I.) of the median were calculated using a bootstrapping method. For the HBAI, the bootstrapped C.I.s do not account for the complex survey design. They are therefore only illustrative and may be wider. The C.I.s calculated for the SHS take into account the survey design (stratification by council areas).Income resultsWe found that overall, average earnings were reasonably similar between surveys, with SHS earnings slightly higher than HBAI earnings. Note though that for lower household incomes, SHS earnings were slightly lower than HBAI earnings. While average earnings for households with employed household heads were comparable between the two surveys, earnings for the small number of households with a self-employed head were much higher in the SHS compared to the HBAI, probably due to differences in survey methodology.Earnings determine total income of employed and self-employed households, the majority of households in Scotland. For example, households where the head is employed have fairly consistent earnings levels across the two surveys, and therefore their total income also matches well. In contrast, households with a self-employed head show very different earnings levels in the two surveys, and therefore also very different income levels.Average benefit income was consistently lower in the SHS compared to the HBAI, due to underreporting and non-reporting of benefits. The impact of under- and non-reporting of the State Pension dwarfed the impact of all other (also underreported) benefits and thereby affected almost all pensioner households. Underreporting of disability benefits affected households where these benefits are a major or the main income source, such as those who are economically inactive due to disability or long-term illness.Benefit income drives total income of economically inactive and unemployed households. For example, pensioners’ incomes largely consist of the State Pension and to a smaller degree, occupational pensions. Both are considerably lower in the SHS, and therefore total income of pensioners is lower. Similarly, a large part of total income of households with a disabled household head come from disability-related benefits, which are equally much lower in the SHS, resulting in too low total incomes.The total income distribution is similar for both surveys, although the SHS distribution is slightly wider than the HBAI equivalent. As a result, a larger proportion of the population sit below the SHS poverty threshold in the SHS income distribution.Within the limited means of comparing regional incomes between surveys, we found no indication that there were any local areas with particularly large income discrepancies, and no reason to expect any.In conclusion, incomes between SHS and HBAI compare reasonably well, and we believe that the SHS produces very good income estimates considering its wide remit. However, there are some types of households where income estimates are problematic due to underreporting of benefits and differences in self-employed earnings: The largest group among them are pensioner households; other groups are households that are economically inactive due to disability, and households with a self-employed household head.The implications of the findings for poverty measurement are as follows:SHS income data enables us to calculate robust before housing costs (BHC) poverty rates. These will be higher than those based on the HBAI due to the shape of the income distribution and due to pensioners’ incomes which are too low.The differences between poverty estimates based on different surveys always need to be clearly communicated when publishing any estimates.It might also be best to exclude pensioners from any SHS based poverty analysis until state pension reporting is improved.Furthermore, when disaggregating statistics by economic status, one should be aware that poverty of self-employed households may be underestimated.And finally, regional poverty breakdowns should be possible where sample sizes allow, although it would be advisable to always include some measure of uncertainty.The members of the QA group found that the income analysis undertaken was thorough, they considered the findings plausible, and agreed with the conclusions. There were a number of comments relating to how the SHS income data could be improved using imputation (council tax, state pension, disability benefits) and the suggestion to allow negative self-employed earnings responses in the SHS, consistent with HBAI.Poverty methodologyWe compared relative poverty estimates before housing costs (BHC) that are based on SHS income data with those based on HBAI income data and Children in Low-Income Families (CiLIF) data. In contrast to the income analysis above, the HBAI income variable used here is the ‘standard’ income used in official statistics, and the HBAI poverty threshold is based on the UK median. The SHS poverty threshold is based on the SHS median.Poverty resultsWhile poverty rates and numbers differed slightly across surveys, the SHS and HBAI identify the same household characteristics to be associated with a high (and a low) poverty risk.More people were in poverty in the SHS compared to the HBAI. This is largely driven by pensioners, whose poverty risk was exaggerated in the SHS. However, we also had slightly more working-age adults and children in poverty in the SHS than in the HBAI, and consequently, higher poverty rates. The composition of children and working-age adults in poverty was similar across surveys.Disaggregating the Scottish child and working-age population by household type, economic status, tenure, council area (where available) and urban/rural area showed good agreement between SHS and HBAI. While poverty rates were generally slightly higher in the SHS, those groups of the population who had an elevated poverty risk in the HBAI also had an elevated poverty risk in the SHS.While there was generally good agreement in the disaggregated poverty estimates between surveys, there were some discrepancies when disaggregating by economic status. These were expected and can be explained by the income data quality issues identified in the SHS income analysis: benefit underreporting contributes to fairly large differences in poverty rates between SHS and HBAI for inactive and retired households, and similarly, the higher self-employed incomes in the SHS lead to much lower poverty rates in the SHS compared to the HBAI for this group.All other discrepancies between disaggregated groups in the SHS and HBAI are likely to be within the measurement uncertainty of the two surveys, with the SHS poverty rates generally slightly higher than the HBAI poverty rate.SHS and CiLIF child poverty estimates for all children also matched well, and were also comparable in the disaggregated data.Disaggregating child poverty numbers by family type, family work status, council area and urban/rural area showed very good agreement between SHS and CiLIF numbers. There were only two discrepancies: the number of children in poverty who live in non-working families (more children in poverty in the SHS), and those who live in East Renfrewshire (fewer children in poverty in the SHS). Non-working families are likely to depend on benefits, which are underreported in the SHS. This contributes to the higher numbers in poverty in the SHS. There is no obvious reason for East Renfrewshire to have fewer children in poverty in the SHS, but the estimate was based on only 40 cases and may be less robust than those for larger council areas.The implications of all findings for poverty measurement are as follows:The SHS income variable can produce reliable (BHC) poverty estimates for children and working-age adults, although some care needs to be taken when looking specifically at economically inactive or self-employed households.SHS-based poverty estimates differ from the Official Statistics and can therefore not be compared directlyGuidance is required that clearly explains when to use which data source, with the Official Statistics taking priority over SHS-based poverty estimates. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download