The Future of Survey Research: Challenges and Opportunities

The Future of Survey Research: Challenges and Opportunities

A Report to the National Science Foundation Based on Two Conferences Held on October 3-4 and November 8-9, 2012

Report Presented By

The National Science Foundation Advisory Committee for the Social, Behavioral and Economic Sciences Subcommittee on Advancing SBE Survey

Research:

Jon A. Krosnick ? Stanford University (chair) Stanley Presser ? University of Maryland

Kaye Husbands Fealing ? University of Minnesota Steven Ruggles ? University of Minnesota

Project Coordinator:

David L. Vannette ? Stanford University

When the National Science Foundation Advisory Committee for the Social, Behavioral and Economic Sciences formed the subcommittee on Advancing SBE Survey Research, Janet Harkness was a member. We dedicate this report to her memory, to recognize her contribution to this subcommittee and the survey research community and the work of the National Science Foundation.

Any opinions, findings, conclusions or recommendations presented in this material are only those of the authors; and do not necessarily reflect the views of the National Science Foundation.

May, 2015

Table of Contents

TABLE OF CONTENTS ....................................................................................................................................... 2

OVERVIEW ............................................................................................................................................................. 4 SECTION 1 ? BEST PRACTICES FOR SURVEY RESEARCH .................................................................................................. 6

SECTION 2 ? RECOMMENDATIONS ............................................................................................................... 12 CONCLUDING REMARKS ? WAYS FORWARD FOR NSF...........................................................................................13 REFERENCES: ....................................................................................................................................................................14

APPENDIX: ........................................................................................................................................................... 18

DOCUMENTATION OF CONFERENCES ................................................................................................. 18

BACKGROUND ................................................................................................................................................... 18

PRESENTERS ...................................................................................................................................................... 22

SUMMARIES OF PRESENTATIONS........................................................................................................... 34 SECTION 1: CONVENTIONAL SURVEY RESEARCH....................................................................................................34 Reasons for Optimism about the Accuracy of Survey Research ? Jon A. Krosnick...................................34 Probability vs. Non-probability Sampling Methods ? Gary Langer ................................................................35 Sampling for Single and Multi-Mode Surveys using Address-based Sampling - Colm O'Muircheartaigh .................................................................................................................................................................39 The Impact of Survey Nonresponse on Survey Accuracy ? Scott Keeter........................................................42 Optimizing Response Rates ? J. Michael Brick ........................................................................................................44 Modes of Data Collection ? Roger Tourangeau.......................................................................................................47 The Use and Effects of Incentives in Surveys ? Eleanor Singer.........................................................................53 Building Household Rosters Sensibly ? Kathleen T. Ashenfelter......................................................................59 Proxy Reporting ? Curtiss Cobb .....................................................................................................................................64 Improving Question Design to Maximize Reliability and Validity ? Jon A. Krosnick..............................67 Perception of Visual Displays and Survey Navigation ? Stephen Kosslyn....................................................71 Cognitive Evaluation of Survey Instruments: State of the Science and Future Directions ? Gordon Willis ...........................................................................................................................................................................................74 Survey Interviewing: Deviations from the Script ? Nora Cate Schaeffer ......................................................77 Challenges and Opportunities in Open-Ended Coding ? Arthur Lupia .........................................................85 What Human Language Technology can do for you (and vice versa) ? Mark Liberman .......................90 Confidentiality and Anonymity ? Roger Tourangeau.............................................................................................93 Respondent Attrition vs. Data Attrition and Their Reduction - Randall J. Olsen.......................................97 Computation of Survey Weights ? Matthew DeBell ................................................................................................99 SECTION 2: OPPORTUNITIES TO EXPAND DATA COLLECTION .......................................................................... 102 Paradata ? Frauke Kreuter............................................................................................................................................ 102 Interviewer Observations ? Brady T. West .............................................................................................................. 106 Leave-behind Measurement Supplements ? Michael W. Link ......................................................................... 109 Experience Sampling and Ecological Momentary Assessment ? Arthur A. Stone .................................. 110 Biomarkers in Representative Population Surveys ? David Weir.................................................................. 114 Specialized Tools for Measuring Past Events ? Robert Belli........................................................................... 117 SECTION 3: LINKING SURVEY DATA WITH EXTERNAL SOURCES .................................................................... 121 Linking Survey Data to Official Government Records ? Joseph W. Sakshaug......................................... 121 Linking Knowledge Networks Web Panel Data with External Data ? Josh Pasek................................. 123

2

Linking Survey Data with the Catalist Commercial Database ? Bob Blaemire ...................................... 125 Election Administration Data ? Michael P. McDonald ..................................................................................... 126 Challenges with Validating Survey Data ? Matthew K. Berent ...................................................................... 128 Improving Government, Academic and Industry Data-Sharing Opportunities ? Robert Groves..... 130 SECTION 4: IMPROVING RESEARCH TRANSPARENCY AND DATA DISSEMINATION ................................... 132 Data Curation ? Steven Ruggles .................................................................................................................................. 132 Evaluating the Usability of Survey Project Websites ? David L. Vannette ................................................ 135 Research Transparency and the Credibility of Survey-Based Social Science - Arthur Lupia............ 139 CONFERENCE WEBSITE AND ORGANIZER CONTACT INFORMATION ..............................142 LIST OF CONFERENCE PRESENTERS .................................................................................................. 143 LIST OF CONFERENCE DISCUSSANTS ................................................................................................. 144 CONFERENCE PROGRAMS........................................................................................................................ 145

3

Overview

For more than thirty years, the National Science Foundation (NSF) has supported data for research on a wide variety of topics by making awards to three major long-term survey efforts, the American National Elections Studies (ANES), the Panel Study of Income Dynamics (PSID), and the General Social Survey (GSS). In February 2012, the Advisory Committee for the Social, Behavioral, and Economic Sciences (SBE) was asked to provide advice about future investments in these surveys and others. The Advisory Committee then charged a subcommittee to provide that advice. The Subcommittee on Advancing SBE Survey Research is comprised of Jon Krosnick (Stanford University, chair), Janet Harkness (University of Nebraska, deceased), Kaye Husbands-Fealing (University of Minnesota), Stanley Presser (University of Maryland), and Steven Ruggles (University of Minnesota).

The Subcommittee submits this report to the Assistant Director of the SBE Directorate, with the purpose of providing advice related to how the Foundation can best use its resources to support research through survey data collection. Specifically, the report addresses the following questions, as requested:

1. What are the challenges facing survey-based data collection today (e.g., falling participation rates, rising costs, or coverage of frames)?

2. What innovations in survey methodology have taken place or are on the horizon?

3. How should SBE think about survey data in the context of the explosion of new digital sources of data? Does the Subcommittee see opportunities for blending data or mixed source methods that integrate existing administrative, commercial, or social media data with existing surveys to answer social science questions?

4. Given current challenges faced by survey research as well as the potential opportunities presented by new approaches to survey research, what types of questions will we be able to address with surveys in the future?

5. What is an overarching strategy for data collection that the Directorate supports (including, but not limited, to the three existing surveys), which could be used to guide planning for NSF-supported data in the future? This might include shared infrastructure across surveys, but should not be limited to that approach.

The report addresses the first four questions ? which are about the current and future status of survey research in general (as opposed to uniquely about NSF funded surveys) ? by drawing on the results of presentations we commissioned from leading experts at two conferences held at NSF in the fall of 2012. An extensive summary of the conferences is included as an appendix to this report. The fifth item in our charge ? the strategic vision for SBE regarding the Big Three NSF surveys ? was developed by the Subcommittee based on the proceedings of the conference and our own deliberations.

4

The two conferences we convened in October 2012 brought together leading scholarly experts on topics that fit into four broad areas. First, discussion on challenges faced in conventional survey research covered a broad landscape, including key topics such as: probability versus nonprobability sampling methods; multi-mode survey techniques; optimizing response rates and how nonresponse affects survey accuracy; use of incentives in survey collection; survey design, visual displays and cognitive evaluation of survey instruments; proxy reporting; interviewing techniques and challenges; confidentiality, respondent attrition and data attrition; and computation of survey weights.

The second category of exploration focused on opportunities to expand data collection, including: paradata; the use of leave-behind measurement supplements and biomarkers; and specialized tools for measuring past events. Third, several methods of linking survey data with external sources were discussed, specifically: improving government, academic and industry data-sharing opportunities; linking survey data to official government records or with the Catalist Commercial Database; linking knowledge networks web panel data with external data; and the use of election administration data with other datasets. Lastly, discussion turned to improving research transparency and data dissemination, with a focus on: data curation; evaluating the usability of survey project websites; and the broader topic of the credibility of survey-based social science. Throughout the proceedings, conference participants explored steps that can be taken to enhance the value of survey methodology to a wide range of users, in academia, government, and the private sector.

Collectively, the conferences yielded several useful outcomes, including: (1) insights about how surveys should be done today to maximize data quality (thereby specifying how major infrastructure surveys should be designed and carried out), (2) important challenges facing the methodology, (3) best practices in data dissemination and data collection procedure documentation, (4) approaches that would be most desirable for large-scale infrastructure surveys to implement, and (5) research questions that merit future investigation.

Our report is organized as follows. Section 1 summarizes best practices for survey research recommended by the conference participants that should be useful to NSF programs as a touchstone in evaluating research proposals involving the collection of survey data. Section 2 offers a set of recommendations in response to the charge's final (fifth) question about an overarching strategy for NSF's major infrastructure projects that involve the collection of survey data. The Appendix is a rich compilation of detailed summaries of presentations at the two conferences sponsored for this study. (For transcripts of each presentation and the visual displays employed during the conferences see .)

5

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download