2020 Census Quantitative Creative Testing Final Report

2020 Census Quantitative Creative Testing

Final Report

A New Design for the 21st Century

Issued November 16, 2022

Version 1.2

Prepared by Steven Scheid, Erica Olmsted-Hawala, Casey Eggleston, Monica Vines

The Census Bureau's Disclosure Review Board and Disclosure Avoidance Officers have reviewed this information

product for unauthorized disclosure of confidential information and have approved the disclosure avoidance

practices applied to this release. (CBDRB-FY22-DSSD007-0041)

2020 Census Research |

2020 Census Quantitative Creative Testing Final Report, Version 1.2

Page intentionally left blank.

2020 Census Research |

2020 Census Quantitative Creative Testing Final Report, Version 1.2

Table of Contents

List of Tables ......................................................................................................................................................... ii

Executive Summary.............................................................................................................................................. iii

1.

Introduction .................................................................................................................................................. 1

2.

Background ................................................................................................................................................... 1

3.

Methodology ................................................................................................................................................ 4

3.1

Research Questions.............................................................................................................................. 4

3.2

Experimental Design ............................................................................................................................ 5

Control and Treatment Groups .................................................................................................................... 6

Random Assignment to Condition ................................................................................................................ 8

Online Panel Design ...................................................................................................................................... 8

Measures of Interest .................................................................................................................................. 10

Analyses ...................................................................................................................................................... 14

3.3

Schedule ............................................................................................................................................. 15

4.

Limitations .................................................................................................................................................. 16

5.

Results ........................................................................................................................................................ 17

5.1 Research Question 1. Television advertisements. ........................................................................................ 20

5.2 Research Question 2. Radio advertisements. ............................................................................................... 23

5.3 Research Question 3. Evaluative measures .................................................................................................. 26

5.4 Research Question 4. Operational Hurdles: Is this type of research useful for future creative testing? ..... 26

6.

Conclusions and Recommendations........................................................................................................... 28

6.1

Conclusions ........................................................................................................................................ 28

6.2

Recommendations ............................................................................................................................. 30

6.3 Future Directions with Open-Ended Questions ............................................................................................ 30

7.

Knowledge Management Resolutions ........................................................................................................ 31

8.

Acknowledgements .................................................................................................................................... 31

9.

References .................................................................................................................................................. 31

10.

Appendix ................................................................................................................................................ 32

Appendix A. Acronyms and Abbreviations ......................................................................................................... 32

Appendix B. List of 2020 Census Diverse Mass Television and Radio Advertisements ...................................... 32

Appendix C. Questionnaire ................................................................................................................................. 33

i

2020 Census Research |

2020 Census Quantitative Creative Testing Final Report, Version 1.2

List of Tables

Table 1. Diverse Mass Advertisements Tested in This Study ........................................................................ 6

Table 2. Advertisements in Experiment ........................................................................................................ 8

Table 3. Quota Targets .................................................................................................................................. 9

Table 4. Wording and Values for Outcome Variables ................................................................................. 10

Table 5. Schedule of Operational and Analysis Milestones for Quantitative Creative Testing .................. 15

Table 6. Demographics of Respondents ..................................................................................................... 18

Table 7. Sample Size per Condition............................................................................................................. 19

Table 8. Means and Standard Errors for Survey Questions About the Surrounding Show (TV) ................. 20

Table 9. Means and Standard Errors for Questions About Attitudes Toward the Census (TV) .................. 21

Table 10. Means and Standard Errors for Questions About Potential Barriers to Responding (TV) .......... 22

Table 11. Means and Standard Errors for Comparisons Between Census Ads (TV) ................................... 23

Table 12. Means and Standard Errors for Questions About the Surrounding Show (Radio) ..................... 23

Table 13. Means and Standard Errors for Questions About Attitudes Toward the Census (Radio) ........... 24

Table 14. Means and Standard Errors for Questions About Potential Barriers to Responding (Radio) ..... 24

Table 15. Means and Standard Errors for Comparisons Between Census Ads (Radio) .............................. 25

ii

2020 Census Research |

2020 Census Quantitative Creative Testing Final Report, Version 1.2

Executive Summary

In February 2020, the Census Bureau tested five television and three radio advertisements for

the 2020 Census using quantitative procedures to better understand the effectiveness of the

ads, as well as how the Census Bureau might use this methodology for ad testing in intercensal

years and in preparation for the 2030 Census. Each ad was 30 seconds long. The five television

ads varied in content including information about census takers, information about how census

data are used featuring different places and people across the U.S., kids explaining who to

count, kids explaining how census data are used, and a grandmother filling out the census

questionnaire online. The three radio ads included a man explaining that the census is

mandated in the Constitution and mentioning what the data is used for, many voices describing

how census data is used, and a computerized voice sharing the message that the census

questionnaire is now online. Each ad was embedded within a four-minute clip from ¡°The Big

Bang Theory¡± television program. Respondents were randomly assigned to either watch a

census ad or a 30-second T-Mobile ¡°control¡± ad before being asked about attitudes and

intended behaviors related to the census and the advertisement. We then compared the

answers by ad shown to evaluate the census ads. In total, 10,000 respondents completed this

survey. Respondents were recruited from a nonprobability panel. Quotas were imposed for

age, sex, race, income, education, and region to obtain an experimental sample more

generalizable to the U.S. population.

Respondents were instructed to watch or listen to a short clip from a television show; no

mention of the 2020 Census or Census Bureau was included in the instructions or the invitation

sent to participants. The ad was inserted into the middle of the clip to mimic a regular

commercial break. After watching or listening to the clip of the show with the commercial

breaks, respondents were asked a series of questions about their attitudes and intended

behaviors toward the census. We compared census ads to the control T-Mobile ad across

several questions including intention to respond to the census, intention to recommend others

to fill out the census questionnaire, Census Bureau corporate image, attitudes toward the

census, and efficacy to respond to the census, among other measures. At the end of the

questionnaire, all respondents were reshown a Census Bureau advertisement and asked to

evaluate the advertisement. We used answers to these questions to compare the likeability,

clarity, offensiveness, and teachability of different census advertisements.

Significance testing was done using a linear model predicting the outcome variable from the ad

shown, with age, sex, Hispanic origin, race, marital status, household size, renter status,

income, and region as covariates. The dependent measure was rescaled between 0 and 1 and

treated as an interval variable. Pairwise comparisons were made using Tukey¡¯s Honest

Significant Difference to control family-wise error rates. Results indicate the following:

iii

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download