Are geographically diverse samples used in the journals ...
How diverse are the samples used in the journals ‘Evolution & Human Behavior’ and ‘Evolutionary Psychology’?Thomas V. Pollet1?& Tamsin K. Saxton1????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????1 Northumbria University?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Author noteThomas V. Pollet & Tamsin K. Saxton, Dept. of Psychology, Northumbria University, Newcastle upon Tyne, UKCorrespondence concerning this article should be addressed to Thomas V. Pollet, NB 165, Northumberland Building, 2 Ellison Place, NE1 8ST, Newcastle Upon Tyne, UK. E-mail: thomas.pollet@northumbria.ac.uk** This is a preprint of a manuscript currently under consideration with a journal, cite at own risk**This version: 26-2-2019?AbstractPsychologists regularly draw inferences about populations based on data from small samples of people, and so have long been interested in how well those samples generalise to wider populations. There is a consensus that psychology probably relies too much on samples from Western Educated Industrialised Rich and Democratic (WEIRD) societies, and among those from university students. Online surveys might be used to increase sample diversity, although online sampling still reaches only a restricted range of participants. Studies from evolutionary psychology often seek to uncover aspects of evolved universal characteristics, and so might demonstrate a particular interest in the use of diverse samples. Here, we empirically examine the samples used in the 2015-2016 volumes of ‘Evolution & Human Behavior’ (104 articles) and ‘Evolutionary Psychology’ (76 articles). Our database consists of 311 samples of humans (median sample size = 186). The majority of samples were either online or student samples (70% of samples), followed by other adult Western samples (19%). 253 (81%) of the samples were classified as ‘Western’ (Europe/North America/Australia). The remaining samples were predominantly from Asia (N= 37; 12%, mostly Japan). Only a small fraction of the samples was taken from Latin American and Caribbean (N = 8) or African (N = 6) countries. The median sample size did not differ significantly between continents, but online samples (both paid and unpaid) were typically larger than samples sourced offline. It seems that the samples used are more diverse than those that have been reported in reviews of the literature from social and developmental psychology, perhaps because evolutionary psychology has a greater inherent need to test hypotheses about an evolved and universal human nature. However, it is also apparent that the majority of samples within contemporary evolutionary psychology research remain WEIRD.Keywords: Cross-cultural samples; WEIRD; sampling; participantsWord count: 5,028 (main text)????How diverse are the samples used in the journals ‘Evolution & Human Behavior’ and ‘Evolutionary Psychology’?A recurrent criticism of psychology as a science is the lack of diversity. This lack of diversity refers not solely to the producers of psychological science ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1080/00207590143000351", "ISSN" : "0020-7594", "author" : [ { "dropping-particle" : "", "family" : "Adair", "given" : "John G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Co\u00ealho", "given" : "Angela E L", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Luna", "given" : "Jesus R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "International Journal of Psychology", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "2002" ] ] }, "page" : "160-170", "publisher" : "Taylor & Francis", "title" : "How international is psychology?", "type" : "article-journal", "volume" : "37" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1080/002075997400908", "ISSN" : "1464-066X", "author" : [ { "dropping-particle" : "", "family" : "Bauserman", "given" : "Robert", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "International Journal of Psychology", "id" : "ITEM-2", "issue" : "2", "issued" : { "date-parts" : [ [ "1997" ] ] }, "page" : "107-112", "publisher" : "Wiley Online Library", "title" : "International representation in the psychological literature", "type" : "article-journal", "volume" : "32" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1037/0003-066X.61.8.904", "ISSN" : "1935-990X", "author" : [ { "dropping-particle" : "", "family" : "Cole", "given" : "Michael", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Psychologist", "id" : "ITEM-3", "issue" : "8", "issued" : { "date-parts" : [ [ "2006", "11" ] ] }, "page" : "904-917", "title" : "Internationalism in psychology: We need it now more than ever.", "type" : "article-journal", "volume" : "61" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Adair et al. 2002; Bauserman 1997; Cole 2006)", "manualFormatting" : "(e.g., Adair, Co\u00ealho, & Luna 2002; Bauserman 1997; Cole 2006)", "plainTextFormattedCitation" : "(Adair et al. 2002; Bauserman 1997; Cole 2006)", "previouslyFormattedCitation" : "(Adair et al. 2002; Bauserman 1997; Cole 2006)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Adair, Coêlho, & Luna 2002; Bauserman 1997; Cole 2006) or to the topics studied ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/a0031246", "ISSN" : "1878-7304", "author" : [ { "dropping-particle" : "", "family" : "Berry", "given" : "John W", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Canadian Psychology/Psychologie canadienne", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "2013" ] ] }, "page" : "55-61", "publisher" : "Educational Publishing Foundation", "title" : "Achieving a global psychology.", "type" : "article-journal", "volume" : "54" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Berry 2013)", "manualFormatting" : "(e.g., Berry 2013)", "plainTextFormattedCitation" : "(Berry 2013)", "previouslyFormattedCitation" : "(Berry 2013)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Berry 2013), but also to the samples that are used as the basis for drawing inferences. This criticism is recurrent and has been voiced approximately once a decade since around 1965 ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/0003-066X.63.7.602", "ISBN" : "0003-066X", "ISSN" : "0003066X", "PMID" : "18855491", "abstract" : "This article proposes that psychological research published in APA journals focuses too narrowly on Americans, who comprise less than 5% of the world's population. The result is an understanding of psychology that is incomplete and does not adequately represent humanity. First, an analysis of articles published in six premier APA journals is presented, showing that the contributors, samples, and editorial leadership of the journals are predominantly American. Then, a demographic profile of the human population is presented to show that the majority of the world's population lives in conditions vastly different from the conditions of Americans, underlining doubts of how well American psychological research can be said to represent humanity. The reasons for the narrowness of American psychological research are examined, with a focus on a philosophy of science that emphasizes fundamental processes and ignores or strips away cultural context. Finally, several suggestions for broadening the scope of American psychology are offered.", "author" : [ { "dropping-particle" : "", "family" : "Arnett", "given" : "Jeffrey J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Psychologist", "id" : "ITEM-1", "issue" : "7", "issued" : { "date-parts" : [ [ "2008" ] ] }, "page" : "602-614", "publisher" : "American Psychological Association", "title" : "The Neglected 95%: Why American Psychology Needs to Become Less American", "type" : "article-journal", "volume" : "63" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1037/0003-066X.47.5.629", "ISSN" : "0003-066X", "author" : [ { "dropping-particle" : "", "family" : "Graham", "given" : "Sandra", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Psychologist", "id" : "ITEM-2", "issue" : "5", "issued" : { "date-parts" : [ [ "1992" ] ] }, "page" : "629-639", "publisher" : "American Psychological Association", "title" : "\"Most of the subjects were White and middle class\": Trends in published research on African Americans in selected APA journals, 1970-1989.", "type" : "article-journal", "volume" : "47" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1017/S0140525X0999152X", "ISSN" : "1469-1825", "author" : [ { "dropping-particle" : "", "family" : "Henrich", "given" : "Joseph", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Heine", "given" : "S J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Norenzayan", "given" : "A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-3", "issue" : "2-3", "issued" : { "date-parts" : [ [ "2010" ] ] }, "page" : "61-83", "publisher" : "Cambridge Univ Press", "title" : "The weirdest people in the world", "type" : "article-journal", "volume" : "33" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "DOI" : "10.1037/h0027880", "ISSN" : "1939-1455", "author" : [ { "dropping-particle" : "", "family" : "Schultz", "given" : "Duane P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Bulletin", "id" : "ITEM-4", "issue" : "3", "issued" : { "date-parts" : [ [ "1969" ] ] }, "page" : "214-228", "publisher" : "American Psychological Association", "title" : "The human subject in psychological research.", "type" : "article-journal", "volume" : "72" }, "uris" : [ "" ] }, { "id" : "ITEM-5", "itemData" : { "DOI" : "10.1080/10478400802049936", "ISSN" : "1047-840X", "author" : [ { "dropping-particle" : "", "family" : "Henry", "given" : "Peter J.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Inquiry", "id" : "ITEM-5", "issue" : "2", "issued" : { "date-parts" : [ [ "2008", "5", "22" ] ] }, "page" : "49-71", "publisher" : "American Psychological Association", "title" : "College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice", "type" : "article-journal", "volume" : "19" }, "uris" : [ "" ] }, { "id" : "ITEM-6", "itemData" : { "DOI" : "10.1037/h0083096", "ISSN" : "0008-4832", "author" : [ { "dropping-particle" : "", "family" : "Smart", "given" : "Reginald G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Canadian Psychologist/Psychologie canadienne", "id" : "ITEM-6", "issue" : "2", "issued" : { "date-parts" : [ [ "1966" ] ] }, "page" : "115-121", "publisher" : "Canadian Psychological Association", "title" : "Subject selection bias in psychological research.", "type" : "article-journal", "volume" : "7a" }, "uris" : [ "" ] }, { "id" : "ITEM-7", "itemData" : { "DOI" : "10.1037/h0086893", "ISBN" : "1878-7304 (Electronic); 0708-5591 (Print)", "ISSN" : "0708-5591", "abstract" : "Gallander Wintre, M., North, C., & Sugar, L. A. (2001). Psychologists' response to criticisms about research based on undergraduate participants: A developmental perspective. Canadian Psychology/Psychologie Canadienne, 42(3), 216.", "author" : [ { "dropping-particle" : "", "family" : "Gallander Wintre", "given" : "Maxine", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "North", "given" : "Christopher", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sugar", "given" : "Lorne A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Canadian Psychology/Psychologie Canadienne", "id" : "ITEM-7", "issue" : "3", "issued" : { "date-parts" : [ [ "2001" ] ] }, "page" : "216-225", "publisher" : "Canadian Psychological Association", "title" : "Psychologists' response to criticisms about research based on undergraduate participants: A developmental perspective.", "type" : "article-journal", "volume" : "42" }, "uris" : [ "" ] }, { "id" : "ITEM-8", "itemData" : { "DOI" : "10.1016/j.jecp.2017.04.017", "ISSN" : "00220965", "author" : [ { "dropping-particle" : "", "family" : "Nielsen", "given" : "Mark", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Haun", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "K\u00e4rtner", "given" : "Joscha", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Legare", "given" : "Cristine H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Experimental Child Psychology", "id" : "ITEM-8", "issued" : { "date-parts" : [ [ "2017", "10" ] ] }, "page" : "31-38", "publisher" : "Elsevier", "title" : "The persistent sampling bias in developmental psychology: A call to action", "type" : "article-journal", "volume" : "162" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Arnett 2008; Gallander Wintre et al. 2001; Graham 1992; Henrich et al. 2010; Henry 2008; Nielsen et al. 2017; Schultz 1969; Smart 1966)", "manualFormatting" : "(e.g., Arnett 2008; Gallander Wintre, North, & Sugar 2001; Graham 1992; Henrich, Heine, & Norenzayan 2010; Henry 2008; Nielsen, Haun, K\u00e4rtner, & Legare 2017; Schultz 1969; Smart 1966)", "plainTextFormattedCitation" : "(Arnett 2008; Gallander Wintre et al. 2001; Graham 1992; Henrich et al. 2010; Henry 2008; Nielsen et al. 2017; Schultz 1969; Smart 1966)", "previouslyFormattedCitation" : "(Arnett 2008; Gallander Wintre et al. 2001; Graham 1992; Henrich et al. 2010; Henry 2008; Nielsen et al. 2017; Schultz 1969; Smart 1966)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Arnett 2008; Gallander Wintre, North, & Sugar 2001; Graham 1992; Henrich, Heine, & Norenzayan 2010; Henry 2008; Nielsen, Haun, K?rtner, & Legare 2017; Schultz 1969; Smart 1966). Following a review of the participants in psychological studies, Schultz (1969:218) wrote: ‘The extremely small percentage of studies sampling the general adult population was particularly disturbing; none of the studies published in the Journal of Experimental Psychology during those years used a sample of the general population’. There is an indisputable geographical bias to the populations sampled by psychologists. For example, a review of articles published between 2006 and 2010 in the three experimental developmental psychology journals with the highest impact factors found that over 90% of the research participants came from Australia, Canada, Europe, the United States, or New Zealand, while under 3% of the participants in the research studies were from Africa, Asia, Central and South America, and the Middle East and Israel ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1016/j.jecp.2017.04.017", "ISSN" : "00220965", "author" : [ { "dropping-particle" : "", "family" : "Nielsen", "given" : "Mark", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Haun", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "K\u00e4rtner", "given" : "Joscha", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Legare", "given" : "Cristine H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Experimental Child Psychology", "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017", "10" ] ] }, "page" : "31-38", "publisher" : "Elsevier", "title" : "The persistent sampling bias in developmental psychology: A call to action", "type" : "article-journal", "volume" : "162" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Nielsen et al. 2017)", "plainTextFormattedCitation" : "(Nielsen et al. 2017)", "previouslyFormattedCitation" : "(Nielsen et al. 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Nielsen et al. 2017). Similarly, in the flagship journal ‘Journal of Personality and Social Psychology’, 96% of the papers published in 2012 were based on WEIRD samples (Kurzban 2013). Further, from its inception, psychology has relied heavily on undergraduate samples, a situation that has not changed substantially over time. For example, Gallander Wintre and colleagues (2001) reviewed 1,179 articles spanning 6 journals across the different subdivisions of psychology and found 68% of the samples to be student samples. They also found that, if anything, the reliance on student samples had increased between 1975 and 1995. A classic paper by Sears (1986) reviewed papers published in 1980 in three mainstream social psychology journals and found that 82% of the samples used students in some form, and 75% used undergraduate students (mainly from the USA) exclusively. Likewise, the 1995 editions of two leading social psychology journals (‘Journal of Experimental Social Psychology’ and ‘Journal of Personality and Social Psychology’) used undergraduate students as participants in 95.8% and 70.6% of all cases respectively ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/h0086893", "ISBN" : "1878-7304 (Electronic); 0708-5591 (Print)", "ISSN" : "0708-5591", "abstract" : "Gallander Wintre, M., North, C., & Sugar, L. A. (2001). Psychologists' response to criticisms about research based on undergraduate participants: A developmental perspective. Canadian Psychology/Psychologie Canadienne, 42(3), 216.", "author" : [ { "dropping-particle" : "", "family" : "Gallander Wintre", "given" : "Maxine", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "North", "given" : "Christopher", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sugar", "given" : "Lorne A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Canadian Psychology/Psychologie Canadienne", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "2001" ] ] }, "page" : "216-225", "publisher" : "Canadian Psychological Association", "title" : "Psychologists' response to criticisms about research based on undergraduate participants: A developmental perspective.", "type" : "article-journal", "volume" : "42" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Gallander Wintre et al. 2001)", "plainTextFormattedCitation" : "(Gallander Wintre et al. 2001)", "previouslyFormattedCitation" : "(Gallander Wintre et al. 2001)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Gallander Wintre et al. 2001), and Arnett (2008) calculated that 74% of the samples in the journal ‘Social Psychological and Personality Science’ were from student populations. The issues relating to sampling are not limited to (social) psychology, and similar concerns have been voiced in other related disciplines such as consumer research ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1086/323732", "ISSN" : "0093-5301", "author" : [ { "dropping-particle" : "", "family" : "Peterson", "given" : "Robert A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Consumer Research", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "2001", "12" ] ] }, "page" : "450-461", "publisher" : "JSTOR", "title" : "On the Use of College Students in Social Science Research: Insights from a Second-Order Meta-analysis", "type" : "article-journal", "volume" : "28" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Peterson 2001)", "plainTextFormattedCitation" : "(Peterson 2001)", "previouslyFormattedCitation" : "(Peterson 2001)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Peterson 2001), education research ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1080/00461520.2018.1442220", "ISSN" : "0046-1520", "author" : [ { "dropping-particle" : "", "family" : "Usher", "given" : "Ellen L", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Educational Psychologist", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "2018", "4", "3" ] ] }, "note" : "doi: 10.1080/00461520.2018.1442220", "page" : "131-144", "publisher" : "Routledge", "title" : "Acknowledging the Whiteness of Motivation Research: Seeking Cultural Relevance", "type" : "article-journal", "volume" : "53" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Usher 2018)", "plainTextFormattedCitation" : "(Usher 2018)", "previouslyFormattedCitation" : "(Usher 2018)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Usher 2018), behavioural economics ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1257/jep.21.2.153", "ISSN" : "0895-3309", "author" : [ { "dropping-particle" : "", "family" : "Levitt", "given" : "Steven D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "List", "given" : "John A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Economic Perspectives", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "2007", "4" ] ] }, "page" : "153-174", "publisher" : "JSTOR", "title" : "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?", "type" : "article-journal", "volume" : "21" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Levitt and List 2007)", "plainTextFormattedCitation" : "(Levitt and List 2007)", "previouslyFormattedCitation" : "(Levitt and List 2007)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Levitt and List 2007), and business research ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1057/jibs.2008.101", "ISBN" : "0047-2506", "ISSN" : "0047-2506", "author" : [ { "dropping-particle" : "", "family" : "Bello", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Leung", "given" : "Kwok", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Radebaugh", "given" : "Lee", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Tung", "given" : "Rosalie L", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Witteloostuijn", "given" : "Arjen", "non-dropping-particle" : "van", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of International Business Studies", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "2009", "4", "11" ] ] }, "page" : "361-364", "publisher" : "Springer", "title" : "From the Editors: Student samples in international business research", "type" : "article-journal", "volume" : "40" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Bello et al. 2009)", "plainTextFormattedCitation" : "(Bello et al. 2009)", "previouslyFormattedCitation" : "(Bello et al. 2009)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Bello et al. 2009). For example, Peterson (2001) reviewed the literature in consumer research and found 86% of the samples to be from students. Online participantsPerhaps in part as a response to these sorts of criticisms, (social) psychologists have increasingly turned to online platforms to recruit participants who are not students ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/12076-000", "ISBN" : "1-4338-0695-9", "ISSN" : "1-4338-0695-9", "abstract" : "The Internet is revolutionizing the way psychologists conduct behavioral research. Studies conducted online are not only less error prone and labor intensive but also rapidly reach large numbers of diverse participants at a fraction of the cost of traditional methods. In addition to improving the efficiency and accuracy of data collection, online studies provide automatic data storage and deliver immediate personalized feedback to research participants-a major incentive that can exponentially expand participant pools. Furthermore, behavioral researchers can also track data on online behavioral phenomena, including Instant Messaging (IM), social networking, and other social media. This book goes beyond the basics to teach readers advanced methods for conducting behavioral research on the Internet. Short chapters offer practical advice by leading experts in key domains of Internet research. Readers are shown, step by step, how to conduct online experiments, surveys, and ability testing, use advanced graphic tools such as drag-and-drop objects, apply automatic text analysis tools, check the validity of protocols, automate the storage and analysis of data, record \"field notes\" on the behavior of online subjects and chatroom or blogging communities, and much more. Chapters also address critical issues such as data security, ethics, participant recruitment, and how to ensure the completion of tests or questionnaires. This volume also features a companion Web site with additional resources, links, scripts, and instructions to further assist readers with their online research. This book is designed for researchers and advanced graduate students in the behavioral sciences seeking greater technical detail about emerging research methods. Readers will be well equipped to implement and integrate these exciting new methods into their own Internet-based behavioral research effectively, securely, and responsibly.", "author" : [ { "dropping-particle" : "", "family" : "Gosling", "given" : "Samuel D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Johnson", "given" : "John a", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "editor" : [ { "dropping-particle" : "", "family" : "Gosling", "given" : "Samuel D.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Johnson", "given" : "John A.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2010" ] ] }, "number-of-pages" : "286", "publisher" : "American Psychological Association", "publisher-place" : "Washington, DC", "title" : "Advanced methods for conducting online behavioral research.", "type" : "book" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1017/S0140525X10000300", "ISSN" : "0140-525X", "author" : [ { "dropping-particle" : "", "family" : "Gosling", "given" : "Samuel D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sandy", "given" : "Carson J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "John", "given" : "Oliver P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Potter", "given" : "Jeff", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-2", "issue" : "2-3", "issued" : { "date-parts" : [ [ "2010", "6", "15" ] ] }, "page" : "94-95", "publisher" : "Cambridge University Press", "title" : "Wired but not WEIRD: The promise of the Internet in reaching more diverse samples", "type" : "article-journal", "volume" : "33" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1037/0003-066X.59.2.93", "ISSN" : "1935-990X", "author" : [ { "dropping-particle" : "", "family" : "Gosling", "given" : "Samuel D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Vazire", "given" : "Simine", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Srivastava", "given" : "Sanjay", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "John", "given" : "Oliver P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Psychologist", "id" : "ITEM-3", "issue" : "2", "issued" : { "date-parts" : [ [ "2004" ] ] }, "page" : "93-104", "publisher" : "American Psychological Association", "title" : "Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires.", "type" : "article-journal", "volume" : "59" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Gosling et al. 2004, 2010; Gosling and Johnson 2010)", "manualFormatting" : "(e.g., Gosling et al. 2004, 2010; Gosling and Johnson 2010)", "plainTextFormattedCitation" : "(Gosling et al. 2004, 2010; Gosling and Johnson 2010)", "previouslyFormattedCitation" : "(Gosling et al. 2004, 2010; Gosling and Johnson 2010)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Gosling et al. 2004, 2010; Gosling and Johnson 2010). Over the past decade, there has been a strong increase in the use of participants crowd-sourced via online platforms, such as Amazon M-Turk or Crowdflower ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1177/1745691610393980", "ISSN" : "1745-6916", "author" : [ { "dropping-particle" : "", "family" : "Buhrmester", "given" : "Michael", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Kwang", "given" : "Tracy", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gosling", "given" : "Samuel D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Perspectives on Psychological Science", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "2011", "1", "1" ] ] }, "page" : "3-5", "publisher" : "Sage Publications", "title" : "Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?", "type" : "article-journal", "volume" : "6" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1177/0963721414531598", "ISSN" : "0963-7214", "author" : [ { "dropping-particle" : "", "family" : "Paolacci", "given" : "Gabriele", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Chandler", "given" : "Jesse", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Current Directions in Psychological Science", "id" : "ITEM-2", "issue" : "3", "issued" : { "date-parts" : [ [ "2014" ] ] }, "page" : "184-188", "publisher" : "Sage Publications", "title" : "Inside the turk understanding mechanical turk as a participant pool", "type" : "article-journal", "volume" : "23" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Buhrmester et al. 2011; Paolacci and Chandler 2014)", "manualFormatting" : "(e.g., Buhrmester, Kwang, & Gosling 2011; Paolacci & Chandler 2014)", "plainTextFormattedCitation" : "(Buhrmester et al. 2011; Paolacci and Chandler 2014)", "previouslyFormattedCitation" : "(Buhrmester et al. 2011; Paolacci and Chandler 2014)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Buhrmester, Kwang, & Gosling 2011; Paolacci & Chandler 2014). Such expansion has benefited psychological research in many ways. For example, results from classic behavioural experiments (e.g., the Stroop Task; ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/h0054651", "ISSN" : "0022-1015", "author" : [ { "dropping-particle" : "", "family" : "Stroop", "given" : "J Ridley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Experimental Psychology", "id" : "ITEM-1", "issue" : "6", "issued" : { "date-parts" : [ [ "1935" ] ] }, "page" : "643-662", "publisher" : "Psychological Review Company", "title" : "Studies of interference in serial verbal reactions.", "type" : "article-journal", "volume" : "18" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Stroop 1935)", "manualFormatting" : "Stroop 1935)", "plainTextFormattedCitation" : "(Stroop 1935)", "previouslyFormattedCitation" : "(Stroop 1935)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Stroop 1935) were shown to replicate well on these online platforms ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1371/journal.pone.0057410", "ISSN" : "1932-6203", "abstract" : "Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.", "author" : [ { "dropping-particle" : "", "family" : "Crump", "given" : "Matthew J C", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "V", "family" : "McDonnell", "given" : "John", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gureckis", "given" : "Todd M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "PLoS ONE", "editor" : [ { "dropping-particle" : "", "family" : "Gilbert", "given" : "Sam", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "2013", "3", "13" ] ] }, "page" : "e57410", "publisher" : "Public Library of Science", "title" : "Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research", "type" : "article-journal", "volume" : "8" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Crump et al. 2013)", "plainTextFormattedCitation" : "(Crump et al. 2013)", "previouslyFormattedCitation" : "(Crump et al. 2013)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Crump et al. 2013). Even when studying political ideologies, it seems that M-Turk is well-suited ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1177/2053168015622072", "ISSN" : "2053-1680", "abstract" : "Amazon?s Mechanical Turk (MTurk) is an increasingly popular tool for the recruitment of research subjects. While there has been much focus on the demographic differences between MTurk samples and the national public, we know little about whether liberals and conservatives recruited from MTurk share the same psychological dispositions as their counterparts in the mass public. In the absence of such evidence, some have argued that the selection process involved in joining MTurk invalidates the subject pool for studying questions central to political science. In this paper, we evaluate this claim by comparing a large MTurk sample to two benchmark national samples ? one conducted online and one conducted face-to-face. We examine the personality and value-based motivations of political ideology across the three samples. All three samples produce substantively identical results with only minor variation in effect sizes. In short, liberals and conservatives in our MTurk sample closely mirror the psychological divisions of liberals and conservatives in the mass public, though MTurk liberals hold more characteristically liberal values and attitudes than liberals from representative samples. Overall, our results suggest that MTurk is a valid recruitment tool for psychological research on political ideology.", "author" : [ { "dropping-particle" : "", "family" : "Clifford", "given" : "Scott", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Jewell", "given" : "Ryan M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Waggoner", "given" : "Philip D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Research & Politics", "id" : "ITEM-1", "issue" : "4", "issued" : { "date-parts" : [ [ "2015", "10", "15" ] ] }, "note" : "doi: 10.1177/2053168015622072", "page" : "205316801562207", "publisher" : "SAGE Publications Ltd", "title" : "Are samples drawn from Mechanical Turk valid for research on political ideology?", "type" : "article-journal", "volume" : "2" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Clifford et al. 2015)", "plainTextFormattedCitation" : "(Clifford et al. 2015)", "previouslyFormattedCitation" : "(Clifford et al. 2015)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Clifford et al. 2015). However, although Amazon boasts >.5 million participants, the actual pools from which participants are sampled are much smaller, with the population estimated at around 7,300 individuals ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1126/science.aag0592", "ISSN" : "0036-8075", "author" : [ { "dropping-particle" : "", "family" : "Bohannon", "given" : "John", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Science", "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2016", "6", "7" ] ] }, "title" : "Psychologists grow increasingly dependent on online research subjects", "type" : "article-newspaper" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Bohannon 2016)", "plainTextFormattedCitation" : "(Bohannon 2016)", "previouslyFormattedCitation" : "(Bohannon 2016)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Bohannon 2016). Moreover, while crowd-sourcing platforms such as Amazon’s M-Turk allow for sampling more diversely than typical student samples, for example with respect to age range, the participants remain predominantly WEIRD, with some notable exceptions ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1371/journal.pone.0073131", "ISBN" : "1932-6203 (Electronic)\\r1932-6203 (Linking)", "ISSN" : "19326203", "PMID" : "23951342", "abstract" : "The decision rules underpinning human cooperative behaviour are often investigated under laboratory conditions using monetary incentives. A major concern with this approach is that stake size may bias subjects' decisions. This concern is particularly acute in online studies, where stakes are often far lower than those used in laboratory or field settings. We address this concern by conducting a Dictator Game using Amazon Mechanical Turk. In this two-player game, one player (the dictator) determines the division of an endowment between himself and the other player. We recruited subjects from India and the USA to play an online Dictator Game. Dictators received endowments of $1, $5 or $10. We collected two batches of data over two consecutive years. We found that players from India were less generous when playing with a $10 stake. By contrast, the effect of stake size among players from the USA was very small. This study indicates that the effects of stake size on decision making in economic games may vary across populations.", "author" : [ { "dropping-particle" : "", "family" : "Raihani", "given" : "Nichola J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Mace", "given" : "Ruth", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lamba", "given" : "Shakti", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "PLoS ONE", "id" : "ITEM-1", "issue" : "8", "issued" : { "date-parts" : [ [ "2013" ] ] }, "page" : "e0073131", "title" : "The Effect of $1, $5 and $10 Stakes in an Online Dictator Game", "type" : "article-journal", "volume" : "8" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Raihani et al. 2013)", "manualFormatting" : "(e.g., Raihani, Mace, & Lamba 2013)", "plainTextFormattedCitation" : "(Raihani et al. 2013)", "previouslyFormattedCitation" : "(Raihani et al. 2013)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Raihani, Mace, & Lamba 2013). Evolutionary Psychology as an exception?Since its inception, evolutionary psychology has stressed the importance of human universals ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1017/S0140525X00023992", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain sciences", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "1989" ] ] }, "page" : "1-49", "publisher" : "Cambridge Univ Press", "title" : "Sex differences in human mate preferences: Evolutionary hypotheses tested in 37 cultures", "type" : "article-journal", "volume" : "12" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "ISBN" : "0465077501", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-2", "issued" : { "date-parts" : [ [ "1994" ] ] }, "publisher" : "Basic books", "publisher-place" : "New York, NY", "title" : "The Evolution of Desire: Strategies of Human Mating.", "type" : "book" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1207/s15327965pli0601_1", "ISSN" : "1047-840X", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological inquiry", "id" : "ITEM-3", "issue" : "1", "issued" : { "date-parts" : [ [ "1995" ] ] }, "page" : "1-30", "publisher" : "Taylor & Francis", "title" : "Evolutionary psychology: A new paradigm for psychological science", "type" : "article-journal", "volume" : "6" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "DOI" : "10.1111/j.1467-6494.1990.tb00907.x", "ISSN" : "0022-3506", "author" : [ { "dropping-particle" : "", "family" : "Tooby", "given" : "John", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Cosmides", "given" : "Leda", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Personality", "id" : "ITEM-4", "issue" : "1", "issued" : { "date-parts" : [ [ "1990", "3" ] ] }, "page" : "17-67", "publisher" : "Wiley Online Library", "title" : "On the Universality of Human Nature and the Uniqueness of the Individual: The Role of Genetics and Adaptation", "type" : "article-journal", "volume" : "58" }, "uris" : [ "" ] }, { "id" : "ITEM-5", "itemData" : { "URL" : "", "author" : [ { "dropping-particle" : "", "family" : "Cosmides", "given" : "Leda", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Tooby", "given" : "John", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-5", "issued" : { "date-parts" : [ [ "1997" ] ] }, "title" : "Evolutionary psychology: A primer", "type" : "webpage" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Buss 1989, 1994, 1995; Cosmides and Tooby 1997; Tooby and Cosmides 1990)", "manualFormatting" : "(e.g., Buss 1989, 1994, 1995; Cosmides & Tooby 1997; Tooby & Cosmides 1990)", "plainTextFormattedCitation" : "(Buss 1989, 1994, 1995; Cosmides and Tooby 1997; Tooby and Cosmides 1990)", "previouslyFormattedCitation" : "(Buss 1989, 1994, 1995; Cosmides and Tooby 1997; Tooby and Cosmides 1990)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Buss 1989, 1994, 1995; Cosmides & Tooby 1997; Tooby & Cosmides 1990). Empirical examples where researchers have evaluated whether universals exist by testing across different populations include studies of homicide ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "ISBN" : "020201178X", "author" : [ { "dropping-particle" : "", "family" : "Daly", "given" : "Martin", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Wilson", "given" : "Margo", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "1988" ] ] }, "publisher" : "Transaction Books", "publisher-place" : "New Brunswick, NJ", "title" : "Homicide", "type" : "book" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Daly and Wilson 1988)", "plainTextFormattedCitation" : "(Daly and Wilson 1988)", "previouslyFormattedCitation" : "(Daly and Wilson 1988)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Daly and Wilson 1988), economic behavior ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1017/S0140525X05000142", "ISSN" : "0140-525X", "author" : [ { "dropping-particle" : "", "family" : "Henrich", "given" : "Joseph", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Boyd", "given" : "Robert", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Bowles", "given" : "Samuel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Camerer", "given" : "Colin", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Fehr", "given" : "Ernst", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gintis", "given" : "Herbert", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "McElreath", "given" : "Richard", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Alvard", "given" : "Michael", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Barr", "given" : "Abigail", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ensminger", "given" : "Jean", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Henrich", "given" : "Natalie Smith", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Hill", "given" : "Kim", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gil-White", "given" : "Francisco", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gurven", "given" : "Michael", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Marlowe", "given" : "Frank W.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Patton", "given" : "John Q.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Tracer", "given" : "David", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-1", "issue" : "06", "issued" : { "date-parts" : [ [ "2005", "12" ] ] }, "page" : "795-815", "publisher" : "Cambridge Univ Press", "title" : "\u201cEconomic man\u201d in cross-cultural perspective: Behavioral experiments in 15 small-scale societies", "type" : "article-journal", "volume" : "28" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Henrich et al. 2005)", "manualFormatting" : "(e.g., Henrich et al. 2005)", "plainTextFormattedCitation" : "(Henrich et al. 2005)", "previouslyFormattedCitation" : "(Henrich et al. 2005)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Henrich et al. 2005), and mate preferences ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1017/S0140525X00023992", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain sciences", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "1989" ] ] }, "page" : "1-49", "publisher" : "Cambridge Univ Press", "title" : "Sex differences in human mate preferences: Evolutionary hypotheses tested in 37 cultures", "type" : "article-journal", "volume" : "12" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1016/S1090-5138(00)00048-9", "ISSN" : "1090-5138", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Shackelford", "given" : "T K", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "LeBlanc", "given" : "G J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Evolution and Human Behavior", "id" : "ITEM-2", "issue" : "5", "issued" : { "date-parts" : [ [ "2000" ] ] }, "page" : "323-331", "publisher" : "Elsevier", "title" : "Number of children desired and preferred spousal age difference: context-specific mate preference patterns across 37 cultures", "type" : "article-journal", "volume" : "21" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1016/j.paid.2005.01.023", "ISSN" : "0191-8869", "author" : [ { "dropping-particle" : "", "family" : "Shackelford", "given" : "Todd K", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Schmitt", "given" : "David P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Personality and Individual Differences", "id" : "ITEM-3", "issue" : "2", "issued" : { "date-parts" : [ [ "2005" ] ] }, "page" : "447-458", "publisher" : "Elsevier", "title" : "Universal dimensions of human mate preferences", "type" : "article-journal", "volume" : "39" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "DOI" : "10.1017/S0140525X0500005", "ISSN" : "1469-1825", "author" : [ { "dropping-particle" : "", "family" : "Schmitt", "given" : "David P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-4", "issue" : "2", "issued" : { "date-parts" : [ [ "2005" ] ] }, "page" : "247-275", "publisher" : "Cambridge Univ Press", "title" : "Sociosexuality from Argentina to Zimbabwe: A 48-nation study of sex, culture, and strategies of human mating", "type" : "article-journal", "volume" : "28" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Buss 1989; Buss et al. 2000; Schmitt 2005; Shackelford et al. 2005)", "manualFormatting" : "(e.g., Buss, Shackelford, & LeBlanc 2000; Buss 1989; Schmitt 2005; Shackelford, Schmitt, & Buss 2005)", "plainTextFormattedCitation" : "(Buss 1989; Buss et al. 2000; Schmitt 2005; Shackelford et al. 2005)", "previouslyFormattedCitation" : "(Buss 1989; Buss et al. 2000; Schmitt 2005; Shackelford et al. 2005)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Buss, Shackelford, & LeBlanc 2000; Buss 1989; Schmitt 2005; Shackelford, Schmitt, & Buss 2005). Cross-cultural universals (see for example those listed in ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "ISBN" : "007008209X", "author" : [ { "dropping-particle" : "", "family" : "Brown", "given" : "Donald E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "1991" ] ] }, "publisher" : "McGraw-Hill", "publisher-place" : "New York, NY", "title" : "Human universals", "type" : "book" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Brown", "given" : "Donald E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Being humans: Anthropological universality and particularity in transdisciplinary perspectives", "editor" : [ { "dropping-particle" : "", "family" : "Roughley", "given" : "N", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-2", "issued" : { "date-parts" : [ [ "2000" ] ] }, "page" : "156-174", "publisher" : "Walter de Gruyter", "publisher-place" : "Berlin", "title" : "Human universals and their implications", "type" : "chapter" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Brown 1991, 2000)", "manualFormatting" : "Brown 1991, 2000)", "plainTextFormattedCitation" : "(Brown 1991, 2000)", "previouslyFormattedCitation" : "(Brown 1991, 2000)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Brown 1991, 2000) are often used by evolutionary psychologists as evidence for adaptive psychological mechanisms (e.g., ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1207/s15327965pli0601_1", "ISSN" : "1047-840X", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological inquiry", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "1995" ] ] }, "page" : "1-30", "publisher" : "Taylor & Francis", "title" : "Evolutionary psychology: A new paradigm for psychological science", "type" : "article-journal", "volume" : "6" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Buss 1995)", "manualFormatting" : "Buss 1995)", "plainTextFormattedCitation" : "(Buss 1995)", "previouslyFormattedCitation" : "(Buss 1995)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Buss 1995). It would thus seem, as ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1016/j.copsyc.2015.08.015", "ISSN" : "2352-250X", "abstract" : "Perhaps no field of psychology is more strongly motivated and better equipped than evolutionary psychology to respond to the recent call for psychologists to expand their empirical base beyond WEIRD (Western Educated Industrialized Rich Democratic) samples. Evolutionary psychologists have historically focused their efforts on identifying species-specific psychological traits, for which evidence often hinged on the extent to which traits were generalizable across human groups. Now, a new generation of researchers is embracing cultural and environmental variation to test evolutionary hypotheses. Here we discuss how comparative research with diverse societies, while challenging, can help inform the complex nature of our species\u2019 psychology and in doing so, we outline best theoretical and methodological practices as well as common pitfalls in cross-cultural investigations. We end with a recommendation for the use of publicly available databases for cataloging psychological variation across the world's many diverse populations. Because of rapid culture change and globalization, it is more important now than ever to document what we know about the world's cultures in ways that can be used by future researchers.", "author" : [ { "dropping-particle" : "", "family" : "Apicella", "given" : "Coren L", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Barrett", "given" : "H Clark", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Current Opinion in Psychology", "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2016", "2" ] ] }, "page" : "92-97", "title" : "Cross-cultural evolutionary psychology", "type" : "article-journal", "volume" : "7" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Apicella and Barrett 2016)", "manualFormatting" : "Apicella and Barrett (2016: p. 92)", "plainTextFormattedCitation" : "(Apicella and Barrett 2016)", "previouslyFormattedCitation" : "(Apicella and Barrett 2016)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Apicella and Barrett (2016: p. 92) have argued, that “perhaps no field of psychology is more strongly motivated and better equipped than evolutionary psychology to respond to the recent call for psychologists to expand their empirical base beyond WEIRD (Western Educated Industrialized Rich Democratic) samples”. Similarly, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "URL" : "", "accessed" : { "date-parts" : [ [ "2016", "11", "14" ] ] }, "author" : [ { "dropping-particle" : "", "family" : "Kurzban", "given" : "Robert", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "EP Journal blog", "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2013" ] ] }, "title" : "Is Evolutionary Psychology WEIRD or NORMAL?", "type" : "webpage" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Kurzban 2013)", "manualFormatting" : "Kurzban (2013)", "plainTextFormattedCitation" : "(Kurzban 2013)", "previouslyFormattedCitation" : "(Kurzban 2013)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Kurzban (2013) argued on the Evolutionary Psychology blog that “adding evolution to psychology makes the science less WEIRD”. He found that for the 2012 volume, 65% of the articles in the journal ‘Evolution & Human Behavior’ were WEIRD, which contrasts favourably with data for other fields as cited above. This initial evidence suggests that evolutionary psychology is indeed less WEIRD than some subdivisions of psychology. Here, we examine the samples used in two leading evolutionary psychology journals in more depth. We have no explicit hypotheses, but rather describe the samples used in these two journals according to their geographical origin, age group (adult or child), student status, and source (online vs. offline). In addition, we test whether the sample sizes vary based on these categories. Our aim is to provide an up-to-date snapshot of contemporary evolutionary psychology sampling practice, while responding to calls to increase description within science ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "URL" : "", "author" : [ { "dropping-particle" : "", "family" : "Scott-Phillips", "given" : "Thomas C", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2018" ] ] }, "title" : "Expression Unleashed", "type" : "webpage" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Scott-Phillips 2018)", "plainTextFormattedCitation" : "(Scott-Phillips 2018)", "previouslyFormattedCitation" : "(Scott-Phillips 2018)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Scott-Phillips 2018); it is easier to move forward if we better know where we currently stand.MethodsCodingAs part of a larger project, data relevant to our research questions were captured from all of the articles published in 2015 and 2016 within the journals ‘Evolutionary Psychology’ (EP, published by Sage) and ‘Evolution & Human Behavior’ (E&HB, published by Elsevier). The 2015 articles were coded by eight coders under the supervision of the first author, and the 2016 articles were coded by the first author. Of course, many key papers on evolutionary psychology are published in other outlets, such as, for example, ‘Journal of Personality and Social Psychology’ ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/0022-3514.72.2.346", "ISSN" : "1939-1315", "abstract" : "Although much research has explored the adaptive problems of mate selection and mate attraction, little research has investigated the adaptive problem of mace retention. We tested several evolutionary psychological hypotheses about the determinants of mate retention in 214 married people. We assessed the usage of 19 mate retention tactics ranging from vigilance to violence. Key hypothesized findings include the following: Men's, but not women's, mate retention positively covaried with partner's youth and physical attractiveness. Women's, but not men's, mate retention positively covaried with partner's income and status striving. Men's mate retention positively covaried with perceived probability of partner's infidelity. Men, mon than women, reported using resource display, submission and debasement, and intrasexual threats to retain their mates. Women, more than men, reported using appearance enhancement and verbal signals of possession. Discussion includes an evolutionary psychological analysis of mate retention in married couples.", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Shackelford", "given" : "Todd K", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Personality and Social Psychology", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "1997", "2" ] ] }, "page" : "346-361", "publisher" : "AMER PSYCHOLOGICAL ASSOC", "publisher-place" : "750 FIRST ST NE, WASHINGTON, DC 20002-4242 USA", "title" : "From vigilance to violence: Mate retention tactics in married couples.", "type" : "article-journal", "volume" : "72" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1037/0022-3514.69.6.1166", "ISSN" : "1939-1315", "author" : [ { "dropping-particle" : "", "family" : "Kenrick", "given" : "Douglas T", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Keefe", "given" : "Richard C", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Bryan", "given" : "Angela", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Barr", "given" : "Alicia", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Brown", "given" : "Stephanie", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Personality and Social Psychology", "id" : "ITEM-2", "issue" : "6", "issued" : { "date-parts" : [ [ "1995" ] ] }, "page" : "1166-1172", "publisher" : "American Psychological Association", "title" : "Age preferences and mate choice among homosexuals and heterosexuals: A case for modular psychological mechanisms.", "type" : "article-journal", "volume" : "69" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Buss and Shackelford 1997; Kenrick et al. 1995)", "manualFormatting" : "(e.g., Buss & Shackelford 1997; Kenrick, Keefe, Bryan, Barr, & Brown 1995)", "plainTextFormattedCitation" : "(Buss and Shackelford 1997; Kenrick et al. 1995)", "previouslyFormattedCitation" : "(Buss and Shackelford 1997; Kenrick et al. 1995)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Buss & Shackelford 1997; Kenrick, Keefe, Bryan, Barr, & Brown 1995), or ‘Psychological Science’ ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1111/j.1467-9280.1992.tb00038.x", "ISSN" : "0956-7976", "abstract" : "In species with internal female fertilization, males risk both lowered paternity probability and investment in rival gametes if their mates have sexual contact with other males. Females of such species do not risk lowered maternity probability through partner infidelity, but they do risk the diversion of their mates' commitment and resources to rival females. Three studies tested the hypothesis that sex differences in jealousy emerged in humans as solutions to the respective adaptive problems faced by each sex. In Study 1, men and women selected which event would upset them more\u2014a partner's sexual infidelity or emotional infidelity. Study 2 recorded physiological responses (heart rate, electrodermal response, corrugator supercilii contraction) while subjects imagined separately the two types of partner infidelity. Study 3 tested the effect of being in a committed sexual relationship on the activation of jealousy. All studies showed large sex differences, confirming hypothesized sex linkages in jealousy activation.", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Larsen", "given" : "Randy J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Westen", "given" : "Drew", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Semmelroth", "given" : "Jennifer", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Science", "id" : "ITEM-1", "issue" : "4", "issued" : { "date-parts" : [ [ "1992", "7", "1" ] ] }, "note" : "10.1111/j.1467-9280.1992.tb00038.x", "page" : "251-255", "title" : "Sex Differences in Jealousy: Evolution, Physiology, and Psychology", "type" : "article-journal", "volume" : "3" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Buss et al. 1992)", "manualFormatting" : "(e.g., Buss, Larsen, Westen, & Semmelroth 1992)", "plainTextFormattedCitation" : "(Buss et al. 1992)", "previouslyFormattedCitation" : "(Buss et al. 1992)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Buss, Larsen, Westen, & Semmelroth 1992). However, it is reasonable to assume that any article published in EP and E&HB is allied to the discipline of evolutionary psychology, broadly conceived. In addition, the choice follows Kurzban 2013’s selection of E&HB for analysis, and the publication of his analysis within the Evolutionary Psychology blog, on the website of the eponymous journal. The coders recorded the geographical region from which the data originated based on the M49 UNDP codes ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "URL" : "", "author" : [ { "dropping-particle" : "", "family" : "United Nations", "given" : "", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2013" ] ] }, "title" : "Composition of macro geographical (continental) regions, geographical sub-regions, and selected economic and other groupings", "type" : "webpage" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(United Nations 2013)", "manualFormatting" : "(United Nations 2013", "plainTextFormattedCitation" : "(United Nations 2013)", "previouslyFormattedCitation" : "(United Nations 2013)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(United Nations 2013: Africa, North America, Latin America and the Caribbean, Asia, Australia, Europe, and Oceania (excluding Australia)). If a paper listed more than five countries, we labelled it ‘Cross-Cultural’; we coded each sample individually for papers with one to four geographical samples. We acknowledge that some papers could have an explicitly ‘cross-cultural’ goal even with just two samples, for example studies establishing measurement invariance. However, our focus here is on the samples being used not the paper, as we believed this to be more easy to assess. After piloting, we settled on the following eight categories for the sample participants: online (paid crowdsourced, such as a sample recruited via M-Turk); online (unpaid crowdsourced, such as a sample recruited via Facebook or Twitter); offline (western child); offline (western student); offline (western non-student adult); offline (non-western child); offline (non-western student); offline (non-western, non-student adult). Online studies were subdivided only into paid and unpaid samples given the focus of our research, together with the difficulty of confirming online participant age, student/non-student status, and location. Samples from Europe, Australia, New Zealand, and North America were coded as Western, while samples from other countries coded as Non-Western, following ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1002/ajhb.22917", "ISSN" : "10420533", "abstract" : "\u00a9 2016 The Authors American Journal of Human Biology Published by Wiley Periodicals, Inc. Objectives: The study of assortative mating for height has a rich history in human biology. Although the positive correlation between the stature of spouses has often been noted in western populations, recent papers suggest that mating patterns for stature are not universal. The objective of this paper was to review the published evidence to examine the strength of and universality in assortative mating for height. Methods: We conducted an extensive literature review and meta-analysis. We started with published reviews but also searched through secondary databases. Our search led to 154 correlations of height between partners. We classified the populations as western and non-western based on geography. These correlations were then analyzed via meta-analytic techniques. Results: 148 of the correlations for partner heights were positive and the overall analysis indicates moderate positive assortative mating (r =.23). Although assortative mating was slightly stronger in countries that can be described as western compared to non-western, this difference was not statistically significant. We found no evidence for a change in assortative mating for height over time. There was substantial residual heterogeneity in effect sizes and this heterogeneity was most pronounced in western countries. Conclusions: Positive assortative mating for height exists in human populations, but is modest in magnitude suggesting that height is not a major factor in mate choice. Future research is necessary to understand the underlying causes of the large amount of heterogeneity observed in the degree of assortative mating across human populations, which may stem from a combination of methodological and ecological differences.", "author" : [ { "dropping-particle" : "", "family" : "Stulp", "given" : "Gert", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Simons", "given" : "Mirre J.P.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Grasman", "given" : "Sara", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "V.", "family" : "Pollet", "given" : "Thomas", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Journal of Human Biology", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "2017", "1" ] ] }, "page" : "e22917", "title" : "Assortative mating for human height: A meta-analysis", "type" : "article-journal", "volume" : "29" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Stulp et al. 2017)", "plainTextFormattedCitation" : "(Stulp et al. 2017)", "previouslyFormattedCitation" : "(Stulp et al. 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Stulp et al. 2017). Where disagreement between coders existed, this was resolved via discussion. Data analysisWe used R (3.5.1, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "R Development Core Team", "given" : "", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2008" ] ] }, "publisher" : "R Foundation for Statistical Computing.", "publisher-place" : "Vienna, Austria", "title" : "R : A language and environment for statistical computing", "type" : "article" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(R Development Core Team 2008)", "manualFormatting" : "R Development Core Team 2008)", "plainTextFormattedCitation" : "(R Development Core Team 2008)", "previouslyFormattedCitation" : "(R Development Core Team 2008)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }R Development Core Team 2008) and, among other packages, the R-packages bindrcpp (0.2.2, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "M\u00fcller", "given" : "Kirill", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017" ] ] }, "title" : "bindrcpp", "type" : "article" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(M\u00fcller 2017)", "manualFormatting" : "M\u00fcller 2017)", "plainTextFormattedCitation" : "(M\u00fcller 2017)", "previouslyFormattedCitation" : "(M\u00fcller 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Müller 2017), broom (0.5.0, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Robinson", "given" : "David", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017" ] ] }, "title" : "broom", "type" : "article" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Robinson 2017)", "manualFormatting" : "Robinson 2017)", "plainTextFormattedCitation" : "(Robinson 2017)", "previouslyFormattedCitation" : "(Robinson 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Robinson 2017), dplyr (0.7.6, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Wickham", "given" : "Hadley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Francois", "given" : "Romain", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017" ] ] }, "title" : "dplyr: A grammar of data manipulation", "type" : "article" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Wickham and Francois 2017)", "manualFormatting" : "Wickham & Francois 2015)", "plainTextFormattedCitation" : "(Wickham and Francois 2017)", "previouslyFormattedCitation" : "(Wickham and Francois 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Wickham & Francois 2015), ggplot2 (3.3.0, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1007/978-0-387-98141-3", "ISBN" : "978-0-387-98140-6", "author" : [ { "dropping-particle" : "", "family" : "Wickham", "given" : "Hadley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2009" ] ] }, "publisher" : "Springer New York", "publisher-place" : "New York, NY", "title" : "ggplot2", "type" : "book" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Wickham 2009)", "manualFormatting" : "Wickham 2009)", "plainTextFormattedCitation" : "(Wickham 2009)", "previouslyFormattedCitation" : "(Wickham 2009)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Wickham 2009), knitr (1.17, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "ISBN" : "1498716970", "author" : [ { "dropping-particle" : "", "family" : "Xie", "given" : "Yihui", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2015" ] ] }, "publisher" : "CRC Press", "title" : "Dynamic Documents with R and knitr", "type" : "book", "volume" : "29" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Xie 2015)", "manualFormatting" : "Xie 2015)", "plainTextFormattedCitation" : "(Xie 2015)", "previouslyFormattedCitation" : "(Xie 2015)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Xie 2015), papaja (0.1.0.9709, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Aust", "given" : "Frederik", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Barth", "given" : "M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2016" ] ] }, "publisher" : "See . com/crsh/papaja", "title" : "papaja: Create APA manuscripts with R Markdown", "type" : "article" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Aust and Barth 2016)", "manualFormatting" : "Aust & Barth 2016)", "plainTextFormattedCitation" : "(Aust and Barth 2016)", "previouslyFormattedCitation" : "(Aust and Barth 2016)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Aust & Barth 2016), plyr (1.8.4, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Wickham", "given" : "Hadley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Wickham", "given" : "Maintainer Hadley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017" ] ] }, "title" : "\u2018plyr\u2019 package", "type" : "article" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Wickham and Wickham 2017)", "manualFormatting" : "Wickham & Wickham 2017)", "plainTextFormattedCitation" : "(Wickham and Wickham 2017)", "previouslyFormattedCitation" : "(Wickham and Wickham 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Wickham & Wickham 2017), readxl (1.010, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Wickham", "given" : "Hadley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Bryan", "given" : "Jennifer", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017" ] ] }, "title" : "readxl: Read Excel Files. R package version 1.0.0.", "type" : "article" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Wickham and Bryan 2017)", "manualFormatting" : "Wickham & Bryan 2017)", "plainTextFormattedCitation" : "(Wickham and Bryan 2017)", "previouslyFormattedCitation" : "(Wickham and Bryan 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Wickham & Bryan 2017), stargazer (5.2.2, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Hlavac", "given" : "Marek", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2014" ] ] }, "title" : "stargazer: LaTeX code and ASCII text for well-formatted regression and summary statistics tables.", "type" : "article" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Hlavac 2014)", "manualFormatting" : "Hlavac 2014)", "plainTextFormattedCitation" : "(Hlavac 2014)", "previouslyFormattedCitation" : "(Hlavac 2014)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Hlavac 2014), and tidyr (0.8.1, ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Wickham", "given" : "Hadley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2014" ] ] }, "title" : "Tidyr: easily tidy data with spread and gather functions.", "type" : "article" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Wickham 2014)", "manualFormatting" : "Wickham 2014)", "plainTextFormattedCitation" : "(Wickham 2014)", "previouslyFormattedCitation" : "(Wickham 2014)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }Wickham 2014) for our analyses. To compare sample sizes, we relied on non-parametric statistics (Siegel & Castellan, 1988), with post-hoc comparisons adjusted for multiple testing ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.2307/2346101", "ISSN" : "00359246", "abstract" : "The common approach to the multiplicity problem calls for controlling the familywise error rate (FWER). This approach, though, has faults, and we point out a few. A different approach to problems of multiple significance testing is presented. It calls for controlling the expected proportion of falsely rejected hypotheses-the false discovery rate. This error rate is equivalent to the FWER when all hypotheses are true but is smaller otherwise. Therefore, in problems where the control of the false discovery rate rather than that of the FWER is desired, there is potential for a gain in power. A simple sequential Bonferroni-type procedure is proved to control the false discovery rate for independent test statistics, and a simulation study shows that the gain in power is substantial. The use of the new procedure and the appropriateness of the criterion are illustrated with examples.", "author" : [ { "dropping-particle" : "", "family" : "Benjamini", "given" : "Yoav", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Hochberg", "given" : "Yosef", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of the Royal Statistical Society. Series B (Methodological)", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "1995" ] ] }, "page" : "289-300", "publisher" : "[Royal Statistical Society, Wiley]", "title" : "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing", "type" : "article-journal", "volume" : "57" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Benjamini and Hochberg 1995)", "plainTextFormattedCitation" : "(Benjamini and Hochberg 1995)", "previouslyFormattedCitation" : "(Benjamini and Hochberg 1995)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Benjamini and Hochberg 1995), given that visual inspection showed that the data were non-normally distributed. We used logarithmic transformations when presenting figures on sample sizes because the largest samples are so much greater than the smallest ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1002/sim.4780140810", "ISSN" : "02776715", "author" : [ { "dropping-particle" : "", "family" : "Keene", "given" : "Oliver N", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Statistics in Medicine", "id" : "ITEM-1", "issue" : "8", "issued" : { "date-parts" : [ [ "1995", "4", "30" ] ] }, "page" : "811-819", "publisher" : "Wiley Online Library", "title" : "The log transformation is special", "type" : "article-journal", "volume" : "14" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Keene 1995)", "plainTextFormattedCitation" : "(Keene 1995)", "previouslyFormattedCitation" : "(Keene 1995)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Keene 1995). The data and analysis document, including a list of all R packages used in the analysis, are available from the Open Science Framework.ResultsDescriptive statisticsThere were 219 papers, of which 180 papers contained codable samples (EP: 76; E&HB: 104). Thirty-nine papers could not be coded because they consisted of, for instance, mathematical models, work on non-humans, or literature reviews. Within the 180 codable papers, there were 311 samples, and the median number of samples per paper was 1. The mean sample size was 4,094 but this was driven by one extremely large sample (N=927,134). The median sample size was 186 but sample sizes varied substantially (minimum: 11; first quartile: 96.5; third quartile: 334.5; maximum: 927,134).Figure 1 shows the distribution of samples by geographical region. The majority of samples were from North America (153), followed by Europe (93) and Asia (37). Of the Asian samples, the majority were from Japan (11), followed by China (7) and Israel (6). There were only 6 samples from Africa (4 from Tanzania, 1 from Namibia, 1 from Nigeria), and only 8 from Latin America and the Caribbean (2 from Guatemala; 2 from Cura?ao; 2 from Bolivia; 1 community from Northern-Brazil/Southern Guyana/East Venezuela; and 1 undefined [Latin American students studying in Germany]). There were 7 samples from Australia, and 1 from Oceania (excluding Australia): Fiji Island. Only 6 samples were Cross-Cultural (containing samples from more than 5 different countries). Combining the figures, we found that around 8 out of 10 samples were from WEIRD populations (81%, Europe/North America/Australia), and that 87% of the samples used were from developed regions (following the UN classification; United Nations, 2013). Figure 1: Origin of samples. N. America= North America, L. America= Latin America and Caribbean.In terms of sample type, 113 of the 311 samples were Western student samples, while 24 were non-Western student samples; 60 samples were online paid crowdsourced, while 20 were unpaid crowdsourced. Thus, 70% of the samples were either online samples or student samples. 25 samples were based on children (21 from Western and 4 from Non-western populations). Only a small fraction of the samples consisted of non-Western adults who were not students (24 out of 311 samples, or 8%).Are samples from certain geographical locations larger than others?Given that there was only one sample from Oceania (excluding Australia) (see Figure 1 and Figure 2), we combined this with Australia for the analysis of variation in sample sizes between regions (see ESM for additional analyses using this combination). Variation in sample size between geographical regions was not statistically significant (Kruskal-Wallis test: χ2(6) = 10.095, p = .12). Following adjustment for multiple testing, the median sample size was found to be significantly larger for cross-cultural samples (which, according to our coding criteria, had to contain data from more than 5 different countries) than for Latin American and Caribbean samples (p = .037). The ESM contains all post-hoc multiple comparisons (all remaining p-values >.09; see ESM).Figure 2 Violin plot for geographical origin and Log. sample size, density distribution (curve), median (horizontal line), interquartile range (IQR, box), whiskers (1.5 times the IQR) and individual samples (dots).Are some types of samples larger than others?The sample sizes differed significantly according to type (Figure 3; Kruskal-Wallis test: χ2(7) = 63.9, p < .0001). Post-hoc comparisons adjusted for multiple testing using the Benjamini-Hochberg procedure showed that online (paid crowdsourced) and online (unpaid crowdsourced) samples tended to be larger than other types of samples (Figure 3; Table 1).Figure 3 Violin plot for geographical origin and Log. sample size, density distribution (curve), median (horizontal line), interquartile range (IQR, box), whiskers (1.5 times the IQR) and individual samples (dots).Table 1: Post-hoc comparisons based on Type of sampleGroup 1Group 2Adjusted p value1W StudentOnline Paid<.000012W StudentOnline Unpaid0.000183W StudentW Adult0.000184N-W StudentOnline Paid0.000965N-W AdultOnline Paid0.005186W ChildOnline Paid0.005827W ChildOnline Unpaid0.007978N-W AdultOnline Unpaid0.008269N-W StudentOnline Unpaid0.0085810W AdultW Child0.0298111N-W StudentW Adult0.0410012W AdultN-W Adult0.0620513N-W ChildOnline Paid0.2254614N-W ChildOnline Unpaid0.3408015N-W StudentW Child0.3408016W StudentN-W Student0.4179917Online PaidOnline Unpaid0.4336918W AdultOnline Unpaid0.4336919W ChildN-W Child0.4336920W AdultN-W Child0.4336921W StudentN-W Child0.4336922N-W AdultW Child0.4336923W StudentW Child0.4336924N-W AdultN-W Child0.6215625N-W StudentN-W Adult0.7033026N-W StudentN-W Child0.7210227W StudentN-W Adult0.7392128W AdultOnline Paid0.94837DiscussionOur survey of papers published in 2015-2016 in two key journals relevant to evolutionary psychology, ‘Evolution & Human Behavior’ and ‘Evolutionary Psychology’, indicated a clear dominance of adult samples from western, developed countries, with a particular preponderance of North American samples. Seventy percent of samples were sourced online or from student populations. Asian samples mainly consisted of samples from Japan. Notable under-representations included samples collected from Africa and Latin America (including the Caribbean). Data collected online typically gave rise to the largest sample sizes.Implications of relying on WEIRD samplesIn our survey, 81% of samples were from WEIRD populations. The main advantages of relying on WEIRD samples arise from the fact that most authors are WEIRD, and so WEIRD samples are more practical and convenient, particularly in respect of ease of access and low costs of data sampling. Requiring costly and time-consuming data collection can stifle scientific research, given that it is often poorly resourced ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1038/s41562-018-0311-x", "ISSN" : "2397-3374", "abstract" : "In response to recommendations to redefine statistical significance to P \u2264 0.005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.", "author" : [ { "dropping-particle" : "", "family" : "Lakens", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Adolfi", "given" : "Federico G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Albers", "given" : "Casper J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Anvari", "given" : "Farid", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Apps", "given" : "Matthew A J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Argamon", "given" : "Shlomo E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Baguley", "given" : "Thom", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Becker", "given" : "Raymond B", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Benning", "given" : "Stephen D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Bradford", "given" : "Daniel E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Buchanan", "given" : "Erin M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Caldwell", "given" : "Aaron R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Calster", "given" : "Ben", "non-dropping-particle" : "Van", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Carlsson", "given" : "Rickard", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Chen", "given" : "Sau-Chin", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Chung", "given" : "Bryan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Colling", "given" : "Lincoln J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Collins", "given" : "Gary S", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Crook", "given" : "Zander", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Cross", "given" : "Emily S", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Daniels", "given" : "Sameera", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Danielsson", "given" : "Henrik", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "DeBruine", "given" : "Lisa", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Dunleavy", "given" : "Daniel J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Earp", "given" : "Brian D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Feist", "given" : "Michele I", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ferrell", "given" : "Jason D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Field", "given" : "James G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Fox", "given" : "Nicholas W", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Friesen", "given" : "Amanda", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gomes", "given" : "Caio", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gonzalez-Marquez", "given" : "Monica", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Grange", "given" : "James A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Grieve", "given" : "Andrew P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Guggenberger", "given" : "Robert", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Grist", "given" : "James", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Harmelen", "given" : "Anne-Laura", "non-dropping-particle" : "van", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Hasselman", "given" : "Fred", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Hochard", "given" : "Kevin D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Hoffarth", "given" : "Mark R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Holmes", "given" : "Nicholas P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ingre", "given" : "Michael", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Isager", "given" : "Peder M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Isotalus", "given" : "Hanna K", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Johansson", "given" : "Christer", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Juszczyk", "given" : "Konrad", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Kenny", "given" : "David A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Khalil", "given" : "Ahmed A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Konat", "given" : "Barbara", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lao", "given" : "Junpeng", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Larsen", "given" : "Erik Gahner", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lodder", "given" : "Gerine M A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lukavsk\u00fd", "given" : "Ji\u0159\u00ed", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Madan", "given" : "Christopher R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Manheim", "given" : "David", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Martin", "given" : "Stephen R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Martin", "given" : "Andrea E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Mayo", "given" : "Deborah G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "McCarthy", "given" : "Randy J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "McConway", "given" : "Kevin", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "McFarland", "given" : "Colin", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Nio", "given" : "Amanda Q X", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Nilsonne", "given" : "Gustav", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Oliveira", "given" : "Cilene Lino", "non-dropping-particle" : "de", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Xivry", "given" : "Jean-Jacques Orban", "non-dropping-particle" : "de", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Parsons", "given" : "Sam", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pfuhl", "given" : "Gerit", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Quinn", "given" : "Kimberly A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sakon", "given" : "John J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Saribay", "given" : "S Adil", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Schneider", "given" : "Iris K", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Selvaraju", "given" : "Manojkumar", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sjoerds", "given" : "Zsuzsika", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Smith", "given" : "Samuel G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Smits", "given" : "Tim", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Spies", "given" : "Jeffrey R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sreekumar", "given" : "Vishnu", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Steltenpohl", "given" : "Crystal N", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Stenhouse", "given" : "Neil", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "\u015awi\u0105tkowski", "given" : "Wojciech", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Vadillo", "given" : "Miguel A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Assen", "given" : "Marcel A L M", "non-dropping-particle" : "Van", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Williams", "given" : "Matt N", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Williams", "given" : "Samantha E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Williams", "given" : "Donald R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Yarkoni", "given" : "Tal", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ziano", "given" : "Ignazio", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Zwaan", "given" : "Rolf A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Nature Human Behaviour", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "2018" ] ] }, "page" : "168-171", "title" : "Justify your alpha", "type" : "article-journal", "volume" : "2" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Lakens et al. 2018)", "plainTextFormattedCitation" : "(Lakens et al. 2018)", "previouslyFormattedCitation" : "(Lakens et al. 2018)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Lakens et al. 2018). There are obvious practical difficulties in collecting data outside of one’s country of residence, or in countries where attitudes and familiarity may vary in relation to the psychological procedures that will be well-known to readers of this journal, such as models of obtaining informed consent, or methods of data elicitation. Indeed, we do not always need to go far to find diverse samples ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "ISBN" : "1783741902", "author" : [ { "dropping-particle" : "", "family" : "Nettle", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017" ] ] }, "publisher" : "Open Book Publishers", "publisher-place" : "Cambridge, UK", "title" : "Tyneside neighbourhoods: Deprivation, social life and social behaviour in one British city", "type" : "book" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1037/h0099131", "ISSN" : "2330-2933", "author" : [ { "dropping-particle" : "", "family" : "Hill", "given" : "Jessica M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Jobling", "given" : "Ruth", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pollet", "given" : "Thomas Victor", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Nettle", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Evolutionary Behavioral Sciences", "id" : "ITEM-2", "issue" : "2", "issued" : { "date-parts" : [ [ "2014" ] ] }, "page" : "59-69", "publisher" : "Educational Publishing Foundation", "title" : "Social capital across urban neighborhoods: A comparison of self-report and observational data.", "type" : "article-journal", "volume" : "8" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "ISBN" : "0316175250", "author" : [ { "dropping-particle" : "", "family" : "Wilson", "given" : "David Sloan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-3", "issued" : { "date-parts" : [ [ "2011" ] ] }, "publisher" : "Little, Brown", "publisher-place" : "New York, NY", "title" : "The Neighborhood Project: Using evolution to improve my city, one block at a time", "type" : "book" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "DOI" : "10.1016/j.evolhumbehav.2008.12.002", "ISSN" : "10905138", "author" : [ { "dropping-particle" : "", "family" : "Wilson", "given" : "David Sloan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "O'Brien", "given" : "Daniel Tumminelli", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sesma", "given" : "Artura", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Evolution and Human Behavior", "id" : "ITEM-4", "issue" : "3", "issued" : { "date-parts" : [ [ "2009", "5" ] ] }, "page" : "190-200", "publisher" : "Elsevier", "title" : "Human prosociality from an evolutionary perspective: variation and correlations at a city-wide scale", "type" : "article-journal", "volume" : "30" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Hill et al. 2014; Nettle 2017; Wilson 2011; Wilson et al. 2009)", "manualFormatting" : "(e.g., Hill et al. 2014; Nettle 2017; Wilson 2011; Wilson et al. 2009)", "plainTextFormattedCitation" : "(Hill et al. 2014; Nettle 2017; Wilson 2011; Wilson et al. 2009)", "previouslyFormattedCitation" : "(Hill et al. 2014; Nettle 2017; Wilson 2011; Wilson et al. 2009)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Hill et al. 2014; Nettle 2017; Wilson 2011; Wilson et al. 2009). WEIRD samples themselves are certainly not homogeneous, and for instance even people from different neighbourhoods within the same city can vary as much as people from entirely different cultures ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "ISBN" : "1783741902", "author" : [ { "dropping-particle" : "", "family" : "Nettle", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017" ] ] }, "publisher" : "Open Book Publishers", "publisher-place" : "Cambridge, UK", "title" : "Tyneside neighbourhoods: Deprivation, social life and social behaviour in one British city", "type" : "book" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1371/journal.pone.0026922", "abstract" : "Human cooperative behaviour, as assayed by decisions in experimental economic dilemmas such as the Dictator Game, is variable across human populations. Within-population variation has been less well studied, especially within industrial societies. Moreover, little is known about the extent to which community-level variation in Dictator Game behaviour relates to community-level variation in real-world social behaviour. We chose two neighbourhoods of the city of Newcastle upon Tyne that were similar in most regards, but at opposite ends of the spectrum in terms of level of socioeconomic deprivation. We administered Dictator Games to randomly-selected residents, and also gathered a large number of more naturalistic measures of cooperativeness. There were dramatic differences in Dictator Game behaviour between the two neighbourhoods, with the mean allocation to the other player close to half the stake in the affluent neighbourhood, and close to one tenth of the stake in the deprived neighbourhood. Moreover, the deprived neighbourhood was also characterised by lower self-reported social capital, higher frequencies of crime and antisocial behaviour, a higher frequency of littering, and less willingness to take part in a survey or return a lost letter. On the other hand, there were no differences between the neighbourhoods in terms of the probability of helping a person who dropped an object, needed directions to a hospital, or needed to make change for a coin, and people on the streets were less likely to be alone in the deprived neighbourhood than the affluent one. We conclude that there can be dramatic local differences in cooperative behaviour within the same city, and that these need further theoretical explanation.", "author" : [ { "dropping-particle" : "", "family" : "Nettle", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Coll\u00e9ony", "given" : "Agathe", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Cockerill", "given" : "Maria", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "PLoS ONE", "id" : "ITEM-2", "issue" : "10", "issued" : { "date-parts" : [ [ "2011", "10", "27" ] ] }, "page" : "e26922", "publisher" : "Public Library of Science", "title" : "Variation in Cooperative Behaviour within a Single City", "type" : "article-journal", "volume" : "6" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Nettle 2017; Nettle et al. 2011)", "plainTextFormattedCitation" : "(Nettle 2017; Nettle et al. 2011)", "previouslyFormattedCitation" : "(Nettle 2017; Nettle et al. 2011)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Nettle 2017; Nettle et al. 2011). Thus, even samples from WEIRD cultures can be sufficiently diverse that they provide some useful evidence in support of generalisability.On the other hand, evolutionary psychologists are often keen to sample outside WEIRD populations because of their interest in assessing and recording the nature of humans as a species. WEIRD populations may experience environments (in terms of novel technology, experience of hunger, exposure to death, and so on) that are particularly dissimilar from those experienced by many of our ancestors, something that needs to be bourne in mind when constructing and testing evolutionary theories of behaviour. Human universals (Brown 1991, 2000) may only be uncovered following assessment of multiple human populations, if not all populations, and assist in developing and evaluating evolutionary theories of behaviour. If we are to test how individual differences may be functionally adaptive ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1098/rstb.1998.0202", "ISSN" : "0962-8436", "abstract" : "David Sloan WilsonDepartment of Biological Sciences, Binghamton University, Binghamton, NY 13902- 6000, USA (dwilson{at}binghamton.edu)Abstract Phenotypic differences can exist between species, between local populations of the same species and between individuals within single local populations. At all scales, phenotypic differences can be either adaptive or non\u2013adaptive. Using natural selection to explain differences between closely related species was controversial during the 1940s but had become common by the 1960s. Similarly, the adaptive nature of differences between local populations was initially controversial but had become widely accepted by the 1980s. The interpretation of differences at the finest scale\u2014between individuals within single populations\u2014is still unresolved. This paper reviews studies of adaptive individual differences in resource use and response to risk. A general conceptual framework for thinking about adaptive individual differences within populations can unite subjects as seemingly different as speciation and personality psychology. adaptationnatural selectionindividual differencestrophic polymorphismspersonality differencesfrequency-dependence", "author" : [ { "dropping-particle" : "", "family" : "Wilson", "given" : "David Sloan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Philosophical Transactions of the Royal Society B: Biological Sciences", "id" : "ITEM-1", "issue" : "1366", "issued" : { "date-parts" : [ [ "1998", "2", "28" ] ] }, "page" : "199-205", "title" : "Adaptive individual differences within single populations", "type" : "article-journal", "volume" : "353" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1037/ebs0000017", "ISSN" : "2330-2933", "author" : [ { "dropping-particle" : "", "family" : "Tybur", "given" : "Joshua M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Frankenhuis", "given" : "Willem E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pollet", "given" : "Thomas Victor", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Evolutionary Behavioral Sciences", "id" : "ITEM-2", "issue" : "4", "issued" : { "date-parts" : [ [ "2014" ] ] }, "page" : "274-283", "publisher" : "Educational Publishing Foundation", "title" : "Behavioral immune system methods: Surveying the present to shape the future.", "type" : "article-journal", "volume" : "8" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Tybur et al. 2014; Wilson 1998)", "plainTextFormattedCitation" : "(Tybur et al. 2014; Wilson 1998)", "previouslyFormattedCitation" : "(Tybur et al. 2014; Wilson 1998)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Tybur et al. 2014; Wilson 1998), we need to compare individuals from different ecological settings ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1016/j.tree.2009.05.013", "ISSN" : "0169-5347", "author" : [ { "dropping-particle" : "", "family" : "Nettle", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Trends in Ecology & Evolution", "id" : "ITEM-1", "issue" : "11", "issued" : { "date-parts" : [ [ "2009", "11" ] ] }, "page" : "618-624", "title" : "Ecological influences on human behavioural diversity: a review of recent findings", "type" : "article-journal", "volume" : "24" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Nettle 2009)", "plainTextFormattedCitation" : "(Nettle 2009)", "previouslyFormattedCitation" : "(Nettle 2009)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Nettle 2009). An awareness of the diversity of worldwide human behaviours would help us understand how human behaviours emerge from an interaction between local ecologies and our evolved brains (Henrich et al. 2010). For these sorts of reasons, classic studies that seek to test adaptive reasoning have taken pains to survey different populations ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1017/S0140525X00023992", "author" : [ { "dropping-particle" : "", "family" : "Buss", "given" : "David M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain sciences", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "1989" ] ] }, "page" : "1-49", "publisher" : "Cambridge Univ Press", "title" : "Sex differences in human mate preferences: Evolutionary hypotheses tested in 37 cultures", "type" : "article-journal", "volume" : "12" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1017/S0140525X00067595", "ISSN" : "1469-1825", "author" : [ { "dropping-particle" : "", "family" : "Kenrick", "given" : "Douglas T", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Keefe", "given" : "R C", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-2", "issue" : "01", "issued" : { "date-parts" : [ [ "1992" ] ] }, "page" : "75-91", "publisher" : "Cambridge Univ Press", "title" : "Age preferences in mates reflect sex differences in human reproductive strategies", "type" : "article-journal", "volume" : "15" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1037/0022-3514.95.1.212", "ISSN" : "1939-1315", "author" : [ { "dropping-particle" : "", "family" : "Schaller", "given" : "M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Murray", "given" : "D R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Personality and Social Psychology", "id" : "ITEM-3", "issue" : "1", "issued" : { "date-parts" : [ [ "2008" ] ] }, "page" : "212-221", "publisher" : "American Psychological Association", "title" : "Pathogens, personality, and culture: disease prevalence predicts worldwide variability in sociosexuality, extraversion, and openness to experience.", "type" : "article-journal", "volume" : "95" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "DOI" : "10.1017/S0140525X0500005", "ISSN" : "1469-1825", "author" : [ { "dropping-particle" : "", "family" : "Schmitt", "given" : "David P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-4", "issue" : "2", "issued" : { "date-parts" : [ [ "2005" ] ] }, "page" : "247-275", "publisher" : "Cambridge Univ Press", "title" : "Sociosexuality from Argentina to Zimbabwe: A 48-nation study of sex, culture, and strategies of human mating", "type" : "article-journal", "volume" : "28" }, "uris" : [ "" ] }, { "id" : "ITEM-5", "itemData" : { "DOI" : "10.1073/pnas.1409643111", "ISSN" : "0027-8424", "abstract" : "A large literature proposes that preferences for exaggerated sex typicality in human faces (masculinity/femininity) reflect a long evolutionary history of sexual and social selection. This proposal implies that dimorphism was important to judgments of attractiveness and personality in ancestral environments. It is difficult to evaluate, however, because most available data come from large-scale, industrialized, urban populations. Here, we report the results for 12 populations with very diverse levels of economic development. Surprisingly, preferences for exaggerated sex-specific traits are only found in the novel, highly developed environments. Similarly, perceptions that masculine males look aggressive increase strongly with development and, specifically, urbanization. These data challenge the hypothesis that facial dimorphism was an important ancestral signal of heritable mate value. One possibility is that highly developed environments provide novel opportunities to discern relationships between facial traits and behavior by exposing individuals to large numbers of unfamiliar faces, revealing patterns too subtle to detect with smaller samples.", "author" : [ { "dropping-particle" : "", "family" : "Scott", "given" : "Isabel M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Clark", "given" : "Andrew P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Josephson", "given" : "Steven C", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Boyette", "given" : "Adam H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Cuthill", "given" : "Innes C", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Fried", "given" : "Ruby L", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gibson", "given" : "Mhairi A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Hewlett", "given" : "Barry S", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Jamieson", "given" : "Mark", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Jankowiak", "given" : "William", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Honey", "given" : "P Lynne", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Huang", "given" : "Zejun", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Liebert", "given" : "Melissa A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Purzycki", "given" : "Benjamin G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Shaver", "given" : "John H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Snodgrass", "given" : "J Josh", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sosis", "given" : "Richard", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sugiyama", "given" : "Lawrence S", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Swami", "given" : "Viren", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Yu", "given" : "Douglas W", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Zhao", "given" : "Yangke", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Penton-Voak", "given" : "Ian S", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Proceedings of the National Academy of Sciences", "id" : "ITEM-5", "issue" : "40", "issued" : { "date-parts" : [ [ "2014", "10", "7" ] ] }, "page" : "14388-14393", "publisher" : "NATL ACAD SCIENCES", "publisher-place" : "2101 CONSTITUTION AVE NW, WASHINGTON, DC 20418 USA", "title" : "Human preferences for sexually dimorphic faces may be evolutionarily novel", "type" : "article-journal", "volume" : "111" }, "uris" : [ "" ] }, { "id" : "ITEM-6", "itemData" : { "ISBN" : "020201178X", "author" : [ { "dropping-particle" : "", "family" : "Daly", "given" : "Martin", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Wilson", "given" : "Margo", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-6", "issued" : { "date-parts" : [ [ "1988" ] ] }, "publisher" : "Transaction Books", "publisher-place" : "New Brunswick, NJ", "title" : "Homicide", "type" : "book" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Buss 1989; Daly and Wilson 1988; Kenrick and Keefe 1992; Schaller and Murray 2008; Schmitt 2005; Scott et al. 2014)", "manualFormatting" : "(e.g., Buss 1989; Daly & Wilson 1988; Kenrick & Keefe 1992; Schaller & Murray 2008; Schmitt 2005; Scott et al. 2014)", "plainTextFormattedCitation" : "(Buss 1989; Daly and Wilson 1988; Kenrick and Keefe 1992; Schaller and Murray 2008; Schmitt 2005; Scott et al. 2014)", "previouslyFormattedCitation" : "(Buss 1989; Daly and Wilson 1988; Kenrick and Keefe 1992; Schaller and Murray 2008; Schmitt 2005; Scott et al. 2014)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Buss 1989; Daly & Wilson 1988; Kenrick & Keefe 1992; Schaller & Murray 2008; Schmitt 2005; Scott et al. 2014). Reliance on WEIRD populations limits discovery of any patterns that might allow us to predict domains where psychological phenomena are more likely to be universal, and domains where psychological phenomena are more likely to show variability (Henrich et al. 2010). As an additional step, WEIRD authors (including ourselves) could usefully reach out to non-WEIRD collaborators to attempt to draw from wider samples. Encouraging greater diversity amongst authors should automatically increase participant diversity ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1038/s41562-017-0088", "author" : [ { "dropping-particle" : "", "family" : "Medin", "given" : "Douglas", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ojalehto", "given" : "Bethany", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Marin", "given" : "Ananda", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Bang", "given" : "Megan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Nature Human Behaviour", "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2017", "4", "18" ] ] }, "page" : "88", "publisher" : "Macmillan Publishers Limited", "title" : "Systems of (non-)diversity", "type" : "article-journal", "volume" : "1" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Medin et al. 2017)", "plainTextFormattedCitation" : "(Medin et al. 2017)", "previouslyFormattedCitation" : "(Medin et al. 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Medin et al. 2017). Henrich et al. ADDIN EN.CITE <EndNote><Cite ExcludeAuth="1"><Author>Henrich</Author><Year>2010</Year><RecNum>1707</RecNum><DisplayText>(2010)</DisplayText><record><rec-number>1707</rec-number><foreign-keys><key app="EN" db-id="t9299aaeg9fdpaesrx5xss0ptrtz2pxfzatd" timestamp="0">1707</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Henrich, Joseph</author><author>Heine, Steven J.</author><author>Norenzayan, Ara</author></authors></contributors><titles><title>The weirdest people in the world</title><secondary-title>Behavioral and Brain Sciences</secondary-title></titles><periodical><full-title>Behavioral and Brain Sciences</full-title><abbr-1>Behav. Brain Sci.</abbr-1><abbr-2>Behav Brain Sci</abbr-2><abbr-3>Behavioral & Brain Sciences</abbr-3></periodical><pages>61-83</pages><volume>33</volume><number>2-3</number><dates><year>2010</year></dates><urls></urls></record></Cite></EndNote>(2010)’s renowned position piece explains that participants from WEIRD (western, educated, industrialised, rich, democratic) populations can be more or less universally representative, dependent upon the area of research, and goes on to detail where reliance on WEIRD populations might not present a complete picture. To summarise Henrich et al.’s findings in as far as they are of particular concern to evolutionary psychologists, behavioural economics games used to assess fairness and co-operation showed that western undergraduate samples behaved very differently from participants from other societies. Similarly, folkbiological reasoning develops differently in rural American children compared to children from other settings. Further, research on moral reasoning has also now shown significant differences between the original data collected from western cultures and data collected later among more diverse cultures. In each case, theories were initially developed that assumed that the results from cultures more familiar to the researchers were universal. On the other hand, some research topics that will be familiar to readers of this journal seem to be those for which we have good evidence of universality, and where a reliance on less diverse samples is less problematic. Such topics can include emotional expression and pride displays, false belief tasks, some mate preferences, personality structure, psychological essentialism, punishment of free-riders, and social relationships ADDIN EN.CITE <EndNote><Cite><Author>Henrich</Author><Year>2010</Year><RecNum>1707</RecNum><DisplayText>(Henrich et al., 2010)</DisplayText><record><rec-number>1707</rec-number><foreign-keys><key app="EN" db-id="t9299aaeg9fdpaesrx5xss0ptrtz2pxfzatd" timestamp="0">1707</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Henrich, Joseph</author><author>Heine, Steven J.</author><author>Norenzayan, Ara</author></authors></contributors><titles><title>The weirdest people in the world</title><secondary-title>Behavioral and Brain Sciences</secondary-title></titles><periodical><full-title>Behavioral and Brain Sciences</full-title><abbr-1>Behav. Brain Sci.</abbr-1><abbr-2>Behav Brain Sci</abbr-2><abbr-3>Behavioral & Brain Sciences</abbr-3></periodical><pages>61-83</pages><volume>33</volume><number>2-3</number><dates><year>2010</year></dates><urls></urls></record></Cite></EndNote>(Henrich et al., 2010).Implications of relying on student samplesWe found that 44% of the samples that we coded were student samples (137 out of 311 samples). The advantages of using student samples are similar to the advantages of using WEIRD samples, together with the additional advantages that student participants should be comfortable within the university setting, and accustomed to following task instructions ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "ISSN" : "0018-7267", "author" : [ { "dropping-particle" : "", "family" : "Rosenthal", "given" : "Robert", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Human relations", "id" : "ITEM-1", "issue" : "4", "issued" : { "date-parts" : [ [ "1965" ] ] }, "page" : "389-406", "publisher" : "Sage Publications Sage CA: Thousand Oaks, CA", "title" : "The volunteer subject", "type" : "article-journal", "volume" : "18" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Rosenthal 1965)", "plainTextFormattedCitation" : "(Rosenthal 1965)", "previouslyFormattedCitation" : "(Rosenthal 1965)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Rosenthal 1965). On the other hand, a reliance on student samples may be particularly problematic when dealing with topics where there is a clear impact of the variables that distinguish students from the general population. These may include broad variables such as age, experience, socio-economic background, and educational level, as well as more specific tendencies including students’ greater level of cognitive ability and obedience to authority, more transient friendships, and still nascent attitudes and sense of self (Sears, 1986). Areas that have received specific criticism due to their reliance on student sampling include research on economic decision making ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1257/jep.21.2.153", "ISSN" : "0895-3309", "author" : [ { "dropping-particle" : "", "family" : "Levitt", "given" : "Steven D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "List", "given" : "John A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Economic Perspectives", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "2007", "4" ] ] }, "page" : "153-174", "publisher" : "JSTOR", "title" : "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?", "type" : "article-journal", "volume" : "21" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Levitt and List 2007)", "plainTextFormattedCitation" : "(Levitt and List 2007)", "previouslyFormattedCitation" : "(Levitt and List 2007)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Levitt and List 2007), socio-political attitudes ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/h0027880", "ISSN" : "1939-1455", "author" : [ { "dropping-particle" : "", "family" : "Schultz", "given" : "Duane P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Bulletin", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "1969" ] ] }, "page" : "214-228", "publisher" : "American Psychological Association", "title" : "The human subject in psychological research.", "type" : "article-journal", "volume" : "72" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Schultz 1969)", "plainTextFormattedCitation" : "(Schultz 1969)", "previouslyFormattedCitation" : "(Schultz 1969)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Schultz 1969), the psychological processes relating to prejudice ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1080/10478400802049936", "ISSN" : "1047-840X", "author" : [ { "dropping-particle" : "", "family" : "Henry", "given" : "Peter J.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Inquiry", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "2008", "5", "22" ] ] }, "page" : "49-71", "publisher" : "American Psychological Association", "title" : "College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice", "type" : "article-journal", "volume" : "19" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Henry 2008)", "plainTextFormattedCitation" : "(Henry 2008)", "previouslyFormattedCitation" : "(Henry 2008)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Henry 2008), and industrial and organisational psychology ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1017/iop.2015.70", "ISSN" : "1754-9426", "author" : [ { "dropping-particle" : "", "family" : "Bergman", "given" : "Mindy E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Jean", "given" : "Vanessa A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Industrial and Organizational Psychology", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "2016" ] ] }, "page" : "84-113", "publisher" : "Cambridge University Press", "title" : "Where have all the \u201cworkers\u201d gone? A critical analysis of the unrepresentativeness of our samples relative to the labor market in the industrial\u2013organizational psychology literature", "type" : "article-journal", "volume" : "9" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Bergman and Jean 2016)", "plainTextFormattedCitation" : "(Bergman and Jean 2016)", "previouslyFormattedCitation" : "(Bergman and Jean 2016)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Bergman and Jean 2016). Effect sizes calculated from student data can differ from other populations not merely in magnitude, but also in direction ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1086/323732", "ISSN" : "0093-5301", "author" : [ { "dropping-particle" : "", "family" : "Peterson", "given" : "Robert A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Consumer Research", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "2001", "12" ] ] }, "page" : "450-461", "publisher" : "JSTOR", "title" : "On the Use of College Students in Social Science Research: Insights from a Second-Order Meta-analysis", "type" : "article-journal", "volume" : "28" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Peterson 2001)", "plainTextFormattedCitation" : "(Peterson 2001)", "previouslyFormattedCitation" : "(Peterson 2001)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Peterson 2001). Further, if researchers are interested in features of a variable (e.g. its range, distribution, mean), then it will not be possible to assess that accurately from a sample that is partially selected in relation to that variable: thus, population-level IQ scores cannot be assessed from student samples.Implications of relying on online samplesOur survey pointed to a substantial reliance (around one quarter of all samples) on online data collection. It has been suggested that the internet offers a practical solution to reliance upon WEIRD samples ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/0003-066X.59.2.93", "ISSN" : "1935-990X", "author" : [ { "dropping-particle" : "", "family" : "Gosling", "given" : "Samuel D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Vazire", "given" : "Simine", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Srivastava", "given" : "Sanjay", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "John", "given" : "Oliver P", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Psychologist", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "2004" ] ] }, "page" : "93-104", "publisher" : "American Psychological Association", "title" : "Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires.", "type" : "article-journal", "volume" : "59" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Gosling et al. 2004)", "plainTextFormattedCitation" : "(Gosling et al. 2004)", "previouslyFormattedCitation" : "(Gosling et al. 2004)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Gosling et al. 2004). Advantages of online sampling include cheap, quick and convenient access to participants who can often be recruited in larger numbers than are readily available for offline studies, and this ease of access to large online samples appears to be reflected in our analyses above. Indeed, online sampling can reach a greater diversity of participants, including some difficult-to-reach and under-represented populations ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1207/S15327590IJHC1602_04", "ISSN" : "1044-7318", "author" : [ { "dropping-particle" : "", "family" : "Andrews", "given" : "Dorine", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Nonnecke", "given" : "Blair", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Preece", "given" : "Jennifer", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "International Journal of Human-Computer Interaction", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "2003", "10" ] ] }, "page" : "185-210", "publisher" : "Taylor & Francis", "title" : "Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users", "type" : "article-journal", "volume" : "16" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Andrews et al. 2003)", "plainTextFormattedCitation" : "(Andrews et al. 2003)", "previouslyFormattedCitation" : "(Andrews et al. 2003)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Andrews et al. 2003). Further, the anonymous setting of an online survey might arguably provoke more honest answers to questions on sensitive topics, compared to lab data collection ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.3758/BF03200723", "ISSN" : "0743-3808", "author" : [ { "dropping-particle" : "", "family" : "Joinson", "given" : "Adam", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavior Research Methods, Instruments, & Computers", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "1999", "9" ] ] }, "page" : "433-438", "publisher" : "Springer", "title" : "Social desirability, anonymity, and internet-based questionnaires", "type" : "article-journal", "volume" : "31" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Joinson 1999)", "plainTextFormattedCitation" : "(Joinson 1999)", "previouslyFormattedCitation" : "(Joinson 1999)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Joinson 1999). From another perspective though, internet sampling is limited in terms of the kinds of research tools that can be used, and in addition internet access itself is only available to a proportion of the population (and in some countries, a smaller proportion of the population than those who constitute undergraduate samples in other countries), meaning that the sample is still restricted (Gosling et al 2010). Researchers sometimes raise concerns that data collected via online sampling might be of lower quality than that collected using more traditional methods ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1016/j.socnet.2009.08.002", "ISSN" : "0378-8733", "author" : [ { "dropping-particle" : "", "family" : "Matzat", "given" : "Uwe", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Snijders", "given" : "Chris", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Social Networks", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "2010", "5" ] ] }, "page" : "105-111", "publisher" : "Elsevier", "title" : "Does the online collection of ego-centered network data reduce data quality? An experimental comparison", "type" : "article-journal", "volume" : "32" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1177/0963721414531598", "ISSN" : "0963-7214", "author" : [ { "dropping-particle" : "", "family" : "Paolacci", "given" : "Gabriele", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Chandler", "given" : "Jesse", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Current Directions in Psychological Science", "id" : "ITEM-2", "issue" : "3", "issued" : { "date-parts" : [ [ "2014" ] ] }, "page" : "184-188", "publisher" : "Sage Publications", "title" : "Inside the turk understanding mechanical turk as a participant pool", "type" : "article-journal", "volume" : "23" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Matzat and Snijders 2010; Paolacci and Chandler 2014)", "manualFormatting" : "(e.g., Matzat & Snijders, 2010; Paolacci & Chandler, 2014)", "plainTextFormattedCitation" : "(Matzat and Snijders 2010; Paolacci and Chandler 2014)", "previouslyFormattedCitation" : "(Matzat and Snijders 2010; Paolacci and Chandler 2014)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Matzat & Snijders, 2010; Paolacci & Chandler, 2014). Online participants do not have easy access to the researcher to raise queries, might enter data carelessly or thoughtlessly, or might have chosen to enter data merely in order to view the study content ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.3758/s13428-012-0265-2", "ISSN" : "1554-3528", "author" : [ { "dropping-particle" : "", "family" : "Aust", "given" : "Frederik", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Diedenhofen", "given" : "Birk", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ullrich", "given" : "Sebastian", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Musch", "given" : "Jochen", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavior Research Methods", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "2013", "6", "10" ] ] }, "page" : "527-535", "publisher" : "Springer", "title" : "Seriousness checks are useful to improve data validity in online research", "type" : "article-journal", "volume" : "45" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Aust et al. 2013)", "plainTextFormattedCitation" : "(Aust et al. 2013)", "previouslyFormattedCitation" : "(Aust et al. 2013)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Aust et al. 2013). Accordingly, to test the quality of online data collection, various studies have compared data collected online and offline, and concluded that in many instances the two sampling methodologies give rise to very similar outcomes ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1016/B978-012099980-4/50003-4", "author" : [ { "dropping-particle" : "", "family" : "Krantz", "given" : "John H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Dalal", "given" : "Reeshad", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Experiments on the Internet", "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2000" ] ] }, "page" : "35-60", "publisher" : "Elsevier", "title" : "Validity of Web-Based Psychological Research", "type" : "chapter" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Krantz and Dalal 2000)", "plainTextFormattedCitation" : "(Krantz and Dalal 2000)", "previouslyFormattedCitation" : "(Krantz and Dalal 2000)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Krantz and Dalal 2000). For instance, judgements of the attractiveness of different female body shapes were similar, irrespective of whether data were collected from laboratory studies of psychology undergraduate students, or online from visitors to psychology webpages hosted by the same university ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.3758/BF03204824", "ISSN" : "1532-5970", "abstract" : "The Internet provides a new method to obtain subjects. Arguments supporting the use of the Internet as an experimental medium include the ease of data collection, limited resources needed, and ability to gain a sample more representative of the population. However, it has not been demonstrated that the Internet provides the conditions necessary to yield valid data. This paper compares results from experiments conducted over the Internet and in a laboratory. Correlating the means from the Internet samples to the laboratory samples on important interactions yielded values near 1.0, indicating that the two data sets are driven by the same psychological variables. Further evidence of the validity of these Web studies comes from regression analyses. However, more studies of this sort are needed to further identify what types of experiments may be validly conducted over the Internet.", "author" : [ { "dropping-particle" : "", "family" : "Krantz", "given" : "John H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ballard", "given" : "Jody", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Scher", "given" : "Jody", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavior Research Methods, Instruments, & Computers", "id" : "ITEM-1", "issue" : "2", "issued" : { "date-parts" : [ [ "1997" ] ] }, "page" : "264-269", "title" : "Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness", "type" : "article-journal", "volume" : "29" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Krantz et al. 1997)", "plainTextFormattedCitation" : "(Krantz et al. 1997)", "previouslyFormattedCitation" : "(Krantz et al. 1997)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Krantz et al. 1997). Nevertheless, data collected online and offline are not identical ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1146/annurev.psych.55.090902.141601", "ISSN" : "0066-4308", "author" : [ { "dropping-particle" : "", "family" : "Birnbaum", "given" : "Michael H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Annual Review of Psychology", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "2004", "2" ] ] }, "page" : "803-832", "title" : "Human Research and Data Collection via the Internet", "type" : "article-journal", "volume" : "55" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1016/S0747-5632(01)00002-4", "ISSN" : "07475632", "author" : [ { "dropping-particle" : "", "family" : "Epstein", "given" : "Joel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Klinkenberg", "given" : "W.D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Wiley", "given" : "D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "McKinley", "given" : "L", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Computers in Human Behavior", "id" : "ITEM-2", "issue" : "3", "issued" : { "date-parts" : [ [ "2001", "5" ] ] }, "page" : "339-346", "publisher" : "Elsevier", "title" : "Insuring sample equivalence across internet and paper-and-pencil assessments", "type" : "article-journal", "volume" : "17" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Birnbaum 2004; Epstein et al. 2001)", "plainTextFormattedCitation" : "(Birnbaum 2004; Epstein et al. 2001)", "previouslyFormattedCitation" : "(Birnbaum 2004; Epstein et al. 2001)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Birnbaum 2004; Epstein et al. 2001). This is unsurprising, given that context and environment can influence behaviour. The demographic differences between people with and without internet access may be particularly stark in developing countries, and so online sampling may not be the most suitable way to reach diverse populations in those countries ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1371/journal.pone.0100966", "ISSN" : "1932-6203", "author" : [ { "dropping-particle" : "", "family" : "Batres", "given" : "Carlota", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Perrett", "given" : "David I", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "PLoS ONE", "editor" : [ { "dropping-particle" : "", "family" : "McCormick", "given" : "Cheryl", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issue" : "7", "issued" : { "date-parts" : [ [ "2014", "7", "9" ] ] }, "page" : "e100966", "publisher" : "Public Library of Science", "title" : "The Influence of the Digital Divide on Face Preferences in El Salvador: People without Internet Access Prefer More Feminine Men, More Masculine Women, and Women with Higher Adiposity", "type" : "article-journal", "volume" : "9" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Batres and Perrett 2014)", "plainTextFormattedCitation" : "(Batres and Perrett 2014)", "previouslyFormattedCitation" : "(Batres and Perrett 2014)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Batres and Perrett 2014), and in some instances of course internet access can contribute to behaviour that we might want to assess; for example, media exposure appears to explain differences in preferences for faces and body types ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1111/bjop.12184", "ISSN" : "00071269", "author" : [ { "dropping-particle" : "", "family" : "Boothroyd", "given" : "Lynda G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Jucker", "given" : "Jean\u2010Luc", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Thornborrow", "given" : "Tracey", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Jamieson", "given" : "Mark A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Burt", "given" : "D Michael", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Barton", "given" : "Robert A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Evans", "given" : "Elizabeth H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Tovee", "given" : "Martin J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "British Journal of Psychology", "id" : "ITEM-1", "issue" : "4", "issued" : { "date-parts" : [ [ "2016", "11" ] ] }, "page" : "752-767", "publisher" : "Wiley Online Library", "title" : "Television exposure predicts body size ideals in rural Nicaragua", "type" : "article-journal", "volume" : "107" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Boothroyd et al. 2016)", "plainTextFormattedCitation" : "(Boothroyd et al. 2016)", "previouslyFormattedCitation" : "(Boothroyd et al. 2016)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Boothroyd et al. 2016). However, despite the differences between online and offline samples, one should not be seen as the poor cousin of the other: both have strengths, and can be used in complementary fashion to rigorously test inferences.Implications of relying on restricted samplesFrom a statistical point of view, restricted sampling can lead to selection bias which in turn can lead to confounding ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1177/2515245917745629", "ISSN" : "2515-2459", "author" : [ { "dropping-particle" : "", "family" : "Rohrer", "given" : "Julia M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Advances in Methods and Practices in Psychological Science", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "2018" ] ] }, "page" : "27-42", "publisher" : "SAGE Publications Sage CA: Los Angeles, CA", "title" : "Thinking clearly about correlations and causation: Graphical causal models for observational data", "type" : "article-journal", "volume" : "1" }, "uris" : [ "", "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1146/annurev-soc-071913-043455", "ISSN" : "0360-0572", "author" : [ { "dropping-particle" : "", "family" : "Elwert", "given" : "Felix", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Winship", "given" : "Christopher", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Annual Review of Sociology", "id" : "ITEM-2", "issued" : { "date-parts" : [ [ "2014" ] ] }, "page" : "31-53", "publisher" : "Annual Reviews", "title" : "Endogenous selection bias: The problem of conditioning on a collider variable", "type" : "article-journal", "volume" : "40" }, "uris" : [ "", "" ] }, { "id" : "ITEM-3", "itemData" : { "author" : [ { "dropping-particle" : "", "family" : "Bareinboim", "given" : "Elias", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pearl", "given" : "Judea", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics", "id" : "ITEM-3", "issued" : { "date-parts" : [ [ "2012" ] ] }, "page" : "100-108", "title" : "Controlling selection bias in causal inference", "type" : "paper-conference" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "DOI" : "10.1037/0033-295X.107.4.659", "ISSN" : "1939-1471", "author" : [ { "dropping-particle" : "", "family" : "Fiedler", "given" : "Klaus", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Review", "id" : "ITEM-4", "issue" : "4", "issued" : { "date-parts" : [ [ "2000" ] ] }, "page" : "659-676", "publisher" : "American Psychological Association", "title" : "Beware of samples! A cognitive-ecological sampling approach to judgment biases.", "type" : "article-journal", "volume" : "107" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Bareinboim and Pearl 2012; Elwert and Winship 2014; Fiedler 2000; Rohrer 2018)", "plainTextFormattedCitation" : "(Bareinboim and Pearl 2012; Elwert and Winship 2014; Fiedler 2000; Rohrer 2018)", "previouslyFormattedCitation" : "(Bareinboim and Pearl 2012; Elwert and Winship 2014; Fiedler 2000; Rohrer 2018)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Bareinboim and Pearl 2012; Elwert and Winship 2014; Fiedler 2000; Rohrer 2018). Recently, statisticians have more explicitly defined the conditions under which causal inferences can be made when combining data under heterogeneous conditions ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1073/pnas.1510507113", "abstract" : "We review concepts, principles, and tools that unify current approaches to causal analysis and attend to new challenges presented by big data. In particular, we address the problem of data fusion\u2014piecing together multiple datasets collected under heterogeneous conditions (i.e., different populations, regimes, and sampling methods) to obtain valid answers to queries of interest. The availability of multiple heterogeneous datasets presents new opportunities to big data analysts, because the knowledge that can be acquired from combined data would not be possible from any individual source alone. However, the biases that emerge in heterogeneous environments require new analytical tools. Some of these biases, including confounding, sampling selection, and cross-population biases, have been addressed in isolation, largely in restricted parametric models. We here present a general, nonparametric framework for handling these biases and, ultimately, a theoretical solution to the problem of data fusion in causal inference tasks.", "author" : [ { "dropping-particle" : "", "family" : "Bareinboim", "given" : "Elias", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pearl", "given" : "Judea", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Proceedings of the National Academy of Sciences", "id" : "ITEM-1", "issue" : "27", "issued" : { "date-parts" : [ [ "2016", "7", "5" ] ] }, "page" : "7345 LP - 7352", "title" : "Causal inference and the data-fusion problem", "type" : "article-journal", "volume" : "113" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Bareinboim and Pearl 2016)", "plainTextFormattedCitation" : "(Bareinboim and Pearl 2016)", "previouslyFormattedCitation" : "(Bareinboim and Pearl 2016)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Bareinboim and Pearl 2016). Depending on the type of inference researchers want to make, they could face confounding, sampling bias, or transportability bias. Importantly, these issues apply to both the decision to focus on (for example) a WEIRD population as well as expanding the research to non-WEIRD populations. An exclusive focus on restricted samples comes at the cost of external validity. In medical research, there has been a repeated call to revalue external validity ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.2105/AJPH.2007.126847", "ISSN" : "0090-0036", "author" : [ { "dropping-particle" : "", "family" : "Steckler", "given" : "Allan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "McLeroy", "given" : "Kenneth R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Journal of Public Health", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "2008", "1", "20" ] ] }, "page" : "9-10", "publisher" : "American Public Health Association", "title" : "The Importance of External Validity", "type" : "article-journal", "volume" : "98" }, "uris" : [ "", "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1177/0163278705284445", "ISSN" : "0163-2787", "abstract" : "Starting with the proposition that ?if we want more evidence-based practice, we need more practice-based evidence,? this article (a) offers questions and guides that practitioners, program planners, and policy makers can use to determine the applicability of evidence to situations and populations other than those in which the evidence was produced (generalizability), (b) suggests criteria that reviewers can use to evaluate external validity and potential for generalization, and (c) recommends procedures that practitioners and program planners can use to adapt evidencebased interventions and integrate them with evidence on the population and setting characteristics, theory, and experience into locally appropriate programs. The development and application in tandem of such questions, guides, criteria, and procedures can be a step toward increasing the relevance of research for decision making and should support the creation and reporting of more practice-based research having high external validity.", "author" : [ { "dropping-particle" : "", "family" : "Green", "given" : "Lawrence W", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Glasgow", "given" : "Russell E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Evaluation & the Health Professions", "id" : "ITEM-2", "issue" : "1", "issued" : { "date-parts" : [ [ "2006", "3", "1" ] ] }, "note" : "doi: 10.1177/0163278705284445", "page" : "126-153", "publisher" : "SAGE Publications Inc", "title" : "Evaluating the Relevance, Generalization, and Applicability of Research: Issues in External Validation and Translation Methodology", "type" : "article-journal", "volume" : "29" }, "uris" : [ "", "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1258/jhsrp.2011.010124", "ISSN" : "1355-8196", "author" : [ { "dropping-particle" : "", "family" : "Burchett", "given" : "Helen", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Umoquit", "given" : "Muriah", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Dobrow", "given" : "Mark", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Health Services Research & Policy", "id" : "ITEM-3", "issue" : "4", "issued" : { "date-parts" : [ [ "2011", "10" ] ] }, "page" : "238-244", "publisher" : "SAGE Publications Sage UK: London, England", "title" : "How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability frameworks", "type" : "article-journal", "volume" : "16" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Burchett et al. 2011; Green and Glasgow 2006; Steckler and McLeroy 2008)", "manualFormatting" : "(e.g., Burchett et al. 2011; Green and Glasgow 2006; Steckler and McLeroy 2008)", "plainTextFormattedCitation" : "(Burchett et al. 2011; Green and Glasgow 2006; Steckler and McLeroy 2008)", "previouslyFormattedCitation" : "(Burchett et al. 2011; Green and Glasgow 2006; Steckler and McLeroy 2008)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Burchett et al. 2011; Green and Glasgow 2006; Steckler and McLeroy 2008). While there had previously been a strong focus on internal validity, for example, focussing on questions such as can confounding be effectively ruled out in randomised controlled trials?; there has been a call to not also forget the importance of external validity (can we generalize the findings from this trial?). An obvious issue with relying on WEIRD, student, and/or online samples would be the degree to which any conclusions would hold in different populations ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1017/S0140525X0999152X", "ISSN" : "1469-1825", "author" : [ { "dropping-particle" : "", "family" : "Henrich", "given" : "Joseph", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Heine", "given" : "S J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Norenzayan", "given" : "A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-1", "issue" : "2-3", "issued" : { "date-parts" : [ [ "2010" ] ] }, "page" : "61-83", "publisher" : "Cambridge Univ Press", "title" : "The weirdest people in the world", "type" : "article-journal", "volume" : "33" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1037/0003-066X.54.12.1070", "ISSN" : "1935-990X", "author" : [ { "dropping-particle" : "", "family" : "Sue", "given" : "Stanley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Psychologist", "id" : "ITEM-2", "issue" : "12", "issued" : { "date-parts" : [ [ "1999" ] ] }, "page" : "1070-1077", "publisher" : "American Psychological Association", "title" : "Science, ethnicity, and bias: Where have we gone wrong?", "type" : "article-journal", "volume" : "54" }, "uris" : [ "", "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1080/10478400802049936", "ISSN" : "1047-840X", "author" : [ { "dropping-particle" : "", "family" : "Henry", "given" : "Peter J.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Inquiry", "id" : "ITEM-3", "issue" : "2", "issued" : { "date-parts" : [ [ "2008", "5", "22" ] ] }, "page" : "49-71", "publisher" : "American Psychological Association", "title" : "College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice", "type" : "article-journal", "volume" : "19" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Henrich et al. 2010; Henry 2008; Sue 1999)", "plainTextFormattedCitation" : "(Henrich et al. 2010; Henry 2008; Sue 1999)", "previouslyFormattedCitation" : "(Henrich et al. 2010; Henry 2008; Sue 1999)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Henrich et al. 2010; Henry 2008; Sue 1999). Even more fundamentally, and before making causal inferences about other populations, researchers face the more basal problem of knowing whether they are measuring the same ‘thing’ in different populations. This issue is well-understood in the field of psychometrics, and has led to the development of measurement techniques and tests to examine the degree to which constructs are measured consistently across cultures ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1080/00207598908247842", "ISSN" : "0020-7594", "author" : [ { "dropping-particle" : "", "family" : "Poortinga", "given" : "Y H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "International Journal of Psychology", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "1989" ] ] }, "page" : "737-756", "publisher" : "Taylor & Francis", "title" : "Equivalence of cross-cultural data: An overview of basic issues", "type" : "article-journal", "volume" : "24" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "ISBN" : "0205160743", "author" : [ { "dropping-particle" : "", "family" : "Vijver", "given" : "F J R", "non-dropping-particle" : "Van de", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Leung", "given" : "K", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Handbook of Cross-Cultural Psychology", "edition" : "2", "editor" : [ { "dropping-particle" : "", "family" : "Berry", "given" : "John W", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Poortinga", "given" : "Ype H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pandey", "given" : "Janak", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-2", "issued" : { "date-parts" : [ [ "1997" ] ] }, "publisher" : "Allyn & Bacon", "publisher-place" : "Boston, MA", "title" : "Methods and data analysis of comparative research.", "type" : "chapter" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1037//0022-3514.82.6.903", "ISSN" : "1939-1315", "author" : [ { "dropping-particle" : "", "family" : "Heine", "given" : "S J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lehman", "given" : "D R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Peng", "given" : "K", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Greenholtz", "given" : "J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Personality and Social Psychology", "id" : "ITEM-3", "issue" : "6", "issued" : { "date-parts" : [ [ "2002" ] ] }, "page" : "903-918", "publisher" : "American Psychological Association", "title" : "What's wrong with cross-cultural comparisons of subjective Likert scales?: The reference-group effect.", "type" : "article-journal", "volume" : "82" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "ISSN" : "0938-8249", "author" : [ { "dropping-particle" : "", "family" : "Nasif", "given" : "E G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Al-Daeaj", "given" : "H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ebrahimi", "given" : "B", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Thibodeaux", "given" : "M S", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "MIR: Management International Review", "id" : "ITEM-4", "issue" : "1", "issued" : { "date-parts" : [ [ "1991" ] ] }, "page" : "79-91", "publisher" : "JSTOR", "title" : "Methodological problems in cross-cultural research: An updated review", "type" : "article-journal", "volume" : "31" }, "uris" : [ "" ] }, { "id" : "ITEM-5", "itemData" : { "DOI" : "10.1177/0022002185016002001", "ISSN" : "0022-0221", "author" : [ { "dropping-particle" : "", "family" : "Hui", "given" : "C H", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Triandis", "given" : "H C", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Cross-Cultural Psychology", "id" : "ITEM-5", "issue" : "2", "issued" : { "date-parts" : [ [ "1985" ] ] }, "page" : "131-152", "publisher" : "Sage Publications", "title" : "Measurement in cross-cultural psychology", "type" : "article-journal", "volume" : "16" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Heine et al. 2002; Hui and Triandis 1985; Nasif et al. 1991; Poortinga 1989; Van de Vijver and Leung 1997)", "plainTextFormattedCitation" : "(Heine et al. 2002; Hui and Triandis 1985; Nasif et al. 1991; Poortinga 1989; Van de Vijver and Leung 1997)", "previouslyFormattedCitation" : "(Heine et al. 2002; Hui and Triandis 1985; Nasif et al. 1991; Poortinga 1989; Van de Vijver and Leung 1997)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Heine et al. 2002; Hui and Triandis 1985; Nasif et al. 1991; Poortinga 1989; Van de Vijver and Leung 1997). We did not explicitly assess how many papers established equivalence of measurement between different samples, as we focused on the samples. Our standard psychological instruments, often developed by researchers working within WEIRD settings, may limit the generalisability of research findings ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1017/S0140525X10000063", "ISSN" : "1469-1825", "author" : [ { "dropping-particle" : "", "family" : "Ceci", "given" : "Stephen J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Kahan", "given" : "Dan M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Braman", "given" : "Donald", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-1", "issue" : "2-3", "issued" : { "date-parts" : [ [ "2010" ] ] }, "page" : "87-88", "publisher" : "Cambridge University Press", "title" : "The WEIRD are even weirder than you think: Diversifying contexts is as important as diversifying samples", "type" : "article-journal", "volume" : "33" }, "uris" : [ "", "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1017/S0140525X10000142", "ISSN" : "0140-525X", "abstract" : "There are many methodological considerations \u2013 some intricately associated with the use of WEIRD samples \u2013 that adversely affect external validity as much as, or even more than, unrepresentative sampling does. Among suspect applications, especially worrisome is the incorporation of WEIRD-based findings regarding moral reasoning and retribution into normative expectations, such as might be held by international criminal tribunals in \u201ccognitively distant\u201d war-torn areas.", "author" : [ { "dropping-particle" : "", "family" : "Kone\u010dni", "given" : "Vladimir J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "edition" : "2010/06/15", "id" : "ITEM-2", "issue" : "2-3", "issued" : { "date-parts" : [ [ "2010" ] ] }, "page" : "98-99", "publisher" : "Cambridge University Press", "title" : "Responsible behavioral science generalizations and applications require much more than non-WEIRD samples", "type" : "article-journal", "volume" : "33" }, "uris" : [ "", "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1017/S0140525X10000233", "ISSN" : "0140-525X", "abstract" : "Catchy acronyms such as \u201cWEIRD population\u201d are good mnemonics. However, they carry the danger of distracting us from deeper issues: how to sample populations, the validity of measuring instruments, the levels of processing involved. These need to be considered when assessing claims of universality regarding how the mind works \u201cin general\u201d \u2013 a dominant and highly rewarded drive in the behavioral and brain sciences.", "author" : [ { "dropping-particle" : "", "family" : "Rochat", "given" : "Philippe", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "edition" : "2010/06/15", "id" : "ITEM-3", "issue" : "2-3", "issued" : { "date-parts" : [ [ "2010" ] ] }, "page" : "107-108", "publisher" : "Cambridge University Press", "title" : "What is really wrong with a priori claims of universality? Sampling, validity, process level, and the irresistible drive to reduce", "type" : "article-journal", "volume" : "33" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Ceci et al. 2010; Kone\u010dni 2010; Rochat 2010)", "plainTextFormattedCitation" : "(Ceci et al. 2010; Kone\u010dni 2010; Rochat 2010)", "previouslyFormattedCitation" : "(Ceci et al. 2010; Kone\u010dni 2010; Rochat 2010)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Ceci et al. 2010; Kone?ni 2010; Rochat 2010). We therefore call for more research explicitly establishing that the same ‘thing’ is measured in different populations. Depending on the sampling scheme, broadening the research to non-WEIRD populations could also give rise to problems such as non-independence ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.2307/2010045", "author" : [ { "dropping-particle" : "", "family" : "Ross", "given" : "Marc Howard", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Homer", "given" : "Elizabeth", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "World Politics", "id" : "ITEM-1", "issue" : "1", "issued" : { "date-parts" : [ [ "1976" ] ] }, "page" : "1-28", "publisher" : "Cambridge Univ Press", "title" : "Galton\u2019s problem in cross-national research", "type" : "article-journal", "volume" : "29" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "ISSN" : "0037-783X", "author" : [ { "dropping-particle" : "", "family" : "Naroll", "given" : "Raoul", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Social Research", "id" : "ITEM-2", "issue" : "4", "issued" : { "date-parts" : [ [ "1965" ] ] }, "page" : "428-451", "publisher" : "JSTOR", "title" : "Galton's problem: The logic of cross-cultural analysis", "type" : "article-journal", "volume" : "32" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.1007/s12110-014-9206-3", "ISSN" : "1936-4776", "PMID" : "25092392", "abstract" : "Many recent evolutionary psychology and human behavioral ecology studies have tested hypotheses by examining correlations between variables measured at a group level (e.g., state, country, continent). In such analyses, variables collected for each aggregation are often taken to be representative of the individuals present within them, and relationships between such variables are presumed to reflect individual-level processes. There are multiple reasons to exercise caution when doing so, including: (1) the ecological fallacy, whereby relationships observed at the aggregate level do not accurately represent individual-level processes; (2) non-independence of data points, which violates assumptions of the inferential techniques used in null hypothesis testing; and (3) cross-cultural non-equivalence of measurement (differences in construct validity between groups). We provide examples of how each of these gives rise to problems in the context of testing evolutionary hypotheses about human behavior, and we offer some suggestions for future research.", "author" : [ { "dropping-particle" : "", "family" : "Pollet", "given" : "Thomas Victor", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Tybur", "given" : "Joshua M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Frankenhuis", "given" : "Willem E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Rickard", "given" : "Ian J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Human nature (Hawthorne, N.Y.)", "id" : "ITEM-3", "issue" : "3", "issued" : { "date-parts" : [ [ "2014", "9", "5" ] ] }, "page" : "410-29", "title" : "What can cross-cultural correlations teach us about human nature?", "type" : "article-journal", "volume" : "25" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "DOI" : "10.1086/204317", "ISSN" : "0011-3204", "author" : [ { "dropping-particle" : "", "family" : "Mace", "given" : "Ruth", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pagel", "given" : "Mark", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Current Anthropology", "id" : "ITEM-4", "issue" : "5", "issued" : { "date-parts" : [ [ "1994", "12" ] ] }, "page" : "549-564", "title" : "The Comparative Method in Anthropology", "type" : "article-journal", "volume" : "35" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Mace and Pagel 1994; Naroll 1965; Pollet et al. 2014; Ross and Homer 1976)", "plainTextFormattedCitation" : "(Mace and Pagel 1994; Naroll 1965; Pollet et al. 2014; Ross and Homer 1976)", "previouslyFormattedCitation" : "(Mace and Pagel 1994; Naroll 1965; Pollet et al. 2014; Ross and Homer 1976)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Mace and Pagel 1994; Naroll 1965; Pollet et al. 2014; Ross and Homer 1976), which then would need to be addressed. We do not discuss these issues in further detail here, as the degree to which they matter could differ on design (experiment/correlational), covariates, and research question. For example, for many psychophysical studies, and also evolutionary psychological studies ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/ebs0000017", "ISSN" : "2330-2933", "author" : [ { "dropping-particle" : "", "family" : "Tybur", "given" : "Joshua M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Frankenhuis", "given" : "Willem E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pollet", "given" : "Thomas Victor", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Evolutionary Behavioral Sciences", "id" : "ITEM-1", "issue" : "4", "issued" : { "date-parts" : [ [ "2014" ] ] }, "page" : "274-283", "publisher" : "Educational Publishing Foundation", "title" : "Behavioral immune system methods: Surveying the present to shape the future.", "type" : "article-journal", "volume" : "8" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Tybur et al. 2014)", "plainTextFormattedCitation" : "(Tybur et al. 2014)", "previouslyFormattedCitation" : "(Tybur et al. 2014)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Tybur et al. 2014), the focus is on within-individual differences. The implicit assumption is that these would not vary depending on the population studied. For such studies, it would be useful for authors to be more explicit to which degree these within-individual differences are expected to generalise to other samples. In some cases, restricted sampling in itself is useful to determine whether a behaviour exists or not, and as such testing a WEIRD, student, and/or online population could constitute a necessary first step ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1111/j.1468-5914.1982.tb00449.x", "ISSN" : "0021-8308", "author" : [ { "dropping-particle" : "", "family" : "Greenwood", "given" : "John D", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal for the Theory of Social Behaviour", "id" : "ITEM-1", "issue" : "3", "issued" : { "date-parts" : [ [ "1982", "10" ] ] }, "page" : "225-250", "publisher" : "Wiley Online Library", "title" : "On the Relation Between Laboratory Experiments and Social Behaviour: Causal Explanation and Generalization", "type" : "article-journal", "volume" : "12" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1037/0003-066X.38.4.379", "ISSN" : "0003-066X", "author" : [ { "dropping-particle" : "", "family" : "Mook", "given" : "Douglas G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "American Psychologist", "id" : "ITEM-2", "issue" : "4", "issued" : { "date-parts" : [ [ "1983" ] ] }, "page" : "379-387", "publisher" : "American Psychological Association", "title" : "In defense of external invalidity.", "type" : "article-journal", "volume" : "38" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Greenwood 1982; Mook 1983)", "plainTextFormattedCitation" : "(Greenwood 1982; Mook 1983)", "previouslyFormattedCitation" : "(Greenwood 1982; Mook 1983)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Greenwood 1982; Mook 1983). Given that every sample is restricted in some way, authors can usefully make a statement pertaining to the constraints on generality, to explain the boundaries of the population that they believe their results to apply to ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1177/1745691617708630", "ISSN" : "1745-6916", "abstract" : "Psychological scientists draw inferences about populations based on samples?of people, situations, and stimuli?from those populations. Yet, few papers identify their target populations, and even fewer justify how or why the tested samples are representative of broader populations. A cumulative science depends on accurately characterizing the generality of findings, but current publishing standards do not require authors to constrain their inferences, leaving readers to assume the broadest possible generalizations. We propose that the discussion section of all primary research articles specify Constraints on Generality (i.e., a ?COG? statement) that identify and justify target populations for the reported findings. Explicitly defining the target populations will help other researchers to sample from the same populations when conducting a direct replication, and it could encourage follow-up studies that test the boundary conditions of the original finding. Universal adoption of COG statements would change publishing incentives to favor a more cumulative science.", "author" : [ { "dropping-particle" : "", "family" : "Simons", "given" : "Daniel J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Shoda", "given" : "Yuichi", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lindsay", "given" : "D Stephen", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Perspectives on Psychological Science", "id" : "ITEM-1", "issue" : "6", "issued" : { "date-parts" : [ [ "2017", "8", "30" ] ] }, "note" : "doi: 10.1177/1745691617708630", "page" : "1123-1128", "publisher" : "SAGE Publications Inc", "title" : "Constraints on Generality (COG): A Proposed Addition to All Empirical Papers", "type" : "article-journal", "volume" : "12" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Simons et al. 2017)", "plainTextFormattedCitation" : "(Simons et al. 2017)", "previouslyFormattedCitation" : "(Simons et al. 2017)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Simons et al. 2017). More broadly, the field would benefit from setting out the conditions under which causal inferences can be made ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "ISBN" : "052189560X", "author" : [ { "dropping-particle" : "", "family" : "Pearl", "given" : "Judea", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2009" ] ] }, "publisher" : "Cambridge university press", "publisher-place" : "Cambridge, UK", "title" : "Causality", "type" : "book" }, "uris" : [ "", "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1214/09-SS057", "ISSN" : "1935-7516", "author" : [ { "dropping-particle" : "", "family" : "Pearl", "given" : "Judea", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Statistics surveys", "id" : "ITEM-2", "issued" : { "date-parts" : [ [ "2009" ] ] }, "page" : "96-146", "publisher" : "The author, under a Creative Commons Attribution License", "title" : "Causal inference in statistics: An overview", "type" : "article-journal", "volume" : "3" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Pearl 2009a, 2009b)", "plainTextFormattedCitation" : "(Pearl 2009a, 2009b)", "previouslyFormattedCitation" : "(Pearl 2009a, 2009b)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Pearl 2009a, 2009b). Limitations and future directionsOur analysis did not cover all of the journals that publish evolutionary psychological studies. Instead, mirroring the work that has been done in other reviews of sample diversity (e.g., Arnett 2008; Gallander Wintre et al. 2001; Sears 1986), we focussed on key journals. Many papers on evolutionary psychology are published outside of those two journals. Similarly, there might be papers in our sample which have a different focus than evolutionary psychology, or which might be better classified as relating to fields such as comparative cognition, behavioural economics, linguistics, demography, or anthropology among others. Future work might compare one journal to the next, and compare across a sequence of different years, to determine the variance in sample diversity. Alternatively, one could define keywords to more clearly delineate articles covering evolutionary psychology. Further, future research might seek to uncover whether different research areas within evolutionary psychology are more or less reliant upon non-diverse samples, and how this corresponds to their development as a research area. Exploratory studies might well choose to focus on easily-accessible samples such as undergraduates to test their initial ideas, whereas more mature research areas ought to seek to diversify their samples further in order to test the generalisability of their findings. Even within specific research areas, some topics have been studied in more diverse worldwide samples than others; for instance, sex differences in partner preferences draws from data from many cultures (e.g., Buss 1989, Shackelford et al. 2005), whereas research on ovulatory shifts in partner preferences very much rests upon studies carried out in English-speaking WEIRD countries ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1037/a0035438", "ISSN" : "1939-1455", "author" : [ { "dropping-particle" : "", "family" : "Gildersleeve", "given" : "Kelly", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Haselton", "given" : "Martie G", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Fales", "given" : "Melissa R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Psychological Bulletin", "id" : "ITEM-1", "issue" : "5", "issued" : { "date-parts" : [ [ "2014" ] ] }, "page" : "1205-1259", "publisher" : "American Psychological Association", "title" : "Do women\u2019s mate preferences change across the ovulatory cycle? A meta-analytic review.", "type" : "article-journal", "volume" : "140" }, "uris" : [ "", "" ] } ], "mendeley" : { "formattedCitation" : "(Gildersleeve et al. 2014)", "plainTextFormattedCitation" : "(Gildersleeve et al. 2014)", "previouslyFormattedCitation" : "(Gildersleeve et al. 2014)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Gildersleeve et al. 2014).Our survey does not present cause for despair. In terms of participant diversity, evolutionary psychology does rely on WEIRD student samples less heavily than some fields ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "URL" : "", "accessed" : { "date-parts" : [ [ "2016", "11", "14" ] ] }, "author" : [ { "dropping-particle" : "", "family" : "Kurzban", "given" : "Robert", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "EP Journal blog", "id" : "ITEM-1", "issued" : { "date-parts" : [ [ "2013" ] ] }, "title" : "Is Evolutionary Psychology WEIRD or NORMAL?", "type" : "webpage" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1016/j.copsyc.2015.08.015", "ISSN" : "2352-250X", "abstract" : "Perhaps no field of psychology is more strongly motivated and better equipped than evolutionary psychology to respond to the recent call for psychologists to expand their empirical base beyond WEIRD (Western Educated Industrialized Rich Democratic) samples. Evolutionary psychologists have historically focused their efforts on identifying species-specific psychological traits, for which evidence often hinged on the extent to which traits were generalizable across human groups. Now, a new generation of researchers is embracing cultural and environmental variation to test evolutionary hypotheses. Here we discuss how comparative research with diverse societies, while challenging, can help inform the complex nature of our species\u2019 psychology and in doing so, we outline best theoretical and methodological practices as well as common pitfalls in cross-cultural investigations. We end with a recommendation for the use of publicly available databases for cataloging psychological variation across the world's many diverse populations. Because of rapid culture change and globalization, it is more important now than ever to document what we know about the world's cultures in ways that can be used by future researchers.", "author" : [ { "dropping-particle" : "", "family" : "Apicella", "given" : "Coren L", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Barrett", "given" : "H Clark", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Current Opinion in Psychology", "id" : "ITEM-2", "issued" : { "date-parts" : [ [ "2016", "2" ] ] }, "page" : "92-97", "title" : "Cross-cultural evolutionary psychology", "type" : "article-journal", "volume" : "7" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Apicella and Barrett 2016; Kurzban 2013)", "plainTextFormattedCitation" : "(Apicella and Barrett 2016; Kurzban 2013)", "previouslyFormattedCitation" : "(Apicella and Barrett 2016; Kurzban 2013)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(Apicella and Barrett 2016; Kurzban 2013). However, this is perhaps in part because of the discipline’s need for cross-cultural surveys to validate theories that claim to purport to humans as a species. Evolutionary psychology has a greater need for cross-cultural replications than other disciplines, such as those focussed around basic psychophysics where we might more easily assume universal underlying mechanics, or more descriptive research approaches that aim to uncover behaviour in culturally-specific environments, such as the workplace or social media sites. We do not mean to imply either that sample diversity should be the only goal; there are many valuable ways to add to our understanding of any phenomenon. Valuable extensions to research on a WEIRD student sample can arise, for instance, from adding methodological diversity, developing theoretical frameworks, creating models of the behaviour, or testing similar behaviours in other species. Developmental approaches can make a useful testing ground for adaptive predictions, given that individuals have different adaptive needs across their lifecourse, but we note that only 8% of the samples that we coded used child participants. Scientists, including evolutionary psychologists, are increasingly recognising the value in replication across multiple labs and samples ADDIN CSL_CITATION { "citationItems" : [ { "id" : "ITEM-1", "itemData" : { "DOI" : "10.1126/science.aaf0918", "ISSN" : "0036-8075", "abstract" : "Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values.Science, this issue p. 1433The replicability of some scientific findings has recently been called into question. To contribute data about replicability in economics, we replicated 18 studies published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014. All of these replications followed predefined analysis plans that were made publicly available beforehand, and they all have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. The replicability rate varies between 67% and 78% for four additional replicability indicators, including a prediction market measure of peer beliefs.", "author" : [ { "dropping-particle" : "", "family" : "Camerer", "given" : "Colin F", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Dreber", "given" : "Anna", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Forsell", "given" : "Eskil", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Ho", "given" : "T.-H.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Huber", "given" : "J\u00fcrgen", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Johannesson", "given" : "Magnus", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Kirchler", "given" : "Michael", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Almenberg", "given" : "Johan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Altmejd", "given" : "Adam", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Chan", "given" : "Taizan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Heikensten", "given" : "Emma", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Holzmeister", "given" : "Felix", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Imai", "given" : "Taisuke", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Isaksson", "given" : "Siri", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Nave", "given" : "Gideon", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Pfeiffer", "given" : "Thomas", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Razen", "given" : "Michael", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Wu", "given" : "Hang", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Science", "id" : "ITEM-1", "issue" : "6280", "issued" : { "date-parts" : [ [ "2016", "3", "25" ] ] }, "page" : "1433-1436", "title" : "Evaluating replicability of laboratory experiments in economics", "type" : "article-journal", "volume" : "351" }, "uris" : [ "" ] }, { "id" : "ITEM-2", "itemData" : { "DOI" : "10.1016/j.jesp.2015.10.012", "ISSN" : "00221031", "author" : [ { "dropping-particle" : "", "family" : "Ebersole", "given" : "Charles R", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Atherton", "given" : "Olivia E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Belanger", "given" : "Aimee L", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Skulborstad", "given" : "Hayley M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Allen", "given" : "Jill M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Banks", "given" : "Jonathan B", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Baranski", "given" : "Erica", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Bernstein", "given" : "Michael J", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Bonfiglio", "given" : "Diane B.V.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Boucher", "given" : "Leanne", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Brown", "given" : "Elizabeth R.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Budiman", "given" : "Nancy I.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Cairo", "given" : "Athena H.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Capaldi", "given" : "Colin A.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Chartier", "given" : "Christopher R.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Chung", "given" : "Joanne M.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Cicero", "given" : "David C.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Coleman", "given" : "Jennifer A.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Conway", "given" : "John G.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Davis", "given" : "William E.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Devos", "given" : "Thierry", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Fletcher", "given" : "Melody M.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "German", "given" : "Komi", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Grahe", "given" : "Jon E.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Hermann", "given" : "Anthony D.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Hicks", "given" : "Joshua A.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Honeycutt", "given" : "Nathan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Humphrey", "given" : "Brandon", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Janus", "given" : "Matthew", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Johnson", "given" : "David J.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Joy-Gaba", "given" : "Jennifer A.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Juzeler", "given" : "Hannah", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Keres", "given" : "Ashley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Kinney", "given" : "Diana", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Kirshenbaum", "given" : "Jacqeline", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Klein", "given" : "Richard A.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lucas", "given" : "Richard E.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lustgraaf", "given" : "Christopher J.N.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Martin", "given" : "Daniel", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Menon", "given" : "Madhavi", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Metzger", "given" : "Mitchell", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Moloney", "given" : "Jaclyn M.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Morse", "given" : "Patrick J.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Prislin", "given" : "Radmila", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Razza", "given" : "Timothy", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Re", "given" : "Daniel E.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Rule", "given" : "Nicholas O.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sacco", "given" : "Donald F.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sauerberger", "given" : "Kyle", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Shrider", "given" : "Emily", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Shultz", "given" : "Megan", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Siemsen", "given" : "Courtney", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Sobocko", "given" : "Karin", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Weylin Sternglanz", "given" : "R.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Summerville", "given" : "Amy", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Tskhay", "given" : "Konstantin O.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Allen", "given" : "Zack", "non-dropping-particle" : "van", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Vaughn", "given" : "Leigh Ann", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Walker", "given" : "Ryan J.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Weinberg", "given" : "Ashley", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Wilson", "given" : "John Paul", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Wirth", "given" : "James H.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Wortman", "given" : "Jessica", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Nosek", "given" : "Brian A.", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Journal of Experimental Social Psychology", "id" : "ITEM-2", "issued" : { "date-parts" : [ [ "2016", "11" ] ] }, "page" : "68-82", "publisher" : "Elsevier", "title" : "Many Labs 3: Evaluating participant pool quality across the academic semester via replication", "type" : "article-journal", "volume" : "67" }, "uris" : [ "" ] }, { "id" : "ITEM-3", "itemData" : { "DOI" : "10.7554/eLife.04333", "ISSN" : "2050-084X", "abstract" : "It is widely believed that research that builds upon previously published findings has reproduced the original work. However, it is rare for researchers to perform or publish direct replications of existing results. The Reproducibility Project: Cancer Biology is an open investigation of reproducibility in preclinical cancer biology research. We have identified 50 high impact cancer biology articles published in the period 2010-2012, and plan to replicate a subset of experimental results from each article. A Registered Report detailing the proposed experimental designs and protocols for each subset of experiments will be peer reviewed and published prior to data collection. The results of these experiments will then be published in a Replication Study. The resulting open methodology and dataset will provide evidence about the reproducibility of high-impact results, and an opportunity to identify predictors of reproducibility.", "author" : [ { "dropping-particle" : "", "family" : "Errington", "given" : "Timothy M", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Iorns", "given" : "Elizabeth", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Gunn", "given" : "William", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Tan", "given" : "Fraser Elisabeth", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lomax", "given" : "Joelle", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Nosek", "given" : "Brian A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "eLife", "id" : "ITEM-3", "issued" : { "date-parts" : [ [ "2014", "12", "10" ] ] }, "publisher" : "eLife Sciences Publications, Ltd", "title" : "An open investigation of the reproducibility of cancer biology research", "type" : "article-journal", "volume" : "3" }, "uris" : [ "" ] }, { "id" : "ITEM-4", "itemData" : { "DOI" : "10.1017/S0140525X17001972", "ISSN" : "0140-525X", "abstract" : "Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component of science. A finding is elevated from single observation to scientific evidence when the procedures that were used to obtain it can be reproduced and the finding itself can be replicated. Recent replication attempts show that some high profile results \u2013 most notably in psychology, but in many other disciplines as well \u2013 cannot be replicated consistently. These replication attempts have generated a considerable amount of controversy, and the issue of whether direct replications have value has, in particular, proven to be contentious. However, much of this discussion has occurred in published commentaries and social media outlets, resulting in a fragmented discourse. To address the need for an integrative summary, we review various types of replication studies and then discuss the most commonly voiced concerns about direct replication. We provide detailed responses to these concerns and consider different statistical ways to evaluate replications. We conclude there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science.", "author" : [ { "dropping-particle" : "", "family" : "Zwaan", "given" : "Rolf A", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Etz", "given" : "Alexander", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Lucas", "given" : "Richard E", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" }, { "dropping-particle" : "", "family" : "Donnellan", "given" : "M Brent", "non-dropping-particle" : "", "parse-names" : false, "suffix" : "" } ], "container-title" : "Behavioral and Brain Sciences", "id" : "ITEM-4", "issued" : { "date-parts" : [ [ "2018", "10", "25" ] ] }, "page" : "e120", "publisher" : "Cambridge University Press", "title" : "Making replication mainstream", "type" : "article-journal", "volume" : "41" }, "uris" : [ "" ] } ], "mendeley" : { "formattedCitation" : "(Camerer et al. 2016; Ebersole et al. 2016; Errington et al. 2014; Zwaan et al. 2018)", "manualFormatting" : "(e.g., Camerer et al. 2016; Ebersole et al. 2016; Errington et al. 2014; Zwaan et al. 2017)", "plainTextFormattedCitation" : "(Camerer et al. 2016; Ebersole et al. 2016; Errington et al. 2014; Zwaan et al. 2018)", "previouslyFormattedCitation" : "(Camerer et al. 2016; Ebersole et al. 2016; Errington et al. 2014; Zwaan et al. 2018)" }, "properties" : { "noteIndex" : 0 }, "schema" : "" }(e.g., Camerer et al. 2016; Ebersole et al. 2016; Errington et al. 2014; Zwaan et al. 2017). In this light, it is of interest that the first study to be accepted by the Psychological Science Accelerator (), a project that uses multiple laboratories to test hypotheses, was proposed by two researchers, Jones and DeBruine, whose work often draws upon a functional framework. For now, we conclude that while two key journals use more diverse samples than many typical (social or developmental) psychology journals, as Kurzban (2013) suggested, it is important to realise that given the glaring underrepresentation of certain regions we still have a long road ahead. AcknowledgmentsPart of the data were collected while the first author was at the University of Leiden and he wishes to thank his bachelor thesis group for their support with the project. We thank the editor and two reviewers for their very helpful input. On behalf of all authors, the corresponding author states that there is no conflict of interest.?ReferencesADDIN Mendeley Bibliography CSL_BIBLIOGRAPHY Adair, J. G., Coêlho, A. E. L., & Luna, J. R. (2002). How international is psychology? International Journal of Psychology, 37(3), 160–170. doi:10.1080/00207590143000351Andrews, D., Nonnecke, B., & Preece, J. (2003). Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users. International Journal of Human-Computer Interaction, 16(2), 185–210. doi:10.1207/S15327590IJHC1602_04Apicella, C. L., & Barrett, H. C. (2016). Cross-cultural evolutionary psychology. Current Opinion in Psychology, 7, 92–97. doi:10.1016/j.copsyc.2015.08.015Arnett, J. J. (2008). The Neglected 95%: Why American Psychology Needs to Become Less American. American Psychologist, 63(7), 602–614. doi:10.1037/0003-066X.63.7.602Aust, F., & Barth, M. (2016). papaja: Create APA manuscripts with R Markdown. See . com/crsh/papaja.Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2013). Seriousness checks are useful to improve data validity in online research. Behavior Research Methods, 45(2), 527–535. doi:10.3758/s13428-012-0265-2Bareinboim, E., & Pearl, J. (2012). Controlling selection bias in causal inference. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (pp. 100–108).Bareinboim, E., & Pearl, J. (2016). Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(27), 7345 LP-7352. doi:10.1073/pnas.1510507113Batres, C., & Perrett, D. I. (2014). The Influence of the Digital Divide on Face Preferences in El Salvador: People without Internet Access Prefer More Feminine Men, More Masculine Women, and Women with Higher Adiposity. PLoS ONE, 9(7), e100966. doi:10.1371/journal.pone.0100966Bauserman, R. (1997). International representation in the psychological literature. International Journal of Psychology, 32(2), 107–112. doi:10.1080/002075997400908Bello, D., Leung, K., Radebaugh, L., Tung, R. L., & van Witteloostuijn, A. (2009). From the Editors: Student samples in international business research. Journal of International Business Studies, 40(3), 361–364. doi:10.1057/jibs.2008.101Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289–300. doi:10.2307/2346101Bergman, M. E., & Jean, V. A. (2016). Where have all the “workers” gone? A critical analysis of the unrepresentativeness of our samples relative to the labor market in the industrial–organizational psychology literature. Industrial and Organizational Psychology, 9(1), 84–113. doi:10.1017/iop.2015.70Berry, J. W. (2013). Achieving a global psychology. Canadian Psychology/Psychologie canadienne, 54(1), 55–61. doi:10.1037/a0031246Birnbaum, M. H. (2004). Human Research and Data Collection via the Internet. Annual Review of Psychology, 55(1), 803–832. doi:10.1146/annurev.psych.55.090902.141601Bohannon, J. (2016, June 7). Psychologists grow increasingly dependent on online research subjects. Science. doi:10.1126/science.aag0592Boothroyd, L. G., Jucker, J., Thornborrow, T., Jamieson, M. A., Burt, D. M., Barton, R. A., et al. (2016). Television exposure predicts body size ideals in rural Nicaragua. British Journal of Psychology, 107(4), 752–767. doi:10.1111/bjop.12184Brown, D. E. (1991). Human universals. New York, NY: McGraw-Hill.Brown, D. E. (2000). Human universals and their implications. In N. Roughley (Ed.), Being humans: Anthropological universality and particularity in transdisciplinary perspectives (pp. 156–174). Berlin: Walter de Gruyter.Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3–5. doi:10.1177/1745691610393980Burchett, H., Umoquit, M., & Dobrow, M. (2011). How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability frameworks. Journal of Health Services Research & Policy, 16(4), 238–244. doi:10.1258/jhsrp.2011.010124Buss, D. M. (1989). Sex differences in human mate preferences: Evolutionary hypotheses tested in 37 cultures. Behavioral and Brain sciences, 12(1), 1–49. doi:10.1017/S0140525X00023992Buss, D. M. (1994). The Evolution of Desire: Strategies of Human Mating. New York, NY: Basic books.Buss, D. M. (1995). Evolutionary psychology: A new paradigm for psychological science. Psychological inquiry, 6(1), 1–30. doi:10.1207/s15327965pli0601_1Buss, D. M., Larsen, R. J., Westen, D., & Semmelroth, J. (1992). Sex Differences in Jealousy: Evolution, Physiology, and Psychology. Psychological Science, 3(4), 251–255. doi:10.1111/j.1467-9280.1992.tb00038.xBuss, D. M., & Shackelford, T. K. (1997). From vigilance to violence: Mate retention tactics in married couples. Journal of Personality and Social Psychology, 72(2), 346–361. doi:10.1037/0022-3514.72.2.346Buss, D. M., Shackelford, T. K., & LeBlanc, G. J. (2000). Number of children desired and preferred spousal age difference: context-specific mate preference patterns across 37 cultures. Evolution and Human Behavior, 21(5), 323–331. doi:10.1016/S1090-5138(00)00048-9Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., et al. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. doi:10.1126/science.aaf0918Ceci, S. J., Kahan, D. M., & Braman, D. (2010). The WEIRD are even weirder than you think: Diversifying contexts is as important as diversifying samples. Behavioral and Brain Sciences, 33(2–3), 87–88. doi:10.1017/S0140525X10000063Clifford, S., Jewell, R. M., & Waggoner, P. D. (2015). Are samples drawn from Mechanical Turk valid for research on political ideology? Research & Politics, 2(4), 205316801562207. doi:10.1177/2053168015622072Cole, M. (2006). Internationalism in psychology: We need it now more than ever. American Psychologist, 61(8), 904–917. doi:10.1037/0003-066X.61.8.904Cosmides, L., & Tooby, J. (1997). Evolutionary psychology: A primer. , M. J. C., McDonnell, J. V, & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE, 8(3), e57410. doi:10.1371/journal.pone.0057410Daly, M., & Wilson, M. (1988). Homicide. New Brunswick, NJ: Transaction Books.Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., et al. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82. doi:10.1016/j.jesp.2015.10.012Elwert, F., & Winship, C. (2014). Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40, 31–53. doi:10.1146/annurev-soc-071913-043455Epstein, J., Klinkenberg, W. ., Wiley, D., & McKinley, L. (2001). Insuring sample equivalence across internet and paper-and-pencil assessments. Computers in Human Behavior, 17(3), 339–346. doi:10.1016/S0747-5632(01)00002-4Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014). An open investigation of the reproducibility of cancer biology research. eLife, 3. doi:10.7554/eLife.04333Fiedler, K. (2000). Beware of samples! A cognitive-ecological sampling approach to judgment biases. Psychological Review, 107(4), 659–676. doi:10.1037/0033-295X.107.4.659Gallander Wintre, M., North, C., & Sugar, L. A. (2001). Psychologists’ response to criticisms about research based on undergraduate participants: A developmental perspective. Canadian Psychology/Psychologie Canadienne, 42(3), 216–225. doi:10.1037/h0086893Gildersleeve, K., Haselton, M. G., & Fales, M. R. (2014). Do women’s mate preferences change across the ovulatory cycle? A meta-analytic review. Psychological Bulletin, 140(5), 1205–1259. doi:10.1037/a0035438Gosling, S. D., & Johnson, J. a. (2010). Advanced methods for conducting online behavioral research. (S. D. Gosling & J. A. Johnson, Eds.). Washington, DC: American Psychological Association. doi:10.1037/12076-000Gosling, S. D., Sandy, C. J., John, O. P., & Potter, J. (2010). Wired but not WEIRD: The promise of the Internet in reaching more diverse samples. Behavioral and Brain Sciences, 33(2–3), 94–95. doi:10.1017/S0140525X10000300Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires. American Psychologist, 59(2), 93–104. doi:10.1037/0003-066X.59.2.93Graham, S. (1992). “Most of the subjects were White and middle class”: Trends in published research on African Americans in selected APA journals, 1970-1989. American Psychologist, 47(5), 629–639. doi:10.1037/0003-066X.47.5.629Green, L. W., & Glasgow, R. E. (2006). Evaluating the Relevance, Generalization, and Applicability of Research: Issues in External Validation and Translation Methodology. Evaluation & the Health Professions, 29(1), 126–153. doi:10.1177/0163278705284445Greenwood, J. D. (1982). On the Relation Between Laboratory Experiments and Social Behaviour: Causal Explanation and Generalization. Journal for the Theory of Social Behaviour, 12(3), 225–250. doi:10.1111/j.1468-5914.1982.tb00449.xHeine, S. J., Lehman, D. R., Peng, K., & Greenholtz, J. (2002). What’s wrong with cross-cultural comparisons of subjective Likert scales?: The reference-group effect. Journal of Personality and Social Psychology, 82(6), 903–918. doi:10.1037//0022-3514.82.6.903Henrich, J., Boyd, R., Bowles, S., Camerer, C., Fehr, E., Gintis, H., et al. (2005). “Economic man” in cross-cultural perspective: Behavioral experiments in 15 small-scale societies. Behavioral and Brain Sciences, 28(06), 795–815. doi:10.1017/S0140525X05000142Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world. Behavioral and Brain Sciences, 33(2–3), 61–83. doi:10.1017/S0140525X0999152XHenry, P. J. (2008). College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology’s View of the Nature of Prejudice. Psychological Inquiry, 19(2), 49–71. doi:10.1080/10478400802049936Hill, J. M., Jobling, R., Pollet, T. V., & Nettle, D. (2014). Social capital across urban neighborhoods: A comparison of self-report and observational data. Evolutionary Behavioral Sciences, 8(2), 59–69. doi:10.1037/h0099131Hlavac, M. (2014). stargazer: LaTeX code and ASCII text for well-formatted regression and summary statistics tables. , C. H., & Triandis, H. C. (1985). Measurement in cross-cultural psychology. Journal of Cross-Cultural Psychology, 16(2), 131–152. doi:10.1177/0022002185016002001Joinson, A. (1999). Social desirability, anonymity, and internet-based questionnaires. Behavior Research Methods, Instruments, & Computers, 31(3), 433–438. doi:10.3758/BF03200723Keene, O. N. (1995). The log transformation is special. Statistics in Medicine, 14(8), 811–819. doi:10.1002/sim.4780140810Kenrick, D. T., & Keefe, R. C. (1992). Age preferences in mates reflect sex differences in human reproductive strategies. Behavioral and Brain Sciences, 15(01), 75–91. doi:10.1017/S0140525X00067595Kenrick, D. T., Keefe, R. C., Bryan, A., Barr, A., & Brown, S. (1995). Age preferences and mate choice among homosexuals and heterosexuals: A case for modular psychological mechanisms. Journal of Personality and Social Psychology, 69(6), 1166–1172. doi:10.1037/0022-3514.69.6.1166Kone?ni, V. J. (2010). Responsible behavioral science generalizations and applications require much more than non-WEIRD samples. Behavioral and Brain Sciences, 33(2–3), 98–99. doi:10.1017/S0140525X10000142Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness. Behavior Research Methods, Instruments, & Computers, 29(2), 264–269. doi:10.3758/BF03204824Krantz, J. H., & Dalal, R. (2000). Validity of Web-Based Psychological Research. In Psychological Experiments on the Internet (pp. 35–60). Elsevier. doi:10.1016/B978-012099980-4/50003-4Kurzban, R. (2013). Is Evolutionary Psychology WEIRD or NORMAL? EP Journal blog. . Accessed 14 November 2016Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., et al. (2018). Justify your alpha. Nature Human Behaviour, 2(3), 168–171. doi:10.1038/s41562-018-0311-xLevitt, S. D., & List, J. A. (2007). What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World? Journal of Economic Perspectives, 21(2), 153–174. doi:10.1257/jep.21.2.153Mace, R., & Pagel, M. (1994). The Comparative Method in Anthropology. Current Anthropology, 35(5), 549–564. doi:10.1086/204317Matzat, U., & Snijders, C. (2010). Does the online collection of ego-centered network data reduce data quality? An experimental comparison. Social Networks, 32(2), 105–111. doi:10.1016/j.socnet.2009.08.002Medin, D., Ojalehto, B., Marin, A., & Bang, M. (2017). Systems of (non-)diversity. Nature Human Behaviour, 1, 88. doi:10.1038/s41562-017-0088Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379–387. doi:10.1037/0003-066X.38.4.379Müller, K. (2017). bindrcpp. , R. (1965). Galton’s problem: The logic of cross-cultural analysis. Social Research, 32(4), 428–451.Nasif, E. G., Al-Daeaj, H., Ebrahimi, B., & Thibodeaux, M. S. (1991). Methodological problems in cross-cultural research: An updated review. MIR: Management International Review, 31(1), 79–tle, D. (2009). Ecological influences on human behavioural diversity: a review of recent findings. Trends in Ecology & Evolution, 24(11), 618–624. doi:10.1016/j.tree.2009.05.013Nettle, D. (2017). Tyneside neighbourhoods: Deprivation, social life and social behaviour in one British city. Cambridge, UK: Open Book tle, D., Colléony, A., & Cockerill, M. (2011). Variation in Cooperative Behaviour within a Single City. PLoS ONE, 6(10), e26922. doi:10.1371/journal.pone.0026922Nielsen, M., Haun, D., K?rtner, J., & Legare, C. H. (2017). The persistent sampling bias in developmental psychology: A call to action. Journal of Experimental Child Psychology, 162, 31–38. doi:10.1016/j.jecp.2017.04.017Paolacci, G., & Chandler, J. (2014). Inside the turk understanding mechanical turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188. doi:10.1177/0963721414531598Pearl, J. (2009a). Causality. Cambridge, UK: Cambridge university press.Pearl, J. (2009b). Causal inference in statistics: An overview. Statistics surveys, 3, 96–146. doi:10.1214/09-SS057Peterson, R. A. (2001). On the Use of College Students in Social Science Research: Insights from a Second-Order Meta-analysis. Journal of Consumer Research, 28(3), 450–461. doi:10.1086/323732Pollet, T. V., Tybur, J. M., Frankenhuis, W. E., & Rickard, I. J. (2014). What can cross-cultural correlations teach us about human nature? Human nature (Hawthorne, N.Y.), 25(3), 410–29. doi:10.1007/s12110-014-9206-3Poortinga, Y. H. (1989). Equivalence of cross-cultural data: An overview of basic issues. International Journal of Psychology, 24(1), 737–756. doi:10.1080/00207598908247842R Development Core Team. (2008). R?: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.Raihani, N. J., Mace, R., & Lamba, S. (2013). The Effect of $1, $5 and $10 Stakes in an Online Dictator Game. PLoS ONE, 8(8), e0073131. doi:10.1371/journal.pone.0073131Robinson, D. (2017). broom. , P. (2010). What is really wrong with a priori claims of universality? Sampling, validity, process level, and the irresistible drive to reduce. Behavioral and Brain Sciences, 33(2–3), 107–108. doi:10.1017/S0140525X10000233Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1(1), 27–42. doi:10.1177/2515245917745629Rosenthal, R. (1965). The volunteer subject. Human relations, 18(4), 389–406.Ross, M. H., & Homer, E. (1976). Galton’s problem in cross-national research. World Politics, 29(1), 1–28. doi:10.2307/2010045Schaller, M., & Murray, D. R. (2008). Pathogens, personality, and culture: disease prevalence predicts worldwide variability in sociosexuality, extraversion, and openness to experience. Journal of Personality and Social Psychology, 95(1), 212–221. doi:10.1037/0022-3514.95.1.212Schmitt, D. P. (2005). Sociosexuality from Argentina to Zimbabwe: A 48-nation study of sex, culture, and strategies of human mating. Behavioral and Brain Sciences, 28(2), 247–275. doi:10.1017/S0140525X0500005Schultz, D. P. (1969). The human subject in psychological research. Psychological Bulletin, 72(3), 214–228. doi:10.1037/h0027880Scott-Phillips, T. C. (2018). Expression Unleashed. , I. M., Clark, A. P., Josephson, S. C., Boyette, A. H., Cuthill, I. C., Fried, R. L., et al. (2014). Human preferences for sexually dimorphic faces may be evolutionarily novel. Proceedings of the National Academy of Sciences, 111(40), 14388–14393. doi:10.1073/pnas.1409643111Shackelford, T. K., Schmitt, D. P., & Buss, D. M. (2005). Universal dimensions of human mate preferences. Personality and Individual Differences, 39(2), 447–458. doi:10.1016/j.paid.2005.01.023Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on Generality (COG): A Proposed Addition to All Empirical Papers. Perspectives on Psychological Science, 12(6), 1123–1128. doi:10.1177/1745691617708630Smart, R. G. (1966). Subject selection bias in psychological research. Canadian Psychologist/Psychologie canadienne, 7a(2), 115–121. doi:10.1037/h0083096Steckler, A., & McLeroy, K. R. (2008). The Importance of External Validity. American Journal of Public Health, 98(1), 9–10. doi:10.2105/AJPH.2007.126847Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18(6), 643–662. doi:10.1037/h0054651Stulp, G., Simons, M. J. P., Grasman, S., & Pollet, T. V. (2017). Assortative mating for human height: A meta-analysis. American Journal of Human Biology, 29(1), e22917. doi:10.1002/ajhb.22917Sue, S. (1999). Science, ethnicity, and bias: Where have we gone wrong? American Psychologist, 54(12), 1070–1077. doi:10.1037/0003-066X.54.12.1070Tooby, J., & Cosmides, L. (1990). On the Universality of Human Nature and the Uniqueness of the Individual: The Role of Genetics and Adaptation. Journal of Personality, 58(1), 17–67. doi:10.1111/j.1467-6494.1990.tb00907.xTybur, J. M., Frankenhuis, W. E., & Pollet, T. V. (2014). Behavioral immune system methods: Surveying the present to shape the future. Evolutionary Behavioral Sciences, 8(4), 274–283. doi:10.1037/ebs0000017United Nations. (2013). Composition of macro geographical (continental) regions, geographical sub-regions, and selected economic and other groupings. , E. L. (2018). Acknowledging the Whiteness of Motivation Research: Seeking Cultural Relevance. Educational Psychologist, 53(2), 131–144. doi:10.1080/00461520.2018.1442220Van de Vijver, F. J. R., & Leung, K. (1997). Methods and data analysis of comparative research. In J. W. Berry, Y. H. Poortinga, & J. Pandey (Eds.), Handbook of Cross-Cultural Psychology (2nd ed.). Boston, MA: Allyn & Bacon.Wickham, H. (2009). ggplot2. New York, NY: Springer New York. doi:10.1007/978-0-387-98141-3Wickham, H. (2014). Tidyr: easily tidy data with spread and gather functions. , H., & Bryan, J. (2017). readxl: Read Excel Files. R package version 1.0.0. , H., & Francois, R. (2017). dplyr: A grammar of data manipulation. , H., & Wickham, M. H. (2017). ‘plyr’ package. , D. S. (1998). Adaptive individual differences within single populations. Philosophical Transactions of the Royal Society B: Biological Sciences, 353(1366), 199–205. doi:10.1098/rstb.1998.0202Wilson, D. S. (2011). The Neighborhood Project: Using evolution to improve my city, one block at a time. New York, NY: Little, Brown.Wilson, D. S., O’Brien, D. T., & Sesma, A. (2009). Human prosociality from an evolutionary perspective: variation and correlations at a city-wide scale. Evolution and Human Behavior, 30(3), 190–200. doi:10.1016/j.evolhumbehav.2008.12.002Xie, Y. (2015). Dynamic Documents with R and knitr (Vol. 29). CRC Press.Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. doi:10.1017/S0140525X17001972 ................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.