Open access Research Top-cited articles in medical professionalism: a ...

BMJ Open: first published as 10.1136/bmjopen-2019-029433 on 31 July 2019. Downloaded from on June 1, 2024 by guest. Protected by copyright.

Open access

Research

Top-cited articles in medical professionalism: a bibliometric analysis versus altmetric scores

Samy A Azer, 1 Sarah Azer2

To cite: Azer SA, Azer S. Top-cited articles in medical professionalism: a bibliometric analysis versus altmetric scores. BMJ Open 2019;9:e029433. doi:10.1136/ bmjopen-2019-029433 Prepublication history and additional material for this paper are available online. To view please visit the journal (http:// dx.10.1136/bmjopen- 2019-029433).

Received 25 January 2019 Revised 26 April 2019 Accepted 1 July 2019

? Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ. 1Professor of Medical Education, Department of Medical Education, King Saud University, College of Medicine, Riyadh, Saudi Arabia 2Senior Robotic Fellow, Department of Urology, Southmead Hospital, Bristol, United Kingdom

Correspondence to Professor Samy A Azer; azer2000@.au, mcqsmcqs@

Abstract Introduction Citation counts of articles have been used to measure scientific outcomes and assess suitability for grant applications. However, citation counts are not without limitations. With the rise of social media, altmetric scores may provide an alternative assessment tool. Objectives The aims of the study were to assess the characteristics of highly cited articles in medical professionalism and their altmetric scores. Methods The Web of Science was searched for top-cited articles in medical professionalism, and the characteristics of each article were identified. The altmetric database was searched to identify report for each identified article. A model to assess the relationship between the number of citations and each of the key characteristics as well as altmetric scores was developed. Results No correlations were found between the number of citations and number of years since publication (p=0.192), number of institutes (p=0.081), number of authors (p=0.270), females in authorship (p=0.150) or number of grants (p=0.384). The altmetric scores varied from 0 to 155, total=806, median=5.0, (IQR=20). Twitter (54%) and Mendeley (62%) were the most popular altmetric resources. No correlation was found between the number of citations and the altmetric scores (p=0.661). However, a correlation was found for articles published in 2007 and after (n=17, p=0.023). To further assess these variables, a model was developed using multivariate analysis; did not show significant differences across subgroups. The topics covered were learning and teaching professionalism, curriculum issues, professional and unprofessional behaviour. Conclusions Altmetric scores of articles were significantly correlated with citations counts for articles published in 2007 and after. Highly cited articles were produced mainly by the USA, Canada and the UK. The study reflects the emerging role of social media in research dissemination. Future studies should investigate the specific features of highly cited articles and factors reinforcing distribution of research data among scholars and non-scholars.

Introduction Citation counts have been used by universities and funding bodies to measure scientific outcomes, make decisions about professional promotion and assess suitability for grant applications.1 2 In this context, it was claimed that the higher number of citations received,

Strengths and limitations of this study

Four searches were conducted in the web of Science database and the altmetric tracks.

The analysis explored a range of bibliometric parameters.

The study was limited to top-cited articles in the English language.

the higher quality of work and the more likely that other researchers cite the work.3 While these claims may not necessarily be true, there is a substantial body of evidence that the number of citations correlates with other research achievements including research awards, honours, nomination for Nobel laureateship,3 4 prestigious research positions5 and academic ranking.6 7 However, there are factors other than scientific quality that may affect the decision to cite.8 For example, there is evidence that early interest in a research publication reflected by online access within a week of publication predicts citations up to 15 years later.9 Also, scientific citations favour positive results and authors tend to cite primarily works by authors with whom they know and personally acquainted.10 11

With these limitations in mind, there is a continuous search for alternatives or metres that can complement the citation counts. Currently, there is a rising interest in the altmetric scores. Contrary to traditional citation-based analysis, the altmetrics reflect the widespread attention to published scientific articles and the rise of social media for dissemination and discussion of scientific information. Therefore, it is possible to quantify discussion of an article on blogs, news media or other social media platforms.12

Considering these two tools, it was decided to assess highly cited articles on medical professionalism.13 14 The top-cited articles were selected because an earlier study revealed a number of attributes of articles on medical professionalism.14 The use of the

Azer SA, Azer S. BMJ Open 2019;9:e029433. doi:10.1136/bmjopen-2019-029433

1

BMJ Open: first published as 10.1136/bmjopen-2019-029433 on 31 July 2019. Downloaded from on June 1, 2024 by guest. Protected by copyright.

Open access

altmetric scores in this study in particular is thought to be useful since articles on professionalism are usually shared on social media.

Therefore, the present study aims at the following: first, identify the most-cited articles in medical professionalism and evaluate their characteristics, and study any correlations between the number of citations and each of their biliometric characteristics. Second, assess the impact of such articles on social media by calculating the altmetric scores and conducting an exploratory analysis examining the altmetric findings compared with citation analysis. The conduction of multivariate analysis model may provide additional insight into such evaluation. The findings from this study may enable researchers to identify common features of articles behind the progress of medical professionalism and key topics discussed over the last two to three decades. The study may provide more insight into any relationships between citation analysis and the altmetric scores. The identified list of publications may be useful to medical educators and those teaching medical professionalism or doing masters or research in these areas.

Methods Study design To achieve the objectives of this study, it was decided to search the Web of Science database of Clarivate Analytics for highly cited articles and track the citation records of publications identified. Although Scopus and Google Scholar databases also provide citation tracking, it was decided to limit the search to the Web of Science. This is because the Web of Science is regularly updated and its 2016-Journal Citation Reports reported over 59million citations in its Science Edition and 7million from its Social Science Edition. In the area of medical education, medical ethics and bioethics, general medicine and surgery, the Web of Science has included 16, 49, 457 and 180 peer-reviewed journals, respectively. Google Scholar was not included in the search because it is difficult to search, and it is not possible to identify the number of citations for each year across the last two to three decades, and the citations in Google Scholar usually include textbooks, monographs, conference proceedings, as well as non-peer-reviewed work. The Scopus database was not included in our search because its records only go back to 1966.

To achieve the first aim, we planned to identify the highly cited articles in medical professionalism and their characteristics using three mechanisms: (1) searching the Web of Science using keywords, (2) searching medical education, ethics, general medicine and surgery journals in the Web of Science, (3) searching the webpage of journals and (4) searching for related resources mentioned in the list of references of articles identified. For the second aim, the altmetric bookmarklet application was used to obtain the altmetric scores and construct exploratory analysis examining the role of social media and the

different resources contributing to altmetrics. At the end, we compared these findings with those obtained from the citation analysis.15?17 A description of the steps used in the search is discussed below.

Searching the web of science database using keywords Searching the Web of Science database was carried out in the 5 April 2017 by two researchers (SAA is a professor of medical education with a 20-year experience in research in the field of medical education and professionalism, and SA a surgical registrar and researcher). The search words used were the following: `Medical professionalism', `Patient safety', `Professional behavior', `Unprofessional behavior', `Role modeling', `Accountability', `Faculty training in professionalism', `Altruism', `Physician code', `Physician charter', `Medical ethics', `Integrity', `Consent', `Defining medical professionalism', `Empathy', `Compassionate doctor', `Professional conduct', `Collaborative doctor', `Self-assessment', `Professional development', `Resilient doctor', `Social justice', `Patient autonomy', `Patient Welfare', `Professional responsibility', `Managing conflict', Patient confidentiality', `Quality of care', `Social contract', `Team work and professionalism', `Personal development', `Public professionalism', `Interpersonal professionalism' and `Intrapersonal professionalism'. These keywords were identified from the terminology and themes used in defining medical professionalism in six resources including.18?23 We also looked at conference proceedings in the field and websites of organisations and agencies responsible for accreditation of medical education worldwide including: The World Federation for Medical Education, the UK's General Medical Council, the Association of American Medical Colleges, the Australian Medical Council, the Liaison Committee on Medical Education, and the Quality Assurance of Basic Medical Education, and documents such as: Tomorrow's doctors, 2003; The New Doctor, 2004; and General Medical Practice, 2001.

For each search word, the results were arranged using a link on the Web of Science database system `sort-by'-- `Time Cited- highest to lowest'. The results showed the articles organised in a descending order with the articles most frequently cited on the top. The findings from each search word were then arranged on one Excel sheet in a descending order based on the number of citations. The results identified by each evaluator were discussed and duplicate articles were excluded.

Searching journals in the Web of Science The second search involved searching all journals in the field of medical education, ethics, general medicine and surgery included in the Web of Science database. These journals are known to publish articles on medical professionalism. They were selected on the basis of the outcomes of the Web of Science search and the references cited by the articles identified. The aims of this second search were to maximise the yield of the search and detect any articles that were possibly missed during the first search.

2

Azer SA, Azer S. BMJ Open 2019;9:e029433. doi:10.1136/bmjopen-2019-029433

BMJ Open: first published as 10.1136/bmjopen-2019-029433 on 31 July 2019. Downloaded from on June 1, 2024 by guest. Protected by copyright.

This search was conducted under the same conditions of the first search, by the two researchers, on the same day, and by using the same keywords used in the first search. The journals in medical education that were searched included Academic Medicine, Medical Education, Medical Teacher, BMC Medical Education, Advances in Health Sciences Education Theory, and Practices, Teaching and Learning in Medicine and the Journal of Continuing Education in the Health Professions. The journals searched in general medicine and surgery were the New England Journal of Medicine, the Lancet, the British Medical Journal, the Journal of the American Medical Association, Journal of General Internal Medicine, Annals of Internal Medicine, Archives of Internal Medicine, Canadian Medical Association Journal, PLOS Medicine, Annals of Surgery, Archives of Surgery, British Journal of Surgery, Perspectives in Biology and Medicine, Mayo Clinic Proceedings and the Australian Medical Journal. The journals in bioethics that were searched included the American Journal of Bioethics, Journal of Medical Ethics and BMC Medical Ethics. The findings from journals were then arranged on one Excel sheet in a descending order based on the number of citations. The results identified by each evaluator were discussed and duplicate articles were excluded.

Searching the webpage of journals To maximise the yield of our search and to ensure that no paper was missed through searching the Web of Science, we conducted a third search using the webpage of the journals mentioned above. We examined the titles of articles listed in each issue of these journals during 2011 and prior years. This search was particularly important as for example, Teaching and Learning in Medicine first appeared in Web of Science in 1996 but the journal was published since 1989. Therefore, any relevant articles from this journal or others prior to 1996 would be included.

Inclusion and exclusion criteria The inclusion criteria were: (1) papers focusing on medical professionalism in the English language and (2) articles, reviews, research papers, reports, editorials on any aspect related to medical professionalism in the English language. The exclusion criteria were: (1) articles on medical professionalism in languages other than English, and (2) articles that focused on education/curriculum or clinical practices and medical professionalism was not the main focus. Articles with identical absolute number of citations were ranked on the basis of the average citation per year (the number of citations obtained divided by the number of years since published).24 A copy of all papers included in the list was obtained and read by the evaluators.

It is interesting to note that none of the articles excluded on the basis of language were qualified for inclusion in the list because they had less citation numbers than those of the article marked number 50 in the list.

Open access

Assessing the articles For each of the identified articles (online supplementary appendix 1), a full text was obtained and a copy was given to each researcher. The following information was collected: (1) the authors' names and their affiliations, and the number of females contributing to authorship (2) the number of institutes involved and the city and country of the origin of the publication, (3) the total number of citations obtained up to the day of searching the database, and the number of yearly citations since publication, (4) the year of publication and the calculated number of years since publication and (5) grants/ funding bodies stated in the publication and (6) the 2016-JIF of the journal that published the work.

We also aimed at grouping the identified top-cited articles into categories. We have not used the categories provided by the Web of Science `study type' because we noted that the Web of Science system does not differentiate between `original research' or `articles' and classified both as `articles'. For consistency and the purpose of this study, we grouped the articles into four categories-- article, review, editorial material and research. A definition of each category is given in the glossary. Using these definitions, two researchers independently allocated each article under a category. For articles that were difficult to classify or not fitting into the same category, a meeting was held to discuss these articles and a final decision was made.

The topics covered in identified articles were created by each researcher independently by generating key words reflecting the main idea covered in an article and using these words to phrase a short statement that could help in grouping more than one article under one topic. The topics were then discussed in a meeting to harmonise the grouping into a logical, simple and practical approach. Articles covering more than one topic were classified on the basis of the aim of the study, the title and the main outcomes.

Identification of author's gender Regarding the data collected for each article, it is important to mention here that the identification of the gender of each author was a challenging task particularly when a journal uses abbreviations of the first and second name rather than the full name, which was the case in three articles. The approach used in order to identify the females in the top-cited articles included (1) searching the Google database to find the university website, personal website of the author, LinkedIn webpage and ResearchGate account. This approach was particularly useful for authors who could have moved to other universities, (2) searching the university websites not only provided the full names but also provided identification photos of these authors, and in many times a list of their publication records, as well as areas of research/ teaching interests (3) Searching the Google Scholar database to identify their accounts, where we can find other publications under their names, the full name or an

Azer SA, Azer S. BMJ Open 2019;9:e029433. doi:10.1136/bmjopen-2019-029433

3

BMJ Open: first published as 10.1136/bmjopen-2019-029433 on 31 July 2019. Downloaded from on June 1, 2024 by guest. Protected by copyright.

Open access

identification photo showing them. Usually, authors of highly cited articles have other publications related to the same topic, or work with the same coauthors, which could also help in identifying them and tracing them and (4) In two difficult cases we emailed the corresponding author of these articles for help.

Altmetric system The altmetric system comprises, but not limited to, policy documents, news, blogs, tweeters, online reference managers (eg, Mendeley, CitULike), postpublication peer reviews (eg, Publons), Social media platforms (eg, Facebook, Google+, Pinterest), citations on Wikipedia, sites running Stack Exchanges (Q&A) and reviews on Faculty 1000 (F1000) and YouTube. Therefore, altmetric scores may reflect interest of the public as well as clinicians and researchers in a publication and the scores may provide information about the geographical and demographic details of those involved in such online/social media discussions.25

The altmetric programme processes raw data collected from the above-mentioned resources and the data are weighted according to a system created by altmetrics to reflect the relative contribution of each source to the total altmetric score. News, Blogs, Wikipedia and policy documents have a relatively higher weighting values.26 While Mendeley and CiteULike are shown in the report, they do not contribute to the total score.

Searching the altmetric system The search of the altmetric system was conducted on the same day (5 April 2017). The scores were identified using the Altmetric bookmarklet provided by the company.27 In summary, the articles were searched on PubMed database (the PMID or DOI are essential for triggering the altmetric bookmarklet to function). By clicking on the LinkOut link, we identified the publisher webpage hosting the original article and by clicking the altmetric bookmarklet application, we can check the attention records for the article. The altmetric attention score and donut help in identifying the relative quantity and the type of attention received by a published article. The meaning of the colours included in altmetric donut is explained in this link.28

Statistical analysis All analyses were conducted using SPSS Software (IBM SPSS Statistics Premium V.22.0 for Mac OS-SPSS) and the results were reported at total, mean, median, IQR and percentage. Pearson's correlation coefficient (r) was calculated to determine if the high citation numbers obtained were related to parameters characteristic of articles. Because of the observed differences in the citations of the top articles in the list compared with those in the bottom of the list, and the variability in the altmetric scores, it was decided to conduct a multivariate analysis model comprising the effect of number of authors and other parameters. The inter-rater agreement between evaluators was calculated using the Fleiss kappa scale.29

Patient and public involvement Patients and the public were not involved in this study.

Results Top-cited papers identified Online supplementary appendix 1 summarises the 50 most-cited articles in medical professionalism identified by searching the Web of Science database,30?79 out of a total of 3500 articles identified on professionalism. The articles are listed in a descending order from 1 to 50 with the highest absolute citation number is ranked 1 and the article with the lowest citation ranked 50 as per the day of the search. Articles with the same number of citations were ranked on the basis of average citation per year (eg, the articles ranked 34 and 35 had the same citation number 97, they were allocated to a ranking order based on the calculated citation per year, 13.86 and 7.46, respectively). Other articles that had the same citation number and were ranked on the basis of their calculated citation per year were articles ranked 36 and 37; 43 and 44; as well as 46 and 47.

Table 1 summarises the year of publication and article category. The articles were published over 17 years (from 1994 to 2011). During the period from 1994 to 1999, only seven articles (14%) were published. However, the number increased significantly from 2000 to 2005 making a total of 24 (48%) articles. The number in the years from 2006 to 2011 dropped to 19 (38%). No correlation was found between the citation counts of these papers and

Table 1 The most-cited papers in medical professionalism, summarised by year of publication and category

Article category

Year of publication: no of articles (reference) 1994?1996 1997?1999 2000?2002 2003?2005

2006?2008 2009?2011

Article

Review

178

Editorial material

Research

259 74 232 50 144

142

435 46 60 67 230 36 157

840 47 49 51 53 61 64 76 455 56 62 70 131

434 37 66 73

333 65 68 158 543 45 48 54 71

269 75 238 52

639 41 63 72 77 79

Total (%)

1 (2)

6 (12)

7 (14)

17 (34)

9 (18)

10 (20)

Total (%) 19 (38) 11 (22) 4 (08)

16 (32) 50 (100)

4

Azer SA, Azer S. BMJ Open 2019;9:e029433. doi:10.1136/bmjopen-2019-029433

BMJ Open: first published as 10.1136/bmjopen-2019-029433 on 31 July 2019. Downloaded from on June 1, 2024 by guest. Protected by copyright.

the number of years since publication (Pearson correlation (r)=0.188, p=0.192).

The distribution of the medical professionalism topics covered in these articles is summarised in table 2. The inter-rater agreement between assessors was in the range 0.758?0.846.

The articles were published in the following journals: Academic Medicine (n=19, 38%), the Journal of the American Medical Association (n=9, 18%), Journal of General Internal Medicine (n=4, 8%), Annals of Internal Medicine (n=4, 8%), the New England Journal of Medicine (n=3, 6%) and Medical Education (n=3, 6%). It is interesting to note that 24 (48%) articles were published in eight general medicine journal and the remaining were published in four medical education journals and one journal specialised in bioethics. Most journals have high journal impact factors and are on the top of their field (table 3). This finding reflects the significance of medical professionalism in undergraduate and postgraduate training. The first author of the top-cited articles was from the USA (n=37, 74%), Canada (n=8, 16%), the UK (n=2, 4%), Germany (n=1, 2%), Israel (n=1, 2%) and New Zealand (n=1, 2%).

Table 4 summarises the 26 authors who have contributed to two or more articles in the list. Of these, five authors were the first authors of two or more papers, 10 were coauthors of two or more papers, and the remaining 11 were the first authors and coauthors of two or more articles. Top authors were Papadakis, M (n=4; first author of all four papers) and Blank, L (n=4; first author of one paper and coauthor of three). Other top authors are shown in table 4.

The leadership of universities and institutes that have contributed to the creation of these publications are shown in online supplementary appendix 1.

Characteristics of the top-cited articles These articles were created by 252 authors, median 4, minimum 1, maximum 19, IQR 4 and the females in authorship were 102, median 2, minimum 0, maximum 11, IQR 2. The institutes involved were 168, median 2, minimum 1, maximum 17, IQR 3; the countries involved were 67, median 1, minimum 1, maximum 9, IQR 0; and the grants/funds received were 35, median 0, minimum 0, maximum 8, IQR 1. Significant correlations were found between the number of citations and the 2016JIF (Pearson correlation (r)=0.318; p=0.024), and the number of countries (r=0.453; p=0.001). No significant correlations were found between the number of citations and the number of years since publication (r=0.188, p=0.192), the number of authors (r=0.159; p=0.270), the number of females in authorship (r=0.343; p=0.150), the number of institutes involved (r=249; P=0.081) or the number of grants received (r=-0.126; p=0.384).

The altmetric scores The altmetric scores and reports were found for 70% of articles. The total scores were 806, median 5, minimum 0.0, maximum 155, IQR 20. No correlation was found

Open access

between the number of citations and the total altmetric scores (r=0.064; p=0.661). A significant correlation was found between number of citations and altmetric scores for articles published in 2007 and after (n=17, r=0.547, p=0.023). No correlation was found for articles published in 2006 or earlier. Only 38% of the articles had readers on CiteULike (mean 1.6, 95%CI 0.4 to 2.7, median 0, minimum 0, maximum 19, IQR 1; while 62% were read Mendeley (mean 72.7, 95%CI 45.2 to 100.3, median 39.5, minimum 0, maximum 499, IQR 120. The coverage of journal articles by Twitter was 54% (mean 7.8, 95%CI 2.7 to 13.0, median 1.5, minimum 0, maximum 117, IQR 10) followed by blogs 38% (mean 1.2, 95%CI 0.6 to 1.8, median 0, minimum 0, maximum 7, IQR 1, then policy sources 24% (mean 0.38, 95%CI 0.15 to 0.61, median 0.0, minimum 0.0, maximum 3, IQR 0, then Facebook 20% (mean 0.3, 95%CI 0.07 to 0.61, median 0, minimum 0, maximum 6, IQR 0). The Wikipedia was the lowest resource. No significant correlation was found between the number of citations and altmetric scores (r=0.064; p=0.661). The geographical breakdown showed that USA had the highest share, followed by UK. Other countries identified for some articles were Canada, Mexico, Spain, Australia, Spain, Chile, Netherlands, Portugal, Japan, Columbia, Italy, France and Brazil. We looked at journals' webpages of the top-cited articles and those hosting the altmetric metre; we did not find significant correlation between hosting the altmetric metre and recorded altmetric scores.

Multivariate analysis Because of the observed differences in the citations of the top articles in the list compared with those in the bottom of the list, and the variability in the altmetric scores, it was decided to conduct a multivariate analysis comprising the effect of number of authors, number of institutes, number of countries, number of females in authorship, number of grants obtained on the citation scores and altmetric scores (table 5).

Discussion The aims of this study were to identify the highly cited papers in medical professionalism and compare their characteristics and citation analysis with the altmetric scores. Currently, there is a great interest to examine if there is a relationship between altmetric indicators and citation counts. The question remains can we measure the impact of scientific publications by measuring their social density effects?

The study has identified key topics related to medical professionalism including: learning/teaching professionalism and curriculum issues, professional and unprofessional behaviour/disciplinary actions, defining and measuring medical professionalism, response to conflict, social responses and social environment, empathy and moral development, online social networking and professionalism, quality improvement and evidence-based

Azer SA, Azer S. BMJ Open 2019;9:e029433. doi:10.1136/bmjopen-2019-029433

5

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download