Www.rogerclarke.com



An Exploratory Study of

Information Systems Researcher Impact

Roger Clarke

Principal, Xamax Consultancy Pty Ltd, Canberra

Visiting Professor at the University of N.S.W.,

the University of Hong Kong, and the Australian National University

Roger.Clarke@.au

Abstract

Citation-counts of refereed articles are a potentially valuable measure of the impact of a researcher's work, in the information systems discipline as in many others. This paper reports on a pilot study of the apparent impact of IS researchers, as disclosed by citation-counts of their works. Citation analysis using currently available indexes is found to be fraught with many problems, and must be handled with great care.

I Introduction

Information systems (IS) is a maturing discipline, with a considerable specialist literature, and relationships with reference disciplines that are now fairly stable and well-understood. In a mature discipline, various forms of 'score-keeping' are undertaken. One reason for this is as a means to distinguish among applicants for promotion, and contenders for senior appointments. An increasingly significant application of score-keeping, however, is as a factor in the allocation of resources to support research.

Two intuitively obvious approaches to score-keeping are to count the number of works that a researcher publishes, and moderate it by the time-span over which they were published, the categories of publication (such as books, conference papers and journal articles), and the quality of the venues; and to count the number of citations of a researcher's publications. The weighted count of publications represents a measure of research quality. The citation-count represents a measure of the researcher's impact.

This paper performs an analysis of citations of IS researchers, in order to examine the extent to which currently available data provides satisfactory measures of researcher impact. It is motivated by the concern that, whether or not such analyses are performed by members of the IS discipline, it appears increasingly likely that others will do it for us.

The paper commences by briefly reviewing formal schemes for appraising researcher quality and impact that have been implemented or proposed in several countries in recent years. It then discusses citation analysis, and its hazards. The research objectives and research method are described. The raw scores are presented and issues arising from the analysis are identified and examined.

II Schemes to Assess Researcher Impact

In a number of countries in recent years, there have been endeavours to implement mechanisms for evaluating the impact of individual researchers and research groups. These have generally been an element within a broader activity, particularly relating to the award of block grants to research centres. Examples include the U.K. Research Assessment Exercise (RAE 2001, 2005), which has been operational since 1986, the New Zealand Performance-Based Research Fund (PBRF 2005), and the emergent Australian Research Quality Framework (RQF – DEST 2005), currently in train.

The procedure in the most recent RAE in 2001 was described as follows: "Every higher education institution in the UK may make [submissions] ... Such submissions consist of information about the academic unit being assessed, with details of up to four publications and other research outputs for each member of research-active staff. The assessment panels award a rating on a scale of 1 to 5, according to how much of the work is judged to reach national or international levels of excellence" (RAE 2001, p. 3). The forthcoming 2008 Exercise also involves evaluation of "up to four items ... of research output produced during the publication period (1 January 2001 to 31 December 2007) by each individual named as research active and in post on the census date (31 October 2007)" (RAE 2005, p. 13). Similarly, in the New Zealand scheme, a major part of the 'quality evaluation' depends on assessment of the 'evidence portfolio' prepared by or for each eligible staff-member, which contains up to four 'nominated research outputs' (PBRF 2005, p. 41).

The Australian RQF seeks to measure firstly "the quality of research", which includes "its intrinsic merit and academic impact", and secondly the "broader impact or use of the research", which refers to "the extent to which it is successfully applied in the broader community". The outcomes of the measurements would be rankings, which would be used in decision-making about the allocation of research-funding. The unit of study is 'research groupings', which are to be decided by each institution, but will be subject to considerable constraints in terms of disciplinary focus and minimum size (DEST 2005, 2006).

The design of all three schemes involves lengthy and bureaucratic specifications of how research groupings and individuals are to fill in forms, including definitions of the publications that can be included, large evaluation panels comprising people from disparate disciplines, and lengthy and bureaucratic assessment processes. The RAE in particular has been a very highly resource-intensive activity. A recent report suggests that itthe RAE is shortly to be abandoned (MacLeod 2006), on the basis that it has achieved its aims.

A conventional indicator of research quality is publications in refereed venues, primarily in 'quality, refereed journals', and perhaps secondarily in the stronger refereed conferences. This may be supplemented by the rankings of those journals and conference proceedings. The RQF, for example, appears to have adopted a modified form of the RAE approach, with each research grouping to be allocated a rating, on a scale of 1 (low) to 5 (high), based on the relevant panel's assessment of the "contribution" of the research grouping's "research work", and the "significance" of the area in which the work is undertaken.

Research impact is distinguished from research quality. A potential indicator of research impact is evidence of the use of refereed publications, in particular citations of them. Citations within journals and refereed conferences would indicate the researcher's impact on other researchers; whereas citations within less formal literatures such as professional magazines and newsletters, government reports, and the trade press, might be used as indicators of impact in the broader community. Other indicators of impact include re-publications, translations and inclusion in collections and anthologies. Researcher impact is, however, largely a cumulative matter, rather than being limited to a single publication. The focus might therefore be more broadly on reputation or 'esteem', in which case further indicators could include international appointments, prizes and awards, membership of academies and editorial boards, keynote-speaker invitations, and collaborations with other highly-reputed researchers. The RQF envisages the impact of each research grouping as being assessed against a 3-point scale of High, Moderate and Limited.

Schemes such as the RAE, PBRF and RQF are political in nature, designed to provide a justification for funds-allocation decisions. There are many aspects of such evaluation processes that could influence the accessibility of research support by IS academics. This paper is concerned with whether the use of citation analysis would provide a reasonable basis for evaluating the impact of IS researchers.

III Citation Analysis

"Citations are references to another textual element [relevant to] the citing article. ... In citation analysis, citations are counted from the citing texts. The unit of analysis for citation analysis is the scientific paper" (Leydesdorff 1998). Leydesdorff and others apply 'citation analysis' to the study of cross-references within a literature, in order to document the intellectual structure of a discipline. This paper is concerned with its use for the somewhat different purpose of evaluating the quality and/or impact of works and their authors by means of the references made to them in refereed journal articles.

Authors have cited prior works for centuries. Gradually, the extent to which a work was cited in subsequent literature emerged as an indicator of the work's influence, which in turn implied significance of the author. Whether the influence of work or author was of the nature of notability or notoriety was, and remains, generally ignored by citation analysis. Every citation counts equally, always provided that it is in a work recognised by whoever is doing the counting.

Attempts to formally measure the quality and/or impact of works, and of their authors, on the basis of the number of citations that they gather, is a fairly recent phenomenon. Indeed, the maintenance of citation indices appears to date only to about 1960, with the establishment of the Science Citation Index (SCI), associated with Garfield (1964). Considerable progress in the field of 'bibliometrics' has ensued.

The advent of the open, public Internet, particularly since the Web exploded in 1993, has stimulated many developments. Individual journal publishing companies such as Elsevier, Blackwell, Kluwer, Springer and Taylor & Francis have developed automated cross-linkage services, at least within their own journal-sets. Meanwhile, the open access movement is endeavouring to produce not only an open eLibrary of refereed works, but also full and transparent cross-referencing within the literature. A leading project in the area was the Open Citation (OpCit) project in 1999-2002. An outgrowth from the OpCit project, the Citebase prototype, was referred to as 'Google for the refereed literature'. It took little time for Google Inc. to discover the possibility of a lucrative new channel. It launched Google Scholar in late 2004.

These need to be regarded as initial forays. All still fall far short of the vision of the electronic library, which was conceived by Vannevar Bush (1945), and articulated by Ted Nelson in the 1960s as 'hypertext'. As outlined in Nelson's never-completed Project Xanadu, the electronic library would include the feature of 'transclusion', that is to say that quotations are included by precise citing of the source, rather than by replicating some part of the content of the source.

In the complex, inter-locking and ambiguous field of learned publishing, it is to be expected that citation analysis will gave rise to a degree of contention. Dissatisfaction with it as a means of evaluating the quality and impact of works and of researchers has a long history. See, for example, Hauffe (1994), MacRoberts & MacRoberts (1997) and Adam (2002).

Within the IS discipline, there is a long history of attention being paid to citations. The primary references appear to be Culnan (1978, 1986), Culnan & Swanson (1986), Culnan (1987), Cheon et al. (1992), Cooper et al. (1993), Eom et al. (1993), Holsapple et al. (1993), Eom (1996), Walstrom & Leonard (2000), Vessey et al. (2002), Schlogl (2003) and Katerattanakul & Han (2003). That is a fairly short list of articles. Moreover, a General Search on the ISI database shows that the most cited among them (Holsapple et al.) has only accumulated a count of 24 (or, if a deep and well-informed analysis is conducted, using the ISI Cited Ref facility, then the most cited appears to be achieved by combining several counts for a total of 61 for Culnan 1987), and on Google Scholar the largest citation-count appears as 63, for each of Culnan (1986) and Culnan (1987). As will be shown, these are not insignificant counts, but they are not large ones either; and the difficulties and ambiguities involved in generating them represent a mini-case study in the application of citation analysis.

The primary purposes of the research reported in the papers listed above have been to develop an understanding of the intellectual structure of the IS discipline, of patterns of development within the discipline, and of the dependence of IS on reference disciplines. In some cases, the impact of particular journals has been in focus (in particular Cooper et al. 1993, Holsapple et al. 1993 and Katerattanakul & Han 2003). In one instance (Walstrom & Leonard 2000), highly-cited articles were the primary concern. The literature search conducted as part of this project did not identify any articles in which the primary focus of the citation analysis was on individual researchers or research groupings.

A number of deficiencies in the use of citation analysis for this purpose are apparent from the outset. In the course of presenting the analysis, more will emerge, and a consolidated list is provided at the end of the paper. Despite these deficiencies, 'score-keeping' is increasingly being applied to the allocation of research resources. The work reported on here is accordingly justified at least as much by political pragmatics as for intellectual reasons.

IV The Research Purposes and Method

Because little prior research has been conducted in this specific area, the essential purpose was to provide insights into the effectiveness of citation analysis applied to individual IS researchers.

Because of the vagaries of databases that are organised primarily on names, considerable depth of knowledge of individuals active in IS research is needed in order to achieve a reasonable degree of accuracy in the analysis. The analysis accordingly focusses primarily on researchers active in the author's country of long-term residence, Australia. This was appropriate not only as a means of achieving reasonable data quality, but also because the scale was manageable (with c. 700 members of the IS discipline, but no more than 150 with well-established publishing records over the last decade), and the issue of researcher impact assessment is current. Balance was sought by complementing the Australian analysis by also assessing the impact of leading researchers in North America and Europe.

One important insight that was sought was the extent to which the publishing venues that are generally regarded by IS researchers as carrying quality refereed articles are reflected in the databases on which citation analysis can be performed. Rather than simply tabulating citation-counts, the research accordingly commenced by establishing a list of relevant journals and conference proceedings.

Consideration was given to generating lists of highly-cited articles, and working down the list, accumulating total counts for each individual. Preliminary experimentation showed, however, that this was impractical. It was concluded that a 'census' approach was to be preferred.

The following method was adopted:

• a list of significant refereed publishing venues was developed;

• available citation indices were identified and evaluated;

• the Thomson / ISI Social Science Citation Index (SSCI) and Science Citation Index (SCI) were assessed to establish the extent to which the relevant refereed publishing venues were covered;

• an extensive set of names of Australian IS researchers was established;

• a small set of names of leading international IS researchers was established;

• statistics were extracted from the Thomson / ISI SSCI and SCI for those names;

• statistics were extracted from Google Scholar;

• further investigations were performed, in order to enhance understanding of the quality of the counts arising from the Thomson/ISI and Google citation analyses.

Further detail on each of these steps is now provided.

The set of venues was developed by reference to the now well-established literature on IS journals and their rankings, for which a bibliography is provided at Saunders (2005). Consideration was given to the lists and rankings there, including the specific rankings used by several universities, and available on that site. Details of individual journals were checked in the most comprehensive of the several collections, which is maintained at Deakin University (Lamp 2005).

The author believes that the set selected, listed in Exhibit 1, represents a fairly conventional view of the key refereed journals on the management side of the IS discipline. It significantly under-represents those journals that are in the reference discpline of computer science, and at the intersection between IS and computer science. The reason is that these fields are themselves highly diverse, and a very large number of venues would need to be considered, and many included, in which the heavy majority of IS researchers neither read nor publish. Less conventionally, the list separates a few 'AA-rated' journals, and divides the remainder into general, specialist and regional journals. There is, needless to say, ample scope for debate on all of these matters.

Consideration could be given to supplementing the journals with the major refereed conferences. In this author's experience, ICIS is widely regarded as approaching AA status, ECIS as a generic A, and AMCIS, PACIS and ACIS may be considered by some as being generic A as well. These are accessible and indexed in the Association for Information Systems' AIS eLibrary, in the case of ICIS since it commenced in 1980, ECIS since 2000, and ACIS since 2002. However, no attempt was made to extend the analysis undertaken in this work to even the strongest of the refereed conferences.

A survey was conducted of available citation indices. It was clear that Thomson / ISI needed to be included, because it is well-known and would be very likely to be used by evaluators. Others considered included:

• Elsevier's Scopus;

• the Computer Science Bibliography, at Universität Trier;

• CiteSeer Scientific Literature Digital Library, at Penn State University;

• The Collection of Computer Science Bibliographies, at Universität Karlsruhe;

• Google Scholar.

Thomson Scientific, previously known as the Institute for Scientific Information (ISI), publishes the Science Citation Index (SCI) and the Social Science Citation Index (SSCI). In January 2006, its site stated that SCI indexes 6,496 'journals' (although some are proceedings), and that SSCI indexes 1,857 'journals'. The company's policies in relation to inclusion (and hence exclusion) of venues is explained at . An essay on the topic is at Thomson (2005), but access is unreliable.

Elsevier's Scopus has only been operational since late 2004. The next three are computer science indexes adjacent to IS, and at the time the research was conducted the last on the list was still experimental. The decision was taken to utilise the Thomson/ISI SCI and SSCI Citation Indices, and to extract comparable data from Google Scholar. A more comprehensive project would be likely to add Scopus into the mix.

Exhibit 1 shows the set of journals that were selected, together with indications as to whether the journal is included in the Thomson/ISI SCI or SSCI Citation Indices. The final column shows the inferences drawn by the author regarding the extent of the Thomson/ISI coverage of the journal.

Exhibit 1: Refereed Venues Selected

|Journal Name |Journal Abbrev.|SSCI |SCI |Issues Included |

| | | | | |

|AA Journals (3) | | | | |

|Information Systems Research |ISR |Y | |Only from 1994, Vol. 4 ? |

|Journal of Management Information Systems |JMIS |Y | |Only from 1999, Vol. 16! |

|Management Information Systems Quarterly |MISQ |Y | |Only from 1984, Vol. 8! |

| | | | | |

|AA Journals in the Major Reference Disciplines (4) | | | | |

|Communications of the ACM (Research Articles only) |CACM | |Y |From 1958, Vol. 1 |

|Management Science |MS |Y | |From 1955, Vol. 1 |

|Academy of Management Journal |AoMJ |Y | |From 1958, Vol. 1 |

|Organization Science |OS |Y | |From 1990, Vol. 1? |

| | | | | |

|A Journals – General (9) | | | | |

|Communications of the AIS (Peer Reviewed Articles only) |CAIS | | |None! |

|Database |Data Base | |Y |Only from 1982 Vol. 14 ? |

|Information Systems Frontiers |ISF | |Y |Only from 2001, Vol. 3 |

|Information Systems Journal |ISJ |Y | |Only from 1995, Vol. 5 |

|Information & Management |I&M |Y | |Only from 1983, Vol. 6 |

|Journal of the AIS |JAIS | | |None! |

|Journal of Information Systems |JIS | | |None! |

|Journal of Information Technology |JIT |Y | |Only 18 articles |

|Wirtschaftsinformatik |WI | |Y |Only from 1990, Vol. 32 |

| | | | | |

|A Journals – Specialist (15) | | | | |

|Decision Support Systems |DSS | |Y |Only from 1985, Vol. 1 |

|Electronic Markets |EM | | |None |

|International Journal of Electronic Commerce |IJEC |Y | |From 1996, Vol. 1 |

|Information & Organization |I&O | | |None |

|Information Systems Management |ISM |Y | |Only from 1994, Vol. 11 |

|Information Technology & People |IT&P | | |None! |

|Journal of End User Computing |JEUC | | |None |

|Journal of Global Information Management |JGIM | | |None |

|Journal of Information Systems Education |JISE | | |None |

|Journal of Information Systems Management |JISM | | |None |

|Journal of Management Systems |JMS | | |None |

|Journal of Organizational and End User Computing |JOEUC | | |None |

|Journal of Organizational Computing and Electronic Commerce |JOCEC | | |None |

|Journal of Strategic Information Systems |JSIS | |Y |From 1992, Vol. 1 ? |

|The Information Society |TIS |Y | |Only from 1997, Vol. 13! - ? |

| | | | | |

|A Journals – Regional (3) | | | | |

|Australian Journal of Information Systems |AJIS | | |None |

|European Journal of Information Systems |EJIS | |Y |Only from 1995, Vol. 4 |

|Scandinavian Journal of Information Systems |SJIS | | |None |

When assembling a list of individuals active in the field, it is challenging to be sure of being comprehensive. People enter and depart from the discipline. There are overlaps with the Computer Science discipline, with various management disciplines, and with the IS profession. When determining the set of IS academics in a particular country, immigration, emigration and expatriates create definitional challenges.

The set of names of Australian academics was developed based on the author's experience in the field since about 1970, but in particular by reference to the Australian IS Academics Directory (Clarke 1988), the Australasian IS Academics Directory (Clarke 1991), the Asia Pacific Directory of IS Researchers (Gable & Clarke 1994 and 1996), and the ISWorld Directory (1997-). Arbitrary decisions were taken as to who was an expatriate Australian, and how long immigrants needed to be active in Australia to be treated for the purposes of this analysis as being Australian. Data was sought in relation to about 100 leading Australian IS researchers, plus 4 well-known and successful expatriates.

Data was extracted from the SCI and SSCI citation indices over several days in late January 2006. Access was gained through the ANU Library Reverse Proxy, by means of the company's 'Web of Science' offering. Both sets of searches were restricted to 1978-2006, across all Citation Indices (Science Citation Index – SCI-Expanded, Social Sciences Citation Index – SSCI and Arts & Humanities Citation Index – A&HCI). Multiple name-spellings and initials were checked, and where doubt arose were also cross-checked with the AIS eLibrary.

In order to provide comparison with leaders in the IS discipline outside Australia, the same process was applied to a small sample of leading overseas researchers. The selection was not intended to be random. It relied on this author's longstanding involvement in the field, and his knowledge of the literature and the individuals concerned. Relatively uncommon surnames were used, in order to reduce the likelihood of pollution through the conflation of articles by multiple academics.

Google Scholar was then searched for a sub-set of the 30 apparently most-highly-cited Australian researchers. Supplementary research was then undertaken within the Thomson/ISI database. These elements were performed in respectively early and late April 2006. During the 3 months between the two rounds of analysis using ISI, the database and hence the citation-counts of course continued to accumulate.

5. Thomson/ISI

This section presents the results of the pilot citation analyses of the Thomson/ISI collections.

Method

The data collected for each of the authors was the apparent count of articles, and the apparent total count of citations. The Thomson/ISI site provides several search-techniques. The search-technique used was the 'General Search'. In each case, the search-terms used were combined with , with the date-range restricted to 1978 onwards. In the case of Australian researchers with common names, this was qualified with . Each list generated by a search was inspected, in order to remove articles that appeared to be by people other than the person being targetted.

The ISI General Search provides a list of articles by all authors sharing the name in question, provided that they were published in venues that are in the ISI database. For each such article, a citation-count is provided, which is defined as the number of other articles in the ISI database that cite it. (It should be noted that although the term 'citation' is consistently used by all concerned, the analysis appears to actually utilise the entries in the reference list provided with the article, rather than the citations that appear within the text of the article).

This measure was selected primarily because it is the most obvious, and hence the most likely to be used by someone evaluating the apparent contribution of a particular academic or group of academics. It is also the most constrictive definition available, and hence could be argued to be the most appropriate to use when evaluating applicants for the most senior appointments, and when evaluating the reputations of the most prominent groups of researchers. Other possible approaches are discussed at the end of this section.

Citation-Counts for Australian IS Researchers

Exhibit 2 provides data for the highest-scoring Australian IS academics, using an arbitrary cut-off of 40 total citations. This resulted in the inclusion in the table of the 4 expatriates and 19 local researchers. For each person, the data shown is the total citation-count, the number of papers in the index, and the citation-count for that person's most-cited paper.

CAVEAT: For reasons that are discussed progressively through the remainder of this paper, there are strong arguments for not utilising the data in this table, and for not utilising the ISI 'General Search', as a basis for assessing the impact of individual articles or individual researchers.

Exhibit 2: Citation-Counts for Australian IS Researchers

| |Citation Count |Number of |Largest |

| | |Articles |Per-Article Count |

|Expatriates | | | |

|Iris Vessey (as I.) |601 |35 |111 |

|Rick Watson (as R.T.) |485 |28 |78 |

|Ted Stohr (as E.A.) |217 |12 |108 |

|Peter Weill (as P.) |178 |13 |47 |

|Locals | | | |

|Marcus O'Connnor (as M.) |354 |31 |66 |

|Ron Weber (as R.) |328 |22 |38 |

|Philip Yetton (as P. and P.W.) |270 |26 |57 |

|Michael Lawrence (as M.) |208 |27 |66 |

|Michael Vitale (as M. and M.R.) |179 |14 |107 |

|Ross Jeffery (as D.R., and as R.) |172 |28 |38 |

|Marianne Broadbent (as M.) |166 |24 |36 |

|Bob Edmundson (as R.H.) |86 |4 |66 |

|Graham Low (as G. and G |83 |19 |38 |

|Peter Seddon (as P. and P.B.) |70 |6 |60 |

|June Verner (as J. and J.M.) |65 |18 |22 |

|Graeme Shanks (as G.) |61 |13 |14 |

|Paula Swatman (P.M |53 |9 |29 |

|Kit Dampney (as C and C.N.G.) |47 |16 |25 |

|John D'Ambra (as J.) |47 |7 |16 |

|Roger Clarke (as R. and R.A.) |44 |22 |16 |

|Michael Rosemann (as M.) |41 |14 |18 |

|Graham Winley (as G.) |40 |13 |13 |

|Chris Sauer (as C.) |40 |18 |13 |

Citation-Counts for Leading International IS Academics

Some form of benchmark is required, against which this measure of each local researcher's impact can be compared. The appropriate comparison is not with people in other disciplines, because their data reflects different numbers of academics, and different numbers of journals, which are differentially indexed. Exhibit 3 shows the same data as for Australian IS academics, but for well-known leaders in the discipline in North America and Europe.

CAVEAT: For reasons that are discussed progressively through the remainder of this paper, there are strong arguments for not utilising the data in this table, and for not utilising the ISI 'General Search', as a basis for assessing the impact of individual articles or individual researchers.

Exhibit 3: Citation-Counts for Leading International IS Academics

| |Citation Count |Number of |Largest |

| | |Articles |Per-Article Count |

|North American | | | |

|Lynne Markus (as M.L.) |1,335 |39 |296 |

|Izak Benbasat (as I.) |1,281 |71 |155 |

|Dan Robey (as D.) |1,247 |45 |202 |

|Sirkka Jarvenpaa (as S.L.) |960 |40 |107 |

|Detmar Straub (as D. and D.W.) |873 |49 |160 |

|Rudy Hirschheim (as R.) |600 |44 |107 |

|Gordon Davis (as G.B.) |428 |48 |125 |

|Peter Keen (as P.G.W.) |427 |21 |188 |

|Eph(raim) McLean (as E. and E.R.) |119 |30 |31 |

|Europeans | | | |

|Kalle Lyytinen (as K.) |458 |55 |107 |

|Leslie Willcocks (as L.) |231 |42 |28 |

|Bob Galliers (as R.D. and R.) |173 |34 |25 |

|Guy Fitzgerald (as G.) |121 |50 |38 |

|Enid Mumford (as E.) |103 |21 |42 |

|Claudio Ciborra (as C. and C.U.) |60 |13 |26 |

|Frank Land (as F.) |59 |18 |19 |

|David Avison (as D.) |51 |37 |44 |

|Ron Stamper (as R. and R.K.) |32 |13 |16 |

|Niels Bjorn-Andersen (as N.) |3 |3 |2 |

Quality Assessment

An important aspect of this exploratory study was the question as to how effectively the available databases reflected the extent to which the individuals concerned were actually cited. Several tests were applied. The first was the ISI database's coverage of the selected publishing venues. Nothing was found on the Thomson/ISI site that declared which Issues of journals were in the database, and it was necessary to conduct experiments in order to infer the extent of the coverage. The examination disclosed that:

• the short list of AA journals in reference disciplines appeared to be included in their entirety;

• the list of AA journals within the IS discipline itself were covered only from quite late in their histories. MISQ was the first and for a long time the primary AA journal, yet articles prior to Volume 8 are omitted. ISR's first 3 volumes are omitted, and only the most recent 6 of the 16 volumes of JMIS are included;

• the list of generalist A journals was also severely inadequate. The first 5 volumes of I&M are missing, as are the first 3 refereed volumes of Database (1979-1981). Others were adopted only some years into their lives. JIS and JAIS are entirely omitted, and JIT is very sparsely represented;

• the specialist A journals are even more poorly covered. This includes both IS sub-disciplines, and journals oriented towards particular research domains (which not only publish research using diverse methods, but also accept articles written from a wide range of disciplinary perspectives – which in the author's view is the appropriate way to define the much-abused expression 'multi-discipinary research');

• of the regional A journals, only EJIS is represented, but omitting the first 3 volumes.

These are not the only reasons why citation analysis based on the Thomson/ISI collection leaves a great deal to be desired. The following problems were detected:

• variability of discoverability depending on the initial(s) used for the author, and on the spelling of the author's surname. Considerable effort was necessary in multiple cases among the c. 130 analyses performed, and the accuracy of the measures is difficult to gauge;

• serious deficiencies in the Thomson/ISO collection. This results in misleadingly low counts. The primary causes are the exclusion of:

• a significant number of important journals;

• almost all conference proceedings;

• all articles from some journals prior to an ISI-selected start-date;

• all articles from some journals after some ISI-selected finish-date;

• some articles that appear to be within the ISI criteria but are nonetheless missing; and

• all books;

• the inappropriate inclusion of some categories of material, which inflates the item-counts and citation-counts of some authors, in particular:

• non-refereed items in listed journals (particularly editorials, but also book reviews). These doubled the article-count for some academics, although it inflated their citation-count to a much smaller extent;

• some non-academic outlets which are edited but are not refereed academic venues in the manner in which the mainstream journals are (in particular, Sloan Mngt. Rev., Harv. Bus. Rev.);

• all articles in a journal that has switched its emphasis away from strongly academic papers (e.g. Commun. ACM); and

• a few (mostly old) purely trade outlets (e.g. Datamation, I-S Analyzer).

Two further quality tests were undertaken. In the European list, I was surprised by the low counts for Ron Stamper and Niels Bjorn-Andersen. I report later on the results of an investigation of their citations as indicated by Google Scholar. In the North American list, I was surprised by the low results for Eph McLean, and consequently I examined his list more closely. An article that, in my view at that time, could have been expected to be among the most highly-cited in the discipline (Delone & McLean's 'Information Systems Success: The Quest for the Dependent Variable') did not appear in Eph McLean's list. It was published in ISR, which would have been AA-rated by most in the discipline from its inception. But ISR is indexed only from 5, 1 (March 1994), whereas the Delone & McLean paper appeared in 3, 1 (March 1992). Using the 'Cited Ref Search' provide by ISI also fails to detect it under McLean E.R., but does detect a single citation when the search is performed on 'INFORM SYST RES' and '1992'. ISI finds it with , with 7 variants of the citation, all of which are misleadingly foreshortened to 'INFORMATION SYSTEMS'. These disclose the very substantial sum of 448 hits. A later section reports on the citations of this paper as indicated by Google Scholar.

The final quality test applied was a comprehensive assessment of the inclusion and exclusion of the refereed works of a single author. A highly convenient sample of 1 was selected: the author of this paper. In the author's view, this is legitimate, for a number of reasons. This was exploratory research, confronted by many challenges, not least the problems of false inclusions and exclusions, especially in the case of researchers with common names. I have a very common surname, I have a substantial publications record, those publications are scattered across a wide range of topics and venues, I have had some impact, and I am well-positioned to ensure accuracy in this particular analysis because I know all of my own works. The outcome of the analysis was as follows:

• of the author's 36 refereed journal articles to the end of 2005:

- 13 were in the SCI and SSCI combined (34%);

- 1 (3%) was in a journal claimed by Thomson to be indexed (JIT), but could not be located in the index;

- 9 (25% were in 4 important IS journals that are listed in Exhibit 1 but are not indexed by Thomson. (They are JSIS, TIS, IT&P and CAIS);

- 2 were in Computer Science and Engineering journals not indexed by Thomson;

- 4 (11%) were in Computers & Law Journals not indexed by Thomson. (Law is especially badly supported by the citation indices);

- 2 were in Commerce and Economics Journals not indexed by Thomson;

- 5 were in other journals not indexed by Thomson (in journals in Public Policy, Ethics, and the Humanities);

• of the author's 27 refereed conference papers to the end of 2005:

- 2 were in the SCI and SSCI combined (7.5%);

• a further 7 papers were in the index (albeit with a total of only 3 citations), which were, in the author's opinion, at best lightly refereed. These included notes from panel sessions, a guest editorial, a bibliography, and a lengthy, edited but unrefereed Opinion in MISQ.

The many deficiencies in the Thomson/ISI database identified from these tests result in differential bias against researchers. Some of the deficiencies would appear to affect all disciplines (e.g. the apparent incompleteness of journals that are claimed to be indexed, and the failure to differentiate refereed from unrefereed content in at least some journals). Many others would appear to mainly affect relatively new disciplines, such as the long delay before journals are included, and refusals to include some journals even when representations are made. When applied to IS researchers, the extent of the under-reporting is not easily predictable, although it does appear to depend heavily on the individual's areas of interest and preferred venues:

• there is good coverage of well-established reference disciplines;

• there is coverage of a little more than half of the most important generalist IS journals but in many cases only since about the mid-to-late 1990s;

• there is very poor coverage of:

• specialisations that have emerged during the last decade;

• journals that focus on research-domains and that accordingly accept papers written from varying disciplinary perspectives; and

• the interfaces between IS and law, and IS and public policy.

These problems suggest that comparisons within a discipline are problematic, and that comparisons between disciplines would be extraordinarily difficult. The following appear to be important factors, with observations about how IS compares with disciplines generally:

• the size of the population of researchers publishing in the discipline. (IS would appear to have a population of c. 2,000-4,000 publishing researchers worldwide, of which c. 100-150 are in Australia. This is small in comparison with some disciplines);

• the size of the population of researchers who subsequently publish in the topic-area(s) in which the researcher has published. (IS comprises a vast array of topics, and hence many topics are addressed by only a very small number of people. For example, citation analysis in IS attracts few researchers, and hence this paper, and its predecessors, will attract only very small numbers of citations in refereed journals. In addition, IS is driven by rapidly-changing technology, topic-areas are volatile, and hence many papers are left behind as fashion sweeps onwards);

• the extent to which subsequent researchers are influenced by, and cite, the researcher's work. (Anecdotally, there appears to be a tendency among IS researchers to over-cite papers that have appeared in high-status venues, and to under-cite more relevant papers that have appeared in lower-status venues);

• norms within the discipline, in particular:

• the extent to which minor contributors are reflected in the list of authors. (The norm in IS is for very short lists of authors, most commonly either one or two, with only those included who made a very substantial contribution to the paper; whereas some disciplines list many authors per paper, resulting in apparently long lists of publications per author, and bloated citation-counts for each author);

• the extent to which the researcher works within a well-established pattern of 'normal science', in which case many, short publications can be generated quickly. (In IS, this is anything but the norm. Some maturation may be feasible, however. For example, Clarke (2006) proposed the codification of 'AIS research technique practice guides', to enable quicker writing and reviewing of articles reporting on research that uses mainstream technqiues);

• the extent to which refereed journals are the primary venue for research reports. (In IS this is generally the case; but some sub-disciplines and some individuals focus more heavily on scholarly books; some sub-disciplines and some individuals produce software; and some sub-disciplines and some individuals publish significant contributions outside the refereed literature, e.g. in government publications);

• the degree of inherent bias in favour of works in the English language, and against works in other languages. (This is fairly common in IS, with Wirtschaftsinformatik the only recognised journal whose primary language is other than in English, and only a few French-language articles published to date in CAIS);

• the extent to which the publishing venues used in the discipline are reflected, and accurately reflected, in the database on which the citation analysis is performed. (In ISI, IS is poorly represented overall, and some sub-disciplines and some research-domains are very poorly represented indeed).

Alternative Methods

This sub-section considers alternative ways in which the Thomson/ISI database could be applied to the purpose. Two other categories of search are available. One is 'Advanced Search', which provides the ability to apply Boolean operators on combinations of fields. This would be valuable when conducting tightly focussed searches. If a common evaluation method were able to be defined, it may be feasible to use 'Advanced Search' to apply it. It was not considered relevant to the pilot study conducted here.

The other category of search is called 'Cited Ref Search'. This is looser than the 'General Search' used in this pilot study. It lists all articles by all authors sharing the name in question, that have been cited by any article in any publishing venue in the database. It therefore includes articles that were published in venues that are not in the ISI database. It is arguable that this scope of citation-count represents a more appropriate measure of the reputation, worth or impact of an individual researcher or group of researchers. The measure counts citations in articles in the ISI database, rather than citations in articles in the ISI database of articles that are also in the ISI database.

In order to test the likely impact of applying this alternative approach to IS researchers, a small number of researchers in each category were selected, and the study repeated. The design of the search-facility creates challenges because it makes only a small number of parameters available. For example, it does not enable restriction to . In addition, very little information is provided for each hit, the sequence provided is alphabetical by short journal title, and common names generate in excess of 1,000 hits. A further problem is that there is enormous variation in citation styles, and in the accuracy of the data in reference lists. This results in the appearance of there being many more articles than there actually are – many articles were found to have 2 or 3 entries, and the most variants detected during the analysis were 5 and 7.

Appendix 1 provides the results of this part of the study. All of the researchers for whom the analysis was undertaken had substantial numbers of articles that were cited in the ISI database, but that were not themselves in the ISI database. In some cases, relaxing the criterion resulted in an increase in the citation-count by 20-30%, but in others in an increase by factors of as much as 5. The data in Exhibit 5 enables the following inferences to be drawn:

• researchers with very high citation-counts for articles published in ISI venues also tend to have substantial numbers of citation-counts within ISI for articles that they have published in non-ISI venues (e.g. within the small, non-random sample, Sirkka Jarvenpaa, and to a lesser extent Iris Vessey);

• researchers with an instrumentalist orientation, much of whose material is relevant to the IS profession and management, may have very large citation-counts within ISI for articles that they have published in non-ISI venues (e.g. Peter Keen);

• some researchers whose within-ISI citation-counts are not all that high have published articles outside ISI journals which are fairly strongly-cited by other articles published in ISI venues (e.g. Ron Stamper and the author of this paper have 4 times as many ISI citations for articles published outside ISI venues as within);

• researchers who came into IS from an established social science or management discipline may have only limited additional ISI citations of papers in non-ISI venues, because most of their publications are already counted by the 'General Search' (e.g. Philip Yetton).

Dependence on the General Search alone provides only a restricted measure of the reputation, worth or impact of an academic. Moreover, it may give a seriously misleading impression of the impact of researchers who publish in non-ISI venues such as books, and journals targetted at the IS profession and management. To the extent that citation analysis of Thomson/ISI data is used for evaluation purposes, a method needs to be carefully designed that reflects the objectives of the analysis. In addition, it would be essential for an opportunity to be provided for the individuals concerned to consider the results and submit supplementary information.

VI Google Scholar

Google Scholar is still an experimental service. From a bibliometric perspective, it is crude, because it is based on brute-force free-text analysis, without recourse to metadata, and without any systematic approach to testing venues for quality before including them. On the other hand, it has the advantages of substantial reach, ready accessibility, and popularity. It is inevitable that it will be used as a basis for citation analysis, and therefore important that it be compared against the more formal Thomson/ISI database.

The analysis presents considerable challenges. The method adopted was to conduct searches using the names of a sample of the researchers whose data from Thomson/ISI appears in Exhibits 2 and 3. Searches generate long lists of hits, each of which is either an article indexed by Google, or is inferred from a citation in an article indexed by Google. (As is the case with ISI, it appears that the 'citations' counted are actually the entries in the reference-list to each article, rather than the citations within the article's text). Each article has a citation-count shown, inferred from the index; and the hits appear to be sorted in approximate sequence of apparent citation-count, most first. (Only limited documentation appears to be available; and the service, although producing interesting and even valuable results, appears to be anything but stable).

Various approaches had to be experimented with, in order to generate useful data. (From a researcher's perspective, Google's search facilities are among the weakest offered by search-engines). For example, 'A' and 'I' are stop-words in the indexing logic, and hence searches for names including those initials required careful construction. The common words 'is' and 'it' are also stopwords, and hence it is difficult to use the relevant expressions 'IS' and 'IT' to restrict the hits to something more manageable. Search-terms of the form appeared to generate the most useful results, and this format was generally applied. Experiments with searching based on article-titles gave rise to other challenges, in particular the need to develop an even richer starting-point for the analysis: a comprehensive list of article-titles for each researcher.

A very small sample was used, because of the resource-intensity involved, and the experimental nature of the procedure. An intentionally biassed sample was selected, firstly because the intention was to test the comparability of the two sets of data, both in terms of coverage and citation-counts, and secondly in order to avoid conflation among multiple authors and the omission of entries. To simplify matters, only the first 10 articles for each author were gathered (roughly, but not necessarily reliably, those with the highest-citation-count).

The intensity of the 'multiple authors with the same name' problem is highly varied. For many of the researchers for whom data is presented below, there was no evident conflation with others, i.e. their Top-10 appeared on the first page of 10 entries displayed by Google Scholar. For a few, it was necessary to skip some papers, and move to the second or even third page. To reach Eph McLean's 10th-ranked paper, it was necessary to check 60 titles, and to reach Ron Weber's 10th, 190 titles were inspected. A test on my own, rather common name on the above basis resulted in 12,700 hits. To reach the 10th-most-cited, it was necessary to inspect 558 entries. The challenges involved in this kind of analysis are underlined by the fact that those first 558 entries included a moderate number of papers by other R. Clarkes on topics and in literatures that are at least adjacent to topics I have published on and venues I have published in. These could have easily been mistakenly assigned to me by a researcher who lacked a detailed knowledge of my publications list. Similarly, false-negatives would have easily arisen. There are many researchers with common names, and hence accurate citation analysis based on name alone is difficult to achieve.

Appendix 2 contains tables (numbered 6A through 6G) which show comparisons between the raw Google citation-count and the Thomson/ISI citation-count. Results are shown for seven researchers. In each case, careful comparison was necessary, to ensure accurate matching of the articles uncovered by Google against those disclosed by Thomson/ISI.

A number of conclusions can be drawn, some of which are intuitive, some less so; and some of which are disturbing. In particular:

• Google generally lists more articles for each author than Thomson/ISI. This is to be expected because Thomson/ISI is selective and 'exclusive' of publishing venues, whereas Google is highly inclusive;

• where both collections include the same article, Google generally provides a higher citation-count than Thomson/ISI. Considering researchers' top few articles as a whole, the Thomson/ISI represented as high as 70% of the count by Google for the same articles (Iris Vessey – 4 instances, and Philip Yetton – 7 instances) and as low as 28% (Paula Swatman – 3 instances) and 17% (the author of this paper – also 3 instances). A higher Google count is an expected pattern, for the reason discussed immediately above; but the degree of variability suggests that a reliable multiplier may be difficult to interpolate;

• in a small proportion of instances, on the other hand, the Thomson/ISI count exceeds the Google count (e.g. Exhibit 6C – Philip Yetton). These circumstances appear to arise in respect of relatively old articles. Because of the youth of the Web, material originating prior to about 2000, and especially prior to 1995, is under-represented. Hence citations that arose prior to that period will also tend to be under-represented;

• Thomson/ISI excludes citations of books and of conference papers, but Google includes at least some. The under-reporting in the Thomson/ISI citation count is highly significant, not merely in the humanities (where learned books are a primary form of publication), but also for some IS researchers. An important example is a very highly cited book by Philip Yetton (Exhibit 6C);

• several instances exist in which a researcher's most highly cited article according to Google, which was published in an A journal as listed in Exhibit 2, was not in ISI. These include Ron Weber, who has 102 for an unlisted JIS article (Exhibit 6B), Paula Swatman with 73, 61 and 43 (Exhibit 6E); and the author of this paper with 60 and 47 (Exhibit 6F);

• multiple instances exist of apparent error in the Thomson/ISI collection, including the omission of articles in AA-rated journals. Iris Vessey (Exhibit 6A) has 92 citations of an unlisted ISR article, 86 and 48 of two unlisted CACM articles, and 41 of an unlisted IEEE Software article; and Guy Gable has 102 citations of an unlisted EJIS article (Exhibit 6G). There are two other instances of missing articles with Google citation-counts in the twenties. If the Thomson/ISI citation-count were to be treated as authoritative, such errors would result in very substantial under-scoring for some researchers.

An important further test was the measure generated for Delone & McLean's 'Information Systems Success: The Quest for the Dependent Variable'. The test was undertaken by keying the search-term into Google Scholar, and critically considering the results. The test was performed twice, in early April 2006 and late April 2006. The results differed in ways that suggested that, during this period, Google was actively working on the manner in which its software counts citations and presents hits. The later, apparently better organised counts are used here. The analysis was complicated by the following:

• there has been a considerable subsequent literature that applies, extends and re-visits the topic addressed by the article, and sets of words that appear in the original article are used in the titles of many of the later papers. This highlights a limitation of the 'brute force' text-analysis approach of Google in comparison with a more refined metadata-analysis approach;

• some links were broken; and

• many of the hits require particular conditions to be fulfilled by the searcher in order to gain access to sufficient information to test the validity of the putative citation.

The raw results comprised 824 citations for the main entry (and a total of 832 hits). Based on a limited pseudo-random sample from the first 40, many appeared to be indeed attributable to the paper. This is a citation-count of a very high order. An indication of this is that the largest Thomson/ISI citation-count for an IS paper that was located during this research was 296, for a paper in CACM by Lynne Markus. In Google, that paper scored 472. The Delone & McLean scored 75% more Google-citations than the Google-citation score of the highest-ranked IS paper so far located in the Thomson/ISI database. In short, it appears that, as a result of what is most likely data capture error, Thomson/ISI denies the authors the benefit of being seen to have co-authored what may be the single most-referenced paper in the entire discipline.

A further test was undertaken to compare the citation-counts generated by ISI and Google Scholar for two European researchers whose ISI counts were lower than had been anticipated:

• Ron Stamper (as R. and R.K.) generated only 32 citations from 13 articles on ISI, whereas on Google Scholar the count was 511 citations of 36 articles (the largest single count being 60), plus 64 citations of 1 book, totalling 575 citations. (The scan that was undertaken found 37 relevant entries among the first 100 hits of a total of 7,970 hits in all, and doubtless somewhat under-counts);

• Niels Bjorn-Andersen (by surname only) generated the tiny total of 3 citations from 3 articles on ISI, whereas on Google Scholar the count was 145 citations from 13 papers (the largest single count being 72), plus 22 citations of 2 books, totalling 167 citations. (The scan that was undertaken found 49 relevant entries among the first 100 of a total of 534 hits in all, and probably somewhat under-counts).

Some perspective on what 100 citations means in the IS discipline can be gained from an assessment of the total citation-count of the top 10 items that are discovered in response to some terms of considerable popularity in recent years. The terms were not selected in any systematic manner. The count of the apparently 10th-most-cited article containing the expression is also highlighted, because it provides an indication of the depth of the heavily-cited literature using that term:

• "technology acceptance model" – 1957 (370, 349, 174, 190, 196, 117, 89, 173, 194, 105)

• "soft systems methodology" – 1474 (1038 – but a book, 79, 123, 49, 32, 32, 34, 34, 31, 22)

• "outsourcing" – 1329 (238 – but a book, 104, 164, 105, 154, 121, 128, 123, 91, 101)

• "information systems success" – 1208 (722, 89, 46, 35, 58, 43, 47, 41, 31, 96)

• "structural equation modelling" – 959 (162, 279, 124, 53, 92, 82, 17, 62, 47, 41)

• "B2B" – 950 (350, 166, 57, 72, 71, 56, 60, 54, 43, 21)

• "key issues in information systems management" – 929 (205, 101, 155, 216, 22, 165, 16, 31, 14, 4)

• "strategic alignment" – 711 (342, 107, 52, 44, 38, 34, 28, 26, 25, 15)

• "information systems failure" – 546 (139, 209, 46, 25, 18, 12, 52, 11, 8, 26)

• "citation analysis" AND "information systems" – 380 (63, 38, 63, 54, 47, 11, 53, 30, 13, 8)

• "reintermediation" – 224 (76, 40, 22, 24, 15, 14, 13, 8, 7, 5)

• "B2C" – 186 (60, 22, 16, 11, 7, 7, 6, 6, 5, 46)

Another aspect of interest is the delay-factor before citations begin to accumulate. Some insight was gained from an informal sampling of recent MISQ articles, supplemented by searches for the last few years' titles of this author's own refereed works. A rule of thumb appears to be that there is a delay of 6 months before any citations are detected by Google, and of 18 months before any significant citation-count is apparent. The delay is rather longer on Thomson/ISI. This is to be expected, because of the inclusion of edited and lightly-refereed venues in Google, which have a shorter review-and-publication cycle than Thomson/ISI's journals, most of which are heavily refereed.

In summary, citation-counts above about 75-100 on Google Scholar appear to indicate a high-impact article, and above 40-50 a significant-impact article. Appropriate threshholds on Thomson/ISI would appear to be somewhat less than half of those on Google. These threshholds are of course specific to IS, and other levels would be likely to be appropriate in other disciplines, and in other countries.

VII Implications

Reputation is a highly multi-dimensional construct, and reduction to a score is morally dubious, intellectually unsatisfying, and economically and practically counter-productive. On the other hand, the impact of a researcher's publications, as measured by the frequency with which they are cited by other authors, is a factor that an assessment of reputation would ignore at its peril.

The research presented in this paper has demonstrated that there are enormous problems to be confronted in applying currently available databases to the purpose. Exhibit 4 summarises them.

Exhibit 4: Deficiencies in Citation Analysis as a Means of Assessing Researcher Impact

Citation Databases, particularly Thomson/ISI

• exclusion of refereed publishing venues. Relevant examples include:

- journals that are highly-rated by researchers, but are not indexed. This appears to involves strong biases against:

- relatively new disciplines

- journals that are oriented towards research-domains, and hence are not closely related to, nor fostered by, any particular discipline

- refereed conference proceedings, particularly those that are highly-rated among practitioners within a discipline or research-domain

- scholarly books

• exclusion of refereed articles even though they appear in an included refereed publishing venue. For example:

- journals may be indexed only after a certain date, or (less commonly) only before a certain date

- some articles may be included but not others (for reasons other than whether or not they were refereed)

• almost no coverage of law journals, despite the acknowledged high quality of many of them, presumably because of the tradition in that professionally-oriented discipline of quite limited formal refereeing

• patchy and unclear inclusion and exclusion of venues that are not refereed, but have standing of some other kind, such as text-books, and bulletins and newsletters of disciplinary, professional and industry associations. Although the quality of the citing works needs to be differentiated, counts in less formal publications are relevant to the extent to which a researcher's work is applied in the broader community

• inclusion of items that appear within an appropriate refereed venue, but that are not themselves refereed. This applies in particular to editorials, notes, book reviews, and letters

• non-transparency of venue inclusions and exclusions and item inclusions and exclusions

• data capture errors, including complete omission of items, and mis-spellings of journal titles, authors' names and keywords

• search capabilities limited by user-interface constraints or arbitrary design decisions about searchable fields

The Articles

• citation errors in articles, including mis-spellings of journal titles, authors' names and keywords, and incorrect and missing volume and issue numbers, page-numbers, and dates

• name-variants. Variations in spelling and transliteration, changes of name (particularly by women, on marriage), and variations in the completeness of the set of initials. These give rise to further risks of conflation and omission

• variations between disciplines in relation to the inclusion of minor contributors in author-lists and the approaches adopted to citations

Contextual Factors

• non-uniqueness of author-name. This gives rise to the risks of:

- conflation of the publications of multiple individuals

- the overlooking of publications of an individual, lost in the flood of publications by one or more other people with closely similar names

• time-delay in the emergence of a citation-count for even the most successful articles

• time-delay in the emergence of any impact at all. Many important ideas are 'ahead of their time', and their impact comes after the originator's career is over, and in many cases after their death. This gives rise to the risk that some researchers, particularly those focussed on 'pure research', will be starved of research funding throughout their careers

• bandwagon effects. These arise as researchers flock to a new 'idea in good standing' or fashion, or sculpt their focus or methods to those currently preferred by funding agencies or their assessors

• self-citation. This is in principle legitimate, but is a potential source of inflated measures

• 'courtesy citation' of works of limited relevance, typically authored by the author's supervisor or colleagues

• existence of several over-lapping services, in particular Thomson/ISI Web of Knowledge, but also Elsevier's Scopus, open access prototypes such as CiteSeer, and most recently Google Scholar

Despite these problems, the scope exists for citation analysis to be used constructively, as an aid in evaluation, with care taken to avoid or overcome the problems. For example, members of search and selection committees responsible for the appointment of key staff-members apply the technique, but incorporate subtlety, sophistication and insight into their work. This could be summed up with the epithet: 'if you can't be good, be careful'. One approach would be to apply both ISI and Google Scholar; and another would be for citation analysis to be undertaken openly, with the opportunity for researchers to submit additional information needed to fill out the picture.

Where impact assessment is institutionalised, however, as in such schemes as the U.K. RAE, the N.Z. PBRF and the Australian RQF, the likelihood of subtlety, sophistication and insight being applied is remote. These are political mechanisms aimed at focussing research funding on a small proportion of research centres within a small proportion of institutions. They are mass-production exercises, and are subject to heavily bureaucratic processes and definitions. Citation analysis used in such processes is inevitably largely mechanical.

Simplistic application of raw citation-counts to evaluate the performance of individual researchers and of research groupings would disadvantage some disciplines, many research groupings, and many individual researchers. The IS discipline is highly exposed to the risk of simplistic application of citation analysis. For the many reasons identified above, citation-counts will suggest that most IS researchers fall short of the criteria demanded for the higher rankings.

The IS discipline in at least some countries is confronted by the spectre of reduced access to research funding, as a result of unreasonable application of citation analysis. Options available include representations to Thomson in order to achieve back-loading into ISI of large volumes of missing publications, and the development of the discipline's own comprehensive database of publication titles and reference lists (possibly building on the AIS eLibrary). At the very least, it appears to be necessary for a defensive measure to be adopted, in the form of a set of guidelines explaining how to utilise citation analysis to evaluate the impacts of articles, individuals and research groupings, including an exhortation that results never be applied without the affected individuals having the opportunity to review the data and submit supplementary information.

VIII Conclusions

There may be a world in which the electronic library envisioned by Bush and Nelson has come into existence, in which all citations are counted, and in which venues are subject to a weighting scheme that reflects differences among disciplines and research-domains, and that is subject to progressive adaptation.

Back in the real world, however, the electronic library is deficient in a great many ways. It is fragmented and very poorly cross-linked. And the interests of copyright-owners (including discipline associations but particularly the for-profit corporations that publish and exercise control over the majority of journals) are currently in building more and more substantial barriers rather than working towards integration. It remains to be seen whether that will be broken down by the communitarian open access movement, or by the new generation of corporations spear-headed by Google.

In the current and near-future contexts, citation analysis is a very blunt weapon, which should be applied only with great care, but which appears very likely to harm the interests of the less politically powerful disciplines such as IS.

Acknowledgements

This paper has benefitted from comments from several colleagues on an earlier draft, in particular from Peter Seddon of the University of Melbourne, who identified a methodological flaw that needed to be addressed. Responsibility for the work rests, however, entirely with the author.

References

In all cases in which URLs are shown, the web-page was last accessed in early April 2006.

Adam D. (2002) 'Citation analysis: The counting house' Nature 415 (2002) 726-729

ARC (2006) 'Research Fields, Courses and Disciplines Classification (RFCD)' Australian Research Council, undated, apparently of 21 February 2006, at

Bush V. (1945) 'As We May Think' The Atlantic Monthly. July 1945, at

Cheon M.J., Lee C.C. & Grover V. (1992) 'Research in MIS - Points Of Work and Reference - A Replication and Extension of the Culnan and Swanson Study' Data Base 23, 2 (September 1992) 21-29

Clarke R. (Ed.) (1988) 'Australian Information Systems Academics: 1988/89 Directory' Australian National University, November 1988

Clarke R. (Ed.) (1991) 'Australasian Information Systems Academics: 1991 Directory' Australian National University, April 1991

Clarke R. (2006) 'Plagiarism by Academics: More Complex Than It Seems' J. Assoc. Infor. Syst. 7, 2 (February 2006)Cooper R.B., Blair D. & Pao M. (1993) 'Communicating MIS research: a citation study of journal influence' Infor. Processing & Mngt 29, 1 (Jan.–Feb. 1993) 113 - 127  Culnan M.J. (1978) 'Analysis of Information Usage Patterns of Academics and Practitioners in Computer Field - Citation Analysis of a National Conference Proceedings' Infor. Processing & Mngt 14, 6 (1978) 395-404

Culnan M.J. (1986) 'The Intellectual-Development of Management-Information-Systems, 1972-1982 - A Cocitation Analysis' Mngt Sci. 32, 2 (February 1986) 156-172

Culnan M.J. (1987) 'Mapping the Intellectual Structure of MIS, 1980-1985: A Co-Citation Analysis' MIS Qtly 11, 3 (September 1987) 341-353

Culnan M.J. & Swanson E.B. (1986) 'Research In Management-Information-Systems, 1980-1984 - Points Of Work And Reference' MIS Qtly 10, 3 (September 1986) 289-302

DEST (2005) 'Research Quality Framework: Assessing the quality and impact of research in Australia: Final Advice on the Preferred RQF Model' Department of Education, Science & Training, December 2005 , at

DEST (2006) 'Research Quality Framework Guidelines Scoping Workshop : Workshop Summary' Department of Education, Science & Training, 9 February 2006 , at

Eom S.B. (1996) 'Mapping The Intellectual Structure Of Research In Decision Support Systems Through Author Cocitation Analysis (1971-1993)' Decision Support Systems 16, 4 (April 1996) 315-338

Eom S.B., Lee S.M. & Kim J.K. (1993) 'The Intellectual Structure Of Decision-Support Systems (1971-1989)' Decision Support Systems 10, 1 (July 1993) 19-35

Gable G. & Clarke R. (Eds.) (1994) 'Asia Pacific Directory of Information Systems Researchers: 1994' National University of Singapore, 1994

Gable G. & Clarke R. (Eds.) (1996) 'Asia Pacific Directory of Information Systems Researchers: 1996' National University of Singapore, 1996

Garfield E. (1964) ''Science Citation Index – A New Dimension in Indexing' Science 144, 3619 (8 May 1964) 649-654 , at

Hauffe H. (1994) 'Is Citation Analysis a Tool for Evaluation of Scientific Contributions?' Proc. 13th Winterworkshop on Biochemical and Clinical Aspects of Pteridines, St.Christoph/Arlberg, 25 February 1994, at

Holsapple C.W., Johnson L.E., Manakyan H. & Tanner J. (1993) 'A Citation Analysis Of Business Computing Research Journals' Information & Management 25, 5 (November 1993) 231-244

Katerattanakul P. & Han B. (2003) 'Are European IS journals under-rated? an answer based on citation analysis' Euro. J. Infor. Syst. 12, 1 (March 2003) 60-71

Lamp J. (2005) 'The Index of Information Systems Journals', Deakin University, version of 16 August 2005, at

Leydesdorff L. (1998) 'Theories of Citation?' Scientometrics 43, 1 (1998) 5-25, at

MacLeod D. (2006) 'Research exercise to be scrapped' The Guardian, 22 March 2006, at

MacRoberts M.H. & MacRoberts B.R. (1997) 'Citation content analysis of a botany journal' J. Amer. Soc. for Infor. Sci. 48 (1997) 274-275

PBRF (2005) 'Performance-Based Research Fund', N.Z. Tertiary Education Commission, July 2005, at

Perkel J.M. (2005) 'The Future of Citation Analysis' The Scientist 19, 20 (2005) 24

RAE (2001) 'A guide to the 2001 Research Assessment Exercise', U.K. Department for Employment and Learning, apparently undated, at

RAE (2005) 'Guidance on submissions' Department for Employment and Learning, RAE 03/2005, June 2005, at

Saunders C. (2005) 'Bibliography of MIS Journals Citations', Association for Information Systems, undated but apparently of 2005, at

Schlogl C. (2003) 'Mapping the intellectual structure of information management' Wirtschaftsinformatik 45, 1 (February 2003) 7-16

Vessey I., Ramesh V. & Glass R.L. (2002) 'Research in information systems: An empirical study of diversity in the discipline and its journals' J. Mngt Infor. Syst. 19, 2 (Fall 2002) 129-174

Walstrom K.A. & Leonard L.N.K. (2000) 'Citation classics from the information systems literature' Infor. & Mngt 38, 2 (December 2000) 59-72

Appendix 1: Thomson/ISI Cited Ref Search

This Appendix provides comparisons of results obtained using the ISI General Search and ISI Cited Ref Search. A small number of academics were selected, from among expatriates, local researchers, North Americans and Europeans. All but one were selected because of their relatively uncommon names, in order to ease the difficulties of undertaking the searches and thereby achieve reasonable quality in the data, and no significance should be inferred from inclusion in or exclusion from this short list. One, this author, was selected because, for researchers with common names, full knowledge of the publications list makes it much easier to ensure accuracy.

The first three columns of the table show the number of citations for each author of articles that are in the ISI database, together with the count of those articles, and the largest citation-count found. (This data should correspond with that for the same researcher in Exhibits 2 and 3, but in practice there are many small variations, some caused by the 3-month gap between the studies that gave rise to those two tables and the study that gave rise to Exhibit 5). The next three columns show the same data for articles by the author that are not in the ISI database. The final two columns show the sum of the two Citation-Count columns, and the apparent Expansion Factor, computed by dividing the Total Citations by the Citation-Count for articles in the ISI database.

Exhibit 5: Thomson/ISI Cited Ref Search

| | | -------- In| | | ------ Not| | | |

| | |ISI Database| | |in ISI | | | |

| | |-------- | | |Database | | | |

| | | | | |------ | | | |

|Researcher |Citation-Coun|Article-Coun|Highest |Citation-Co|Article-Cou|Highest |Total |Expansion |

| |t |t |Cite-Count |unt |nt |Cite-Count |Citations |Factor |

|Sirkka Jarvenpaa (as S.L.) |973 |34 |110 |575 |118 |88 |1548 |1.6 |

|Peter Keen (as P.G.W.) |425 |14 |190 |1625 |325 |463 |2050 |4.8 |

|Eph McLean (as E. and E.R.)|132 |14 |41 |84 |29 |42 |216 |1.6 |

|(Eph Mclean, corrected) |132 |14 |41 |532 |20 |448 |664 |5 |

| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |

|David Avison (as D.) |66 |9 |46 |99 |37 |16 |165 |2.5 |

|Ron Stamper (as R. and |59 |13 |16 |255 |149 |19 |314 |5.3 |

|R.K.) | | | | | | | | |

|Frank Land (as F.) |74 |16 |19 |161 |88 |25 |235 |3.2 |

| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |

|Iris Vessey (as I.) |622 |32 |114 |186 |52 |76 |808 |1.3 |

| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |

|Philip Yetton (as P. and |278 |20 |59 |65 |51 |6 |343 |1.2 |

|P.W.) | | | | | | | | |

|Peter Seddon (as P. and |81 |4 |69 |67 |30 |22 |148 |1.8 |

|P.B.) | | | | | | | | |

|Graeme Shanks (as G.) |66 |10 |15 |51 |32 |7 |117 |1.8 |

|Paula Swatman (as P.M |43 |4 |31 |102 |39 |23 |145 |3.4 |

|Roger Clarke (as R. and |41 |11 |17 |176 |131 |8 |217 |5.3 |

|R.A.) | | | | | | | | |

|Guy Gable (as G.G.) |35 |4 |29 |73 |24 |39 |108 |3.1 |

Appendix 2: Thomson/ISI cf. Google Comparisons for Selected Researchers

This Appendix provides detailed comparisons of results for both ISI and Google Scholar. Seven Australian academics were selected, from among both expatriates and local researchers. All but one were selected because of their relatively uncommon names, in order to ease the difficulties of undertaking the searches and thereby achieve reasonable quality in the data, and no significance should be inferred from inclusion in or exclusion from this short list. One, this author, was selected because, for researchers with common names, full knowledge of the publications list makes it much easier to ensure accuracy.

In each of the following tables:

• the first column shows the citation-counts for the top 10 works found for the relevant author by Google Scholar;

• the second column shows the count for each of those works as disclosed by Thomson/ISI;

• the third column shows the nature of the venue in which the work was published, including the name of the journal where the work is not indexed by Thomson/ISI;

• in the bottom row, the total Thomson/ISI count shows the sum of the citation-counts for those 10 works. The figure in brackets shows the total for that author as displayed in Exhibit 2;

• in the bottom row, the total Google count for the 10 works is shown. The second figure is the total for the sub-set of papers that is also indexed by Thomson/ISI, and is therefore comparable with the first figure in the Thomson/ISI total.

Exhibit 6A: Thomson/ISI cf. Google – Iris Vessey

|Google Count |Thomson Count |Venue |

|145 |111 |Journal |

|92 |Unindexed (!!) |Journal (ISR) |

|88 |83 |Journal |

|86 |Unindexed (!!) |Journal (CACM) |

|56 |26 |Journal |

|52 |Unindexed |Conference (ICIS) |

|52 |Unindexed |Journal (IJMMS) |

|48 |Unindexed (!!) |Journal (CACM) |

|41 |Unindexed (!!) |Journal (IEEE Software) |

|31 |9 |Journal |

|691 or 320 |229 (of 601) |Totals |

Exhibit 6B: Thomson/ISI cf. Google – Ron Weber

|Google Count |Thomson Count |Venue |

|125 |38 |Journal |

|102 |Unindexed |Journal (JIS) |

|106 |30 |Journal |

|87 |36 |Journal |

|72 |26 |Journal (Commentary) |

|65 |20 |Journal (Commentary) |

|45 |Unindexed |Book |

|34 |Unindexed |Journal (JIS) |

|34 |22 |Journal |

|31 |24 |Journal |

|701 or 520 |196 (of 328) |Totals |

Exhibit 6C: Thomson/ISI cf. Google – Philip Yetton

|Google Count |Thomson Count |Venue |

|302 |Unindexed |Book |

|55 |11 |Journal |

|42 |12 |Journal |

|32 |12 |Journal |

|31 |34 |Journal (1988) |

|27 |Unindexed |Book |

|26 |57 |Journal (1982) |

|20 |23 |Book |

|18 |6 |Journal (1985) |

|18 |Unindexed |Government Report |

|571 or 224 |155 (of 270) |Totals |

Exhibit 6D: Thomson/ISI cf. Google – Peter Seddon

|Google Count |Thomson Count |Venue |

|133 |60 |Journal |

|47 |Unindexed |Journal (CAIS) |

|43 |Unindexed |Conference (ICIS) |

|33 |Unindexed |Conference |

|22 |Unindexed (!) |Journal (DB, 2002) |

|24 |2 |Journal (I&M, 1991) |

|18 |2 |Journal |

|18 |Unindexed |Journal (JIS) |

|13 |Unindexed |Conference (ECIS) |

|9 |0 |Journal (JIT, Editorial) |

|360 or 184 |64 (of 70) |Totals |

Exhibit 6E: Thomson/ISI cf. Google – Paula Swatman

|Google Count |Thomson Count |Venue |

|117 |29 |Journal |

|73 |Unindexed |Journal (Int'l Mkting Rev) |

|61 |Unindexed |Journal (TIS) |

|43 |Unindexed |Journal (JSIS) |

|29 |Unindexed (!) |Journal (IJEC) |

|26 |12 |Journal |

|26 |Unindexed |Journal (JIS) |

|24 |6 |Journal |

|22 |Unindexed |Conference |

|20 |Unindexed |Journal (EM) |

|441 or 167 |47 (of 53) |Totals |

Exhibit 6F: Thomson/ISI cf. Google – Roger Clarke

|Position |Google Count |Thomson Count |Venue |

|57 |81 |14 |Journal |

|59 |85 |16 |Journal |

|102 |60 |Unindexed |Journal (IT&P) |

|148 |47 |Unindexed |Journal (TIS) |

|253 |33 |Unindexed |Conference |

|325 |28 |Unindexed |Conference |

|373 |25 |3 |Journal |

|407 |23 |Unindexed |Journal (JSIS) |

|539 |18 |Unindexed |Journal |

|558 |17 |Unindexed |Conference |

| |417 or 191 |33 (of 44) |Totals |

Exhibit 6G: Thomson/ISI cf. Google – Guy Gable

|Google Count |Thomson Count |Venue |

|102 |Unindexed (!) |Journal (EJIS, 1994) |

|56 |Unindexed (!) |Journal (ISF, 2000) |

|40 |Unindexed |Journal (JGIM) |

|27 |6 |Journal (MS) |

|27 |Unindexed |Conference |

|24 |23 |Journal (I&M, 1991) |

|23 |Unindexed |Conference |

|14 |Unindexed |Conference |

|13 |Unindexed |Conference |

|10 |Unindexed |Conference |

|336 or 51 |29 (of 34) |Totals |

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download