Summary of reports on TurnItIn and other plagiarism programs



Summary of reports on TurnItIn and other plagiarism programs

Craig Nathanson

May 24, 2005

The issue of the reliability and validity of Turnitin has been the subject of numerous reports and studies across universities worldwide. By extension, one might argue that the problem of plagiarism and the hope of rectifying this problem via improved ease of detection are global or international problems. This international approach proved rather useful in addressing the various pros and cons of Turnitin for numerous reasons: Not only were a number of perspectives resulting from myriad approaches to assessing Turnitin’s reliability and validity offered, but more notable is the overall consensus reached by these myriad approaches. Moreover, the pros and cons offered and consensus reached by these reports match in large part our experiences in and recommendations for using Turnitin.

When taken together, these reports ultimately favor the use of Turnitin. However, this endorsement is not without notable criticisms and limitations.

Why should we use Turnitin?

Turnitin clearly has several impressive features, which became more pronounced when compared against its major (and lesser) competitors. Probably the biggest advantage to Turnitin is its constantly growing database, comprising literally billions of entries from all over the Internet including previously submitted student essays (e.g., Plagiarism Software Evaluation, 2004). For this reason, Turnitin is at present the only program that allows for the simultaneous analysis and detection of so-called “cut-and-paste” plagiarism (i.e., copying material from the Internet and placing it in one’s essay)[1] and between-student collusion (Bull, Collins, Coughlin, & Sharp, 2001). All other plagiarism detectors are either cut-or-paste detectors or collusion detectors.

When studies were conducted in which identical essays were submitted to various plagiarism programs and the outputs of these programs compared, Turnitin seems to emerge as the most powerful detector, in general (Bull et al., 2001; Sattherwhite & Gerein, n.d.; Tittenberger, 2004). That is, Turnitin was most successful at identifying copied material or essays found on some paper mills.

However, as described below, Turnitin is, in practice, a better cut-and-paste detector than a collusion detector (Barrett, Malcolm, & Lyon, 2003). Yet this apparent con may arguably be a pro. Recall the fundamental problem with multiple-choice cheating detection programs like S-Check: Only cheating dyads are identified thereby transforming the ideally simple discernment of cheater from cheatee into a complicated ordeal. In contrast, Turnitin’s ability to compare every student against external sources -- that is, shifting the “unit of analysis,” so to speak, from dyad to individual -- allows for a clearer index of who the cheaters are.

At a more technical level, Turnitin was found to be very user friendly and simple to use: Short of a computer with a Web browser and Internet access, the user needs little else (Bull et al., 2001; Humes, Stiffler, & Malsed, 2003; Plagiarism Software Evaluation, 2004). That is, unlike many other plagiarism detectors, no additional software is required above and beyond what is likely already found on most computers.

Why should we not use Turnitin?

The criticisms leveled against Turnitin were numerous, although not all were specific to Turnitin. Among the more consistent and general criticisms -- that is, comments about all electronic methods of plagiarism detection -- was that no single plagiarism detector is (1) foolproof, (2) completely comprehensive, or (3) excuses the instructor from further investigation. Of course, these criticisms are not mutually exclusive.

For several reasons, clearly no program is all-powerful or -knowing about whether students actually plagiarized. Aside from technical limitations about whether a program is designed primarily as a cut-and-paste or collusion detector, instances where a given student changed enough of the words to avoid detection will be nearly impossible to identify by program alone (e.g., Barrett et al., 2003; Crisp, 2003; Savage, 2004; turnitin @ fsu, n.d.). Indeed, in several studies where faculty and staff were asked to identify a major clue to suspected plagiarism, abrupt changes in style, including shifts in language and terminology, was repeatedly endorsed heavily (e.g., Crisp, 2003; Savage, 2004). For this reason, along with others described below, the human part of plagiarism detection cannot nor should not be removed.

Despite being the most comprehensive plagiarism detector, Turnitin is nowhere close to being truly completely comprehensive. One domain that is absent from Turnitin’s databank is all material that is either not found online or difficult to access online. That is, hard-copy only books, articles, or any other material that has no online counterpart will not be detected by Turnitin or, for that matter, any other plagiarism detector (Savage, 2003). In addition, Turnitin may not detect material that is found on the Web for only a short period of time (turnitin @ fsu, n.d.). Similarly, Turnitin appears better able to detect papers from free paper mills rather than those originating from pay paper mills (Satterwhite & Gerein, n.d.).

Moreover, unlike those detectors that do not require Web access (i.e., those that compare current essays against those on the user’s computer), Turnitin will not necessarily have access to all papers from previous instances of the course. Put differently and more practically, Turnitin is better suited to cut-and-paste detection rather than collusion detection (Barrett et al., 2003). In fact, a comparison of Turnitin against a program called Ferret noted that Ferret was better able to detect between-student collusion than Turnitin (Barrett et al., 2003). However, these results must be taken with a grain of salt given, in part, the different search algorithms used in each program (Turnitin: at least seven words in a row; Ferret: three words in a row).

In this sense, Turnitin is fraught with false negatives. However, as we discovered, Turnitin is also fraught with false positives (Lau et al., 2005; see also Chandran & Tan, 2002; turnitin @ fsu, n.d.). Indeed, the comment and finding that Turnitin was unable to correctly distinguish quoted or cited material from that which had been truly plagiarized was noted repeatedly, although not unilaterally. Interestingly, while some took this problem as a shortcoming, others saw it as a useful tool for teaching students about proper citation or even identifying those students who are unaware of proper citation techniques (Barrett et al., 2003; Savage, 2003).

The bottom line

The overall message from these various reports appears to be that Turnitin is the best all-around plagiarism detector currently available. The program appears to have improved over time given that some of the criticisms against Turnitin offered in earlier reports, such as those from 2001 (e.g., Bull, 2001), no longer reflect the current Turnitin program. Perhaps, then, the Turnitin people are “listening”, or at least aware, of the international scrutiny it has come under and may be adjusting accordingly. It is also notable than of the many programs compared over several years, Turnitin is one of a handful that has stood the test of time. This persistence does not mean, of course, that the program is without its flaws, many of which, in fact, seem to have remained persistent over the years. Nevertheless, we can and should feel confident about our conclusions and recommendations about the use, pros, and cons of Turnitin.

References

Barrett, R., Malcolm, J., & Lyon, C. (2003). Are we ready for large scale use of plagiarism detection tools? 4th Annual LTSN-ICS Conference, NUI Galway, 79-84.

Bull, J., Collins, C., Coughlin, E., & Sharp, D. (Eds). (2001). Technical review of plagiarism detection software report. Luton, England: University of Luton, Joint Information Systems Committee.

Crisp, G. (2003). Report on the trial of plagiarism detection service: Turnitin. The University of Adelaide, Learning and Teaching Development Unit.

Chandran, R., & Tan, I. (2002). Selecting and implementing plagiarism detection software in a large university. National University of Singapore, Centre for Instructional Technology.

Humes, C., Stiffler, J., & Malsed, M. (2003). Examining anti-plagiarism software: Choosing the right tool - The CMC anti-plagiarism software survey. Claremont, CA: Claremont McKenna College.

Lau, K.S.L, Nathanson, C., Williams, K.M., Westlake, B., & Paulhus, D.L. (2005, May). Investigating academic dishonesty with concrete measures. Presented at the meeting of the American Psychological Society, Los Angeles.

Plagiarism Software Evaluation (May 2004). Retrieved online on May 23, 2005, from .

Satterwhite, R., & Gerein, M. (n.d.). Downloading detectives: Searching for on-line plagiarism. Retrieved May 23, 2005, from Colorado College, Tutt Library Web site: .

Savage, S. (2004, July). Staff and student responses to a trial of Turnitin plagiarism det ection software. Paper presented at the 3rd annual Australian Universities Quality Forum, Adelaide, Australia.

Tittenberger, P. (2004). Technology and plagiarism: Can the problem provide the solution? Paper presented at the 3rd annual Teaching and Learning Symposium at the University of Manitoba, Winnipeg, MB.

turnitin @ fsu (n.d.). Retrieved May 23, 2005, from

Appendix 1: Available abstracts from references

Barrett et al. (2003):

One strategy in the prevention and detection of plagiarism and collusion is to use an automated detection tool. We argue that for consistent treatment of all students, we should be applying these tools to ALL written submissions in a given assignment rather than merely using a detection tool to confirm suspicions that a single text has been plagiarised. In this paper, we describe our investigations into two plagiarism detection tools: the widely used commercial service Turnitin, and an in-house tool, Ferret. We conclude that there are technical and practical problems, first in the large scale use of electronic submission of assignments and then in the further submission of these assignments to a plagiarism detector. Nevertheless, the reporting mechanisms of both tools are fast and easy to use. Turnitin is more useful in detecting plagiarism from web sources, Ferret for detecting collusion within a group of students.

Humes et al. (2003):

Claremont McKenna College has conducted a survey of faculty, students, and relevant support staff to determine satisfaction levels with several anti-plagiarism tools. Students were asked to compare learning products from Glatt, , and Prentice Hall, and high-profile plagiarism education websites at Indiana University, Purdue, and the University of Michigan. Faculty and support staff were asked to compare plagiarism detection services from , Canexus (Eve2), and the University of Virginia (WCopyFind). Students reported a statistically significant preference for learning tools offered by Prentice Hall, Purdue, Michigan, and , over those offered by Glatt and Indiana. With less statistical significance, students preferred the tools available at Prentice Hall overall. Faculty reported a significant preference for detection software through over those offered by the University of Virginia software, and by the Eve2 program.

Parallel to the collection of survey data regarding the software, instructors in the Literature Department have been testing a combination of ’s detection services, and Prentice Hall’s educational software. Results of this program are still preliminary, but indicate a pattern of support for the findings of the survey results. After one semester of employment, the program does appear to be helping faculty to detect student plagiarism, and students seem comfortable both with the detection program, and with efforts to better teach them the rules of academic honesty.

Savage (2004):

An evaluation of student and staff responses to the trial of the plagiarism detection service Turnitin indicates that both students and staff consider Turnitin () to be a useful but limited tool for combating Internet-assisted plagiarism. The evaluation found that Turnitin is thought to be most useful as a deterrent rather than as a solution to Internet-assisted plagiarism, and that it would be wise to concurrently pursue other methods to reduce the problem of plagiarism in higher education. This paper charts the views expressed by staff and students participating in the trial of Turnitin.

Teaching staff reported broad support for the use of Turnitin though expressed reservations about its capacity and the complexity of its use. While students were also generally supportive, the research raised two warnings in regard to student objections. The first, that some students hold objections to Turnitin that relate to legal issues concerning privacy, copyright, and ownership of labour. The second, that senior students are more likely to seriously object than junior students. The latter tendency is particularly noteworthy as it forewarns of a problem for the introduction of Turnitin in postgraduate courses. In conclusion, the paper offers a number of recommendations that respond to student and staff consideration of Turnitin.

Appendix 2: Descriptions of other (i.e., non-Turnitin) plagiarism detectors

Note: I chose what I thought were the best and most comprehensive sources for each program, which is why each programs does not have summaries from the same sources.

Widely used plagiarism detectors:

▪ Essay Verification Engine 2 (Eve2) ().

Availability: Download from internet

Price: USD 20 for unlimited use

from Humes et al. (2003):

The Essay Verification Engine, version 2 (Eve2), is a software package which runs a search similar to that conducted by , but does so on your local machine. The software first sorts through a very wide internet search, then refines its search to perform more direct comparisons between the student essay, and online sources with similar text.

When a search is finished, Eve2 generates a report consisting of a list of URLs pointing to websites that may contain plagiarized materials. If the instructor has prepared a plain-text version of the student essay, Eve2 will also generate a color-coded report displaying suspect text.

from Plagiarism Software Evaluation (2004):

EVE2 is a tool that is installed on the user’s workstation, but connects to the Internet while searching the document. The tool is very user friendly and can process several documents at once. The problem with this tool is that it requires a strong computer with extensive processing power. It requires enormous amounts of processor capacity, which slows down the user’s operating system. Also, the fact that the tool itself is not very stable sometimes leads to runtime errors or reports without an outcome. However, the results that are generated lead to good indications, summing up the sources of the (underlined) matches that are found. The report is an RTF-document, which does not provide the opportunity to directly click on the links that are found.

▪ WCopyFind ()

Availability: Download from internet

Price: Free

from WCopyFind website:

This program examines a collection of document files. It extracts the text portions of those documents and looks through them for matching words in phrases of a specified minimum length. When it finds two files that share enough words in those phrases, WCopyfind generates html report files. These reports contain the document text with the matching phrases underlined.

What WCopyfind can do: It can find documents that share large amounts of text. This result may indicate that one file is a copy or partial copy of the other, or that they are both copies or partial copies of a third document.

What WCopyfind cannot do: It cannot search the web or internet to find matching documents for you. You must specify which documents it compares. Those documents can be local ones—on your computer or a file server—or, with versions 2.1 and higher, web-resident documents that are pointed to by localinternet shortcuts. If you suspect that a particular web page has been copied, you must create an internet shortcut to that page and include this shortcut in the collection of documents that you give to WCopyfind.

from Plagiarism Software Evaluation (2004):

WCopyFind is an extremely useful tool for evaluating papers among students. The simple package is free of charge and directly checks a list of documents for matching parts. Very convenient and useful, indicating the percentage of match between 2 articles. The result report is a HTML-document which underlines the parts that are matched.

▪ CopyCatch ()

Availability: Download from the Internet, but may be for use in the UK only

Price: 30-day free demo, full price unclear aside from “Campus Licencing Cost 10p per student per annum”

from Bull et al. (2001):

Designed to compare textual data across multiple submissions of assignments…CopyCatch detects collusion between students by comparing submitted documents and calculating the proportion of words held in common. CopyCatch does not detect plagiarism from the Internet.

Less used plagiarism detectors:

▪ Ferret ()

Availability: Contact Carolyn Lyon (C.M.Lyon@herts.ac.uk)

Price: Unclear (likely free)

from Barrett et al. (2003, p. 82):

Ferret takes a set of submissions from a cohort of students, and compares each with each. One advantage of using Ferret is that the software is installed on the lecturer’s own PC and there are none of the batch submission problems [like those seen in Turnitin]. Ordinary file copy can be used to gather the files to be examined into a folder.

The software converts each file to the set of all the three word phrases, or trigrams, that it contains. The probability of a significant proportion of trigrams occurring in two independently written texts is very low [Lyon & Malcolm, 2002]. To illustrate this a search on Google for the phrase “temporary utility” produced 1070 hits, but a search on “temporary utility that” produced 6 hits, all of which were versions of the same file.

…The file-pairs are then displayed in a table (Figure 3) ranked a chosen similarity measure, which takes into account the number of matches of common triples scaled by document size. A short piece of work, plagiarised from a much larger one, is not missed because of a small number of matches [Lyon, Malcolm, & Dickerson, 2001].

The user can then select a file pair and display these files side by side. Common trigrams are highlighted in blue. Typically there will be a scattering of these throughout each document, but large sections of blue indicate that plagiarism or collusion should be investigated. Individual trigrams which strike the user as unusual can be selected from the right hand column, as in Figure 4, and the sections in each document containing this trigram will be displayed.

▪ Glatt Plagiarism Screening Program ()

Availability: Software (CD) ordered from Internet

Price: USD 300

from Tittenberger (2004):

(*Note: These passages were taken directly from the Glatt website.)

The Glatt Plagiarism Screening Program is the first comprehensive computer software program specifically designed for detecting plagiarism. The procedure assumes that each person has an individual style of writing, i.e., writing styles are as unique as fingerprints. Furthermore, we know and can remember our own writing style far more accurately than anyone else.

Based on Wilson Taylor's (1953) cloze procedure, the Glatt Plagiarism Screening Program eliminates every fifth word of the suspected student's paper and replaces the words with a standard size blank. The student is asked to supply the missing words. The number of correct responses, the amount of time intervening, and various other factors are considered in assessing the final Plagiarism Probability Score.

from Glatt Plagiarism Screening Program website (n.d.):

The Glatt Plagiarism Screening Program is a valid and sensitive measure for successfully discriminating plagiarists from non-plagiarists. It is especially useful in situations where the original source material cannot be located, e.g., papers purchased from research paper writing mills, fraternity files, papers written by friends, etc. The Test results provide teachers with objective evidence regarding plagiarism guilt or innocence.

The Plagiarism Screening Service provides the scoring for all Tests submitted. Based on a proprietary database, statistical variables, and probability theory, each Plagiarism Test is evaluated individually. The Screening Service determines the final Test score and the resulting plagiarism assessment.

-----------------------

[1] For the sake of simplicity, I have included paper mills, both fee and free, along with cut-and-paste plagiarism. Although paper mills will generate an entire paper rather than specific selections (i.e., “traditional” cut-and-paste), I am considering the two together given each is reliant on the Internet as their source.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download