Report on SafeAssign - Trying to think aloud online



Summary

SafeAssign is a plagiarism detection tool that is fully integrated with Blackboard. It is based on safeassignment from ‘’ a Canadian based company which has been an alternative to Turnitin North American for some time. Blackboard acquired the company and the products and now offers SafeAssign free of charge as an extra Blackboard service.

Given our experience with using Turnitin and its integration with Blackboard, a trial comparing the two services was conducted. A repeat of the blind trial that was used to test Turnitin in 2004 was conducted, using the same student work. A new sample piece of material created by copying work from open internet sources was also created to test the two systems in real time side by side.

Only the sample piece of material solicited matches from the internet, all other work matched internally (i.e. to student papers in the same batch of submissions) only.

A major issue with migrating current staff from Turnitin to SafeAssign is the very different way that percentage scores for non-originality are calculated and interpreted. Turnitin uses a percentage matching score, which takes the best match to a source and counts the number of words that match between the student work and the source expressed as a percentage of the total number of words in the piece. These scores are collated to give a final score, so that a non-originality score of 42% means that of 100 words written by the student, 42 match to another source. However, SafeAssign uses a percentage probability score. This means that scores given for each sentence highlighted as containing matching text is a measure of the probability of these two sentences matching, the overall score given to the piece as a whole is an ‘indicator’ of what percentage matches have been found. This will lead to major confusion for staff in interpretation of these different systems.

SA Results and reports

General comments on reports:

Pros:

• Simple format, clear option to re-submit for rescanning without particular sources

• Offers direct email link - URL for emailing to share with colleagues

• Once colour highlighting turned on, straightforward to see how matching text is placed (layout similar to old classic report of Turnitin)

• Print version is in black and white using numbers and italics to identify plagiarised passages

Cons:

• Matching text initially highlighted all in the same colour – bright blue. To highlight individual sources in ‘highlighter’ style colour, must click on each source

• No option to remove quotes or references

• ‘Side by side’ equivalent view limited, puts both paragraphs on top of one another in pop box, need to compare by eye, without colour or other indication to point out matching.

[pic]

Gross level of copying is shown initially in blue text:

[pic]

Can highlight sources in colour:

[pic]

Note, in this case there were two sources, both highlighted in the same pale blue.

The whole sentence is highlighted, regardless of the percentage matching text, so a whole sentence may be highlighted but only a few words may be matching with it.

For example the whole sentence written by the student was

‘Rett Syndrome affects 1 in 15,000 liveborn girls and it is thought that between 70-95% of all cases are sporadic and so will only occur once in the family and are caused by spontaneous mutation’ and the whole sentence was marked out in blue text or highlighted.

|URL: |/view-paper-display.do?paperId=1648585&inst_id=1636028 |[pic] |

| |Matching: 64% | |

|Uploaded Manuscript: |Rett Syndrome affects 1 in 15,000 liveborn girls and it is thought that between 70-95% of |

| |all cases are sporadic and so will only occur once in the family and are caused by |

| |spontaneous mutation |

|Internet Source: |Between 70-95% of Rett syndrome cases are sporadic therefore will only occur once in the |

| |family |

In fact the only matching words are:

Rett Syndrome affects 1 in 15,000 liveborn girls and it is thought that between 70-95% of all cases are sporadic and so will only occur once in the family and are caused by spontaneous mutation

So of the original sentence of 35 words, 15 words matched the source (not an internet source, but another student paper – this is identified in the source listing, and by the complex URL in the box). I calculate this to be 42% matching, not 64% as stated.

Another example:

|URL: |/view-paper-display.do?paperId=1648585&inst_id=1636028 |[pic] |

| |Matching: 64% | |

|Uploaded Manuscript: |Mutations in MeCP2 are lethal in males because males only have one copy of the X |

| |chromosome so compensation cannot occur like in does in females |

|Internet Source: |Mutations in MeCP2 are lethal in males because unlike females (XX), males have X and Y |

| |hence lack a second copy of the X-chromosome that compensates for a defective one |

Mutations in MeCP2 are lethal in males because males only have one copy of the X chromosome so compensation cannot occur like in does in females

26 words, 17 matching = 65%

Note – safeassign information on their reports states that percentage are not percentage f matching text but probability scores…. From

Sentence matching scores are the percentage probability that two sentences have the same meaning.

This number can also be interpreted as the reciprocal to the probability that these two sentences are similar by chance. For example, a score of 90 percent means that there is a 90 percent probability that these two sentences are the same and a 10 percent probability that they are similar by chance and not because the submitted paper includes content from the existing source (whether or not it is appropriately attributed).

Overall score is an indicator of what percentage of the submitted paper matches existing sources. This score is a warning indicator only and papers should be reviewed to see if the matches are properly attributed.

• Scores below 15 percent: These papers typical include some quotes and few common phrases or blocks of text that match other documents. These papers typically do not require further analysis, as there is no evidence of the possibility of plagiarism in these papers.

 

• Scores between 15 percent and 40 percent: These papers include extensive quoted or paraphrased material or they may include plagiarism. These papers should be reviewed to determine if the matching content is properly attributed.

 

• Scores over 40 percent: There is a very high probability that text in this paper was copied from other sources. These papers include quoted or paraphrased text in excess and should be reviewed for plagiarism.

Test piece created to see what would be detected by both systems:

Sample 1: Contained 6 paragraphs, 5 wholly copied from online sources, 1 original (written by me). The copied sources were:

|Source used |Number of words|Percentage of total |

| | |piece (594 words) |

|1 (20/8/07) |63 |11% |

|2. (20/8/07) |93 |16% |

|3. (20/8/07) |45 |8% |

|4. (story on plagiarism from March 2007) |88 |14% |

|5. (story on plagiarism from 2006) |210 |35% |

|6. original paragraph plus title |97 |16% |

Sources 2-5 were likely to be referenced or re-used elsewhere, or have quotations or matching text elsewhere online.

Turnitin

Results from submission of sample 1: Picked up some plagiarism in all but one of the plagiarised pieces, though limited in places (overall score 35%). Got both wiki pieces correctly.

|Turnitin |Overall score |35% |

|Paragraph number |Matching text found? | |

|1 |3% match (publications) | |

| |"Blackboard Unveils Plagiarism Prevention Service; SafeAssign, Available as | |

| |New Blackboard Beyond Ser", Internet Wire, July 10 2007 Issue | |

|2 |16% match (internet from 16/08/07) |All of paragraph copied |

| |(8-16-07) |was found, plus two |

| | |extra words by chance |

| | |from next para |

|3 |7% match (internet from 15/07/07) |All except first 4 four |

| |(7-15-07) |words found |

|4 |No matches | |

|5 |5% match (publications) |Three different sources |

| |"NRI teen novelist 'sorry' beyond words.", The Times of India, April 27 2006|listed here, total of |

| |Issue | |

| | | |

| |2% match (publications) | |

| |"Columbia U.: Harvard U. writer says copying from Columbia U. alum's book | |

| |'unintentional'.", The America's Intelligence Wire, April 25 2006 Issue | |

| | | |

| |1% match (internet from 13/01/07) | |

| |(1-13-07) | |

| | | |

|6 |No matches | |

Safeassign

Each time the report is created, it seems to be done afresh. Therefore if the report is resubmitted with sources excluded, new sources are found as expected, however there is no way to then re-include the original sources again or return to the original report. Here, I removed the two original sources, which picked up a further 3 sources, then trying to re-include the first 2 sources, resubmitted again, only this time to come up with three sources.

On the first run, Safeassign correctly attributed and highlighting two of five the pieces. However, of the three it missed, one was it’s own wiki and two were from wikipedia (this would be my first target for a place to crawl…).

As I cannot return to the original version, these results are from the third refresh, however they represent the best overall score of all three reports.

|Safeassign |Overall score |68% (note: on third |

| | |refresh) |

|Paragraph number |Matching text found? | |

|1 |None | |

|2 | |Picked up all |

| | |sentences, listed as|

| | |96%, 97% and 100% |

| | |probability of |

| | |matchin |

|3 |None | |

|4 | |Correct source, |

| | |fully picked up but |

| | |noted as 82% (first |

| | |sentence on source |

| | |contained title), |

| | |100%, 100% , 100% |

| | |for each sentence |

|5 | |All correct, all |

| | |matching text |

| | |identified |

|6 |none | |

Note, used the option to save at html, but when re-opened, was very difficult to read. Image files were missing and there was no highlighting.

A question has been raised as to whether papers/ peer review journals can be detected by Safeassign. All testing so far suggests not. The Human Genome Paper in Science, copied form science was picked up but on another open site (eurochromatin) not the science site, as detected by Turnitin.

Technical notes

SA Building block installed on test server 12 July 2007, Bb version Bb 7.2.383.0 in testing.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download