Efficent Plagiarism Detection Using a Rolling Hash Scheme

1
Efficent Plagiarism Detection Using a Rolling
Hash Scheme

• Todd Neal

2
Manual Plagiarism Detection
3
Select a Sentence Fragment
4
Search with Google
5
Search with Google
6
Repeat
7
Downsides to Manual Detection
- Boring
8
Downsides to Manual Detection
- Boring - Time Consuming
9
Downsides to Manual Detection
20 Students 10 minutes per paper 3
hours 20 minutes
- Boring - Time Consuming
10
Downsides to Manual Detection
- Boring - Time Consuming - Repetitive
11
Downsides to Manual Detection
Select Sentence Copy Switch to Browser Paste Searc
h Repeat
- Boring - Time Consuming - Repetitive
12
Adler32 Hash
32 Bit Integers
Adler32 Function
All Strings of Binary Data
13
Adler32 Hashes
Variable Length String
Adler32
0,4294967295
Now is the time for all good men
3099790154
Now is the time for all good den
3098020673
3099593546
Now is teh time for all good men
14
Adler32 Hashes
Variable Length String
Adler32
0,4294967295
Now is the time for all good men
3099790154
Now is the time for all good den
3098020673
3099593546
Now is teh time for all good men
fztxagbjnyqaeohwlizbfvmprm
2505378581
wewocbyecgurazvhjnkxdjrvli
2505378581
15
Rolling Hashes
Now is the time for all good men to come to the
aid of their country
Hash Function
1234123
16
Rolling Hashes
Now is the time for all good men to come to the
aid of their country
5617241
17
Rolling Hashes
Now is the time for all good men to come to the
aid of their country
18
Rolling Hashes
Now is the time for all good men to come to the
aid of their country
19
Normalizing Text
Forward, the Light Brigade!' Was there a man
dismay'd? Not tho' the soldier knew Some one
had blunder'd Theirs not to make reply, Theirs
not to reason why, Theirs but to do and die Into
the valley of Death Rode the six hundred.
20
Convert to Lowercase
forward, the light brigade!' was there a man
dismay'd? not tho' the soldier knew some one
had blunder'd theirs not to make reply, theirs
not to reason why, theirs but to do and die into
the valley of death rode the six hundred.
21
Remove All Punctuation
forward the light brigade was there a man dismay
d not tho the soldier knew some one had
blunder d theirs not to make reply theirs not to
reason why theirs but to do and die into the
valley of death rode the six hundred
22
Remove Short Words
forward the light brigade was there man
dismay not tho the soldier knew some one had
blunder theirs not make reply theirs not reason
why theirs but do and die into the valley of
death rode the six hundred
23
Chunk into Six Word Groups
forward the light brigade was there man dismay
not tho the soldier knew some one had blunder
theirs not make reply theirs not reason why
theirs but do and die into the valley of death
rode the six hundred
24
Compute Hashes of Word Groups
103892 forward the light brigade was there
192831 man dismay not tho the soldier 032910
knew some one had blunder theirs 492012 not make
reply theirs not reason 582101 why theirs but do
and die 281923 into the valley of death rode
192893 the six hundred
25
Database Storage
26
Searching for Plagiarism
The author then said, Was there a man dismay'd?
Not tho' the soldier knew. It was with this
statement
Normalize
the author then said was there man dismay not tho
the soldier knew was with this statement
27
Searching for Plagiarism
The author then said, Was there a man dismay'd?
Not tho' the soldier knew. It was with this
statement
Normalize
the author then said was there man dismay not tho
the soldier knew was with this statement
28
Searching for Plagiarism
The author then said, Was there a man dismay'd?
Not tho' the soldier knew. It was with this
statement
Normalize
the author then said was there man dismay not tho
the soldier knew was with this statement
29
Searching for Plagiarism
The author then said, Was there a man dismay'd?
Not tho' the soldier knew. It was with this
statement
Normalize
the author then said was there man dismay not tho
the soldier knew was with this statement
30
Searching for Plagiarism
The author then said, Was there a man dismay'd?
Not tho' the soldier knew. It was with this
statement
Normalize
the author then said was there man dismay not tho
the soldier knew was with this statement
31
Searching for Plagiarism
The author then said, Was there a man dismay'd?
Not tho' the soldier knew. It was with this
statement
Normalize
the author then said was there man dismay not tho
the soldier knew was with this statement
32
Searching for Plagiarism
The author then said, Was there a man dismay'd?
Not tho' the soldier knew. It was with this
statement
Normalize
the author then said was there man dismay not tho
the soldier knew was with this statement
33
Searching for Plagiarism
The author then said, Was there a man dismay'd?
Not tho' the soldier knew. It was with this
statement
Normalize
the author then said was there man dismay not tho
the soldier knew was with this statement
34
Results