Project Knowledge based Search Engine

1
Project Knowledge based Search Engine

2
Project Components

• Web Crawler (9/21 Guest Lecture)
• Text Files
• http//xanadu.cs.sjsu.edu/drtylin/classes/cs267/F
  all2011/data
• http//xanadu.cs.sjsu.edu/drtylin/classes/cs267/p
  roject_20data/
• http //kdd.ics.uci.edu/databases/20newsgroups/20n
  ewsgroups.html

3
Project Components

• Tokenizer
• Portable system Primitive
• System/user View Positive/negative thinking
• Complexity Analysis
• TFIDF
• Stemming (call system dictionary)

4
Project Components

• TFIDF(token_i, document_j)
• tf(ti dj) log Tr/Tr(ti)

5
Important point about Token

• TFIDF(token_i, document_j)
• tf(ti dj) log Tr/Tr(ti)
• Only consider
• (threshold2) gtTr(ti) gt threshold1
• Improve the formula? Discuss come properties
  about this numerical values