Title: Interactive Focused Crawler: Setup, Monitoring and Control Through User Feedback
1Interactive Focused Crawler Setup, Monitoring
and Control Through User Feedback
M. Tech. Project First Stage Presentation Roger
Menezes
under the guidance of Prof. Soumen Chakrabarti
2Focus Crawler
- Hypertext resource discovery agent
- Selectively seeks out pages relevant to
pre-defined set of topics - Motivation
- General purpose crawlers too much to do,
expensive - Topic specific search engines, portals
-
3Basic Framework
- Pages link to related pages
- Train classifier to recognize relevant topics
- Maintain a crawl frontier
- Prioritize all neighbors v of a fetched page u
based on the score of u - Pick an unvisited link v of the highest priority
4Existing Architecture
- All links may not link to related pages
- Web pages are noisy and multi-topic
- Neighborhood textual content
- Learn two classifiers
- Baseline scores each fetched page
- Apprentice scores each link in the crawl
frontier
5Existing Architecture..contd
6Existing Architecture..contd
- Baseline classifier trained over a standard
taxonomy - Need for a standard taxonomy
- Characterization of the negative class
- Users example URLs routed to categories in
taxonomy - User modifies and confirms selection
- Interest topics may not cleanly fit into the
taxonomy - Bad User Interface
7Existing Architecture..contd
- Resource Discovery
- Lack of monitoring tools
- Topic Specificity
- Incorporate User Feedback
- Control of the crawler
8Project Aim
- Build the focus crawler as an application for the
end-user - Better user interface
- Obviating the standard taxonomy
- Increasing user interactivity
- More control over the application
- Extensions to crawl frontier monitoring
9Focus Crawler Proposed Phases
- Setting Up
- Learning the Baseline Classifier
- Monitoring the crawl
10Setting Up
- Compensate for Standard Taxonomy
- Learn classifier from only positive labeled
examples and mixed unlabeled examples - Need sufficient positive examples
- Identify good positive features
- Web pages are noisy and multi-topic
- Segment into different coherent regions
11Gathering Related Pages
- User supplies keywords/phrases and URLs
- Fetch pages and segment into coherent pages
- User selects relevant segments
- Extract keywords/phrases
- Query search engine and gather more URLs
- Repeatedly run Related Pages algorithm and
collect more pages
12Block Diagram
Extract/ Analyze
URLs
Fetch
More words/phrases
Approve
Words/phrases
Search Engine
URLs
User
Fetch
Approve
Related Pages Algo.
13User Interface
- Screen prompting user to select relevant regions
in the segmented web pages
14User Interface
Screen displaying the user the list of
keywords/phrases to be used for Search Engine
querying.
15User Interface
Screen displaying the user the list of top ranked
URLs gathered from the search engine.
16Learning the classifier
- Training using labeled positive examples
- PEBL (Positive Example Based Learning)
- User tests the classifier and provides feedback
- User ignorant of the standard taxonomy
17User Interface
Screen during the time the classifier is being
trained. The progress bar at the right top corner
indicates the remaining time.
18Monitoring the Crawl Frontier
- Two modes
- Resource Discovery
- Maintenance
- Topic Specific websites based on different
criteria. - Maintaining a window of top 10 relevant pages.
- Maximum link depth traversed by the crawler
19Monitoring..contd.
- Running HITS and returning the top 10 hubs
- Links for the different web pages crawled.
- A graph depicting the average relevance of pages
fetched against time. - Retraining of the apprentice.
- Start/Stop the crawler
20Review of Existing Algorithms
- PEBL
- Related Pages Algorithm
- Page Segmentation Algorithm
21PEBL
- Exploits marginal property of SVMs
- Algorithm(Positive P, Unlabeled M)
- Distinguish strong positive features
- Separate the most negative points N
- Learn SVM based on P and N
- Classify other examples in M using the SVM
- Add all negatively classified points to N
- Go to step 3, if points were added to N
22PEBL..contd.
- Can use the same random sample M for different
topics - May not necessarily use SVMs but use similar
technique for other learning algorithm
23Related Pages Algorithm
- Technique discussed in Dean and Henzinger
- Connectivity based algorithms
- Cocitation Algorithm
- Siblings are related
- Degree of Cocitation Number of common parents
two web pages have. - Two pages u and v having high degree of
cocitation are related.
24Related Pages Algorithmcontd
- Companion Algorithm
- Given a URL u, build a vicinity graph for it.
- B parents and BF children of them
- F children and FB parents of them
- Contract duplicates and near duplicates
- Compute edge weights
- Run HITS and return the top 10 authorities
25Related Pages Algorithmcontd
- Extensions
- Transitive degree of cocitation
- Page v having high degree of cocitation with u
- A page w having high degree of cocitation with v
but not with u. - Use of DOM content analysis along with the above
algorithms - Need an extension for more than one URLs
26Page Segmentation Algorithm
- Similar algorithm to VIPS (VIsion-based Page
Segmentation) - Web pages are multi-topic
- Segments pages using visual cues like color,
images, font sizes, etc. - Proceeds in three phases
- Outputs a content structure tree whose nodes are
regions in page having coherent content
27VIPS
- Visual Block Extraction
- A Visual Block is a single unit of semantic
coherent content. - Given a DOM node check for coherency
- Iteratively replace node by child till
permissible coherency achieved - Visual Separator Detection
- Separator for the visual block retrieved
- Differences in font sizes, color, presence of HR
tag.
28VIPS..contd.
- Content Structure Construction
- Merge blocks on either side of separator into
regions - Check for the regions, if the Degree of coherency
is below a threshold - If not, repeat the process for the region
29Referencescontd
- Krishna Bharat and Monika R. Henzinger. Improved
algorithms for topic distillation in a
hyperlinked environment. In Proceedings of
SIGIR-98, 21st ACM International Conference on
Research and Development in Information
Retrieval, pages 104-111, Melbourne, AU, 1998. - S. Chakrabarti, K. Punera, and M. Subramanyam.
Accelerated focused crawling through online
relevance feedback. In WWW, Hawaii. ACM, May
2002. - Soumen Chakrabarti, Martin van den Berg, and
Byron Dom. Focused crawling A new approach to
topic-specific Web resource discovery. Computer
Networks (Amsterdam, Netherlands 1999),
31(1116)16231640, 1999.
30Referencescontd
- Jeffrey Dean and Monika R. Henzinger. Finding
related pages in the World Wide Web. Computer
Networks (Amsterdam, Netherlands 1999), 31(11
16)14671479, 1999. - Jon M. Kleinberg. Authoritative sources in a
hyperlinked environment. Journal of the ACM,
46(5)604632, 1999. - H. Yu, J. Han, and K. C-C. PEBL Positive
example-based learning for web page
classification using SVM, 2002. - Shipeng Yu, Deng Cai, Ji-Rong Wen, and Wei-Ying
Ma. Improving pseudorelevance feedback in web
information retrieval using web page
segmentation. In Proceedings of WWW2003, The
Twelfth International World Wide Web Conference,
Budapest, HUNGARY, 2003.