Title: WHOWEDA%20:%20Warehouse%20of%20Web%20Data
1- WHOWEDA Warehouse of Web Data
- Sanjay Kumar Madria
- Department of Computer Science
- Purdue University, West Lafayette, IN 47907
- skm_at_cs.purdue.edu
2www.is.a.mess
3WWW
- Huge, widely distributed, hetreogenous collection
of semi-structured multimedia documents in the
form of web pages connected via hyperlinks.
4Characteristics of WWW
- WWW is a set of directed graphs
- data in the WWW has a heterogeneous nature
- unstructured versus structured information
- no central authority to manage information
- Dynamic verses static information
- Web information discoveries - search engines
5As WWW grows, more chaotic it becomes
- Web is fast growing, distributed,
non-administered global information resource - WWW allows access to text, image, video, sound
and graphic data - more business organizations creating web servers
- e-commerce - more chaotic environment to locate information of
interest - lost in hyperspace syndrome
6WWW data - Does it affect the corporate world?
- Lack of credibility of data
- Different sites with different data
- Same site different data
- Historical information is not available
- Previous versions of web data
- How does web data change with time
- Summarization over time
- Data to information
- Reduction in productivity
- Analysis is manual
7How users find web sites
- Indexes and search engines 75
- UseNet newsgroups 44
- Cool lists 27
- New lists 24
- Listservers 23
- Print ads 21
- Word-of-mouth and e-mail 17
- Linked web advertisement 4
8Limitations of Search Engines
- Do not exploit hyperlinks - recently google
- search is limited to string matching
- key-world oriented search queries are evaluated
on archived data rather than up-to-date data no
indexing on current data - low accuracy
- replicated results
- no further manipulation possible
9Continue .
- ERROR 404!
- No efficient document management
- Query results cannot be further manipulated
- No efficient means for knowledge discovery
10Current Research Projects
- Web Query System
- W3QS, WebSQL, AKIRA, NetQL, RAW,
- WebLog, XML-QL
- Semistructured Data
- LOREL, UnQL, WebOQL, Website Management System
- STRUDEL
- Web Warehouse
- - WHOWEDA
11WHOWEDA -Key Objectives
- Design a suitable data model to represent web
information - development of web algebra and query language
- Maintenance of Web data
- Development of knowledge discovery and web mining
tools - Web warehouse
12WHOWEDA - What?
- WareHouse Of Web Data
- Subject - oriented
- Integrated
- Temporal
- Granularity - Lower, higher
- Some summary
- Not updatable
- Alternative information sources
13Web Warehouse?
- Subject-oriented, integrated, time-variant,
non-volatile repository of web data for direct
querying and analysis for some sort of decision
making - A process whereby organizations or individuals
extract value from their Web informational assets
through the use of special stores called web
warehouses
14WHOWEDA! www.cais.ntu.edu.sg8000/whoweda
- A WareHouse Of WEb DAta
- Web Information Coupling Model (WICM)
- Web Objects
- Web Schema
- Web Information Coupling Algebra
- Web Information Maintenance
- Web Mining and Knowledge discovery
15User
WWW
Warehouse Concept Mart
Web Querying Analysis Component
Web Information Coupling System
Web Information Maintenance System
Web Information Mining System
Web Mart
Web Mart
Web Warehouse
Web Mart
Web Mart
16User
WWW
Web Query Display
Warehouse Concept Mart
Global Web Manipulation
Global Web Coupling
Pre processing
Global Ranking
Data Visualization
Schema Tightness
Web Warehouse
Data Visualization
Web Union
Web Select
Web Intersection
Web Project
Local Web Manipulation
Local Web Coupling
Schema Tightness
Local Ranking
Schema Search
Web Join
Schema Match
17Web Objects
- Node - url, title, format, size, date, text
- Link - source-url, target-url, label, link-type
- Web tuple
- Web table
- Web schema
- Web database
18Web Schema
- Metadata in the warehouse
- Structural summary of web table
- Information Coupling using a Query graph
- Query graph -gtWeb schema
- directed graph represented by Ordered 4-tuple
- Set of node variables
- Set of link variables
- Connectivities
- Predicates
19(No Transcript)
20(No Transcript)
21url contains headlines
22(No Transcript)
23Schema- example
- Node variables Xn x, y, z, w
- Link variable Xl e, f, g
- Connectivities C xltegty and xltfg-gtz and
xltfh-gtw - The symbol represents an unbound node variable
or link variable a variable not restricted by
any predicate. - - represents one unbound links
- - represents more than one unbound links
24 - Predicates
- Px.urlhttp//www.mediacity.com.sg/i-square,
- y.url CONTAINS headlines
- e.target_url CONTAINS "article",
- f.target.url CONTAINS "newshub/specials",
- g.label CONTAINS "Local News",
- z.url CONTAINS "local",
- h.label CONTAINS "World News",
- w.url CONTAINS "world"
25Query Graph - Example 1
- Query graph - same as schema except that it has
one more parameter to control the results
returned. - Informally, it is directed connected graph
consists of nodes, links and keywords imposed on
them. - Produce a list of diseases with their symptoms,
evaluation procedures and treatment starting from
the web site at http//www.panacea.org/ - Web table Diseases
26Treatment list
q
Treatment
g
http//www.panacea.org/
Issues
Symptoms list
f
y
x
z
Symptoms
List of Diseases
e
Evaluation
Evaluation
w
p
27Treatment list
q1
g1
Treatment
http//www.panacea.org/
Issues
f1
Symptoms list
x0
z1
y1
Symptoms
AIDS
List of Diseases
e1
Evaluation
Evaluation
w1
p2
Elisa Test
28Example 2
- Produce a list of drugs, and their uses and side
effects starting from the web site at
http//www.panacea.org/ - Web table Drugs
29(No Transcript)
30Side effects of Indavir
Drug list
http//www.panacea.org/
Issues
r1
AIDS
a0
b1
c1
d1
Indavir
Side effects
List of Diseases
Use
s1
k1
Uses of Indavir
31Query Language
- Starting from the CS dept. home page at NTU, find
all documents that are linked through paths of
length less than two containing only local links,
and have in their text database.
32- COUPLE WEBTABLE W FROM WWW
- SUCH THAT NODE I, J IN WWW and LINK e,f,g IN WWW
AND Iltef,ggtJ WHERE I.url EQUALS
http//www.ntu.edu.sg AND J.text CONTAINS
database AND f.link-type EQUALS local AND
g.link-type EQUALS local
33Web Algebra
- Formal foundation of data representation and
manipulation in a web warehouse - Web operators
- Information access operator
- Information manipulation operators
- Web schema operators
- Data visualization operators
34Information access operator
35Information Manipulation
- - Web select
- Web project
- Local web coupling
- Web join
- Web cartesian product
- Web union
- Web intersect
- Local Web coupling
36Web Select
- Extracts web tuples from web tables satisfying
certain conditions on node and link variables and
on connectivities - Input is select Schema
- Output is a web table satisfying the select schema
37 - select W1 tuples that contain world news about
Indonesia since May 1 1998. - sMsW1 where
- Ms lt Xsn, Xsl, Cs, Ps gt,
- Xsn x, w , Xsl ,
- Cs ,
- Ps x.date gt "1May1998", w.text CONTAINS
Indonesia
38- Xn x, y, z, w ,Xl e, f, g
- C xltegty and xltfg-gtz and xltfh-gtw
- Px.urlhttp//www.mediacity.com.sg/i-square,
x.date gt "1May1998", - e.target_url CONTAINS "article", f.target.url
CONTAINS "newshub/specials", - g.label CONTAINS "Local News",
- z.url CONTAINS "local",
- h.label CONTAINS "World News",
- w.url CONTAINS "world",
- w.text CONTAINS Indonesia
39Web Information Coupling System
- A database system to couple related web
information - Global web Coupling and Local Web Coupling
40Global Coupling - Information Access
- To integrate data from the Web
- To create historical data
- To couple related information from the WWW
satisfying a query graph - Operator to create web tables
- From web with no schema to web table with web
schema
41Why local web coupling?
- Directly querying the WWW to gather these
information is an expensive and repetitive affair
- Web documents containing similar information can
reside in different web tables in a web warehouse
- A mechanism to gather these similar information
by additional manipulation of the materialized
web tables
42Local Web Couple operator
- Two web tuples and can be coupled if
there exist atleast one pair of nodes from
and which contains similar information.
43Local Web Couple operator
- The web couple operator is basically a web
cartesian product followed by web select - We denote web couple by the symbol
44Web Coupling
45Example 1
- Produce a list of diseases and their symptoms
starting from the web site at http//www.panacea.o
rg/ - Web table Diseases
46Issues
http//www.panacea.org/
symptoms
e
z
x
y
symptoms
List of Diseases
Web Schema or Query Graph of Diseases
47Web table Diseases
48Example 2
- Produce a list of drugs, and their side effects
starting from the web site at http//www.panacea.o
rg/ - Web table Drugs
49Drug list
Side effects
http//www.panacea.org/
Issues
r
c
a
b
d
Side effects
List of Diseases
Web Schema or Query Graph of Drugs
50Web table Drugs
51Issues
http//www.panacea.org/
Symptoms of AIDS
e0
AIDS
z0
x0
y0
symptoms
List of Diseases
Side effects of Ritonavir
Drug list
http//www.panacea.org/
Issues
r2
AIDS
a0
b1
c2
d2
Ritonavir
Side effects
Issues
http//www.panacea.org/
Symptoms of Cancer
e1
Cancer
z1
x0
y1
symptoms
List of Diseases
http//www.panacea.org/
Issues
Heart Disorder
r4
a0
b4
c4
d4
Side effects
Beta Carotene
Symptoms Side effects
52- M2 lt Xn, Xl, C,P gt for W2
- Xn s, t, u, Xl k, l, m, n ,
- C sltklgtt and sltmngtu ,
- Ps.url http//www.asia1.com.sg/straitstimes/,
- k.label REGION,
- l.target_url http//www.asia1.com.sg/straitstime
s/pages/sea.html, m.label WORLD, - n.target_urlhttp//www.asia1.com.sg/straitstimes
/pages/wrld.html
53- W1 qq W2 where
- q (x.dates.date) (w.text CONTAINS
Indonesia) (t.text CONTAINS Indonesia) - Schema of the coupled table is
54- Xn x, y, z, w, s, t, u , Xl e, f,
g, k, l, m, n , C xltegty and xltfg-gtz and
xltfh-gtw and sltklgtt and sltmngtu - P x.urlhttp//www.mediacity.com.sg/i-square
, e.target_url CONTAINS "article", - f.target.url CONTAINS "newshub/specials",
- g.label CONTAINS "Local News",
- z.url CONTAINS "local",
- h.label CONTAINS "World News",
- w.url CONTAINS "world",
- s.url http//www.asia1.com.sg/straitstimes/,
55- k.label REGION, l.target_url
http//www.asia1.com.sg/straitstimes/pages/sea.h
tml, - m.label WORLD,
- n.target_url http//www.asia1.com.sg/straitstim
es/pages/wrld.html, - x.date s.date,
- w.text CONTAINS Indonesia,
- t.text CONTAINS Indonesia"
56Local Web Coupling
- Initiated explicitly by the user
- User provides the pair of node variables and the
keyword set based on which coupling is to be
performed - Coupling nodes in each pair of web tuples in the
input web tables must satisfy one of the coupling
conditions
57Types of web coupling
- System driven web coupling system to decide the
coupling nodes. If atleast a pair of coupling
nodes cannot be identified then the web tables
cannot be coupled. - User driven web coupling user decides the
coupling nodes. - Coupling is performed only on those user
specified node variable(s).
58Attribute driven web coupling
- Attribute driven web coupling user specifies the
coupling attributes and coupling is performed
only on those user specified coupling
attribute(s). - COUPLE TABLE3
- FROM TABLE1 AND TABLE 2
- ON ATTRIBUTE TEXT
- AT SCHEMA/TUPLE(optional)
59Value Driven web coupling
- Value driven web coupling user specifies the
values of the attributes of the nodes on which
coupling should be performed. - COUPLE TABLE3
- FROM TABLE1 AND TABLE 2
- ON VALUE Software Agents
- AT SCHEMA/TUPLE(optional)
60Schema level web coupling
- We inspect the schemas to decide whether the two
web tables can be coupled. - If coupling conditions cannot be identified then
the two web tables cannot be coupled. - We do not inspect the web tuples in the web
table. - Number of web tuples coupled will be nm.
61Tuple level web coupling
- We inspect the web tuples of the two input web
tables to identify nodes with similar
information. - The number of web tuples in the coupled web table
ltnm
62Why two levels?
- A schema does not capture all the information of
the web documents in a web table not always
possible to identify coupling condition by
inspecting the schemas. - possible to find existence of coupling nodes
which are not defined in the schemas.
63Why two levels?
- Tuple level coupling gives us a mean to correlate
web documents containing similar information from
the web tables (that cannot be identified from
their schemas) at the expense of additional
processing.
64Join Processing in Web Databases
65Web Join
- Concatenate tuples based on identical nodes or
documents - Input are two web tables and their schemas
- Output is a joined table
- Types
- Pi-web join, theta-web join, outer joins, web
composition, semi web join
66Web Join
- Used for combining related data from various web
tables - Mechanism to detect changes
- Mechanism to find alternative web document in
case of Document Not Found error
67Web Join Operator
- Information manipulation operator
- Manipulate information residing in a web database
to derive additional information - Harness useful, composite information from two
web tables - Capitalize on the reuse of retrieved data from
the WWW in order to reduce execution time of
queries
68Joinable Nodes
- Node variables participating in the web join
process - Expressed as a pair
- Each node in the pair should have identical URLs
69Web Join
- Combine two web tables by concatenating a web
tuple of one web table with a web tuple of other
web table whenever there exist joinable nodes - Joinable nodes are identified from the schemas of
the two web tables - URLs of the joinable nodes are identical
70Treatment list
q
Treatment
g
http//www.panacea.org/
Symptoms list
Issues
List of Diseases
f
y
x
z
Symptoms
e
Evaluation
Evaluation
Drug list
w
p
Issues
r
Side effects
b
c
d
Side effects
Use
s
Uses
k
71AIDS treatment
q1
g1
Symptoms of AIDS
http//www.panacea.org/
f1
y1
x0
z1
AIDS
e1
AIDS
Evaluation
Elisa Test
w1
p2
r1
Side effects of Indavir
b1
c1
d1
Indavir
s1
Uses of Indavir
k1
72Pi-Web Join
73Example 1
- Produce a list of diseases with their symptoms,
evaluation procedures and treatment starting from
the web site at http//www.panacea.org/ - Web table Diseases
74http//www.panacea.org/
z
Query Graph (Web Schema) for Example 1
75Treatment list
q1
http//www.panacea.org/
Symptoms list
x0
z1
AIDS
List of Diseases
Evaluation
p2
Elisa Test
A web tuple in Diseases
76Example 2
- Produce a list of drugs, and their uses and side
effects starting from the web site at
http//www.panacea.org/ - Web table Drugs
77Query Graph (Web Schema) of Drugs
78A web tuple in Drugs
79Web Project
- Eliminate nodes from web tuples which are
irrelevant - Based on project conditions
- Set of node variables
- Start node variable and end-node variable
- Node variable and depth of links
- Used to isolate data of interest in a web table,
allowing subsequent web queries to run over
smaller, more structured web table
80http//www.panacea.org/
Symptoms list
x0
z1
AIDS
List of Diseases
Evaluation
p2
A web project on Diseases
81treatment
q
http//www.panacea.org/
z
x
symptoms
Disease List
p
evaluation
Side effects
Drug list
b
d
Joined schema
k
Uses
82Treatment list
q1
http//www.panacea.org/
Symptoms list
x0
z1
AIDS
List of Diseases
AIDS
Evaluation
p2
Side effects of Indavir
Drug list
Elisa Test
b1
d1
Indavir
Side effects
Use
k1
Uses of Indavir
Joined Tuple
83Motivation of Pi-web Join
- Quite often web join operation couples irrelevant
nodes - In a complex web query with several web join
operation, the size of the resultant web table
can become very large with many contaminated
nodes - Pi-web join resolves the above limitation by
eliminating contaminated nodes - Reduces the size of joined web table
84Pi-web Join
- Web join followed by web project
- The projection conditions are specified by the
user conditions are similar to web project - We do not eliminate the joinable nodes
- By retaining the joinable nodes we preserve the
correlation between the information captured from
two web tables - Pi-web join may result in a web bag
85Example 3
- Produce a list of diseases with their symptoms
and side-effects starting from the web site at
http//www.panacea.org/
86Procedure
- Perform web join on Diseases and Drugs
- Project node variables b, k, q, p, node variables
between a and q, node variables between b and k,
node variables between b and d
87http//www.panacea.org/
z
x
symptoms
Disease List
Side effects
d
Pi-joined schema
88http//www.panacea.org/
Symptoms list
x0
z1
AIDS
List of Diseases
Side effects of Indavir
d1
Pi-joined Tuple
89Benefits of Pi-web Join
- Minimize the amount of data transmitted over the
network in distributed web join processing - Reduction in storage cost associated with a
joined web table - Reduces cognitive overhead associated with
locating relevant nodes - Improve completeness of schema by removing
unbound nodes and links
90Web Bags
- Existence of identical web tuples.
- Created due to web project operation.
- Structure based mining
- Used for discovering
- Visible nodes
- Luminous nodes
- Luminous paths
91Definitions
- Visibility of a web document or node D in a web
table W measures the number of different web
documents in W that have links to D - Luminosity - Reverse of visibility, the number of
other distinct documents that are linked from D - Luminous paths - a set of inter-linked nodes
which occurs number of times in a web table
92Steps to find visible nodes
- Input Web table W, node variable x, visibility
threshold v - Output Set of visible nodes and inter-site
support for each node - Create a web table from W where each web tuple
contains distinct instances of node x and the
preceeding node which is linked to x (use project
and create distinct tuples if node x has more
than 1 incoming edge) - Eliminate the nodes linked to x in each tuple of
the web table using web project
93Continue...
- Check if the collection of web tuples of node x
thus created is a web bag by comparing their URLs - Create multiplets for each collection of
identical nodes - For each multiplet calculate the node visibility
(using the mathematical formula defined, see
FODO-98) - Determine the multiplets with node visibility
greater than the threshold - Create the visible node set and calculte the
inter-site support using the formula
94Steps to find luminous nodes
- Input Web table W, node variable x, luminosity
threshold l - Output Set of luminous nodes with inter-site
support - Steps are similar to that of visible node
discovery - We consider the nodes linked from x in place of
nodes linked to x
95Steps to find luminous paths
- Input - web table W, nodes x and y
- Output - threshold value for luminous path
- Project nodes between x and y and check for web
bag else go to next slide - Create the collection of multiplets
- Compute path luminosity for each multiplet using
the formula - If the path luminosity value of a multiplet is
greater than or equal to threshold then a path
in the multiplet is a luminous path
96Steps to find luminous paths
- Otherwise, we create a collection of linear web
tuples from the above collection of web tuples - This is to identify if there exist a subset of
inter-linked nodes between x and y that are
luminous paths - We repeat the procedure to compute path
luminosity for these set of inter-linked nodes
97Web Schema
Cancer
http//www.panacea.org/
e
f
x
y
z
Cancer
Diseases
98Cancer
http//www.panacea.org/
Diseases
f0
x0
y0
z1
Cancer
e0
http//www.cancer.org/desc.html
Cancer
Diseases
f0
z1
x0
y0
Cancer
e0
http//www.cancer.org/desc.html
Cancer
Diseases
f0
z2
x0
y0
Cancer
e0
Cancer
Diseases
f0
x0
y0
z1
Cancer
e0
http//www.cancer.org/desc.html
Cancer
Diseases
f0
z4
x0
y0
Cancer
e0
Web Table
99Projected schema
100Cancer
Web Table after eliminating x and y
101Projected schema
Cancer
http//www.panacea.org/
e
z
x
y
Diseases
102http//www.cancer.org/desc.html
http//www.cancer.org/desc.html
http//www.disease.com/cancer/skin.htm
http//www.cancer.org/desc.html
http//www.jhu.edu/medical/research/cancer.htm
http//www.panacea.org/
Diseases
Cancer
x0
y0
z4
Web Bag
103After removal of identical tuples
http//www.cancer.org/desc.html
104Cancer
z1
http//www.cancer.org/desc.html
Cancer
http//www.cancer.org/desc.html
z1
http//www.disease.com/cancer/skin.htm
http//www.cancer.org/desc.html
http//www.jhu.edu/medical/research/cancer.htm
105http//www.cancer.org/desc.html
106Visible Nodes
Cancer
http//www.cancer.org/desc.html
z1
Cancer
z2
http//www.disease.com/cancer/skin.htm
Cancer
z1
http//www.cancer.org/desc.html
Cancer
z4
http//www.jhu.edu/medical/research/cancer.htm
107Luminous Paths
108Change Management
109More Operators . . .
- Web schema operators
- Schema tightness operator, Schema match operator,
Schema search operator - Data visualization operators
- Ranking operators (Global Local), Web Nest, Web
Un-nest, Web Coalesce, Web Expand, Web Pack, Web
Unpack, Web Sort
110Partitioning of web tables
- Partitioning web tables
- restructured easily
- indexed easily
- monitored easily
- reorganized easily
- By
- time
- schema tree structure
- keywords
111Warehouse Concept Mart (WCMart)
- Subject oriented
- Concept generation.
- Manually -gt Autonomous.
- Used for
- Ranking tuples
- Global web coupling
- Content based mining
112Mining in Web Warehouse
- Web Structure Mining
- Web Content Mining
- Web usage Mining
113Web Data Refinement
- Improve web schema - schema tightness operator
- Partition web tables based on content and
structure
114Partitioning of web tables
- Partitioning web tables
- restructured easily
- indexed easily
- monitored easily
- reorganized easily
- By
- time
- schema tree structure
- keywords
115WWW
Warehouse Concept Mart
Global Web Coupling
Webtable (Jan)
Webtable (Feb)
Webtable (Mar)
Webtable (Apr)
116Webtable (Jan)
Webtable (Feb)
Webtable (Mar)
Webtable (Apr)
Lower-level Granularity
Web Information Manipulation Operators
Higher level Granularity
Summarized data
117User
WWW
Warehouse Concept Mart
Web Querying Analysis Component
Web Information Coupling System
Web Information Mining System
Web Warehouse
118What type of information can be summarized?
- Structural
- Content-based
- time-variant analysis
- snapshot analysis
- compare one period with another
- trend analysis
119Structural Summarization
- Most volatile documents
- Sites which change frequently
- Rate of change over time
- a pointer to directly access documents which
change rapidly - Most visible nodes, luminous nodes, luminous
paths - Change with time
- Decrease or increase - Analyze the reason
120Content Summarization
- What can be aggregrated in a web page?
- Number of links with identical labels
- Number of keywords
- Changes in content with time
- Comparing the changes
- Open question
- XML will improve the ability of analysis of web
data
121Summary
- Current status
- Mechanism for accessing and manipulating web
information in WHOWEDA - Implementing various web operators and query
language - Future research
- What types of information can be summarized?
- What types of knowledge can be mined?
- Refine web warehouse architecture
- www.cais.ntu.edu.sg8000/whoweda