Storage, Indexing and Joins Slides taken from Database Management Systems, R. Ramakrishnan

1
Storage, Indexing and Joins Slides taken
fromDatabase Management Systems, R.
Ramakrishnan
2
Disks and Files

• DBMS stores information on (hard) disks.
• This has major implications for DBMS design!
• READ transfer data from disk to main memory
  (RAM).
• WRITE transfer data from RAM to disk.
• Both are high-cost operations, relative to
  in-memory operations, so must be planned
  carefully!

3
Why Not Store Everything in Main Memory?

• Costs too much. 1000 will buy you either 0.5GB
  of RAM or 50GB of disk today.
• Main memory is volatile. We want data to be
  saved between runs. (Obviously!)
• Typical storage hierarchy
• Main memory (RAM) for currently used data.
• Disk for the main database (secondary storage).
• Tapes for archiving older versions of the data
  (tertiary storage).

4
Disks

• Secondary storage device of choice.
• Main advantage over tapes random access vs.
  sequential.
• Data is stored and retrieved in units called disk
  blocks or pages.
• Unlike RAM, time to retrieve a disk page varies
  depending upon location on disk.
• Therefore, relative placement of pages on disk
  has major impact on DBMS performance!

5
Components of a Disk
Spindle
Disk head

• The platters spin (say, 90rps).

• The arm assembly is moved in or out to position
  a head on a desired track. Tracks under heads
  make a cylinder (imaginary!).

Sector
Platters

• Only one head reads/writes at any one time.

• Block size is a multiple of sector size (which
  is fixed).

6
Accessing a Disk Page

• Time to access (read/write) a disk block
• seek time (moving arms to position disk head on
  track)
• rotational delay (waiting for block to rotate
  under head)
• transfer time (actually moving data to/from disk
  surface)
• Seek time and rotational delay dominate.
• Seek time varies from about 1 to 20msec
• Rotational delay varies from 0 to 10msec
• Transfer rate is about 1msec per 4KB page
• Key to lower I/O cost reduce seek/rotation
  delays! Hardware vs. software solutions?

7
Arranging Pages on Disk

• Next block concept
• blocks on same track, followed by
• blocks on same cylinder, followed by
• blocks on adjacent cylinder
• Blocks in a file should be arranged sequentially
  on disk (by next), to minimize seek and
  rotational delay.
• For a sequential scan, pre-fetching several pages
  at a time is a big win!

8
Disk Space Management

• Lowest layer of DBMS software manages space on
  disk.
• Higher levels call upon this layer to
• allocate/de-allocate a page
• read/write a page
• Request for a sequence of pages must be satisfied
  by allocating the pages sequentially on disk!
  Higher levels dont need to know how this is
  done, or how free space is managed.

9
Buffer Management in a DBMS
Page Requests from Higher Levels
BUFFER POOL
disk page
free frame
MAIN MEMORY
DISK
choice of frame dictated by replacement policy

• Data must be in RAM for DBMS to operate on it!
• Table of ltframe, pageidgt pairs is maintained.

10
When a Page is Requested ...

• If requested page is not in pool
• Choose a frame for replacement
• If frame is dirty, write it to disk
• Read requested page into chosen frame
• Pin the page and return its address.

• If requests can be predicted (e.g., sequential
  scans)
• pages can be pre-fetched several pages at a
  time!

11
More on Buffer Management

• Requestor of page must unpin it, and indicate
  whether page has been modified
• dirty bit is used for this.
• Page in pool may be requested many times,
• a pin count is used. A page is a candidate for
  replacement iff pin count 0.
• CC recovery may entail additional I/O when a
  frame is chosen for replacement.

12
Buffer Replacement Policy

• Frame is chosen for replacement by a replacement
  policy
• Least-recently-used (LRU), Clock, MRU etc.
• Policy can have big impact on of I/Os depends
  on the access pattern.
• Sequential flooding Nasty situation caused by
  LRU repeated sequential scans.
• buffer frames lt pages in file means each page
  request causes an I/O. MRU much better in this
  situation (but not in all situations, of course).

13
DBMS vs. OS File System

• OS does disk space buffer management why
  not let OS manage these tasks?
• Differences in OS support portability issues
• Some limitations, e.g., files cant span disks.
• Buffer management in DBMS requires ability to
• pin a page in buffer pool, force a page to disk
  (important for implementing CC recovery),
• adjust replacement policy, and pre-fetch pages
  based on access patterns in typical DB operations.

14
Files of Records

• Page or block is OK when doing I/O, but higher
  levels of DBMS operate on records, and files of
  records.
• FILE A collection of pages, each containing a
  collection of records. Must support
• insert/delete/modify record
• read a particular record (specified using record
  id)
• scan all records (possibly with some conditions
  on the records to be retrieved)

15
Record Formats Fixed Length
F1
F2
F3
F4
L1
L2
L3
L4
Base address (B)
Address BL1L2

• Information about field types same for all
  records in a file stored in system catalogs.
• Finding ith field requires scan of record.

16
Record Formats Variable Length

• Two alternative formats ( fields is fixed)

F1 F2 F3
F4
Fields Delimited by Special Symbols
Field Count
F1 F2 F3 F4
Array of Field Offsets

• Second offers direct access to ith field,
  efficient storage
• of nulls (special dont know value) small
  directory overhead.

17
Page Formats Fixed Length Records
Slot 1
Slot 1
Slot 2
Slot 2
Free Space
. . .
. . .
Slot N
Slot N
Slot M
N
M
1
. . .
1
1
0
M ... 3 2 1
number of records
number of slots
PACKED
UNPACKED, BITMAP

• Record id ltpage id, slot gt. In first
  alternative, moving records for free space
  management changes rid may not be acceptable.

18
Page Formats Variable Length Records
Rid (i,N)
Page i
Rid (i,2)
Rid (i,1)
N
Pointer to start of free space
20
16
24
N . . . 2 1
slots
SLOT DIRECTORY

• Can move records on page without changing rid
  so, attractive for fixed-length records too.

19
Alternative File Organizations

• Many alternatives exist, each ideal for some
  situation, and not so good in others
• Heap files Suitable when typical access is a
  file scan retrieving all records.
• Sorted Files Best if records must be retrieved
  in some order, or only a range of records is
  needed.
• Hashed Files Good for equality selections.
• File is a collection of buckets. Bucket primary
  page plus zero or more overflow pages.
• Hashing function h h(r) bucket in which
  record r belongs. h looks at only some of the
  fields of r, called the search fields.

20
Unordered (Heap) Files

• Simplest file structure contains records in no
  particular order.
• As file grows and shrinks, disk pages are
  allocated and de-allocated.
• To support record level operations, we must
• keep track of the pages in a file
• keep track of free space on pages
• keep track of the records on a page
• There are many alternatives for keeping track of
  this.

21
Heap File Implemented as a List
Data Page
Data Page
Data Page
Full Pages
Header Page
Data Page
Data Page
Data Page
Pages with Free Space

• The header page id and Heap file name must be
  stored someplace.
• Each page contains 2 pointers plus data.

22
Heap File Using a Page Directory

• The entry for a page can include the number of
  free bytes on the page.
• The directory is a collection of pages linked
  list implementation is just one alternative.
• Much smaller than linked list of all heap file
  pages!

23
Analysis of file organizations

• We ignore CPU costs for simplicity, and use the
  following parameters in our cost model
• B The number of data pages
• R Number of records per page
• D (Average) time to read or write disk page
• Measuring number of page I/Os ignores gains of
  pre-fetching blocks of pages thus, even I/O cost
  is only approximated.
• Average-case analysis based on several
  simplistic assumptions.

• Good enough to show the overall trends!

24
Assumptions in Our Analysis

• Single record insert and delete.
• Heap Files
• Equality selection on key exactly one match.
• Insert always at end of file.
• Sorted Files
• Files compacted after deletions.
• Selections on sort field(s).
• Hashed Files
• No overflow buckets, 80 page occupancy.

25
Cost of Operations

• Several assumptions underlie these (rough)
  estimates!

26
Indexes

• A Heap file allows us to retrieve records
• by specifying the rid, or
• by scanning all records sequentially
• Sometimes, we want to retrieve records by
  specifying the values in one or more fields,
  e.g.,
• Find all students in the CS department
• Find all students with a gpa gt 3
• Indexes are file structures that enable us to
  answer such value-based queries efficiently.
• This will be topic of our next lecture!

27

• Indexing

28
Indexes

• An index on a file speeds up selections on the
  search key fields for the index.
• Any subset of the fields of a relation can be the
  search key for an index on the relation.
• Search key is not the same as key (minimal set of
  fields that uniquely identify a record in a
  relation).
• An index contains a collection of data entries,
  and supports efficient retrieval of all data
  entries k with a given key value k.

29
Alternatives for Data Entry k in Index

• Three alternatives
• Data record with key value k
• ltk, rid of data record with search key value kgt
• ltk, list of rids of data records with search key
  kgt
• Choice of alternative for data entries is
  orthogonal to the indexing technique used to
  locate data entries with a given key value k.
• Examples of indexing techniques B trees,
  hash-based structures
• Typically, index contains auxiliary information
  that directs searches to the desired data entries

30
Index Classification

• Primary vs. secondary If search key contains
  primary key, then called primary index.
• Unique index Search key contains a candidate
  key.
• Clustered vs. unclustered If order of data
  records is the same as, or close to, order of
  data entries, then called clustered index.
• Alternative 1 implies clustered, but not
  vice-versa.
• A file can be clustered on at most one search
  key.
• Cost of retrieving data records through index
  varies greatly based on whether index is
  clustered or not!

31
Clustered vs. Unclustered Index

• Suppose that Alternative (2) is used for data
  entries, and that the data records are stored in
  a Heap file.
• To build clustered index, first sort the Heap
  file (with some free space on each page for
  future inserts).
• Overflow pages may be needed for inserts. (Thus,
  order of data recs is close to, but not
  identical to, the sort order.)

Index entries
UNCLUSTERED
CLUSTERED
direct search for
data entries
Data entries
Data entries
(Index File)
(Data file)
Data Records
Data Records
32
Index Classification (Cont.)

• Dense vs. Sparse If there is at least one data
  entry per search key value (in some data
  record), then dense.
• Alternative 1 always leads to dense index.
• Every sparse index is clustered!
• Sparse indexes are smaller however, some useful
  optimizations are based on dense indexes.

Ashby, 25, 3000
22
Basu, 33, 4003
25
Bristow, 30, 2007
30
Ashby
33
Cass, 50, 5004
Cass
Smith
Daniels, 22, 6003
40
Jones, 40, 6003
44
44
Smith, 44, 3000
50
Tracy, 44, 5004
Sparse Index
Dense Index
on
on
Data File
Name
Age
33
Range Searches

• Find all students with gpa gt 3.0
• If data is in sorted file, do binary search to
  find first such student, then scan to find
  others.
• Cost of binary search can be quite high.
• Simple idea Create an index file.

Index File
kN
k2
k1
Data File
Page N
Page 3
Page 1
Page 2

• Can do binary search on (smaller) index file!

34
Comments on ISAM
Data Pages
Index Pages

• File creation Leaf (data) pages allocated
  sequentially, sorted by search key
  then index pages allocated, then space for
  overflow pages.
• Index entries ltsearch key value, page idgt
  they direct search for data entries, which
  are in leaf pages.
• Search Start at root use key comparisons to go
  to leaf. Cost log F N F entries/index
  pg, N leaf pgs
• Insert Find leaf data entry belongs to, and put
  it there.
• Delete Find and remove from leaf if empty
  overflow page, de-allocate.

Overflow pages

• Static tree structure inserts/deletes affect
  only leaf pages.

35
B Tree The Most Widely Used Index

• Insert/delete at log F N cost keep tree
  height-balanced. (F fanout, N leaf pages)
• Minimum 50 occupancy (except for root). Each
  node contains d lt m lt 2d entries. The
  parameter d is called the order of the tree.
• Supports equality and range-searches efficiently.

36
Example B Tree

• Search begins at root, and key comparisons direct
  it to a leaf.
• Search for 5, 15, all data entries gt 24 ...

Root
17
24
30
13
39
3
5
19
20
22
24
27
38
2
7
14
16
29
33
34

• Based on the search for 15, we know it is not
  in the tree!

37
Inserting a Data Entry into a B Tree

• Find correct leaf L.
• Put data entry onto L.
• If L has enough space, done!
• Else, must split L (into L and a new node L2)
• Redistribute entries evenly, copy up middle key.
• Insert index entry pointing to L2 into parent of
  L.
• This can happen recursively
• To split index node, redistribute entries evenly,
  but push up middle key. (Contrast with leaf
  splits.)
• Splits grow tree root split increases height.
  
• Tree growth gets wider or one level taller at
  top.

38
Inserting 8 into Example B Tree
Entry to be inserted in parent node.

• Observe how minimum occupancy is guaranteed in
  both leaf and index pg splits.
• Note difference between copy-up and push-up be
  sure you understand the reasons for this.

(Note that 5 is
s copied up and
5
continues to appear in the leaf.)
3
5
2
7
8
appears once in the index. Contrast
39
Example B Tree After Inserting 8
Root
17
24
30
13
5
2
3
39
19
20
22
24
27
38
7
5
8
14
16
29
33
34

• Notice that root was split, leading to increase
  in height.

• In this example, we can avoid split by
  re-distributing entries however,
  this is usually not done in practice.

40
Deleting a Data Entry from a B Tree

• Start at root, find leaf L where entry belongs.
• Remove the entry.
• If L is at least half-full, done!
• If L has only d-1 entries,
• Try to re-distribute, borrowing from sibling
  (adjacent node with same parent as L).
• If re-distribution fails, merge L and sibling.
• If merge occurred, must delete entry (pointing to
  L or sibling) from parent of L.
• Merge could propagate to root, decreasing height.

41
Example Tree After (Inserting 8, Then) Deleting
19 and 20 ...
Root
17
27
30
13
5
2
3
39
38
7
5
8
22
24
27
29
14
16
33
34

• Deleting 19 is easy.
• Deleting 20 is done with re-distribution. Notice
  how middle key is copied up.

42
... And Then Deleting 24

• Must merge.
• Observe toss of index entry (on right), and
  pull down of index entry (below).

30
39
22
27
38
29
33
34
Root
13
5
30
17
3
39
2
7
22
38
5
8
27
33
34
14
16
29
43
Summary (cont.)

• Data entries can be actual data records, ltkey,
  ridgt pairs, or ltkey, rid-listgt pairs.
• Choice orthogonal to indexing technique used to
  locate data entries with a given key value.
• Can have several indexes on a given file of data
  records, each with a different search key.
• Indexes can be classified as clustered vs.
  unclustered, primary vs. secondary, and dense vs.
  sparse. Differences have important consequences
  for utility/performance.

44
Summary (cont.)

• Tree-structured indexes are ideal for
  range-searches, also good for equality searches.
• B tree is a dynamic structure.
• Inserts/deletes leave tree height-balanced log F
  N cost.
• High fanout (F) means depth rarely more than 3 or
  4.
• Almost always better than maintaining a sorted
  file.

45
Implementation of Relational Operations
46
Relational Operations

• We will consider how to implement
• Selection ( ) Selects a subset of rows
  from relation.
• Projection ( ) Deletes unwanted columns
  from relation.
• Join ( ) Allows us to combine two
  relations.
• Set-difference ( ) Tuples in reln. 1, but
  not in reln. 2.
• Union ( ) Tuples in reln. 1 and in reln. 2.
• Aggregation (SUM, MIN, etc.) and GROUP BY

47
Schema for Examples
Sailors (sid integer, sname string, rating
integer, age real) Reserves (sid integer, bid
integer, day dates, rname string)

• Reserves
• Each tuple is 40 bytes long, 100 tuples per
  page, 1000 pages.
• Sailors
• Each tuple is 50 bytes long, 80 tuples per page,
  500 pages.

48
Equality Joins With One Join Column
SELECT FROM Reserves R1, Sailors S1 WHERE
R1.sidS1.sid

• In algebra R S. Common! Must be
  carefully optimized. R S is large so, R
  S followed by a selection is inefficient.
• Assume M pages in R, pR tuples per page, N pages
  in S, pS tuples per page.
• In our examples, R is Reserves and S is Sailors.
• We will consider more complex join conditions
  later.
• Cost metric of I/Os. We will ignore output
  costs.

49
Simple Nested Loops Join
foreach tuple r in R do foreach tuple s in S
do if ri sj then add ltr, sgt to result

• For each tuple in the outer relation R, we scan
  the entire inner relation S.
• Cost M (pR M) N 1000 1001000500
  I/Os (a lot!)
• Page-oriented Nested Loops join For each page
  of R, get each page of S, and write out matching
  pairs of tuples ltr, sgt, where r is in R-page
  and S is in S-page.
• Cost M MN 1000 1000500 501,000
• If smaller relation (S) is outer, cost 500
  5001000 500,500

50
Block Nested Loops Join

• Use one page as an input buffer for scanning the
  inner S, one page as the output buffer, and use
  all remaining pages to hold block of outer R.
• For each matching tuple r in R-block, s in
  S-page, add ltr, sgt to result. Then read
  next R-block, scan S, etc.

R S
Join Result
Hash table for block of R (k lt B-1 pages)
. . .
. . .
Input buffer for S
Output buffer
51
Examples of Block Nested Loops

• Cost Scan of outer outer blocks scan of
  inner
• outer blocks
• With Reserves (R) as outer, and blocksize100
• Cost of scanning R is 1000 I/Os a total of 10
  blocks.
• Per block of R, we scan Sailors (S) 10500
  I/Os.
• TOTAL 6,000 I/Os
• With 100-page block of Sailors as outer
• Cost of scanning S is 500 I/Os a total of 5
  blocks.
• Per block of S, we scan Reserves 51000 I/Os.
• TOTAL 5,500 I/Os

52
Index Nested Loops Join
foreach tuple r in R do foreach tuple s in S
where ri sj do add ltr, sgt to result

• If there is an index on the join column of one
  relation (say S), can make it the inner and
  exploit the index.
• Cost M ( (MpR) cost of finding matching S
  tuples)
• For each R tuple, cost of probing S index is
  about 1.2 for hash index, 2-4 for B tree. Cost
  of then finding S tuples depends on clustering.
• Clustered index 1 I/O (typical), unclustered
  up to 1 I/O per matching S tuple.

53
Examples of Index Nested Loops

• Hash-index on sid of Sailors (as inner)
• Scan Reserves 1000 page I/Os, 1001000 tuples.
• For each Reserves tuple 1.2 I/Os to get data
  entry in index, plus 1 I/O to get (the exactly
  one) matching Sailors tuple. Total
  221,000 I/Os.
• Hash-index on sid of Reserves (as inner)
• Scan Sailors 500 page I/Os, 80500 tuples.
• For each Sailors tuple 1.2 I/Os to find index
  page with data entries, plus cost of retrieving
  matching Reserves tuples. Assuming uniform
  distribution, 2.5 reservations per sailor. Cost
  of retrieving them is 1 or 2.5 I/Os depending on
  whether the index is clustered. Total 500
  40,000(1.22.5) 148,500

54
Sort-Merge Join (R S)
ij

• Sort R and S on the join column, then scan them
  to do a merge (on join column), and output
  result tuples.
• R is scanned once each S group is scanned once
  per matching R tuple. (Multiple scans of an S
  group are likely to find needed pages in buffer.)

55
Example of Sort-Merge Join

• Cost M log M N log N (MN)
• The cost of scanning, MN, could be MN (very
  unlikely!)
• With 35, 100 or 300 buffer pages, both Reserves
  and Sailors can be sorted in 2 passes total join
  cost 7500.

(BNL cost 2500 to 15000 I/Os)
56
Hash-Join

• Partition both relations using hash fn h R
  tuples in partition i will only match S tuples in
  partition i.

• Read in a partition of R, hash it using h2 (ltgt
  h!). Scan matching partition of S, search for
  matches.

57
Cost of Hash-Join

• In partitioning phase, readwrite both relns
  2(MN). In matching phase, read both relns MN
  I/Os.
• In our running example, this is a total of 4500
  I/Os.
• Sort-Merge Join vs. Hash Join
• Given a minimum amount of memory (what is this,
  for each?) both have a cost of 3(MN) I/Os. Hash
  Join superior on this count if relation sizes
  differ greatly. Also, Hash Join shown to be
  highly parallelizable.
• Sort-Merge less sensitive to data skew result is
  sorted.

58
General Join Conditions

• Equalities over several attributes (e.g.,
  R.sidS.sid AND R.rnameS.sname)
• For Index NL, build index on ltsid, snamegt (if S
  is inner) or use existing indexes on sid or
  sname.
• For Sort-Merge and Hash Join, sort/partition on
  combination of the two join columns.
• Inequality conditions (e.g., R.rname lt S.sname)
• For Index NL, need (clustered!) B tree index.
• Range probes on inner matches likely to be
  much higher than for equality joins.
• Hash Join, Sort Merge Join not applicable.
• Block NL quite likely to be the best join method
  here.

59
Simple Selections
SELECT FROM Reserves R WHERE R.rname lt
C

• Of the form
• Size of result approximated as size of R
  reduction factor.
• With no index, unsorted Must essentially scan
  the whole relation cost is M (pages in R).
• With an index on selection attribute Use index
  to find qualifying data entries, then retrieve
  corresponding data records. (Hash index useful
  only for equality selections.)

60
Using an Index for Selections

• Cost depends on qualifying tuples, and
  clustering.
• Cost of finding qualifying data entries
  (typically small) plus cost of retrieving records
  (could be large w/o clustering).
• In example, assuming uniform distribution of
  names, about 10 of tuples qualify (100 pages,
  10000 tuples). With a clustered index, cost is
  little more than 100 I/Os if unclustered, up to
  10000 I/Os!
• Important refinement for unclustered indexes
• 1. Find qualifying data entries.
• 2. Sort the rids of the data records to be
  retrieved.
• 3. Fetch rids in order. This ensures that each
  data page is looked at just once (though of
  such pages likely to be higher than with
  clustering).

61
General Selection Conditions

• (daylt8/9/94 AND rnamePaul) OR bid5 OR sid3

• Such selection conditions are first converted to
  conjunctive normal form (CNF)
  (daylt8/9/94 OR bid5 OR
  sid3 ) AND (rnamePaul OR bid5 OR
  sid3)
• We only discuss the case with no ORs (a
  conjunction of terms of the form attr op value).
• An index matches (a conjunction of) terms that
  involve only attributes in a prefix of the search
  key.
• Index on lta, b, cgt matches a5 AND b 3, but not
  b3.

62
Two Approaches to General Selections

• First approach Find the most selective access
  path, retrieve tuples using it, and apply any
  remaining terms that dont match the index
• Most selective access path An index or file scan
  that we estimate will require the fewest page
  I/Os.
• Terms that match this index reduce the number of
  tuples retrieved other terms are used to discard
  some retrieved tuples, but do not affect number
  of tuples/pages fetched.
• Consider daylt8/9/94 AND bid5 AND sid3. A B
  tree index on day can be used then, bid5 and
  sid3 must be checked for each retrieved tuple.
  Similarly, a hash index on ltbid, sidgt could be
  used daylt8/9/94 must then be checked.

63
Intersection of Rids

• Second approach (if we have 2 or more matching
  indexes that use Alternatives (2) or (3) for data
  entries)
• Get sets of rids of data records using each
  matching index.
• Then intersect these sets of rids (well discuss
  intersection soon!)
• Retrieve the records and apply any remaining
  terms.
• Consider daylt8/9/94 AND bid5 AND sid3. If we
  have a B tree index on day and an index on sid,
  both using Alternative (2), we can retrieve rids
  of records satisfying daylt8/9/94 using the first,
  rids of recs satisfying sid3 using the second,
  intersect, retrieve records and check bid5.

64
The Projection Operation
SELECT DISTINCT R.sid,
R.bid FROM Reserves R

• An approach based on sorting
• Eliminate unwanted fields in first pass of
  external sort. Thus, runs of about 2B pages are
  produced, but tuples in runs are smaller than
  input tuples. (Size ratio depends on and size
  of fields that are dropped.)
• Modify merging passes to eliminate duplicates.
  Thus, number of result tuples smaller than input.
  (Difference depends on of duplicates.)
• Cost In first pass, read original relation
  (size M), write out same number of smaller
  tuples. In merging passes, fewer tuples written
  out in each pass. Using Reserves example, 1000
  input pages reduced to 250 in first pass if size
  ratio is 0.25

65
Discussion of Projection

• Sort-based approach is standard it is often
  useful that the result is sorted.
• There is also an approach based on hashing (see
  book).
• If an index on the relation contains all wanted
  attributes in its search key, can do index-only
  scan.
• Apply projection techniques to data entries (much
  smaller!)
• If an ordered (i.e., tree) index contains all
  wanted attributes as prefix of search key, can do
  even better
• Retrieve data entries in order (index-only scan),
  discard unwanted fields, compare adjacent tuples
  to check for duplicates.

66
Set Operations

• Intersection and cross-product special cases of
  join.
• Union (Distinct) and Except similar well do
  union.
• Sorting based approach to union
• Sort both relations (on combination of all
  attributes).
• Scan sorted relations and merge them.
• Hash based approach to union
• Partition R and S using hash function h.
• For each S-partition, build in-memory hash table
  (using h2), scan corresponding R-partition and
  add tuples to table while discarding duplicates.

67
Aggregate Operations (AVG, MIN, etc.)

• Without grouping
• In general, requires scanning the relation.
• Given index whose search key includes all
  attributes in the SELECT or WHERE clauses, can do
  index-only scan.
• With grouping
• Sort on group-by attributes, then scan relation
  and compute aggregate for each group. (Can
  improve upon this by combining sorting and
  aggregate computation.)
• Similar approach based on hashing on group-by
  attributes.
• Given tree index whose search key includes all
  attributes in SELECT, WHERE and GROUP BY clauses,
  can do index-only scan if group-by attributes
  form prefix of search key, can retrieve data
  entries/tuples in group-by order.

68
Summary

• A virtue of relational DBMSs queries are
  composed of a few basic operators the
  implementation of these operators can be
  carefully tuned (and it is important to do
  this!).
• Many alternative implementation techniques for
  each operator no universally superior technique
  for most operators.
• Must consider available alternatives for each
  operation in a query and choose best one based on
  system statistics, etc. This is part of the
  broader task of optimizing a query composed of
  several ops.