Data%20mirrors:%20Techniques%20and%20Issues

1
Data mirrors Techniques and Issues

• By
• Sarah Ponrathnam
• sarah_at_iucaa.ernet.in
• http//www.iucaa.ernet.in/

2
Overview

• Introduction
• What is Data Mirroring?
• Techniques of Mirroring
• Motives for Mirroring
• Mirror site Issues
• Popular Mirroring Tools in Unix
• Mirroring Experience at IUCAA

3
What is data mirroring?

• Creating an exact duplicate copy in real-time.
• In terms of web sites, sites are often mirrored
  to reduce the traffic on one server.

4
Techniques of Mirroring

• Replication is one way to solve availability
• problem.
• Distributed Servers
• Cluster Servers
• Web site Mirrors

5
Techniques of Mirroring

• Distributed Servers
• Large web destinations such as google,
• Yahoo have enough capital to set up
• and to support distributed servers.
• Eg. www.google.com, www.google.co.in

6
Techniques of MirroringCluster servers
7
Techniques of Mirroring

• Website Mirrors
• Mirrors simply compare and pull the contents
• from a single master web site at a regular
• intervals and make identical contents
• available on another computer, ideally closer
• to the users making use of it.

8
Motives for Mirroring

• Load Balancing - yahoo.co.uk yahoo.co.in
• High Availability - ADS mirrors
• Multilingual replication - www.debian.org
• Database Sharing
• www.desert.net www.tucksonweekly.com
• Franchise / Local Versions -
• quicken.excite.com quicken.com
• Virtual Hosting - sports.catalogue.com
  
• 
  www.accesports.com

9
Mirroring Issues

• Maintenance of mirror-sites in different
  geographical locations.
• Integrity of the mirrored contents
• Providing Host-Independent URLS.
• Initial transfer of data
• Optimization
• Economical constraints
• Providing Efficient routing

10
Popular Mirroring tools on Unix

• RSync
• http//samba.anu.edu.au/rsync/
• Wget
• http//www.gnu.org/
• Mirror
• http//www.wehlus.de/mirror/download.html

11
Mirroring Experience

• IUCAA (Inter University Centre
• For Astronomy and Astrophysics)
• works closely with ERNET
• (Educational and Research
• NETwork) to make this
• network a content oriented
• networkhttp//www.iucaa.ernet.in/

12
ERNET India ?

• In 1986, then Department of Electronics
• had initiated a project "ERNET" with the
• funding from UNDP. The objective was to
• establish and operate a nationwide
• Internet for the Indian academic and
• research community. Now it has become
• a full pledged ISP known as ERNET India.

13
ERNET partners
14
ERNET Network
15
ERNET NOC at IUCAA
16
Mirrors available in IUCAA

• VizieR provides access to the most complete
  library
• of published astronomical catalogues and data
  tables
• available on line, organized in a
  self-documented
• database. http//urania.iucaa.ernet.in

17
Mirrors available in IUCAA

• The NASA Astrophysics Data System (ADS)
  maintains four bibliographic databases containing
  more than 4 million records. http//ads.iucaa.ern
  et.in

18
VOI Data Archives at IUCAA

• SDSS Sloan Digital Sky Survey 1 Tb
• 2MASS 2 Micron All Sky Survey 194 Gb
• 2dfGRS 2 degree field Galaxy Redshift Survey
  5.5 Gb
• 2QZ 2 Degree field QSO Survey 630 Mb
•  FIRST Survey Faint Images of Radio Sky at
  Twenty centimeters 226 Gb

19
VOI Hardware
20
Mirror-site under construction

• Chandra Data Archive (CDA)
• 424 GB data is available through ftp service.
  ftp//cdaftp.iucaa.ernet.in
• Incremental update is being done regularly.
• Web based CDA service will be provided,
• once the new release of CXCDS software
• is made available.

21
Acknowledgement

• Dr. Francois Ochsenbein - VizieR
• Dr. Guenther Eichhorn and Dr.Alberto Accomazzi -
  ADS
• Dr. Ramadurai Padmanabhan - CDA

22

Questions or Comments?