Dynamic Circuit Services in US LHCNet

1
Dynamic Circuit Servicesin US LHCNet

• Artur Barczyk, Caltech
• Joint Techs Workshop
• Honolulu, 01/23/2008

2
US LHCNet Overview
Mission oriented network Provide trans-Atlantic
network infrastructure to support the US LHC
program
SARA
Starlight
CERN
Manlan

• Four PoPs
• CERN
• Starlight (? Fermilab)
• Manlan (? Brookhaven)
• SARA

2008 30 (40) Gbps trans-Atlantic
bandwidth (roadmap 80 Gbps by 2010)
3
Large Hadron Collider _at_ CERN
Start in 2008

• pp ?s 14 TeV L1034 cm-2 s-1
• 27 km Tunnel in Switzerland France

6000 Physicists Engineers 250 Institutes
60 Countries
Atlas
LHCb
ALICE
CMS
Higgs, SUSY, Extra Dimensions, CP Violation, QG
Plasma, the Unexpected
Challenges Analyze petabytes of complex data
cooperativelyHarness global computing, data
network resources
4
The LHC Data Grid Hierarchy
CERN/Outside Ratio 14 T0/(?T1)/(?T2)
12240 of Resources in Tier2sUS T1s and T2s
Connect to US LHCNet PoPs
Online
GEANT2NRENS
USLHCNet ESnet
10 40 Gbps
Germany T1
BNL T1
10 Gbps
Outside/CERN Ratio Larger Expanded Role of
Tier1s Tier2s Greater Reliance on Networks
Emerging Vision A Richly Structured, Global
Dynamic System
5
The Roles of Tier Centers
11 Tier1s, over 100 Tier2s ? LHC Computing will
be more dynamic network-oriented
Defines the dynamism of data transfers

• Prompt calibration and alignment
• Reconstruction
• Store complete set of RAW data

• Reprocessing
• Store part of processed data

Requirements for Dynamic Circuit Services in US
LHCNet

• Monte Carlo Production
• Physics Analysis

Tier 0 (CERN)
Physics Analysis
Tier 1
Tier 1
Tier 2
Tier 3
6
CMS Data Transfer Volume (May Aug. 2007)
10 PetaBytes transferredOver 4 Mos. 8.0 Gbps
Avg.(15 Gbps Peak)
7
End-system capabilities growing
88 Gbps Peak 80 Gbps Sustainable for Hours,
Storage-to-Storage
8
Managed Data Transfers

• The scale of the problem and the capabilities of
  the end-systems require a managed approach with
  scheduled data transfer requests
• The dynamism of the data transfers defines the
  requirements for scheduling
• Tier0 ? Tier1, linked to duty cycle of the LHC
• Tier1 ? Tier1, whenever data sets are reprocessed
• Tier1 ? Tier2, distribute data sets for analysis
• Tier2 ? Tier1, distribute MC produced data
• Transfer Classes
• Fixed allocation
• Preemptible transfers
• Best effort
• Priorities
• Preemption
• Use LCAS to squeeze low(er) priority circuits
• Interact with End-Systems
• Verify and monitor capabilities

All of this will happen on demand from
Experiments Data Management systems
Needs to work end-to-end collaboration in GLIF,
DICE
9
Managed Network ServicesOperations Scenario

• Receive request, check capabilities, schedule
  network resources
• Transfer N Gigabytes from A to B with target
  throughput R1
• Authenticate/authorize/prioritize
• Verify end-host rate capabilities R2 (achievable
  rate)
• Schedule bandwidth B gt R2 estimate time to
  complete T(0)
• Schedule path with priorities P(i) on segment
  S(i)
• Check progress periodically
• Compare rate R(t) to R2, update time to complete
  T(i) to T(i-1)
• Trigger on behaviours requiring further action
• Error (e.g. segment failure)
• Performance issues (e.g. poor progress, channel
  underutilized, long waits)
• State change (e.g. new high priority transfer
  submitted)
• Respond dynamically to match policies and
  optimize throughput
• Change channel size(s)
• Build alternative path(s)
• Create new channel(s) and squeeze others in class

10
Managed Network Services End-System Integration
Required for a robust end-to-end production system

• Integration of network services and end-systems
• Requires end-to-end view of the network and
  end-systems, real-time monitoring
• Robust, real-time and scalable messaging
  infrastructure
• Information extraction and correlation
• e.g. network state, end-host state, transfer
  queues-state
• Obtain via network services ? end-host agent
  (EHA) interactions
• Provide sufficient information for decision
  support
• Cooperation of EHAs and network services
• Automate some operational decisions using
  accumulated experience
• Increase level of automation to respond to
  increases in usage, number of users, and
  competition for scarce network resources

11
Lightpaths in US LHCNet domain
Dynamic setup and reservation of lightpaths has
been successfully demonstrated by the VINCI
project controlling optical switches
Control Plane
Data Plane
(Virtual Intelligent Networks for Computing
Infrastructures in Physics)
12
Planned Interfaces

• Most, if not all, LHC data transfers will cross
  more than one domain
• E.g. in order to transfer data from CERN to
  Fermilab
• CERN ? US LHCNet ? ESnet ? Fermilab
• VINCI Control Plane for intra-domain,
• DCN (DICE/GLIF) IDC for inter-domain provisioning

I-NNI VINCI (custom) protocols
UNI DCN IDC? LambdaStation? TeraPaths?
UNI VINCI custom protocol, client EHA
E-NNI Web Services (DCN IDC)
13
Protection Schemes

• Mesh-protection at Layer 1
• US LHCNet links are assigned to primary users
• CERN Starlight for CMS
• CERN Manlan for Atlas
• In case of link failure cannot blindly use
  bandwidth belonging to the other collaboration
• Carefully choose protection links, e.g. use the
  indirect path (CERN-SARA-Manlan)
• Designated Transit Lists, and DTL-Sets
• High-level protection features implemented in
  VINCI
• Re-provision lower priority circuits
• Preemption, LCAS

Needs to work end-to-end collaboration in GLIF,
DICE
14
Basic Functionality To-Date

• Semi-automatic intra-domain circuit provisioning
• Bandwidth adjustment (LCAS)
• End-host tuning by the End-Host Agent
• End-to-End monitoring

Pre-production (RD) setup Local domain routing
of private IP subnets onto tagged VLANs Core
network (TDM) VLAN based Virtual Circuits
US LHCNet routers
Ultralight routers
Ciena CoreDirectors
High performance servers
15
MonALISA Monitoring theUS LHCNet Ciena CDCI
Network
SARA
CERN Geneva
USLHCnet
Starlight
Manlan
16
Roadmap Ahead

• The current capabilities include
• End-to-End monitoring
• Intra-domain circuit provisioning
• End-host tuning by the End-Host Agent
• Towards a production system (intra-domain)
• Integrate existing end-host agent, monitoring and
  measurement services
• Provide a uniform user/application interface
• Integration with experiments Data Management
  Systems
• Automated fault handling
• Priority-based transfer scheduling
• Include Authorisation, Authentication and
  Accounting
• Towards a production system (inter-domain)
• Interface to DCN IDC
• Work with DICE, GLIF on IDC protocol
  specification
• Topology exchange, routing, end-to-end path
  calculation
• Extend AAA infrastructure to multi-domain

17
Summary and Conclusions

• Movement of LHC data will be highly dynamic
• Follow LHC data grid hierarchy
• Different data sets (size, transfer speed and
  duration), different priorities
• Data Management requires network-awareness
• Guaranteed bandwidth end-to-end (storage-system
  to storage-system)
• End-to-end monitoring including end-systems
• We are developing the intra-domain control plane
  for US LHCNet
• VINCI project, based on MonALISA framework
• Many services and agents are already developed or
  in advanced state
• Use Internet2s IDC protocol for inter-domain
  provisioning
• Collaboration with Internet2, ESNet,
  LambdaStation, Terapaths on end-to-end circuit
  provisioning