AisMonitor

1
AisMonitor
Giovanni Chierico Camilla Stenersen
2
Background

• Multiple remarks and demands from both IT-AIS and
  IT-DES after various system failures which often
  had cascade effects on other systems
• Difficult to monitor(everybody has his own
  monitoring system)
• Difficult to know potential cascade effects
  (were there two separate problems or one causing
  the other)
• IT-AIS GL and DGL triggered the project using
  available TECH

3
Goals

• Create a common framework to monitor AIS
  applications
• Simple to integrate in the existing AIS IT
  environments
• Reliable
• Easy to extend
• Generic
• Keep history

4
Process

• Gather requirements
• Interviews with all the AIS Section leaders
• Meetings with IT-DES (Artur Wiecek)
• Meetings with IT-FIO (Sebastian Lopienski)
• Meeting with IT-AIS and IT-DES GLs
• Implement full app stack to monitor some sample
  apps.
• DB
• Messaging
• Applications hooks
• Interface
• Refine the application and implement hooks in
  AIS apps

5
Suggestions
-

• Keep it simple
• Dont reinvent the wheel

• Push Pull monitoring
• Communicate via DB APIs, HTTP, HTTPS, POP, SMTP,
  JMS,
• Aware of scheduled downtimes
• Digest alarms
• Monitor batch jobs
• Map dependencies
• Monitor business processes

6
Architecture

• Everything is a service
• Services are defined by two parameters
• The service level (L)
• The expected service level (E)
• The quality of service is QL-E
• Service hierarchy defines an oriented weighted
  graph, with no closed loops.
• Services communicate with the world via messages
  to modify their actual (L) and expected (E)
  service level.
• Level changes are propagated to the children
• An alarm defines a Q threshold for a given
  service, and is triggered accordingly

7
Architecture

• Message contains
• Payload
• Service level
• Expected service level
• Comment
• Authorization
• Service name
• Auth password

alarm
DB API
Service
Service
HTTP
Service
Service
Service
Service
Service
WS
Message
Service
Service
HTTPS
Service
Service
POP
8
Internals
Pull Http(s), POP
Alerts SMTP
Java BusLogic
Push HTTP, WS, SMTP
Export XML
Push DB APIs
Repository
MetaData Interface
Display Interface
9
Example
level
Expected level (E)
Actual Level (L)
Trigger alarm if threshold equals zero
time
10
Architecture Key Points

• Low entry cost
• No big initial effort to map dependencies, or
  modify applications. Monitor and map what you
  need.
• Positive feedback
• The more you invest (map deps, create app hooks),
  the more you get in return
• The more value we give back, the more likely the
  mapped structure will stay up-to-date
• Give control to the developers
• The developer knows best! No need to depend on
  others to monitor your app.
• Easily manage the alerts and the downtimes

11
For the AIS Developer

• PULL (AisMonitor checks the service) web apps
• Expose the test (e.g. on HTTP)
• Create AisMonitor service
• Configure AisMonitor PULL (where, when, )
• PUSH (App sends events to AisMonitor) batch jobs
• Create AisMonitor service
• Write the app specific test, or hook into
  existing process
• Sending the message can be usually done in 2-3
  lines of code

12
Push examples
Java
MuleClient client new MuleClient()
client.dispatch(axishttp//monitor/?",
Payload, null) or HttpClient client new
HttpClient() GetMethod method new
GetMethod(http//monitor/) //set
params client.executeMethod(method)
PL/SQL
MonitorPkg.push_at_link(name, pwd, level, expected,
comment) or SELECT utl_http.request('http//m
onitor/?params') FROM dual
13
Roadmap

• What we have today
• Push Run servers and receive messages over
  HTTP, HTTPS, and file system.
• Pull messages from HTTP and HTTPS, even behind
  AisLogin
• Propagate messages on Service hierarchy
• Persistence (Services, Dependencies, Messages,
  Statuses saved on DB)
• Export service status to xml, accessible over
  HTTP (needed by IT monitor)
• In a week
• Basic web interface (ApEx) to manage the services
  and the pulling params
• Pull messages via POP from ais.monitor_at_cern.ch
• Send alerts (SMTP)
• In two weeks
• Monitor implemented in production apps (CTA
  GAD) to test it with real data

14
?