Challenges

1
(No Transcript)
2
Challenges

• For large-scale applications, development is only
  half the battle
• Applications must be deployed (installed) to
  multiple servers
• Servers must be restarted when updates are made
• Restarts must be coordinated if end-users arent
  to be affected
• With many servers, this must be automated
• This must be managed in multiple environments
• QA environments, staging environments, and
  production environments
• Environment specific settings need to be
  maintained

3
More Challenges

• Server-side applications have many different
  components to deploy
• Java code
• JHTML or JSP files
• Third-party libraries
• Configuration Files (property files, rule files,
  repository descriptors, etc.)
• Content (images, style sheets, static text)
• Content is often managed and deployed on a
  separate schedule using 3rd party content
  management tools
• Operations staff must be able to roll-back to
  previous versions of an application in case of
  problems

4
AA.com Solutions

• Packaging
• Package applications as Dynamo modules
• Separate certain functionality into different
  modules (Web applications, administrative
  functions, and flight status notification)
• Shell modules for versioning
• Scripted Deployment
• Environment specific configurations using ATGs
  configpath capability
• Use of JMS messaging to coordinate updates from
  the Content Management System and other
  administrative functions

5
Packaging

• As mentioned, we package applications as Dynamo
  Modules
• Structure of a Dynamo Module
• Located in ltD5dirgt
• Directory which contains the resources needed by
  that module
• List of required modules
• Third-party libraries
• Custom Java classes (.jar)
• Classpath and Configpath settings
• Config file for the application
• A manifest that describes the module
  ltD5dirgt/ltmodulenamegt/META-INF/MANIFEST.MF
• We include other things in the module
• The Docroot (See Defining Modules slide)
• Scripts that are needed such as install scripts
  and SQL scripts
• For more details see Section 8 Application
  Modules of the Dynamo Deployment Guide

6
Building Modules

• AA.com uses ANT to build and package all code
• We built a set of ANT tasks to simplify this
  build-module.xml
• Using build-module.xml task, a module can be
  built by simply setting a few ANT properties
• The sample application presented later in the
  presentation is built using this ANT task
• The sample can be downloaded at
• www.atg.com/developernetwork

7
Defining Modules

• Most applications will need more than one module
• How do you decide what goes in a module ?
• Its best to have all pages (JHTML or JSP) in one
  module
• If there are administrative tasks that need to be
  done, for example, scheduled maintenance
  activities
• If the code will run only on a subset of servers,
  for example, Scenario Server
• If certain pieces need to scale or be deployed
  independently, for example, our Flight Status
  Notification (FSN) module
• Be careful about having more than one module
  serving pages (JHTML or JSP)
• The FileFinderServlet will handle multiple
  docroots, but it can lead to unexpected results
• Therefore, we split any page-handling instances
  out into separate modules

8
AA.com Modules

• AA.com consists of 5 modules (in addition to the
  standard Dynamo modules)
• AACom The Web application that runs the DRP
  service for AA.com
• AAdmin Loads content on a schedule and helps
  administrate the other AA.com instances through
  JMS Messages
• FSN Sends Flight Status Notifications (FSN) to
  customers via cell phones, email, etc.
• BX The Web application that runs the DRP service
  for BusinessExtrAA.com
• BXAdmin Helps administrate BX through JMS
  Messages
• Most server instances handle end-user requests
  and run the AACom and BX modules
• A separate server pool runs the AAdmin and
  BXAdmin module
• A third server pool runs the FSN Module

9
AA.com Modules Deployed
AACom/BX Servers
JMS
FSN Servers
AAdmin/BXAdmin Servers
10
Versioned Modules

• One of the goals of this scheme was to allow for
  scripted deployment
• By itself, packaging doesnt meet this goal
• File locking or overwrite problems when new
  modules are deployed
• Sometimes this is handled via a symbolic link,
  but this can also be problematic, for example,
  when the running application needs to load a new
  class before the restart
• Synchronization issues, for example, there is an
  error in the file transfer
• Rollback Issues
• We handle this via versioned modules and
  shell modules

11
Versioned Modules cont.

• Each module is named using a ltModuleNamegt-ltversio
  n_numbergt pattern, for example AACom-1_36
• The ltD5dirgtdirectory looks like this
• ltD5dirgt
• ---AACom
• ---AACom-1_36
• ---AACom-1_34
• ---AACom-1_32
• ---
• Each versioned module contains an install
  script(s) that performs any necessary actions
• The AACom module is a shell that only creates a
  dependency on the actual version of the site,
  i.e. the MANIFEST.MF contains just the following
• ATG-Required AACom-1_36

12
Scripted Deployment

• Our versioned modules allow us to completely
  automate (script) deployments
• The deployment script
• Copies the module files
• Stops a portion of the servers
• Runs the install script(s)
• Starts Dynamo
• Repeat process for remaining servers
• Written in perl
• Allows us to
• Upgrade without manual intervention
• Avoid inconsistencies during the restart
• Handle failed file transfers (we just dont start
  those servers)
• Rollback in case of failure

13
Environment Specific Configurations

• Being able to script deployment to multiple
  environments raises another problem many things
  are different in different environments
• Database connection information
• HTTP Server names
• We do not want the script to have to maintain
  this information
• This is typically handled via JNDI, DNS, or some
  other property file-based solutions
• ATG has a well-known method for creating
  different Configuration Layers Nucleus and
  CONFIGPATH
• Environment specific settings, for example, the
  liveconfig layer
• More robust than typical solutions, for example
  it allows administrators to enable per component
  debugging or logging

14
Understanding Dynamo Configuration Layers

• Nucleus creates and manages Dynamo Components
• CONFIGPATH acts much like the Java CLASSPATH,
  looking for property files in certain directories
• Property files used by the Nucleus to set values
  on Dynamo Components
• Default layers are found in
• ltD5dirgt/DAS/config
• ltD5dirgt/home/localconfig
• ltD5dirgt/home/servers/ltserver_namegt/localconfig
• DPS and DSS have several configpaths set in the
  MANIFEST.MF for those modules
• Configpath can be set by one of 3 mechanisms
• environment.sh (on UNIX) or environment.bat (on
  Windows)
• The MANIFEST.MF file for any module
• postEnvironment.sh (on UNIX) or
  postEnvironment.bat (on Windows)

15
Limitations with Dynamos Default Configuration
Layers

• When running multiple instances on a single
  application server, you must manage different
  postEnvironment.sh files
• Could be handled via symbolic links, if all
  Dynamo instances are identical, but in this case,
  just use ltD5dirgt/home/localconfig
• Usually cant do this because some Dynamo
  services are managed strictly via property files
  
• Session Backup Server
• Load Manager
• These instances wont be able to use the same
  postEnvironment.sh
• MANIFEST.MF doesnt solve this problem these
  settings are for the module only, not the server
  instance

16
AA.com Configuration Issues

• For AA.com, different instances may have
  different configurations
• DRPs
• Session Backup Servers (SB)
• Load Manager (LM)
• Different instances may run different
  applications (AA.com vs. FSN) in certain
  situations
• Airport closing due to weather may require
  additional FSN instances
• Developers need to develop against the exact same
  Dynamo module structure as in QA, Staging and
  Production
• Cant modify the MANIFEST.MF for environment
  specific configurations

17
More AA.com Configuration Issues

• We need to make sure that environment specific
  properties appear in the CONFIGPATH in a certain
  order
• Before ltD5dirgt/home/servers/ltinstance_namegt/localc
  onfig, but after everything else (global
  settings, module definitions, etc.)
• We want to be able to control this via command
  line parameters in the startup
• If the automated script cant start one instance
  with a certain service (like a Session Backup
  Service), it will try to start that service on a
  different instance
• Cant modify every postEnvironment.sh just to
  start a different application module

18
AA.com Configuration Solution

• We name instances with with DRP, SB, or LM as
  a prefix
• ltD5dirgt/home/servers/DRP1
• ltD5dirgt/home/servers/DRP2
• ltD5dirgt/home/servers/SB1
• ltD5dirgt/home/servers/LM1
• Manage overlay directories in each environment
• ltAAConfigDirgt/ltModuleNamegt
• ---global
• ---DRP
• ---SB
• ---LM
• Each directory contains the environment-specific
  configs for that instance type, global contains
  configs used by all instances
• Modify startDynamo.sh to include configuration
  directories based on the server name passed to
  script
• startDynamo DRP1 l f n m ltmodule_namegt

19
startDynamo.sh Modifications

• ) if "SERVERNAME" "." then
• case 1 in
• All the AACom instance types
  here
• DRP) SERVERCONFIGPATH"localc
  onfigINSTANCE_CLASS_DIR/AACom/global/localconfig
  INSTANCE_CLASS_DIR/AACom/DRP/localconfigservers
  /1/localconfig"
• LM) SERVERCONFIGPATH"localc
  onfigINSTANCE_CLASS_DIR/AACom/global/localconfig
  INSTANCE_CLASS_DIR/AACom/LM/localconfigservers/
  1/localconfig"
• SB) SERVERCONFIGPATH"localc
  onfigINSTANCE_CLASS_DIR/AACom/global/localconfig
  INSTANCE_CLASS_DIR/AACom/SB/localconfigservers/
  1/localconfig"
• ) SERVERCONFIGPATH"localconfigservers/1/
  localconfig"
• esac
• SERVERNAME"servers/1

20
startDynamo.sh Modifications

• Load instance class variables (primarily
  CLASSPATH)
• case ARG1 in
• DRP) . INSTANCE_CLASS_DIR/AACom/DRP
  /localconfig/postEnvironment.sh
• LM) . INSTANCE_CLASS_DIR/AACom/LM/
  localconfig/postEnvironment.sh
• SB) . INSTANCE_CLASS_DIR/AACom/SB/loc
  alconfig/postEnvironment.sh

21
AA.com Solution Pros

• Any changes to an application are made local to
  the environment in the configuration overlay
  directory, not the deployed application module
• All configuration overlays are located in one
  location per environment managed under source
  control
• Environment specific configurations are separate
  from general application configurations
• Flexibility to add/remove Dynamo instances of
  each type as needed with little
  setup/configuration
• Ensures development settings dont propagate to
  production
• Configuration settings are layered at the
  appropriate levels of granularity
• Stack global
• Instance Type (LM, SB, DRP)
• Application (AA.com)
• Individual instance (home/servers/DRP1/localconfig
  )

22
AA.com Solution Cons

• Must modify startDynamo.sh with each new ATG
  patch or upgrade
• Environment specific configurations exist in
  multiple locations (QA, Staging and Production),
  changes can get missed
• When new components are added, developers must
  communicate any property files that must be
  added to the overlays in the various environments
  
• Changes or additions are manual our automated
  deployment scripts dont yet handle the situation
  where new property files are added

23
Content Management

• By default, Dynamo supports 3 types of content
  repositories
• HTML
• XML
• SQL Database
• Dynamo also provides the SQL/File System
  Connector, a component that loads a SQL Content
  Repository with repository items from XML or HTML
  files
• For more information on content repositories, see
  Sections 18 SQL Content Repositories of the
  Dynamo Programmers Guide
• This doesnt solve all content management
  problems
• If using a SQL repository, who/what loads the
  content ?
• How are incomplete updates isolated?
• What about time sensitive content?
• How is cache invalidation handled?

24
Cache Invalidation

• Dynamo 5.1.1 provides 4 caching modes
• No caching
• Simple Caching Each server maintains its own
  cache and must be manually invalidated
• Locked Caching Central Lock Manager maintains
  locks to each item. Caches are invalidated when
  transaction commits
• Distributed Caching Messages broadcast whenever
  a change is made
• Most content is read only, so using Locked or
  Distributed caching has a lot of unnecessary
  overhead
• Simple caches may manually be invalidated with
  the command line utility GSAInvalidatorClient
• This fires a JMS Messages to all Dynamo instances
  and informs them to invalidate the caches

25
Content Synchronization

• During a load, content may be queried before all
  content is updated
• Some content must appear online at certain times
• ATG provides a component to help solve these
  issues called the SwitchingDataSource
• Class atg.service.jdbc.SwitchingDataSource which
  can be used anywhere a DataSource can be used
• Creates 2 copies of the database and allows
  content to be loaded into an inactive copy, then
  switched with the active copy
• Admin Server contains a user-interface to switch
  the instances
• For more information, please see Section 24
  Database Connection Pools of the Dynamo
  Programmers Guide

26
AA.com Content Management

• AA.com uses a 3rd party Content Management System
  (CMS)
• As mentioned, we built a separate module (AAdmin
  Module) to handle the interface with the CMS
• The CMS deploys content to the Server with the
  AAdmin Module
• The AAdmin Module then
• Uses the SQL/File System Connector to load
  content into the database
• Fires a custom message to invalidate the caches
  and/or switches the DataSource using an extension
  to the method used by the GSAInvalidatorClient
• Having a separate module lets us isolate this
  functionality from the site itself

27
Fare Sale Example

• Problem
• Fare sale data is published to the site via the
  CMS
• Fare sales are usually published in advance and
  made available at a specific date and time
• Several sales may be published, all at different
  times, but need to appear as one logical unit on
  the site
• Using the solution outlined on the previous
  slide
• The AAdmin Module loads the content using the
  SQL/File System Connector when it is published
• A number of rules are enforced on the data
• A scheduled activity in the AAdmin module then
  switches the DataSource and invalidates the
  caches for all request handling servers at a
  specific date/time
• The site serves only the active data online and
  doesnt worry about the management of this data

28
Custom Messaging

• We mentioned that we built a custom method to
  perform these tasks rather than use the standard
  Dynamo utilities
• Two examples we have seen are Cache Invalidation
  and Switching data sources
• Why?
• The standard utilities were too limited for our
  applications, for example, switching data sources
  must be done through the administration console,
  which is manually intensive
• We have many custom extensions to Dynamo classes
  and they require their own management

29
JMS Administration Facility

• Extending these facilities is straightforward
  using Dynamos SQL/JMS Implementation and
  Patchbay
• The messages these clients send are simple Java
  objects that are Serialized and sent to an
  appropriate listener
• For example
• The cache invalidate command is a serialized
  instance of atg.adapter.gsa.GSAInvalidationMessage
  .
• This message is sent (via SQL/JMS) to the
  component atg.adapter.gsa.GSAInvalidationReceiver
• In general, any command may be Serialized and
  sent to any component listening on the
  appropriate topic

30
AA.coms Custom Administration Messaging

• Commands are Serializable Java classes that
  extend a published interface
• public interface CommandMessage extends
  Serializable
• public void performCommand(ApplicationLogging
  logger)
• For example, this CommandMessage simply writes
  Hello World! to the Dynamo log
• public class HelloWorldCommand implements
  CommandMessage
• public void performCommand(ApplicationLogging
  logger)
• logger.logInfo("\"Hello World!\"")
• Now we need a way to send the messages and to
  invoked the performCommand() method

31
AA.coms Custom Administration Messaging cont.

• As mentioned, we use Dynamos SQL/JMS
  implementation and Patchbay
• Patchbay allows JMS components to be written
  easily and deployed by extending the
  MessageSource and MessageSink interfaces
• An XML file describes how these components are
  connected to send messages
• This file is placed in the Configpath and is
  named /atg/dynamo/messaging/dynamoMessagingSyste
  m.xml
• For more information, please see Section 45
  Dynamo Messaging System of the Dynamo Developers
  Guide

32
AA.coms Custom Administration Messaging cont.

• CommandMessages are sent by a component that
  implements the patchbay MessageSource interface
• public class CommandMessageSender extends
  GenericService
• implements MessageSource
• public void sendCommandMessage(CommandMessage
  event, String port)
• ObjectMessage message
• getMessageSourceContext().createObjectMessage(port
  )
• message.setJMSType(event.getType())
• message.setObject(event)
• getMessageSourceContext().sendMessage(
  port, message)

33
AA.coms Custom Administration Messaging cont.

• CommandMessages are received and invoked by
  CommandReceivers, which are an implementation of
  the patchbay MessageSink interface
• public class CommandMessageReceiver extends
  GenericService implements MessageSink
• public void receiveMessage(String port,
  Message message)
• CommandMessage command
  (CommandMessage) obj
• command.performCommand(this)
• These two components are connected by creating
  a JMS Topic and configuring DMS via the .xml file

34
AA.coms Custom Administration Messaging cont.

• In our case, we also needed to be able to execute
  these CommandMessages from a command line as well
  as from a server page
• We created a simple RMI server that calls
  CommandMessageSender.sendCommandMessage()
• A working example of this administration facility
  is available for download at
• www.atg.com/developernetwork

35
Summary

• Plan for deployment early
• Separate logical applications into separate
  Dynamo Modules
• Develop a versioning scheme that allows for
  scripting server upgrades and restarts
• Use configuration layers to handle environment
  specific configurations
• Separate Content Management and other
  administration functions into a separate module
• Investigate the use of DMS and patchbay (or JMS
  in a pure J2EE environment) to update the site
  with administrative messages
• QA