Grid Packaging Technology Technical Talk

1
Grid Packaging TechnologyTechnical Talk
University of Wisconsin Condor/GPT Meeting January
3,4 2002 Michael Bletzinger mbletzin_at_ncsa.uiuc.e
du Specialist, Software Developer Advanced
Computational Environment and Security, NCSA
2
GPT Purposes

• Give developers an easy way to package software
• All packaging data is specified from a source
  perspective.
• Packaging data is strongly typed.
• Provides a means of building and installing
  collections of packages
• Provides standard version negotiation method for
  managing updates to packages.

3
GPT and Globus

• GPT was designed to re-implement the deployment
  requirements of Globus plus shared libraries.
• Globus most of Globus is libraries.
• Globus has deployment requirements for client
  tools, service daemons, and development/build
  environments.
• Globus wants to release individual components
  separately
• Globus wants to add outside software (ie.
  openssl, openldap) to these deployments.
• Globus wants to work on multiple platforms.

4
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

5
Splitting up the Source Code

• Source Package Types
• Source packages contain source code and
  documentation.
• Setup packages contain configuration/localization
  scripts and files.
• Test Packages contain component test code.

6
Source Packages

• Contains source code for programs, scripts, and
  libraries.
• All libraries are added to the link line.
• Contains documentation and data files.
• All of these share the same version number.

7
Other Packages

• Setup Packages
• Contains scripts and templates.
• These are assumed to have no build flavor.
• Generate files outside of the control of the
  packaging system.
• Test Packages
• Contains test harnesses and code.
• Installed in a special directory.

8
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

9
Binary Package Types
10
Package Transformations
Source Code
Source Package
Setup Package
Test Package
pgm
doc
pgm
doc
pgm
doc
dev
data
rtl
data
data
localized files
Installed Files
11
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

12
Dependencies

• A Dependency is a need relationship to another
  package
• RPM's "Requires" does the same thing.
• Different Dependency Types for Source an Binary
  Packages.
• Types include both build and runtime dependencies.

13
Runtime Dependencies

• Source Runtime dependencies are transferred to
  the specific binary package.
• Runtime dependencies require a binary package
  type in the specification .

14
Build Dependencies

• Are transferred to the binary dev packages
• Are used in the configure stage of the source
  package build.

15
Source Runtime Dependencies

• pgm_runtime
• Dependencies for the programs.
• lib_runtime
• Dependencies for the libraries.
• doc_runtime
• Dependencies for the documentation.
• data_runtime
• Dependencies for the data files.

16
Source Build Dependencies

• compile
• Requires header file from another package
• pgm_link
• Programs require libraries from another package
• lib_link
• Libraries depend on libraries from another package

17
Dependencies
Header File
Script
Source Package
Program Source Code
Package Foo
Package Foe
Library Source Code
Library
Documentation
Doxygen Page
Package Fee
Package Fum
18
Binary Dependencies

• Compile
• Only in dev packages
• Build_Link
• Only in dev packages
• Runtime_Link
• rtl and pgm packages
• Regeneration
• Only in pgm packages
• Runtime
• in any binary package

19
Source to Binary Dep Conversion
Binary Dependencies
Source Dependencies
Pass Thru
compile
compile
Dynamic Linking
runtime
runtime
Static Linking
build_link
lib_link
runtime_link
pgm_link
regeneration
20
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

21
Build Flavors

• A Build Flavor is a set of compilation and
  linking options which have to be defined to
  produce binaries that can be linked with each
  other.
• A development environment issue for older
  versions of Globus.
• Now a runtime issue with GT 2.0 because of
  shared libraries and plugins.

22
Build Flavors

• Build Flavor Choices
• Compiler Choice gcc, vendor cc
• Size 32 bit, 64 bit
• Debugging Debug Symbols included or stripped
• Thread Package pthreads, solaris, sproc, none
• message passing library mpi, none
• Standard Flavors in green
• Example
• gcc32dbgpthr gcc compiler, 32 bit, debugging
  symbols, pthreads

23
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

24
Versioning Purposes

• Source code change identifier.
• Distribution identifier.
• Marketing tool.
• Interface compatibility identifier
• for APIs.
• for protocols.
• for configuration files.

25
Version Negotiation.

• Agreement on compatibility between package
  developer and package user.
• Package developer makes a compatibility
  recommendation.
• Package user choose to accept or override the
  recommendation.

26
Aging Version

• Version scheme used by the package developer.
• Based on libtool and other shared library schemes.

minor version number - indicates a binary
compatible change
major version number
4.1.3
age - indicates backward compatibility to
the major version number
27
Version Requirements

• Simple Version Requirement.
• Package user specifies major version number.
• Trusts the developer compatibility
  recommendation.
• Version Range Requirement.
• Package user specifies a range of major and minor
  version numbers.
• Does not use the compatibility recommendation
  from the package developer.
• Combinations of these can be used for more
  complex requirements.

28
Versioning Example
Version for the package is 5.1.3
29
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

30
Installation Objectives

• Standard structure for both runtime and build
  installations.
• Multiple build flavors can exist in the same
  location.
• Packaging data co-located with the installation.

31
Installation Structure
bin
programs
sbin
lib
libraries
libexec
gcc32 headers
include
GLOBUS_LOCATION
gcc32pthr headers
packaging metadata
etc
setup
test
32
(No Transcript)
33
How to move an installation.

• Copy Directory.
• Re-run setup scripts.

34
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

35
Bundling Objective

• Be able to install a collection of packages or
  even an entire distribution with one command.
• Packages are not installed if compatible ones
  already exist.
• All setups are consolidated into one post install
  script.
• Be able to build a binary bundle with one command.

36
Bundling Data

• Name and version of the package.
• Binary package types that are needed.
• Build options to be avoided ie. no threads.
• Build Flavors are chosen during builds for each
  platform.

37
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

38
Setup Objectives

• Allow installed packages to be configured with a
  script.
• Each script packaged in a setup package.
• Script execution is recorded in the packaging
  data.
• Setup packages can be replaced by distributing
  organizations to tailor a distribution.

39
Setup dependencies

• Source package indicate a setup need by a special
  setup dependency.
• Setup packages have two names and versions.
• One name/version for the specific package.
• different setup packages have different
  name/version.
• One name/version to fulfill the setup dependency.
• All setup packages will have the same
  name/version for this.

40
Setup Package Example
Setup Package Foe
Regular Package Foo
Setup Package Fum
Setup Package Fie
41
Outline

• Splitting up the source code
• Binary Package Types
• Dependencies
• Build Flavors
• Versioning
• Installations
• Bundling
• Setup
• The GPT Tools

42
What does GPT offer?

• An easy way for developers to package their
  software.
• An easy way for people to build large sets of
  source packages.
• Provides a package manager for those systems that
  don't have one.
• Provides packaging metadata in an XML format.
• Compatible with other package managers

43
An easy way for developers to package their
software

• gpt_setup - Sets up Makefiles and autoconf to use
  packaging system.
• gpt_edit - GUI to allow developers to edit the
  metadata.
• patch-n-build - for those developers that want to
  keep their own build system.
• globus-makefile-header - for those who want to
  build applications using Globus components.

44
An easy way for people to build large sets of
source packages.

• globus-build -swiss army knife for building
  packages, bundles, and development directories.

globus-build \ -verbose \
-help \ -installdirpath_to
_installation \ -builddirpath_to_b
uild directory \ -loglogfile \
-srcfilesource_metadata_file \
-srcdirsource_directory \
-staticltbuild numbergt \
-install-only \ -force \
-std-flavors \ -all-flavors
\ \ macrovalue list of
flavors to build list of source
packages/bundles
45
Provides a package manager for those systems that
don't have one.

• globus-install - Installs/uninstalls binary
  packages
• gpt_verify - Verifies that a Globus installation
  is complete.

46
NMI Changes to GPT.

• Changes for April NMI release.
• New schema/dtd for packaging data.
• conversion script to update existing packages
• Add package signatures and other installation
  verification mechanisms.
• Add tools for installation management.
• Centralized file which records installations and
  installed bundles.
• De-globus-ize the tools
• replace "globus" with something else for script
  names variables etc.
• probably keep old globus names for compatibility
  purposes.