Beyond the simple Pipeline managing processes over time

1
Beyond the simple Pipeline - managing processes
over time

• XML Prague , 17/06/2007Geert Bormans,
  independent consultantgeert_at_gbormans.telenet.be

2
Introduction

• Who am I?
• Background
• 10 years team-lead and researcher
• researchgroup document architectures Catholic
  University Leuven
• 5 years VP professional services
• British/Canadian XML software vendor
• Independent Consultant
• Mainly projects that surface the intrinsic value
  of (yet existing) content
• Migrations, legacy conversions, XML
  transformations
• Content Management, content reuse
• e-publishing

3
Introduction

• What to expect?
• A quick introduction of the problem space
• Introduction of the project
• Introduction of the solution
• A word about pipelines
• Challenges for building pipeline solutions
• The solutions Architecture
• Wider applicability and future plans
• A quick demo

4
So we all love XML pipelines by now

• Ever thought about managing them?

5
Managing Pipelines - Problem space
6
The project

• The customer
• A financial institution working for the goverment
• Publishing yearly statements
• New legislation requires archiving the statements
• Current Architecture
• Complex Monolithic Code
• Growing with yearly changes

7
The project Current Architecture
8
Example Statement
9
Next years Example Statement
Publisher company rebrand
ID added to statement
10
Original Request for Proposal
11
Managing XML instead of PDF ...
12
... and version managing the pipeline
Each asset in the CMS is linked (through
metadata) to a specific version of the pipeline,
hence assuring the possibility to reconstruct a
statement as it was printed before
13
Pipeline benefits

• Break up complexity
• Simpler to develop
• Easier to debug and maintain
• Clear assignment of responsibilities
• Mix technologies easily
• Reuse components

14
Pipeline solutions - challenges

• Execution
• Fit components together and Compile Pipeline
• Execute with high performance in a robust way
• Avoiding intermediate (de)-serialisation
• Allow for mixed technologies
• Different technology per component possible
• Easy creation of pipeline
• Manageability

15
Solutions Architecture
16
Pipeline Component Managment System

• Processing steps are URI-addressable resources
• Steps pulled together at run time to create
  pipeline
• Execution by Pipeline Execution Engine
• Datamodel for simple pipeline

17
Pipeline Execution Engine

• Thin layer on top of NetKernel
• Resource oriented computing platform
• Microkernel that schedules resource requests
• Dynamic compilation of various scripting
  languages
• Strong support for XML processing tasks
• Pipeline language DPML

18
DPML Example
19
Meeting the challenges

• Pipeline execution by NetKernel
• Unnecessary serialisation/deserialisation avoided
  by notion of aspects and transreptors
• NetKernel supports pipeline components developed
  in various technologies
• PCMS serves as abstraction layer between pipeline
  development and execution
• PCMS takes care of pipeline version managment

20
Wider Applicability

• Architecture provides version managment of
  pipeline components
• Most important factor of complexity of XML
  processing is variation in time
• Applications layered on this architecture will be
  better protected against damage by changes
• A solution worthwhile considering for any complex
  XML processing application

21
Future Plans

• Non lineair pipelines (forking, looping)
• Dashboard for monitoring
• Timings, breakpoints
• Import filters for
• XProc
• Stylus Studio pipes
• ...

22
Questions and Discussion

• Thanks for your attention