Declarative%20Programming%20and%20the%20Business%20Unit

1
Declarative Programming andthe Business Unit

• Strategies for Custom Application Cost Reduction
• Dan McCreary
• Dan McCreary Associates
• September 11, 2007

2
DISCLAIMER

• The following views are my own and do not
  necessarily reflect the views of the Minnesota
  Department of Revenue or the State of Minnesota

3
Presentation Abstract

• Title Declarative Programming and the Business
  Unit
• Description For the last few years software
  development has seen a dramatic growth in the use
  of XML in all aspects of information systems. 
  What began as a convenient way to store
  structured documents and server configuration
  files has morphed into complete systems that are
  built using only XML and XQuery.
• The amount of procedural code written in new
  systems is dropping dramatically.  This has had a
  large impact on the ability of business units to
  create and maintain their own requirements and
  business logic without the need for an extensive
  software development staff.
• This presentation discusses how XML has allowed
  business units to precisely capture the "what"
  (sometime graphically) without the need to be
  concerned with the "how" systems are
  implemented.  We will present a full example of a
  rich-client web application that has been defined
  and built using only a pure "declarative" style
  of model-driven software development.  These
  technologies include CSS, XForms, XML Schemas,
  XML Transforms, XQuery and business rules.  We
  will also discuss how pushing this power out to
  the business unit requires new data stewardship
  and business semantics skills to be emphasized.

4
Audience Survey

• How many people here
• Work for organizations that have web sites?
• Are currently creating web applications?
• Are familiar with XML syntax?
• Have heard of the term declarative?
• Are moving toward service orientation?
• Have an enterprise service bus?
• Have a metadata registry?
• Regularly do ROI analysis on emerging
  technologies?

5
Incoming!
Has this web thing gone away yet?
XML
6
How did we get here?

• From punch cards
• to tables
• to trees
• to cubes and graphs

7
In the Beginning

• There were punch cards

and there were 80 columns and the widths were
fixed and batches of cards were called
"decks and they were loaded into tables in
core memory
8
And Codd and Date said

• Let there be relationships

Person
Project
Role
person_id
project_id
person_id
first_name
project_nm
project_id
last_name
description
role_nm
birth_date
status
time_pct
and we had the relational databases and we had
SQL and we had joins and we had ER
diagrams and we had PowerBuilder and all seemed
right with the worlduntil
9
Sir Berners-Lee Said

• Let there be HTML

lthtmlgt ltheadgt lttitlegtDans
Homepagelt/titlegt lt/headgt ltbodygt
lth1gtWelcome!lt/titlegt ltpgtWelcome to my web
site!lt/pgt lta href"about.htm"gtAbout Melt/agt
lt/bodygt lt/htmlgt
and we had URIs and we had HTTP and we had we
links and resources and we had trees (not
tables) and we had searchuntil
10
Customers Said..

• We wanted to access our tableson the webin HTML!

html
?
head
body
h1
p
a
and there were objects and there was Java and
JSP and there was ASP.Net and there were
models, views, controllers and there was
conversion from forms to objects to tables and
tables to objects to forms and there were rooms
and rooms of programmers
11
The Translation Pain Chain
Web Forms
Objects
RDBMS

• From web forms to objectsto SQL insertsto
  selectsto objects and back to web forms
• Many format translations
• Impedance mismatch

12
And there Was Chaos
HTML Forms
html
Objects
JSP/ASP
head
body
Hibernate
Serialization
h1
p
a
100 Frameworks
OR Mapping
Struts
JavaScript
XML
JDBC
SQL
.Net
AJAX
Sessions
Stored Procedures
And the number of languages you need to learn
and the headcount of the IT departmentwent
upand up
RDBMS
13
And the W3C Said

• Let their be simplicitywhat if weallowed you to
  put XML in the browserand database

html
Database
head
body
model
input
label
14
Database Vendors that Support XQuery

• IBM DB2 Version 9 PureXML
• Microsoft SQL Server 2005
• Oracle 10g Release 2 Enterprise Edition
• eXist (open source)
• 50 others

15
Evolution Metaphors

• Specialization of Languages
• Generalization of Languages

16
Evolution Specialization

• Darwins Galapagos Finches
• Beaks are highly adapted to different food
  sources
• Finches adapted to specific ecological "niches
  over millions of years of isolated evolution

• Similar to domain-specific declarative languages

See Wikipedia "Darwin's Finches"
17
Evolution Generalization

• Generalization The Raccoon
• The world has a higher population of raccoons
  today due to their ability to quickly adapt to
  changing urban environments
• Similar to highly adaptive procedural languages

18
Computer Science Abstractions
Higher abstractions
time
19
The Software Development Process
Declarative Languages
Requirements (BA)
Test (QA Staff)
Design (Architect)
Build (Programmer)

• Requirements are about What
• Design and Build is about How

20
A Declarative System Is

• A software development system, tailored to a
  specific domain (such as web applications), used
  to capture precise business requirements within
  the context of a problem domain (the implicit
  context)
• Declarative systems do not specify how
  requirements are implemented to build working
  systems. Declarative systems only define the
  requirements
• Declarative systems document requirements in
  specialized vocabularies and can be used to
  generate entire working systems including user
  interfaces, persistence and test data
• Declarative systems specifically omit some
  assumed requirements (such as system
  availability, performance, reliability, security
  etc)
• A Declarative system is a set of "little
  languages" with precise semantics that fit
  together like a puzzle to solve a problem

21
HTML, CSS and SQL
xquery version "1.0" lthtmlgtltbodygt lth1gtOld
Expensive Bookslt/h1gt ltulgt for book in
doc("books.xml")//book order by
book/title return ltligt book /title,
book/author, book/price,
book/pubyear lt/ligt lt/ulgtlt/bodygtlt/htmlgt
/ global CSS used by all web pages / body
font-family Arial, Helvetica, sans-serif
font-size 75 margin 0 padding 0
width 1000px h1 color blue padding 0
15px
lt?xml version"1.0" encoding"UTF-8"?gt lthtml
xmlns"http//www.w3.org/1999/xhtml"gt ltheadgt
lttitlegtDeclarative Systemslt/titlegt lt/headgt
ltbodygt lth1gtIntroduction to Declarative
Systemslt/h1gt ltp class"author"gtDan
McCrearylt/pgt ltp class"date"gtApril 2007lt/pgt
lt/bodygt lt/htmlgt
HTML
CSS
XQuery

• HTML, CSS, XQuery and SQL are declarative
  languages using these definitions
• The semantics or meaning of each tag in the
  file is determined by an external organization
• The syntax does not have to be XML

-- Old expensive books SELECT title, author,
price, pubyear FROM books WHERE price gt 100
AND pubyear BEFORE 1960 ORDER BY title
SQL
22
XML Schema Sample

• Screen capture of Altova XML Spy
• 30 minutes to learn graphical notation

See http//www.altova.com/products/xmlspy/graphic
al_xml_schema_editor.html
23
XML Schema (XMLSpy Diagram)
Solid linemeans required
Data Element
Sequence of items
Dashed linemeans optional
Cardinality
Definition
Graphical notation that most non-programmers can
learn in 20 minutes.
24
Use Case Electronic Forms

• User fills out a web-based form
• Typical requirements may include listing of data
  to be gathered, data types, validation codes and
  data repetition patterns
• Examples
• HTML Forms
• XForms
• InfoPath

25
Forms Players
lthtmlgt ltheadgt ltxfmodelgt
ltxf/modelgt ltheadgt ltbodygt ltxfinputgt
lt/xfinputgt lt/bodygt lt/htmlgt
My-XForm.xhtml
See Wikipedia XForms
26
What Is Declarativeness for A Context?

• Efficiency at capturing the testable business
  requirements in a semantically precise and
  concise manner
• Example Is a field optional? Will validation of
  the data fail if the field is missing?
• Ability of each data element to have precise
  meaning over time and within organizations

Schema Drawing Tool
XML Schema File (.xsd)
XForms
xfinputrequired font-weight bold
xfinputrequired .xf-value
background-colorfff6af requiredafter
font-weightbold font-size1.5em
content "" color red
XForms CSS tags
27
Declarative Spectrum
HTML
CSS
Apache Ant
C
AssemblyLanguage
Python
Java
C
C
XSL
XML Schema
XQuery
XForms
XPath
Groovy
XUL
JavaScript
Ruby
perl
XQuery Update
SQL

• For any given context different languages have
  different levels of declarativeness
• General purpose languages are less abstract but
  can solve a wide variety of problems
• Declarative languages have a more narrow purpose
  and target a specific problem like styling a web
  page or selecting data

28
Middle-Tiers Remain Procedural
declarativeness
user
database
Presentation/style (client tier)
Data definition, insert,selection and
update (persistence tier)
Business Logic (middle tier)

• When the interface is consistent, declarative
  languages flourish
• Middle tiers tend to have the most variation
• Wikipedia lists over 200 web application
  frameworks
• CMSMatrix.org lists over 700 content management
  systems

See http//en.wikipedia.org/wiki/List_of_web_appl
ication_frameworks and http//www.cmsmatrix.org
29
The Application Development Puzzle
30
Filling In Each PieceWith A Declarative Language
Metadata Registry
CSSHTML
XMLSchema
XQuery
BPEL
XForms
XQuery Update
Cocoon
ApacheAnt
XSL
31
Semantically Precise Vocabularies
Standard Domain
Atom/RSS Syndication/Notification
ACORD Insurance Industry Forms Data
FpML Financial Products Markup Language
FixML Banking (Financial Information eXchange)
HL7 Healthcare (now using XML)
MISMO Mortgage Industry
NIEM National Information Exchange Model (registry)
OTA OpenTravel Alliance (Travel industry)
SIF School Interoperability Framework
GJXDM US Department of Justice (registry)
XBRL XML Business Reporting Language
See http//en.wikipedia.org/wiki/CategoryXML-bas
ed_standards
32
Metadata Shopping Tools
Phone
Address
FirstName

• You dont need to know about 100,000 SKUs to
  purchase 10 items from a grocery store
• Sub-schema generation tools give you exactly what
  you need and nothing more

See http//niem.gtri.gatech.edu/iepd-ssgt/SSGT-Sea
rchSubmit.do
33
(No Transcript)
34
If You Use Industry StandardsYou Could Be
Almost Done

• If you use industry standards
• and these standards publish their documents in
  XML Schema format
• and these standards have been transformed from
  XML Schema to XForms
• and you use native XML databases to store and
  XQuery to report on the data
• then sample applications have been created and
  do not require additional procedural code
• just change the constraints in the XML Schema and
  rerun the transforms

See http//www.exist-db.org
See also http//www.alphaworks.ibm.com/tech/purex
ml
35
Accessibility
Lower costs by moving routine logic maintenance
to lower levels in the pyramid
36
Java Libraries
10,000 class and 100,000 methods available but
which ones are relevant to your business problem?
37
Less is More

• XForms 1.1 has only 21 XML elements
• Much of the presentation of XForms is deferred to
  CSS
• Event management is deferred to the XML Events
• XML binding is deferred to the XBL standard

Mies van der RoheReconstruction of theGerman
Pavilion in Barcelona
See Wikipedia Minimalism
38
XForms is Only 21 Elements

1. Message
2. Model
3. Repeat
4. Secret
5. Select
6. Select1
7. Switch
8. Submission
9. Submit
10. Textarea
11. Trigger

1. Bind
2. Case
3. Input
4. Instance
5. Group
6. Help
7. Hint
8. Label
9. Load
10. Output

Color coding limited vocabularies can increase
the speed of pattern matching. Look for advanced
text editors to provide custom element coloring.
39
The New Semantics of "Nutshell"

• The 1.4 release of Java 2 Standard edition
  increases the size of the platform by 50, to
  2,757 classes in 135 packages
• 1.5 and 1.6 add additional classes

992 pages
nutshell something of small size, amount, or
scope in a nutshell in a very brief statement
40
Procedural Programming is Not All Poison

• It would be a mistake to tell all your procedural
  programmers that the programs they are creating
  are fundamentally evil
• The relevant questions are
• How closely does it fit the problem domain?
• Can BAs, SMEs and other non-programmers maintain
  the business rules?
• What are the chances that others will be able to
  maintain it in future years?
• How good are the development tools for your
  system?

41
If You Give a Kid a Hammer
the whole world becomes a nail

• People solve problems using familiar tools
• People develop specific Cognitive Styles based
  on training and experience
• What are we teaching the next generation of
  developers?

Source Shoshana Zuboff In the Age of the
Smart Machine (1988)
42
Semantics

• The science of meaning
• What you mean when you say cat
• How do you associate meaning with symbols
  (verbal, physical, textual)
• How do we know if we both mean the same thing
  when we use a word?
• What if a word has multiple meanings?

Reference WordNet
43
Semantic Triangle
concept
cat
symbol
referent

• Symbols can only link to referents through
  concepts
• You can not link directly from a symbol to a
  referent

Wikipedia Semiotic triangle
44
Communication
cat

• Domestic feline
• House pet
• Has fur
• Has whiskers
• Sometimes has fleas
• Chases mice

• Domestic feline
• House pet
• Has fur
• Has whiskers
• Sometimes has fleas
• Chases mice

• Communication involves exchanging symbols that
  describe common attributes
• A one-to-one match of attributes that describe a
  common symbol match implies a high precision match

45
Same symbol different meaning
customer

• Little match of common attributes
• Low precision semantic match

46
High and Low Precision

• The ideal is to have a precise standard and to
  use the data elements exactly as they were
  intended

47
Semantic Precision

• Semantic mappings are relative in time and
  between groups of people (organization)
• Semantic variability over time
• Something that has precise meaning to you today
  may not have the same precise meaning a year from
  now
• Our memories are imperfect and change over time
• Semantic variability across organizations
  (project and organization)
• A customer to one organization may denote a
  person but to another organization it may denote
  a company

48
Semantics of an XML Data Element
ltcodegt47lt/codegt

• A developer puts an XML data element in an xml
  file
• The tag has some meaning and the data within the
  tag has some meaning when it was created by the
  developer
• What is the probability
• That the developer will know the same meaning of
  the code 47 one year later
• That another project that opens file will
  understand and be able to use the meaning of the
  tag
• Vague standards often trigger vague
  interpretations of the meaning of data

49
The Heart of the Enterprise
The Metadata Registry
A metadata registry is a central location in an
organization where metadata definitions are
stored and maintained in a controlled method.
http//en.wikipedia.org/wiki/Metadata_registry
50
Semantic Precision in Space and Time
space (projects, organizations)
Large SemanticFootprint (long lifetimesystems)
world
enter-prise
dept.
team
Small SemanticFootprint (rapid prototype)
person
time
weeks
months
years
10 years
51
Use Procedural Glue
XForms XML(REST)
Procedural Glue
XQuery Update (JDBC)

• XForms uses a REST interface to send XML to a
  server
• Some XML databases (DB2 v9) sill use JDBC
  interfaces to insert XML documents
• Use procedural glue to build custom interfaces
  between systems with incompatible interfaces

52
How Quickly Can We Create New Declarative
Languages?

• Configuration files are really just small
  languages
• Configuration files are easy to parse, validate
  data using XML tools and build custom forms to
  use
• Easy to teach non-programmers to graphically
  build XML Schemas to validate XML files
• Answer About a week

53
How Quickly Can We Create Consensus?

• Example XForms standard
• Work started in 2001
• XForms 2.0 is still a two to three years away
• How can we accelerate this consensus building
  process?

What factors impact the rate that species
evolve? What impact does life span have on a
species? How are design ideas exchanged between
species?
54
Solution Wikis and Collaboration

• How long does it take to build consensus on the
  semantics of a new data element?
• How many people might use this declarative
  language?
• The larger the stakeholder group, the longer it
  takes

55
In the Can vs. On The Wire

• Make my data insert and update quickly
• Extract, Transform and Load (ETL)
• SQL Skills

• Create shared meaning between systems
• Register the meaning in a metadata registry
• XML Skills

Enterprise Service Bus
Metadata
56
Strategic Consequences

• Empowerment of business units to build and
  maintain their own applications with much less
  involvement of IT staff
• Dramatically reduces the need for middleware IT
  staff using traditional procedural languages like
  Java and C

57
Strategic Impact

• Consider this
• 50 of Applications are simple CRUD Applications
• Create, Read, Update, Delete
• Complex business logic can be stored in an
  XML-based rules engine such as Schematron
• How will this transform an organization?
• How will it allow business units to build and
  maintain IT applications?
• How will it allow the business to be responsive
  to customer needs?

58
Change Where the Line is Drawn
Requirements
Requirements
BAs
SME
Developers
vs.

• Shorten the distance between the business unit
  and the IT staff

59
Many Small vs. One Large
CSS
XForms
XSLT
XPath
XQuery
XML Schema
Business Rules

• It is easier to build a GUI on a small, precise
  language than a single large language

60
Data Stewardship/Governence

• Importance of the business units role in defining
  data elements semantics on-the-wire
• How to write and maintain enterprise definitions
• Concise
• Precise
• Distinct
• Non-circular
• Unencumbered

61
Parker Projection
RelativeCode Base
100
Time
Source Jason Parker, Minnesota Department of
Revenue, November 2006
62
Recommendations

1. Use semantically-precise declarative systems over
   procedural systems when the business problem is
   in the right context
2. Aggressively promote data stewardships within
   business units
3. Use controlled vocabularies and metadata shopping
   tools to allow non-programmers to select data
   elements from lists
4. Separate semantics (leaf elements) from
   constraints (branches)
5. Leverage the pattern matching portion of the
   right brain using graphical development tools and
   element-based syntax coloring editors
6. Use procedural code to build interfaces between
   incompatible systems
7. Isolate procedural by providing XML/REST service
   wrappers code behind semantically precise web
   services
8. Dont expect your procedural programmers to jump
   on the declarative bandwagon without some
   training
9. Use social networking tools (voting, wikis) to
   rank and find appropriate data elements

63
Next Steps For IT Strategists

• Read some articles on the semantic web
  (Scientific American)
• Read a book on business semantics
• Convince yourself it works
• Google for XForms Examples
• Google for XQuery Example
• Create a pilot project
• Pick a "friendly" business unit
• Find a medium-sized problem
• Train staff with XPath-centric tools
• Roll out to the enterprise in stages

64
XForms Wikibook

• http//en.wikibooks.org/wiki/Xforms
• Over 75 working XForms example programs

65
Devx Article

• Semantics and the Evolution of Specialized
  Languages
• http//www.devx.com/semantic/Article/34805

66
Thank You!

• Dan McCreary
• Dan McCreary Associates
• Enterprise Data Architecture Consulting
• dan_at_danmccreary.com
• http//www.danmccreary.com
• Cell [phone]
• Home [phone]