Te Ara and Tawa

1
Te Ara and Tawa

• designing an improved workflow and publishing
  system
• Ross Somerville, Production Manager, Te
  AraMinistry for Culture Heritage Te Manatu
  Taonga

2
Background government-funded reference publishing

• McLintock Encyclopaedia (1966)
• Dictionary of New Zealand Biography (1990)
• New Zealand Historical Atlas (1997)
• NZHistory.net.nz (1997)
• DNZB online (2002)
• Te Ara (2005)

3
Te Ara

• Government funding from 2002
• Long-term (10 years ) project
• 9 themes overview (New Zealand in Brief) 22
  Places
• Technology scalable and future-proofed

4
Te Ara Themes

• 2005 New Zealanders arrival and settlement
• 2006 Earth, Sea and Sky marine life, people
  and the sea, natural resources, and shaping
  forces such as geology and climate
• 2007 The Bush landforms, fauna and flora
• 2008 The Settled Landscape farming, rural
  life, and peoples impact on the land

5
Future themes

• Trade and Exchange economy, business and city
  life
• Connections social groups, families and
  communities
• Nation systems of government and symbols of
  national identity
• Daily Life the customs, leisure activities and
  beliefs that make New Zealand unique
• Creativity arts, culture, invention and
  innovation
• from 2005 Places 22 regions, 50 iconic places

6
Big ideas

• a comprehensive guide to the natural
  environment, history, culture, economics,
  institutions, peoples and social development of
  the country
• bilingual (English and Maori)
• c. 2,000 entries 4.5 million words, 25,000
  non-text illustrative resources

7
Flexibility and future proofing

• Aimed at multiple audiences through layered
  presentation
• High accessibility
• Content separate from presentation (heavy use of
  CSS)
• Extensible, flexible (content stored in XML)
• Sustainable, secure (non-proprietary)
• Single source, multiple outputs (Web, print,
  WAP...)

8
Rich content, micro content?

• Text, images, audio, video, interactives (Flash)
• Layering of content
• Short story a quick fix
• Image Media Trail / Galleries
• Further sources books and websites
• Related content in Te Ara and the 1966
  Encyclopaedia
• Multiple navigation options

9
Content status 2007

• Two topic themes published (c. 200 entries 1.3
  m. words, 7,500 resources)
• Third theme (106 entries 300,000 words, 2,500
  resources) in preparation
• NZ In Brief published 6 Places published (more
  in prep.)
• 1966 Encyclopaedia online
• Two books published (based on new Zealanders
  theme) another in preparation

10
Monthly statistics

• 150,000 unique visitors a month
• 50 GB data per month
• c. 1m. page views monthly
• 5,000 to 8,000 daily visits
• c. 40 overseas traffic
• 34 direct 60 search engines (esp. Google)

11
Future goals

• More media
• Richer interactives
• Integrated and linked databases
• Integration of Dictionary of NZ Biography
• Interoperability with other web resources
• Data sharing

12
Te Ara content structure

• Themes
• SubThemes
• Entries
• SubEntries
• Resources
• see www.TeAra.govt.nz

13
Units of production

• Entry (100 per Theme 2 per week)
• BUT Entry is composed of multiple parts
• Blurb... Short Story... SubEntries... Topic
  Boxes... Footnotes... Further Reading... External
  Websites... Related Entries... Non-text
  resources... Captions... Credits... Metadata...
• Sub-units may move independently through various
  workflows
• Up to 10 SubEntries and 50 Resources in an Entry

14
The compilation process

• Selection and commissioning
• External and internal writers
• Stringent fact checking, reshaping and
  restructuring
• Identifying and acquiring non-text resources
• Designing, creating and optimising resources
• Captioning resources

15
More process

• Editing text for consistency and style
• Writing the Short Story for a younger audience
• Formatting texts for structure
• Conversion to XML
• Merging of data sources to create site structure
• Collation of design elements and content elements
• Publishing upload to staging

16
Process review

• Examined after publication of each theme
• Minor tweaks, clearer definitions of roles
• Sub-process management workflow coversheets
• No major changes to overall content preparation
  sequence or process
• Identified increasing need for better tools to
  manage marked-up content
• Has proven robust, but new requirements emerge

17
Underlying technologies and tools

• Microsoft Word
• SharePoint Portal Server repository
• Microsoft Access database / MS SQL database
• MS WordML
• Apache Cocoon pipeline
• .NET upload to Microsoft CMS
• ltoXygen/gt XML editor

18
Problems and irritants

• Reception and production out of synch
• Multiple workflow streams
• Multiple data sources
• Proxy and placeholder solutions
• Multiple revisions
• Batch publishing
• Technology work-arounds
• Legacy bugs

19
More problems

• Long publishing cycle
• Early conversion, late revision
• Unsupported exceptions, manual fixes
• XML interface clunky
• Multiple outputs required (web print)
• Increasing need for updates
• Increasing volumes to update

20
Effects

• Repetition, duplication
• Inconsistency
• Lack of content control
• Increasing fragmentation of work units

21
Towards a solution

• Review of publishing process
• Original driver editing interface ambulance at
  foot of cliff
• Commissioned analysis of current practices
• Uncovered broader issues of workflow process
• Mending some fences at the top of the cliff...

22
Drivers

• Requirement to maintain source and compile
  content outside CMS
• Reusable fragments
• Flexibility ability to add new components
  cost-effectively
• Repository interface
• Software upgrade path
• Remote access (future)

23
Tawa

• Te Ara Workflow Assistant
• Goals
• Improve version control
• Integrate interface to database and file content
• Provide real-time feedback for entries
• Search and replace operations across a variety of
  sources
• Share resources with other related projects

24
Content types

• MS SQL Server database
• MS Word and MS Word ML
• Image and A/V files (Flash)
• Te Ara XML
• TEI Lite
• dSpace

25
Structure
Web interface
Workflow
Resources
Members/teams
Projects/entries
Layout
Administration
Subversion repository
File systemdocuments
Tawa database
Resource databases
26
Technology

• Apache or IIS
• PHP using Symfony framework
• Javascript using Prototype and YUI libraries
• Subversion
• Tawa database
• Ancillary applications HTML Tidy, ImageMagick,
  etc.

27
Version control

• Subversion file system repository enables atomic
  changes using a number of interfaces
• XML rendering of database views enables
  database version control with rollback features
• Subversion stores the differences between file
  revisions less space and faster response times

28
Tawa database

• Supports MS SQL Server, MySQL, PostgreSQL and
  Oracle
• Mostly stores associations
• Also stores summaries performance increase
• Logs actions

29
Structure workflow
Web interface
Workflow
Resources
Members/teams
Projects/entries
Layout
Administration
Subversion repository
File systemdocuments
Tawa database
Resource databases
30
Workflow

• Members and teams
• Streams of work grouping certain types of
  workflow
• Stages and tasks based on the Dependency
  Structure Matrix/Method
• Notifications messages and email

31
Tawa dashboard
32
Structure projects entries
Web interface
Workflow
Resources
Members/teams
Projects/entries
Layout
Administration
Subversion repository
File systemdocuments
Tawa database
Resource databases
33
Projects and entries

• Provide a framework for publications
• Allows workflow management of distinct areas of
  the publication
• Entries are just content-focused sub-projects
• Sub-entries, again, are a convenient subset for
  workflow tasks

34
Projects interface
35
Structure resources
Web interface
Workflow
Resources
Members/teams
Projects/entries
Layout
Administration
Subversion repository
File systemdocuments
Tawa database
Resource databases
36
Resources

• Allows access to database records, file system
  objects and, eventually, more sophisticated
  source types (such as dSpace, eXist and other XML
  databases)
• Permits editing of some document types (at
  present just XHTML)
• Allows tasks, teams and members to be associated
  with assets

37
Resource interface
38
Access database records
39
Edit a database record
40
Writing and editing

• MS Word editing rounds and final transformations
  using Word ML
• Rich Text Editors TinyMCE and XHTML
• Remote editing of current version
• Missing track changes
• Search and replace operations easier in XHTML

41
XHTML editing interface
42
Structure layout
Web interface
Workflow
Resources
Members/teams
Projects/entries
Layout
Administration
Subversion repository
File systemdocuments
Tawa database
Resource databases
43
Layout

• Provides a working copy of a page
• Composed of a number of assets and components
• Dependent on AJAX and Javascript features for
  sizing and positioning use of YUI-ext

44
Structure members teams
Web interface
Workflow
Resources
Members/teams
Projects/entries
Layout
Administration
Subversion repository
File systemdocuments
Tawa database
Resource databases
45
Members and teams

• Members and teams are tied to workflow
  structures streams and stages
• Any team or member can be associated with a Tawa
  object task, project or resource
• A team may have a team leader, responsible for
  adding or removing members and making associations

46
Structure administration
Web interface
Workflow
Resources
Members/teams
Projects/entries
Layout
Administration
Subversion repository
File systemdocuments
Tawa database
Resource databases
47
Administration

• Adding and editing new projects/publications
• Adding and editing members and teams
• Adding and editing new sources
• Adding and editing streams and stages

48
Administration interface
49
Select a source
50
Progress to completion

• Repository, version control in place
• Resources and projects interfaces complete
• Workflow and Admin functions in revision
• Layout interface in development
• Specifications and end-user documentation
• GPL2 Open Source License

51
Why Tawa?

• Native support for XML
• Integrated workflow and compilation
• Readily extensible and configurable
• Open Source / GPL
• Economical

52
Pitfalls

• Authentication and Network issues
• Corporate IT policies
• Fragmentation of IT support
• Page layouts tail wagging dog?
• Remote contractor and lack of remote server access