Toward a Guideline Authoring Workbench

1
Toward a Guideline Authoring Workbench

• Session 1
• May 6, 1999
• Barry G. Silverman, PhD

2
Penn People

• Penn Database Group
• Peter Bunneman, PhD
• Val Tannen, PhD
• Arnaud Sahuguet
• Realtime Systems
• Performance Lab
• Insup Lee, PhD
• Oleg Sikolski, PhD

• Lab for Informatics
• Intelligent Systems Tech
• (LIIST)
• Barry Silverman, PhD
• Chen
• Tony Yuan

3
Outline

• Purpose Challenges
• Authoring Process
• 1. Protocol Eligibility
• 2. Wellness/Prevention
• 3. Chronic (Ambulatory Care) Guidelines
• 4. Acute Care Guidelines
• 5. Critical Pathways
• 6. Education Decision Support Materials
• Maintenance Requirements
• Summary

4
Purpose Challenges

• Purpose of Todays Session
• Hypothetical Tour of finished CADSE
• Challenges to be Resolved Today

5
Challenges for Today

• Expose the Dream
• Work Toward a Unified DSS View
• Define Guideline Workbench Components
• Breadth
• Analyze Alternative Implementations (Depth)
• Make vs. Buy
• Usefulness, Scalability, Usability
• Synthesize a number of Workbench Requirements

6
Figure 1 - A Proposal to Merge Knowledge Life
Cycle Management, Open Standards, and Integrated
Environment
Authoring
Test Eval
Maintaining
Elicitation Wizards
Debugging Agents
Guideline Project Mgt Support Sys.
KB Models, Skeletons
Clarity Checks
Coherence Verifiers
HL7 RIM, XML/Kona
Milestone WBS Viewer
IRM
Correspon- dence Validators
Rqmts Mapper
Module History, Documentation
Common Medical Term Server
Workability Testers
Knowl. Grapher
Performance Enhancers
Change Control
Parser
Open-SW Canonical Form Guideline Repository
Rules for Workflow
Templates for Notes
Alerts/Remdrs for Orders
Execution Environment
Semantic Messaging (Publish, Notify, Remind)
Data Handline
Presentation
7
Deployment Map
Enterprise Guideline Repository
Kn. Mgt. Workbench
Enter- prise
Kn. Administrators
Local
Clinic- Specific Guideline Repos.
Division- Specific Guideline Repos.
Clinic- Access to KM Toolkit, Guideline
Changes, OLTP System
HFHS Vendor Components
Clinic-Specific
Division-Wide
8
CADSE Ingredients (ATP Diagram)
9
KNOWLEDGE MANAGEMENT GRAND CHALLENGE
National repository to capture/share practice
decision rules and electronic guidelines (open,
reusable, machine form) JIT Filtered Push, Not
Pull NOTE1 Need intuitive editing yet guaranteed
reliability for knowledge-ware. NOTE2 Local user
toggles/adapts desired widgets, NOTE3 The
Infrastructure Challenge (NCIS is assumed)

10
RQ - Guidelines-in-Use (tackling the authoring
component)

• Can a terminology-enabled, visual editor/wizard
  be created that
• Is easy for medical professionals to use
  (cognitive fit)?
• Supports guideline knowledge representations
  (e.g, eligibilities, algorithms, rules, decision
  tables, etc.)
• Causes guidelines to be better written
  (coherence, clarity, completeness, workability,
  etc.)
• Permits local revising/adapting to prevailing
  practice
• Helps authors produce guidelines that make the
  right thing the easiest thing

11
Knowledge Engineering Paradox

• Expert clinicians know what they dont know, but
  dont know what they know.
• Expertise is situationally triggered
• Its hard to do a data dump a priori
• We need to spend this time together to discover
  your knowledge of guideline authoring
• there aint no shortcut -- clinician time is
  essential to design CADSE right

12
Work Toward a Unified DSS View(requires
clinicians to specify it)

• What are you trying to put in front of the
  clinician-author ???????
• What should they put in front of
  clinician-practitioners ??????
• AGAIN KE theory says you arent able to
  elaborate this fully a priori (so lets walk
  through some situations together)

13
The CADSE Dream

• A Tour of what it could be.
• (with apologies to Asymetrix Multimedia Toolbook
  Instructor)

14
Knowledge Management Workbench - Prototype
Implementationver. 1.0 alpha (CADSE)
15
For the rest of this talk

• Try to picture yourself as a clinician-AUTHOR
  using the CADSE workbench

16
Scenario
Tom, the head of Guideline Development for Best
HMO, purchased CADSE in order to develop a suite
of guidelines including but not limited to 1.
Protocol Eligibility 2. Wellness/Prevention 3.
Chronic (Ambulatory Care) Guidelines 4. Acute
Care Guidelines 5. Clinical Pathways 6. Education
Decision Support Materials Tom found that CADSE
ate up these tasks like a knife through
butter. One thing he particularly liked was
CADSEs breadth as a toolbox
17
(No Transcript)
18
This interface allowed Tom to quickly complete
his range of authoring needs

• Using Templates of Guideline Types
• Selecting Objects From Library (add new ones)
• Setting Objects Properties
• Filling Out Object Content
• Adding Behavior (via Script)
• Getting Wizard Help (where needed)

19
Working with templates

• Tom began his guideline authoring using
• one of the many templates. A template
• is a prebuilt book that you customize
• by adding your own content. Templates
• save time and lend a consistent look and
• feel to your guidelines.
• In addition to using one of the templates
• provided with CADSE, you can create
• your own, custom template to distribute
• to other guideline authors.

TOP DOWN GUIDANCE PRINCIPLE
20
Author your Guideline

• After Tom chose a template, he then
• added his content. CADSEs easy-to-
• use authoring and development tools
• make this task simple.
• And CADSE is powerful enough to
• handle a wide variety of technical
• tasksfrom creating and managing
• new guideline objects to adapting those
• in a reuse library.

PRINCIPLE OF REUSE
21
Select objects from CADSEs Library

• He added features to his guideline by
• selecting from the wide variety of
• objects available in the CADSE
• Library.
• Library objects have built-in behavior
• to handle such tasks as eligibility,
• decision tables, workflow and logic
• charts, and clinical paths, among
• others.

ECOLOGICAL ARTIFACTS PRINCIPLE
22
(No Transcript)
23
Setting object properties

• Tom customized objects in his
• guideline by setting their properties.
• Each object, such as an eligibility
• assessment or clinical path, has
• properties that define its appearance
• and behavior.
• Tom could easily set an objects
• properties using the Properties
• dialog box or by responding to a
• wizard.

POLYMORPHISM PRINCIPLE (Cmd Keys, DMI or Agents)
24
Customize guideline behavior without limit
using JavaScript (ECMA)

• With CADSEs easy-to-learn
• JavaScript, Tom further customized
• guidelines to meet unique needs. He
• created new objects, customize existing
• objects, and programmed guidelines to
• exhibit unique behaviors or to simulate
• special processes.
• If he doesnt want to write script the
• wizard copies his actions and produces
• script for him

ADAPTIVITY PRINCIPLE
25
How to Implement Toms CADSE Dream

• Analyze Alternative Implementations?
• Make vs. Buy( integrate)
• Usefulness, Scalability, Usability
• Make Option Case Study
• Protégé dozens of person-years of effort
  (scalability usability still issues!!)
• One could spend all the budget and not be done!

26
The Buy ( integrate) Option

• Identify Best of Breed DSS Authoring Tools
• Scalable, usable, generic, open-standards based
• Identify Next Release Change Requirements
• Lobby effort needed, evolvable API concept
• Assure Workbench is an Integration Platform
• Interface Definitions
• Common Presentation Layer Issues
• Fill In Missing Components

27
Goals of a Knowledge Management Workbench

• Top Down Support
• Templates
• Reusables
• Sharing Existing Objects
• Libraries

• Bottom Up Support
• Free-form Creations
• Custom Designs
• Local Adapting
• Capture of New Objects
• Learning
• Indexing

• Group Development, Version Tracking,
  Administration

NOTE Guideline Checking is an Extensive Topic
for the Workshop
28
Power vs. Learning Tradeoff Curve
Shallow Workbench (visual, with wizards)
Learnability
Hybrid
Powerful Workbench (programming langs)
Power of Expression
29
How should a workbench support guideline creation
efforts?
ANSWER A mix of approaches is required (HYBRID)
30
Figure 1 - Overview of the Three Layers of the
Knowledge Management Tasks Ring, Tools Ring, and
Repository Core
Level 3 Registration
Guidelines Development Cycle
Level 2
Test/Execution Environment
VGR Administration

• Ontology
• CBR /Search
• Version

• Workability
• Benckmarking/Quality Ass.
• Integration (R2Do2,CDR, Careweb)

Level 1 Registration
Guidelines
Deployment
Development
Tools (HOLON)
Tools (COPE)
3
Level 2 Registration
See/Fill In Textual Annotations Explanations
Canonical Guidelines Repository (CGR)
Critics

• Clarity
• Completeness
• Logic
• Lexicon

Syntactic and Semantic Tools (Servers)
Semantics, Syntax, Structure Models
Exchange Protocol Definition Models

• Knowledge Mediators
• Info Agents

TOOLS LAYER

• ARDEN syntax
• HL7 RIM
• GLIF EGO

Vocabulary Server

• SNOMED RT
• LOINC
• CPT 4
• NDC

TASK LAYER
31
Outline

• Purpose Challenges
• Authoring Process
• 1. Protocol Eligibility
• 2. Wellness/Prevention
• 3. Chronic (Ambulatory Care) Guidelines
• 4. Acute Care Guidelines
• 5. Clinical Pathways
• 6. Education Decision Support Materials
• Maintenance Requirements
• Summary

32
(No Transcript)
33
Starting a New Guideline

• Pick a Type
• Automatically Create a Container
• Wizard Asks Author for Boilerplate
• Authors, Date, etc.
• Target population, needs, cost implications
• Rate of change Practice variation
• Evidence (degree of vs. opinion)
• Difficulty of development, maintenance,
  implementation
• - KPNWs CPG Policy Procedures Paper
  (7/98), p. 8

34
The Knowledge Model

• IoMs 5 Guideline Types (others?)
• screening prevention,
• diagnosis pre-diagnosis mgt of patients,
• indications for use of surgical procedures
• approp. use of techs/tests for clinical care
• guidelines for care of clinical conditions

35
The Knowledge Model (contd)

• For each Generic Type of Guideline, Need to
  Model
• Skeletal Plans/Chapters Sections
• Sorted by Workflow Element (Person-type,
  Sequential Step, Timing Points, etc.)
• Locally Adaptable

36
(No Transcript)
37
Importance of a Layout/eForms Manager

• Adapt all eforms (container GUI, eligibility
  objects, reusable components)
• Use to Create Many Items
• Progress Note Forms
• Clinical Path Tables
• Educational Materials
• Other
• Use to Create Other Tools (e.g., PocketCards)

38
Container Layout Manager/Visual Forms Designer

• REQUIRED Highly Scalable/Usable/Adaptable Forms
  Layout Manager
• Design electronic forms in a Visual Forms
  Designer fill them out using
• Web-based Visual Forms Builder
• Visual Forms Application Builder
• Visual Forms Webpage Plug-in
• Workflow for eForm Artifact
• Support for other Form Designer Tools and
  interchange standards
• e.g., Visual Forms Designer can also read
  FormFlow, OmniForm and
• JetForm forms after they are converted via the
  Visual Converter.

39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
Uses Advanced Basic
44
Sample Visual Designer Screens
Examine some of the screens to assess role of
clinician-authors Www.mmacorp.comvfftw.htm
45
Make vs. Buy - Layout Manager
46
(No Transcript)
47
Authoring Process1. Protocol Eligibility

• At 1st Blush Eligibility Seems Straightforward
• Inclusion Criteria
• Exclusion Criteria (Mutable, Immutable)
• BUT, .

48
There seem to be numerous eligibility type
problems

• Standard inclusion/exclusion lists (simple
  branching)
• Level of care/risk stratification (multiple entry
  points, multi-workflows, more probabilistic?)
• Caremap progression checking (temporal issues)
• Health risk assessments (consumer-run
  pre-eligibility checks lifestyle/self-care
  advice)
• Tools may differ somewhat for each

49

• Assessment During Rehab
• Assessment After Discharge from Rehab

50
What kind of authoring will clinicians do?

• Decision tree? (e.g., Lumina/Spreadsheet)
• has a well-grounded semantics
• probability handled readily
• fuzzy membership
• Visual Object Questionnaire Programming (e.g.,
  Toolbook CBT)
• may be the ECOLOGICAL artifact
• convert to HTML/XML
• what about script?

51
Case Study Uncomplicated Cystitis Protocol
Definition

• Eligibility Criteria
• Female at or 18 and at or
• History previous uncomplicated UTIs only
• Classic Sx of nocturia
• Exclusion Criteria
• Concurrent Coumadin Therapy
• Pregnant (confirmed or suspicious) by history
• Diabetes Mellitus, Renal insufficiency
• UTI within last 6 months, or freq. post-coital
  UTI
• Post-partum, less than 2 weeks ..

52
Decision Tree Buildup Process(I)Decision
Analysis by TreeAge (http//www.treeage.com)
53
Decision Tree Buildup Process(II)
54
Decision Tree Buildup Process(III)
6 kinds of nodes
55
Eligibility Decision Tree
56
Advantages and Disadvantageswith Decision Tree
(I)

• Advantages
• Easy to create (reusable objects wizards)
• Table type input by user is efficient
• Probability rollback procedure is a GROUNDED
  SEMANTICS (extends GLIF)
• Values can be queried from Patient Record
• Can be implemented in certain ES shells

57
Advantages and Disadvantageswith Decision Tree
(I)

• Disadvantages
• Might not work with missing values
• Might not interview at screen for missing values
• Visual GUI is feature-rich (FEATURE EXPLOSION) --
  but less tedious than GLIF
• How to parse Decision Tree output into GLIF (ES
  shell rules are easier to parse)?
• Often no visual developer GUI if in an ES shell

58
Same Case Study - Questionnaire Approach

• Asymetrix Multimedia Toolbook II
• Used for Computer Based Training Kiosks
• Competitors - MacroMedia, HyperCard
• Output - HTML and Java

59
(No Transcript)
60
SCORE
61
(No Transcript)
62
Eligibility Tree - Make vs. Buy Alternatives
63
Authoring Process2a. Wellness

• Whats involved here? Any more than
• Handout materials (education DSS)
• Show/Discuss Counseling pages
• Provide Links to support chat groups
• What examples?
• If the above process is correct, do authors need
  website wrapping tools? (eg W4F later)

64
Authoring Process 2b. Prevention

• This is like the Governments Put Prevention
  Into Practice Kits????
• Lots of schedules (physicals, regular exams)
• Immunization calls/timelines/reminding
• Workflow and timelines
• call center
• case manager
• appointment clerk
• nurse
• doc

65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
Visual Workflow - e.g., Filenet
69
(No Transcript)
70
(No Transcript)
71
While this may seem intuitive,

• HFHS Complains
• Not intuitive
• Too hard for their clinicians (WHY? Need to
  investigate)
• Possibility No model of clinical world to map
  to
• Outpatient clinic metaphor missing (ecologic
  artifact)
• Work Performers unmanaged spaghetti code

72
Alternative Timeline Management Metaphors

• Alternative 1 - Find a visual metaphor that
  cognitively fits each type of situation (DMI)
• Alternative 2 - Drive the interview for workflow
  info via a knowledge model of what needs to be
  collected (Agent I/F)
• A few ideas from R2Do2

73
The R2Do2 Agents from a Workflow View
Triggers, Event Server
Parser Broker
Remote Records
Guideline KB Repository
Guideline Agents (Short-lived, multi-thread)
Reminder Agents (Long-lived, wake/sleep)
User Adds Notification Preferences
Users, Patient, Doctor, Nurse, etc.
Plan(ToDo Items) UnPlan(Items)
Ongoing Alerts Reminders
Email, beeper, ITV, etc.
74
Case Study The R2Do2 Immunization Agents
Knowledge Base -- DTP Vaccine
75
(No Transcript)
76
Metaphor that uses graphics, 2-D seq, table
format, and color to capture temporalities (still
need to add personnel resp.)
77
Preceding Picture Conveys Much More Info Than a
Table Format
78
The semantics are of a temporal logic
79
ALT 2 - Agent Uses A Model to Drive the
Interview CDCs HL7 Spec for Immun. Data
80
(No Transcript)
81
Given a Generic Model, a Wizard Should Work for
New Vaccines

• Wizard will interview for all the temporal
  semantical elements, and whatever else
• Can a generic model be extracted for all
  vaccines? (one works for all 7 current ones)
• How many other generic wizards must be created? -
  a research task (cost-plus, not FFP).

82
vCalendar The Electronic Calendaring and
Scheduling Exchange Format

• Consortium Members
• IBM, Lucent, Lotus, Microsoft, Novell, Netscape,
  Oracle, etc.
• Open grammar for personal organizers,
  professional schedulers, calendar software
• GUIs for this grammar have already sold millions
  (HIGH USABILITY)

83
R2Do2 Reminding Calendar GUI http//hera.seas.gw
u.edu/prevention/prevention.html http//bartok.sea
s.upenn.edu8080/
84
Markov Chain Oriented
Recur ComputeNext
(IAction)
DONE(PNk)
IAction
User Input Collect Scheme Update 2Do
Launcher Launch(PAS) Monitor(k)
User Preferences
DONE(Pk) - NULL(ISeparation)
Kill
Mk
-HOLDS(Pk) or DONE(Pk) NULL(ISeparation)
ISleep
Set Alarm ComputeNext(ISleep)
Sleep Monitor
AFTER(Now,ISleep), Monitor
Awaken Check
Archive
ISleep
Fi
Remind Consumer Provider
- DONE(Pk)
OODB Consumer Data
85
Make vs. Buy - Maintenance Tools
86
Authoring Process3. Chronic (Ambul. Care)
Guidelines

• What is the Chronic Guideline Generic
  Structure?
• Show an example Guideline Form
• (eg guideline-form widget from Protégé)
• Walkthrough Jetform Demo for Layout Mgt
• Walkthrough all Parts/Layouts
• Textual instructions sheets
• Visual Annotated Algorithm (GLIF Pallette,
  Risk-Costs-Benefits Table)
• Drill Down to Charting Forms
• Other

87
(No Transcript)
88
Typical Annotated Algorithm Table

• Visual flowchart usefule for elicitation
• (wizard assisted, terminology-enabled, GLIF
  semantics)
• Visual flowchart vital for display to users
• (save as a GIF or JPEG?)
• Annotations include extensive supplemental
  material
• clinician advice pages
• patient educational materials (tables, graphs,
  etc.)
• etc.
• Parse flowchart into machine executable (behind
  scenes)

89
Role of GLIF

• GLIF is an object oriented proposal for the
  interchange of machine executable guidelines
• Guideline Interchange Format

90
Intended Benefits of GLIF

• Store guidelines in repositories and eliminate
  duplication of effort between institutions
• Ease of dissemination (and maintenance)
• Deployment as executables

91
Basic GLIF Model
92
Strengths of GLIF

• Focuses on Types of Steps (bottom LHS)
• Well-Formed Semantics of Step Constructs
• Very Good for Sequential Algorithms

93
Example Annotated Algorithm in GLIF
94
(No Transcript)
95
Illustrative GLIF Pallet and Flowchart Drawing
Tool
96
Poor Criterion Logic
97
R2Do2 Criterion Logic as a Model Can Authors
work with this?
http//hera.seas.gwu.edu/knowledge_factory/login.h
tml
98
(No Transcript)
99
(No Transcript)
100
(No Transcript)
101
(No Transcript)
102
Limitations of GLIF (I)

• Drawbacks mentioned by the authors
• Representation of medical concepts in absence of
  a standard clinical vocabulary
• Lack of a formal syntax for the representation of
  conditional expressions
• Need to express temporal information in clinical
  guidelines
• Recognize and handle uncertainties about patient
  data

103
Limitations of GLIF (II)

• Class hierarchy is not clear
• All the objects are inherited from Guideline
  Model, but they share very few similarities
• Categorization of different steps
• Easier to model simple algorithms than
  annotations and complex guideline models
• Many guidelines are loosely organized and lack
  obvious causal effects between parts

104
Potential Improvements for GLIF(I)

• Revise the class structure and add more
  relationship representation between objects
• basically GLIF can only model steps and their
  links (fares poorly on many other relations)
• guideline must offer a containers to hold all the
  relevant information, definitions, etc. (EGO)

105
Potential Improvements for GLIF(II)

• Increase data structures
• Table Structures (Clinical Paths, C/B Risk)
• Equations, Continuous Graphs
• Graphics

106
But all of this is presumes a NARROW
INTERPRETATION of Guideline Semantics

• What is a BROADER INTERPRETATION?????
• Well for one thing, look at what CANNOT be
  supported by GLIF semantics

107
Weaknesses, What GLIF Omits
108
CADSE requires research on high level clusters of
GLIF semantics
109
What is the Chronic Disease Guideline Generic
Structure?

• SOAP(IE) too individualized
• EM Coding too high level
• Allan Khourys hospital progress note
• diseases, problems, complaint
• HPI
• Review of Systems
• Physical Exam
• Impression/Plan
• Orders
• Other? (eg., the above ignores workflow)

110
(No Transcript)
111
Design GUI for each step of interviewing
process. (Customizable)
112
Insert Charting Objects (Using an eForm Layout
Manager)
PATIENT CHART TEMPLATE Patient Name
_______________________________ Acct. Number
_____________________ Date _____ /_____ /_____
PATIENT
HISTORY ALLERGIES _______________________________
__________________________________________________
_____ CC ________________________________________
__________________________________________________
___ HPI _________________________________________
__________________________________________________
__ ______________________________________________
__________________________________________________
(location/duration/quality/context/severity/timi
ng/modifying factors/associated signs and
symptoms) or (three chronic or inactive
conditions) PFSH Past __________________________
__________________________________________________
______________ Family ___________________________
__________________________________________________
_________________ Social ________________________
__________________________________________________
____________________
113
REVIEW OF SYSTEMS (, -, na ) NOTES
_____Const _____Musc/skel _____Eyes _____Skin
_____ENT _____Neuro _____Cardio _____Psych
_____Resp _____Endocrine _____GI _____Hem/Lymph
_____GU _____Allergy/Immuno _____ See
additional notes
COMPLEXITY OF MEDICAL DECISION MAKING CODABLE 1
_____________________________________ 2
_________________________________ Diagnoses/ sym
ptoms 3 _____________________________________ 4
______________________________________ RULE
OUT Diagnosis 1 _________________________________
____ 2______________________________________ Mana
gement Plan/Orders/Medications Tests ordered
(reference codeable DX for each), performed,
reviewed and old records reviewed Instructions/C
ounseling/Coordination of Care Time in
_________ out _________ with pt _________
Physicians Signature ____________________________
________
114

• Assessment of pain intensity and
  character(QUESTIONNAIRE)
• 1.Onset and temporal pattern
• When did your pain start? How often
  does it occur? Has its intensity changed?
• 2.Location
• Where is your pain? Is there more than
  one site?
• 3.Description
• What does your pain feel like?
• What words would you use to describe
  your pain?
• 4.Intensity
• (see next page)
• 5.Aggravating and relieving factors
• What makes your pain better? What
  makes your pain worse?
• 6.Previous treatment
• What types of treatments have you
  tried to relieve your pain?
• Were they and are they effective?
• 7.Effect
• How does the pain affect physical and
  social function?

115
Interactive Graphic
116
PATIENT INSTRUCTION
117
Make vs. Buy - Annotated Algorithm
We must build Assure Open-std - XML
118
Authoring Process4. Acute Care Guidelines

• Triage in the ER
• Indications for Use of Surgical Procedures
• Appropriate Use of Specific Technologies and
  Tests as Part of Clinical Care
• Outcomes capture and analysis

119
(No Transcript)
120
Website Wrapper Tools
Also for Extraction of Text/Data
121
(No Transcript)
122
(No Transcript)
123
(No Transcript)
124
Building a wrapper for theNational Guideline
Clearinghouse

• We present how to use the our Wrapper Factory
  (W4F) to write a wrapper for the National
  Guideline Clearinghouse.
• Our wrappers consist of
• a retrieval layer (to fetch the remote Web
  document)
• an extraction layer to extract information from
  the HTML source
• a mapping layer to export the data to
  higher-level applications
• As a concrete illustration of extraction, we want
  to build an XML document that briefly describes
  the guideline.

125
Step 1 grab a guideline sample

• Identify a Web page that describes a guideline
• Guidelines should all have (more or less) the
  same structure

126
Step 2 Feed the guidelineinto the extraction
wizard
127
The extraction wizard

• The extraction wizard returns the Web document
  with some invisible annotations.
• The display is identical to the original
  document.
• Text areas get highlighted when the user puts the
  mouse over them
• it helps the user identify information boundaries
• it tells the user how to reach this piece of
  information using the hierarchy of the document

128
Step 3 Define some extraction rules

• Using the information provided by the extraction
  wizard, one can now write the wrapper.

EXTRACTION_RULES all html (
-ai-pcdata2.txt
-aj-pcdata2.txt
-ak-pcdata2.txt -pl (
.txt -a1.getAttr(href) -pm.txt )
) where html-ai.getAttr(name)
"'title'" and html-aj.getAttr(name)
"'references'" and html-ak.getAttr(name)
"'release_date'" and html-pl.b0.txt
"GUIDELINE AVAILABILITY" and
html-pl-pm.txt "Print copies"
129
The output XML document and its DTD

• ILABILITY)

plague recommendations of the Advisory
Commit tee on Immunization Practices (ACIP)."
MMWR Morb Mortal Wkly Rep 1996 Dec
1345(RR-14)1-15 37 references
1996 Dec 13
GUIDELINE
AVAILABILITYElectronic copies Available from
the Centers for Disease Control and Prevention
(CDC). http//..
Print copies Available from the Centers
for Disease Control and Pre vention, MMWR,
Atlanta, GA 30333. Additional copies can be
purchased from the Su perintendent of Documents,
U.S. Government Printing Office, Washington, DC
20402 -9325 [phone].

130
Work to be completed on Penns W4F

• Classic Power vs. Usability Tradeoff
• Working and Scalable (wrapper is for programs)
• No GUI yet available (not usable by clinicians)
• Other Research Issues ...

131
(No Transcript)
132
Authoring Process5. Clinical Pathways

• Reusable Tables and Other Objects
• Several Options to Alter Objects
• Wizard interview to change them
• Control Preference Screens to change them
• Direct manipulation
• Objects self-parse into class hierarchies, XML
  sheets, precedence relations, workflows

133
(No Transcript)
134
Typical TableBuilder Screenshot
135
Authoring Process6. Education/DSS material
136
(No Transcript)
137
Educational Materials Generation

• Education Materials Text/Table Construction
• Clinician DSS Pages (eg, evidence tables)
• Patient Education Handouts
• Useful Toolset
• Type and format
• Cut Paste from other sources
• Webpage authoring (e.g., Netscape Composer)
• Also Publishing the Concept Map/Website Semantic
  Net (RDF) is helpful for maintenance

138
(No Transcript)
139
Outline

• Purpose Challenges
• Authoring Process
• 1. Protocol Eligibility
• 2. Wellness/Prevention
• 3. Chronic (Ambulatory Care) Guidelines
• 4. Acute Care Guidelines
• 5. Clinical Pathways
• 6. Education Decision Support Materials
• Maintenance Requirements
• Summary

140
Maintenance/Admin Requirements

• Guideline Keeper KPNW PP pages 13, 14, 23
• consulting expert on that guideline
• exception outcome tracking
• literature surveying/detect evidence shifts
• recommends guideline updating
• Want tools to help him/her

141
Maintenance/Admin Tools

• General literature tracking tools
• User/Exception/Outcome reports and fixes tracking
• Administrative Tools
• set access privileges/maintain security keys
• track usage/adapting of guideline
  objects/versions
• manage library (checkin, checkout) of diverse
  containers, components, reusable objects, BLOBs
  (unlike Ontyxs homog. term graph)
• inter- and intra-guideline error checking

142
Traditional Maintenance Paradox

• Project Leaders and Organizations pressure
  project staff to complete software tasks with
  minimal budget and delay
• This savings in the development cycle is the very
  thing that leads to unmaintainable code
  downstream
• Maintenance is 50-90 of most SW project budgets

143
What are the Maintenance Needs and How can We
Predict Them?

• Examine Case Studies
• Study literature
• Find Principles/Methods (e.g., SEI Capability
  Maturity Model)

144
Case Study DECs XCON Expert System

• XCON VAX config expert system (1983-present)
• One of the worlds 1st expert systems (largest)
• 12,000 rules, 69 locations, released 3x/year
• Vastly successful (configuration quality and
  sales)
• All production operations70 mission-critical ESs
• 10,000 users, 250 staff, 200 million/year
  savings
• kept DEC afloat for many years

145
XCON Maintenance Nightmares

• Domain misunderstood - VAX change rate
  overlooked, clusters unhandlable, rules not
  reusable for other than VAX 11/780s
• Poor Make-Buy Decision - OPS Shell was
  inexpensive. They allowed it to drive design
• Employee turnover -- no one knows entire KB,
  cant remove anothers rule easily, little
  documentation (as always)
• Irony - unreadable, high level KB

146
XCONs Maintenance Solution Basic Reading
Riting (2Rs)
CLARITY ENFORCING EDITOR Master Structural
Template One Partition, One Problem Domain-Specifi
c Problem Solving Methods One Function Per
Rule Legal Syntax for Each Rule
OPS5 Knowledge Base
Authors
147
Maintenance Case Study 2 The COPE Kn. Management
System

• Automatically help Army to generate, critique,
  index, and maintain 600 new guideline type
  documents/year
• Use of DDN (classified Internet, pre-Web)
• Started 1989 just ahead of C and Windows
• Fielded 1991 (1992 IAAI awardee)
• Major code in C Windows Precursors
• Code Maintenance Issue How to convert seamlessly
  to C and Windows

148
How Did COPE Handle Maintenance?

• 5,000 Chunk Static KB basic guidelines on
  document critiquing document gen. Wizards
• Published index/concept map (automated)
• On-line updating tutorials
• On-line updating editors/critics
• Direct manipulation GUI at XCONs R2 level
• Color and textual cuing (critiquing, feedback)

149
How Did COPE Handle Maintenance (continued)?

• Dynamic KB of 600 new documents/year parsed and
  stored as OO annotated decision algorithms
• World Model concept map editor (like CMT server
  and tools)
• Automated indexing (down to reusable sentence
  structures) via Case Based Reasoning (CBR)
  algorithm
• Password protected signature chain for KB version
  updating
• Use of CBR to push reusables to users

150
Make vs. Buy - Maintenance Tools
Guideline Debug Check
151
Outline

• Purpose Challenges
• Authoring Process
• 1. Protocol Eligibility
• 2. Wellness/Prevention
• 3. Chronic (Ambulatory Care) Guidelines
• 4. Acute Care Guidelines
• 5. Clinical Pathways
• 6. Education Decision Support Materials
• Maintenance Requirements
• Summary

152
What we covered today

• Workbench challenges
• usefulness, scalability, usability
• power vs. learnability
• make vs. buy
• Dream CADSE scenario
• Workbench Requirements
• What are you trying to put in front of the
  clinician-author
• What should they put in front of
  clinician-practitioners to make right thing
  easiest thing

153
What we covered today (continued)

• Common, Ecological Artifacts Need to be Authored
  for a Range of Guideline Types (chronic, acute,
  prevention, wellness, educ, etc.)
• eForms, GUIs
• workflows and calendar/temporal metaphors
• algorithms their annotations
• eligibility/assessment trees questionnaires
• table structures (ClinPath, Risk, Dosage, etc.)
• educational materials

154
What we covered today (continued)

• 15 Tools of the workbench
• Breadth -- which ones to include
• Depth -- how to ground
• implement them

155
Make vs. Buy - Summary
156
Hardest Research Challenges Identified Today

• PRAGMATICS Discovering ecological artifacts
• How to best help authors create guidelines
  materials
• Clinician epistemology (dont know what they
  know)
• In lieu of 1st princ, at regular intervals need
  interviewing time, situational experiments,
  feedback discussions
• SEMANTICS Enhancing GLIF Layers
• Guidelines, Skeletal Parts, Steps, Logic
• SYNTAX Creating OO implementations (tools) with
  direct manipulation GUIs and wizards
• useful, scalable, usable

157
What Did we Skip Today
- Topics to be covered at length in the 2-day
Phila. workshop

• Integrating the Term Picker
• Groupware Editor
• Guideline Checking (lengthy topic)
• Guideline Executables (EGO Class Library)
• Order Feedback Rules -- Author, Make/Buy
• Knowledge Mediation Layer (lengthy topic)
• Update on Make vs Buy Investigations
• Other?