Title: Arial 28pt'

1
MES914 Driving Mobile Application Usage with
Natural Language Processing
Babak Hodjat Sr. Director Engineering bhodjat_at_syba
se.com August 15-19, 2004
2
iAnywhere at TechWave2004

• Ask the iAnywhere Experts on the Technology
  Boardwalk (exhibit hall)
• Drop in during exhibit hall hours and have all
  your questions answered by our technical experts!
• Appointments outside of exhibit hall hours are
  also available to speak one-on-one with our
  Senior Engineers. Ask questions or get your
  yearly technical review ask us for details!
• TechWave ToGo Channel
• TechWave To Go, an AvantGo channel providing
  up-to-date information about TechWave classes,
  events, maps and more also, keep up to date
  with the TechWave Newsletter now available via
  your handheld device!
• www.ianywhere.com/techwavetogo
• Mobile and Wireless Email using Pylon Anywhere
• iAnywhere is providing access to your corporate
  email at TechWave using Pylon Anywhere. You can
  keep up-to-date with your latest email, calendar,
  contacts, and tasks from your PDA or any
  Web-client! Visit the iAnywhere demo station in
  the Sybase booth or our Ask the Experts area in
  the Technology Boardwalk (Exhibit Hall) for
  details on how you can evaluate Pylon Anywhere
  yourself!

3
iAnywhere at TechWave2004

• Wi-Fi Hotspots brought to you by Intel
• You can enjoy wireless internet access via Wi-Fi
  hotspots provided by Intel. Using either a
  laptop or PDA that is Wi-Fi 802.11b
  wirelessly-enabled, visitors can access personal
  email, the internet and TechWave ToGo
• Developer Community
• A one-stop source for technical information!
• Access to newsgroups,new betas and code samples
• Monthly technical newsletters
• Technical whitepapers,tips and online product
  documentation
• Current webcast,class,conference and seminar
  listings
• Excellent resources for commonly asked questions
• All available express bug fixes and patches
• Network with thousands of industry experts
• http//www.ianywhere.com/developer/

4
The Enterprise. Unwired.
5
The Enterprise. Unwired.
Industry and Cross Platform Solutions
Unwire People
Unwire Information
Manage Information

• Adaptive Server Enterprise
• Adaptive Server Anywhere
• Sybase IQ
• Dynamic Archive
• Dynamic ODS
• Replication Server
• OpenSwitch
• Mirror Activator
• PowerDesigner
• Connectivity Options
• EAServer
• Industry Warehouse Studio

• Unwired Accelerator
• Unwired Orchestrator
• Unwired Toolkit
• Enterprise Portal
• Real Time Data Services

• SQL Anywhere Studio
• M-Business Anywhere
• Pylon Family (Mobile Email)
• Mobile Sales
• XcelleNet Frontline Solutions
• PocketBuilder
• PowerBuilder Family
• AvantGo

Sybase Workspace
6
Contents

• Introduction
• Mobile Usability Issues
• Mobile Integration Issues
• Natural Language Interfaces (NLI)
• Why NLI?
• Requirements for Mobile NLI
• Example NLI systems in Mobile (demos)
• Dejima Direct and the AAOSA NLI
• What is Dejima Direct?
• What is AAOSA
• Building an AAOSA Agent Network
• Why AAOSA NLI?

7
Introduction

• Mobile Usability Issues
• Screen Size
• Connectivity issues
• Input modality Restrictions
• Mobile Integration Issues
• How to provide many applications on a mobile
  device?
• The One application-set per modality problem
  (i.e., Application Redundancy)
• How to make different applications work together?

8
Natural Language Interfaces (NLI)

• Why NLI?
• Everybody knows how to express their intent in NL
• Requires less sophisticated interface
• Provides a unified interface across multiple
  applications
• Accommodates intelligent dialoguing to hone in on
  user intent
• Requirements for Mobile NLI
• Easy/Fast implementation for different
  applications
• Support for rapid/on-the-fly update/extension of
  application functionality
• Support for personalization
• Support for colloquial, non grammatical,
  abbreviated, and context sensitive NL

9
Example NLI Systems in Mobile (demos)

• Salesforce.com Wireless Edition
• Orange France SMS Services
• 21212 Orange Pages
• 20220 Information Portal
• Toyota ITC LBS System

10
Salesforce.com Wireless Edition (I)
Requests written in ones own words are all that
is required to access any layer of the solution
Change the Kodak deal to closed won
Change the Kod
11
Salesforce.com Wireless Edition (II)
In a single step, users can execute critical
transactions and locate hard to find data
You Requested Change the Kodak deal to closed
won Update Successful!
12
SFDC WE Email Modality (I)
A simple email sent to the service enables users
to get information with minimal connectivity
To Subject  
access_at_wireless.salesforce.com
get all opps closing this qtr gt200k gt30  
Sending
13
SFDC WE Email Modality (II)
Users get precise answers to their direct
questions

• From access_at_wireless.salesforce.com
• Subject REget all opps closing this
• qtr gt200k gt30
•  
• You Requested get all opps closing this
• qtr gt200k gt30
• Located OPPORTUNITIES with
• Owner Any
• Close Date 10/01/03 - 12/31/03
• Amount gt 200,000.00
• Probability gt 30

14
SFDC WE Quarterly Usage Statistics
5519
3992
Number of Requests
2322
1610
More than 360 increase between Jan, 03 and Nov,
03
15
Enterprise App UI
Traditional Method
Task Query Accounts by Owner and Area Code
16
NLI as Complement to Enterprise App UI
With Dejima Direct
Task Query Accounts by Owner and Area Code
17
Ambiguity and Contextual Follow-ups
18
Ambiguity and Contextual Follow-ups
AAOSA prompts the user to resolve ambiguity
Ambiguity Which John?
19
Ambiguity and Contextual Follow-ups
When the user clicks on a link corresponding to a
contact, the system uses context to take the user
directly to the target page, filling in required
fields
20
Orange France Information Portal
Users interact with applications using their own
words slang, abbreviations, symbols, acronyms,
SMS-Speak

cosy italian w/ bay view valet
ALBONA Zagat, valet, view, northern italian,
casual wear add 5721 Spear St, SF reply 1 or DIR
for directions reply 2 or RES for
reservations

• without memorized keywords
• without menu structures

21
Wireless Carriers Case Studies
gt 85 of requests were made using NL simple
user enters free form text instead of keyword
user experience enhancement complex request
not possible with existing keyword-based based
protocol functional enhancement help help
request or outside app scope triggering dialogue
to guide user to goal feedback to
carrier Usage traffic increased 300 by end of
3 month trial.
22
Toyota ITC Demo

• Speaker will
• Make request casually, not following a narrow
  grammar
• Deviate from flow suggested by system
• Follow-up assuming context
• Barge-in
• System will
• Preserve context allowing for follow-ups
• Search for requests within vicinity of user, a
  given location, or along the selected route
• Construct dialogues to resolve ambiguities
• Gracefully respond to out-of-scope requests
• Present help and tips

23
Banks around here
24
Dejima Direct Features

• Carrier grade, Enterprise ready
• Real-time self customization to customers
  specific meta-data structure
• Any modality to any back-end application(s)
• email, SMS/MMS, IM, web, WAP, speech
• Runs on Windows, Linux, Solaris
• Built-in load balancing and fail over mechanisms
• Embeddable on J2ME

25
Dejima Direct
Dejima Direct? is a robust, carrier-grade
software platform, enabling command and control
via ones own words, bypassing multi-layered
menus, enabling users to access information and
execute transactions in a single step.
26
Dejima Direct Mobility where it fits

• PIM and E-mail application
• Email
• Contacts
• Calendar
• Tasks

Carrier Network
Voice
Voice
WAP
STT/TTS
Messaging Portal/Gateway

• Content Servers, e.g. LBS
• Carrier own content
• Carrier portal

Data
Gateway
WAP
SMSC

• 3rd party WASP Servers
• Picture Apps
• Ring tones
• Games.

MMS
Integrated Infrastructure
Option 3
27
Dejima Direct Application Command and Control
Input
Interpretation
Actuation
Applications
Modality

• Text modalities
• Web
• WAP
• SMS
• Email
• IM

ACTION
Task Execution

• Drive App Logic
• Drive App Interface
• Transact
• Search

ooooo Patterns ooooo
TEXT
SRE Nuance Speechworks IBM ViaVoice Distributed Em
bedded

• API
• XML
• SQL
• VoiceXML

INTENT XML
Speech

• Context-Aware
• History
• App. State
• User Profile
• External Info

INTERACTION
Dialoging
CONTEXT

• Ambiguities
• Follow-ups
• Help

28
Benefits

• Higher task completion rate
• Faster time-to-success
• Reduced training requirements
• Leverage more of investment (breadth and depth)
• Direct user feedback understand what users want

29
Dejima Direct Underlying Technology

• The Agent-Oriented Approach
• Use of Agent Networks to model applications
• Agents collaborate and coordinate using messages
• Agents encapsulate determination of relevance to
  overall system process by making claims and
  proposing actions
• No centralized control (distributed)

30
Main Operating Principles

• Function oriented
• based on what the application can do
• Competition for claims
• Agents compete for the intent behind the user
  input
• Context awareness
• Agents maintain the context of the interaction
  and can dialogue with the user for clarification

31
Definition

• Objects
• Can deal with problems from a requestor that does
  not know if it is relevant to solving the
  problem.
• Adaptive Agent-Oriented Software Architecture
  Agents (AAOSA Agents)

32
Responding to a Service Request

• Requestor is aware of the agents relevance to
  the request and is expecting a certain manner of
  response
• Agent is reactive to service request

Same as Objects
33
Determining relevance to a posed problem

• Posing problem or parts of the problem to other
  agents
• Up-chain agents pose problems to down-chain agents

Unique to AAOSA
34
Offering Solutions

• Providing solution by means of capabilities
  internal to the agent and service requests
• Selection and Combination of solutions received
  from down-chains
• Determining a confidence in the solution
• Determining irrelevance,
• Establishing irrelevance for all down-chains

Unique to AAOSA
35
When is AAOSA Useful?

• Problems can be subdivided, and
• Solutions can be aggregated.

36
Why is it Better?

• Preserves locality of hierarchical information
• There is no need for centralized facilitation,
• In other words
• Agents exist within a hierarchy
• Agents are only aware of and communicate with
  agents that are directly connected to them.

37
Problem posing
Problem rerouting
Rerouting
Problem Delivery
Feedback

• Problem Solving
• Relevance Assignment

Failure reports

• Problem Composition
• Failure recovery

Problem posing
Feedback
Problem Solving Logic

• Solution Composition
• Failure recovery

Learning
Request / Response
Solutions
Programming / Enquiry
Solution
Solution Delivery
Solution Receiver
Failure reports
Service Request Processing
History
Service request / response
State
Local Objects
38
An Agent Network
39
Sample Dialog

• Find contacts in San Jose
• 5 contacts found
• and Sunnyvale
• 7 contacts found
• Not San Jose, San Francisco
• 9 contacts found
• Change Joes city to Mountain View
• Billing address or shipping address?
• Jill moved to France
• Country -gt France, city -gt
• New contact named Bill
• Whats Bills billing address?

40
Application Development Process
Deploy
41
Dejima Direct Tools

• iAnywhere provides more than 10 different tools
  covering the entire design, development and
  deployment cycle.

• Design Corpus Management Suite

• Develop Agent Network Development Environment

• Test Agent Network Test Environment

• Report Log/Edit manages statistics and reports

• Tune Web-based Wizards for Administration

• Deploy App Server and Management Tools

42
Comparison Use your own words

• Natural Language Grammar approaches
• Search Engines
• Statistical Methods

43
Natural Language and Linguistics

• Syntax
• Conditionals
• Questions
• Imperatives
• Assertions
• Active/passive
• Semantics
• boy active, noun, singular, human, male, age lt
  15 yrs,
• Pragmatics
• Baseball bat (sense wooden stick), park (sense
  playing field), objective score runs, hit a ball
  with a bat -gt score runs,

Sentence Noun Phrase Verb Phrase Det ADJs
Noun Verb NP PrepPhrases
The young boy hit the ball out of the park
44
Linguistics Grammar Approaches

• Approach
• Model rules for the language, and then fit an
  application domain into this model.
• Strengths
• Attempts to understand what user is saying to
  provide useful actions and results
• Challenges
• Costly Modeling language is complicated
• Brittle to non-syntactically correct inputs
• Requires specialized knowledge of linguistics
• Doesnt port across languages, most work not
  reusable
• Black box Hard to maintain, understand, because
  of many textual specialized rules with unclear
  scope and relationship

45
Search Engines

• Approach
• Index document keywords, returns weighted
  importance. No domain or linguistic modeling
  (except stemming)
• Strengths
• returning multiple scored hits from
  unstructured documents
• Fast setup for new domain/data set
• Challenges
• No understanding of query, just list of matches
• Returns many results
• Not suitable for answering precise questions,
  performing transactions, dialog

46
Statistical Methods

• Statistical Methods
• Require huge sample collection
• Long and costly development
• Difficult to change or extend application domain
• No dynamic customizations
• Unpredictable
• Good complement to AAOSA
• SLMs are the preferred Speech Recognition
  language models for Dejima Direct

47
Dejima Direct Scalability

• Meta-data driven
• Self-adapts to back-end customizations in real
  time without re-engineering
• Proven scalability to
• Carrier requirements
• Embedded requirements
• Rapid development cycle
• Architecture and tools scale quickly to new
  domains
• Language portability
• Agent Networks primarily model application
  semantics
• Existing products in English, French, Japanese