RETS Software Development

1
RETS Software Development
New Topic

• RETS the standard
• RETS transactions
• Identify issues, concerns and recommendations
• Sample session workflow

2
RETS Software Development Topic Objectives

• Improve understanding of the RETS architecture
• Build familiarity with the RETS document
• Enable more productive RETS software development

3
RETSPrinciples - 1

• Stateless communications protocol
• Carry state using a cookie
• Basically a token that points to a persistent
  store on a server somewhere
• RETS defines an optional cookie for this purpose
• Recommendations
• Return all cookies received
• Client applications using browserswill need to
  accept cookies

4
RETSPrinciples - 2

• Uses Http-based transport
• Uses DMQL - an open query language
• Supports XML and site-specific output data
  formats
• Supports site-specific metadata to describe the
  local MLS system
• Recommendation
• Effort to support metadata produces
  interoperability benefits

5
RETS Document

• Organization follows a typical real estate
  industry workflow
• Initiate single session
• Multiple data requests
• Terminate session
• References standards produced by other standards
  organizations

RETS document can be challenging to read
6
Simple Collaboration Diagram
7
Login Transaction Making the Initial Connection

• Authentication
• Response with authorized resources
• Types
• Basic
• Digest using MD5 or SHA1
• SSL
• Recommendations
• Implement Digest using MD5
• SHA1 if you can

8
Login TransactionAuthentication

• Most common authentication type is digest using
  MD5
• Somewhat tricky to implement algorithm
• Easiest to borrow the implementation from one of
  the open source projects
• When in doubt
• Ask the RETS-DEV list
• Ask your software vendor

9
Login Transaction Activity Diagram
gt
lt
10
Login TransactionDigest Authentication

• Message Digest algorithms are one-way functions
• Easy to perform the calculation
• Difficult to reverse the calculation
• Both sides calculate the same message digest
• Transmit only the digest
• Compare your calculated digest with the received
  digest

11
Login TransactionUserAgent

• Application name accessing the MLS server
• All client requests MUST include this field
• This is a standard HTTP header field as defined
  in RFC 2616
• MLS server should validate application name
  presented against list of allowable applications
  maintained on a table
• Application name and userid is a useful key to a
  table that tracks MLS server usage

12
Login TransactionGotchas

• Using operating system libraries for MD5 can
  cause authentication failure
• Proxies and tunnels can be a problem
• Re-authenticate is part of the RFC but few
  vendors enforce it
• Case may matter

13
Sample LoginChallenge Response Header 1

• HTTP/1.1 401 Unauthorized xxx
• Date Mon, 02 Aug 2004 052354 GMT
• Server WebLogic XMLX Module 8.1 SP1 Fri Jun 20
  230640 PDT 2003 271009 with
• WWW-Authenticate Digest realm"users_at_ftc2.com",
  nonce"313035", opaque"6e"

14
Sample LoginChallenge Response Header 1 cont.

• Content-Length 0
• Content-Type text/html
• RETS-Version RETS/1.0
• Cache-Control private

15
Sample LoginNonce

• nonce
• "313035393731353435373630342058dd631265e3360724c45
  d15f23aec7a"

16
Sample LoginOpaque

• opaque"6e6f742075736564"

17
Sample LoginChallenge Response Header 2

• HTTP/1.1 200 OK
• Date Mon, 02 Aug 2004 053103 GMT
• Server WebLogic XMLX Module 8.1 SP1 Fri Jun 20
  230640 PDT 2003 271009 with
• Content-Type text/plain

18
Sample LoginChallenge Response Header 2 cont.

• Set-Cookie RETS-Session-ID20000439936
• RETS-Version RETS/1.0
• Transfer-Encoding Chunked
• Cache-Control private
• 0230

19
Sample LoginChallenge Response Body 2

• ltRETS ReplyCode"0" ReplyText"Success"gt
• MemberNameMark Crawford
• User632344, NULL, NULL, 632344
• BrokerNational,1
• MetadataVersion1.2.0
• MinMetadataVersion1.1.7
• OfficeListNational1
• TimeoutSeconds900

20
Sample LoginChallenge Response Body 2 cont.

• Searchhttp//rets.ftc2.com6103/search
• GetObjecthttp//rets.ftc2.com6103/get object
• Loginhttp//rets.ftc2.com6103/login
• GetMetadatahttp//rets.ftc2.com6103/getmetadata
• Logouthttp//rets.ftc2.com6103/logout
• lt/RETSgt
• 0000

21
Login Transaction Capability URL List

• Response from the Login transaction includes a
  list of known resources on the server
• Each URL points to a RETS transaction type
• Capability URL list may be different based on the
  user and their authorization level
• Required that the server return a list containing
  at least Search
• The URL references are not permanent locations
• the standard allows the location to change

22
Action-URL and Get Transaction

• Included in the Capability URL List is an
  Action-URL
• Points to a human readable message
• Most useful if changed regularly perhaps daily
• Immediately following successful login, may need
  to do a Get on the Action-URL
• Get Transaction is a file retrieval from the
  server

23
Metadata

• Description of the structure of the data on a
  specific server
• Unique to each site
• There will be common elements
• Most powerful feature of the RETS specification

Perhaps the most challenging part of the RETS
specification
24
GetMetadata Transaction Overview

• Transaction to retrieve system metadata
• Can select a specific level, the level and
  descendants or a specific metadata ID
• Formats supported are COMPACT and Standard-XML
• Metadata id value of 0
• Request is for all types at the same level
  breadth
• Metadata id value of
• Request is for all types at the same level and
  all child types - depth

25
Metadata FormatID

• Hierarchy of types of metadata
• See Figure 11.1 on page 11-2of the RETS
  specification
• When referencing hierarchy as ID
• Parent Child is the form
• Example
• Type METADATA-UPDATE_TYPE
• ID Property RES Add

26
Metadata FormatVersioning - 1

• Hierarchy of metadata is versioned
• Changes to lower level of the hierarchy propagate
  upward
• Example Changed METADATA-TABLE attribute will
  propagate up through CLASS, RESOURCE and SYSTEM

27
Metadata FormatVersioning - 2

• Caching of metadata is permitted
• Use the login information to determine metadata
  changes
• Recommendation
• If possible, get all the metadata on a change
  instead of attempting to get a specific type

28
Metadata FormatVersion 1.5 changes

• Added the Foreign Keys metadata
• Provides relationships between the offered
  resources of the form parentchild
• Clarified a number of issues with Metadata Format
• Recommendation
• Use 1.5 as a reference even when implementing
  1.0.1 compliant software

29
Application Responses to Metadata Changes

• Ignore metadata
• Change breaks the application, intervention
  required to fix
• Static metadata
• Change is invisible, applications keep working,
  intervention to required view change
• Dynamic metadata
• Change is visible to applications without
  intervention

30
Search Transaction Overview

• Most important RETS transaction
• Can use either http GET method or http POST
  method
• Example GET method requesthttp//rets.server.co
  m6103/search?ClassRES
• Issue GET method is limited in the character
  length of the request
• Recommendation POST method

31
Search TransactionClient Diagram
32
Search TransactionServer Diagram
33
Sample SearchSearch Request Header

• HTTP/1.1 200 OK
• Date Fri, 31 Jul 2004 232243 GMT
• Server WebLogic XMLX Module 8.1 SP1 Fri Jun 20
  230640 PDT 2003 271009 with
• Content-Type text/xml

34
Sample SearchSearch Request Header cont.

• Set-Cookie RETS-Session-ID20000439934
• Set-Cookie RETS-Request-ID"42"
• RETS-Version RETS/1.0
• Transfer-Encoding Chunked
• Cache-Control private

35
Sample SearchSearch Response Body

• 0fe8
• lt?xml version"1.0" ?gt
• lt!DOCTYPE RETS SYSTEM "http//www.ftc2.com/rets/dt
  d/RETS-20010812.dtd"gt
• ltRETS ReplyCode"0" ReplyText"Success"gt
• ltCOUNT Records"2507" /gt

36
Sample SearchSearch Response Body cont.

• ltREDatagt
• ltPropertiesgt
• ltResidentialPropertygt
• ltBuildinggt
• ltBuildingDescriptiongt
• ltStoriesgt1lt/Storiesgt
• ltPropertyConditiongtShows Welllt/PropertyConditiongt

RETS Software Development
37
Search TransactionSearchType Required

• SearchType The resource id to search on
• Example SearchTypeProperty

38
Search TransactionClass Required

• The class of data within the resource to search
  against
• Example ClassRES
• Issues
• Some of the data may be in system specific Class
  names rather than the Class name that represents
  an XML Standard Name

39
Search TransactionQueryType Required

• QueryType The query language
• Only two types DMQL and DMQL2
• Example QueryTypeDMQL2
• Issues
• Mainly stylistic difference, DMQL2 supports
  quoted text
• Not suitable for highly complex searches
• Not possible to perform cross Class
  searchesExample NOT Residential and Lot Land

40
Search TransactionQuery Required

• Query
• Key value pairs
• Uses field name based on metadata or on standard
  names
• Example
• Query(ListPrice200000-300000),(StatusA)
• Issues
• Lookup type queries can execute for a long time

41
Search TransactionOptional Arguments

• Optional argument are just that
• A vendor MAY choose to implement none, one or
  more
• Software is RETS compliant without any of the
  optional arguments implemented
• Issue
• Mismatch between client server feature sets
• Recommendation
• Do NOT assume, verify the feature set

42
Search TransactionCount

• Default is no count
• Behavior outside of given three values not defined

43
Search TransactionFormat

• Return records in XML or column header/row values
  format
• XML is more verbose, but contains structure
  information
• Some issues with site specific XML
  clarification in RETS needed
• Default is Standard-XML
• Issue
• Site specific XML is not defined in the current
  standard

44
Search TransactionFormat Sidebar XML - 1

• XML is a meta-language that defines a markup
  language
• RETS is an example of an XML application
• An application consists of a DTD and documents
  that obey the rules embodied in the DTD
• The XML DTD is a document that describes the
  structure and elements of an XML document

45
Search TransactionFormat Sidebar XML - 2

• The XML document takes data and applies the DTD
• RETS will eventually move from a DTD based
  vocabulary to a Schema-based vocabulary

46
Search TransactionFormat Sidebar Compact Formats

• Provides a delimited column description
• Provides delimited data in the same position as
  the column description
• Field delimiter is chosen by the server
• Gotcha
• Delimiter tag is a two character representation
  of the ASCII character
• Delimiter value may collide with data values in
  earlier implementations

47
Search TransactionLimit

• Limits the number of records returned
• The NONE well known name requests a suspension of
  limit
• Behavior is implementation dependent
• Requesting a Limit should make the query faster
• May still search for all records but only
  transmit records to a Limit

48
Search TransactionOffset

• Return records starting from the specified point
  in the result set
• One method to handle very large data pulls
• Given that the underlying transport mechanism is
  not guaranteed, very large results can fail with
  transport errors
• Given that the data volumes can be very large,
  out of memory errors can occur

49
Search Transaction StandardNames

• Indicates that the query uses the standard names
  or the system names
• Servers are not required to implement standard
  names to meet RETS compliance criteria
• Standard Names are intended to be common across
  all RETS systems

Refining the list of Standard Names is an ongoing
process
50
Search TransactionRestricted Indicator

• Certain fields may have restricted visibility
• Client may request fields that are restricted be
  replaced by a specific character string
• Example - compensation amount field
• Client may request that this amount be masked by
  
• Server default is to return a null value

51
Search TransactionSelect

• Provides the ability to create a subset of the
  fields accessible to the client
• Default is to return all accessible fields
• Gotcha Select field names are not specified as
  either the system names or the standard names
• Ambiguity in RETS standard
• Working assumption is that the server will
  understand both

52
Search TransactionDMQL

• Defines the search criteria for the search result
  set
• Based on a BNF grammar to provide a neutral
  representation
• RETS Server parses the DMQL and creates a server
  specific query
• Trade off between domain specific query forms and
  more complex query grammars like SQL-99
• Issues around the lexical parsing of the grammar

53
Update TransactionOverview

• Allows addition of records to the RETS system
• Supports Auto-population of fields
• References only by System Name, not Standard Name
• Validation based on rules described in the
  metadata
• Single record per update no bulk update

54
Update TransactionValidation

• Basic validation on data types
• Additional validation for single lookup,
  multi-lookup and range
• Additional validation for Boolean expression
• Additional validation using external validation
• Certain validation steps may be server only

55
Update TransactionError Block

• Validation tests that fail result in an error
  block that provides some information to assist in
  correction
• One error block per failed validation test
• Means a single field may trigger multiple errors

56
Update TransactionGotchas and Recommendations

• Gotcha
• RETS 1.7d1 adds a third state to the
  validate-flag 2 meaning validate the record
  but do not save it
• Be aware of this coming specification change
• Recommendation
• Client-side validation is fast fail use it

57
GetObject Transaction Overview

• Provides a way for users to get objects defined
  in the metadata
• Main use is to return pictures although there are
  other media types
• Returns the object in the appropriate mime type
  format - this may be a binary format
• Recommendation
• Ordering of the object ID changed in 1.5D2
• Implement using the most recent RETS version

58
ChangePassword Transaction Overview

• Allows a user to change their password
• Some ambiguity in the 1.5 and earlier standard
• Cryptographically weak
• Recommendations
• Be very careful when implementing
• Use the 1.7 standard addition as described in RCP
  041

59
Logout Transaction Overview

• Courtesy transaction
• Communicates that a user has finished the session
• A mechanism for the server to
• dispose of any local resources
• communicate session and billing information
• Recommendation
• Clients should log out
• Servers should be aware that clients may never
  log out

60
Extensions to the RETS Standard

• RETS allows vendors and sites to extend the
  standard to meet the specific needs of a site
• Additional Capability List URLs have no suggested
  form
• Additional parameters to an existing transaction
  and additional metadata have should have prefix
  X
• Example Add sort direction to Search
  XSortDirAsc
• Almost everything else is covered by metadata

61
Role of Parser Generators

• Used to produce parsers for
• DMQL
• Validation Expressions
• DMQL Java code will be used by RETS server to
  generate SQL from DMQL
• Validation expression Java code will be used by
  RETS server to execute the validation expression

62
Benefits for using a Parser Generator

• Improves productivity for software developers
• Not easy to write a good parser
• Relatively easy to write the grammar input file
  for the parser generator
• Avoids having to develop a parser
• Difficult, demanding, time-consuming
• Makes use of available parser generators
• Examples javacc, ANTLR, lexx
• Produces more robust Java source code

63
Generating Java Source Codefrom a BNF Grammar
File
BNF grammar for DMQL - specified in RETS
specification - see example on next two slides
BNF Grammar File (produced by developer)
Grammar file syntax - specified in parser
generator specification - see example on next two
slides
Javacc Parser Generator
Java Source code
64
BNF Grammar File Fragment DMQL Javacc Input File
- 1

• PARSER_BEGIN(DMQL)
• // The DMQL Javacc file which is used to generate
  the the primary class for parsing DMQL queries
• public class DMQL
• ltYour Java code heregt
• PARSER_END(DMQL)
• // Parse the DMQL statement.
• void Input()
• lt Java attributesgt
• ltJavacc expressions your Java codegt
• // Match the the DMQL Query at the highest level.
• void Query()
• lt Java attributesgt

65
BNF Grammar File Fragment DMQL Javacc Input File
- 2

• Search() (ltEOFgt ltTERMINATORgt)
• ... BNF and Javacode Productions and additional
  Java code for processing tokens for parsing DMQL
  based on the RETS DMQL Grammar...
• // Terminals (Tokens)
• TOKEN / The DMQL Tokens (See RETS
  Specification) /
• lt PLUS "" gt
• lt SINGLE_CHAR "?" gt
• lt MULTI_CHAR "" gt
• .
• lt OR "" gt

66
Application ofParser Output
Input rules for update (produced by MLS)
Input DMQL (produced by client software)
Java Validation Expression Code to evaluate Rules
Java code to process DMQL
Validation Outcome
Parser Generator Output
SQL
67
Resources forParser Generators

• ANTLR, ANother Tool for Language Recognition
• http//www.antlr.org
• Java Compiler Compilertm JavaCCtm
• https//javacc.dev.java.net
• Lexers and Parser Generators
• http//catalog.compilertools.net/lexparse.html

68
TopicSummary

• RETS provides a rich set of features for
  implementing a service or client
• Careful design will result in a successful
  project
• Work with the vendor or customer to ensure that
  the correct features are built
• Read the RETS document and watch for forward and
  reverse references
• Use RETS 1.5 as asupplement to RETS 1.0.1
• it includes many clarifications

Discussion Time
69
SoftwareQuality
70
Top 10 signs TechnologyHas Taken Over Your Life

• 10. You need to fill out a form that must be
  typewritten, but you can't because there isn't
  one typewriter in your house - only computers
  with laser printers.
• 9. You would rather get more dots per inch than
  miles per gallon.
• 8. You rotate your screen savers more frequently
  than your automobile tires.
• 7. You think of the gadgets in your office as
  "friends," but you forget to send your father a
  birthday card.
• 6. You know Bill Gates' e-mail address, but you
  have to look up your own social security number.
• 5. On vacation, you are reading a computer manual
  and turning the pages faster than everyone else
  who is reading John Grisham novels.
• 4. The thought that a CD could refer to finance
  or music never enters your mind.
• 3. Your letterhead lists a phone number, a fax
  number, e-mail addresses for two online services,
  and your web site address but no street address.
• 2. You have never sat through an entire movie
  without having at least one device on your body
  beep or buzz.
• 1. You sign Christmas cards by putting -) next
  to your signature.

71
TypicalSession Workflow
Login
Update Metadata ?
Get Metadata
Search
Get Objects ?
Get Object
Logout