FutureBostonMashupCamp

1
FutureBoston_at_MashupCamp

• Data Web Services

Dr. Michael Flaxman, MIT Dept. Urban Studies
Planning
2
Beyond Base Maps

• Next-Gen Geospatial Mashups

3
Overview

• The End of Geographyis a new beginning
• Whats New Important
• Whats Old Important
• Data and Protocols to Get It

4
The End of Geography

• First generation of web featured exploration
  transcending geographic limits
• for example of bricks and mortar
• Second generation rediscovering importance of
  location
• 80 of all information has geospatial component
• Most all forms of media can be geospatial either
  directly, or implicitly (i.e. addresses)

5
New ImportantFrom Top-Down to Bottom-Up

• Traditional Geospatial Planning Methods
• Systematically collect uniform data
• Employ formal quality control
• Careful, rigorous, expert-based evaluations
• Top-down Data Sources
• Airphotos satellite imagery with only least
  possible field sampling
• Usually very aggregated, in terms of space
  attributes
• Official data avoids quality measures
  subjective opinions

6
Limitations of Top-Down Approaches

• Systematic data collection
• Very expensive
• Very slow
• i.e. decadal census updates
• Requires still-rare GIS expertise
• A big problem in most of the developing world
• Quality judgment really are important
• Fixed scale Plan view perspective
• Spatial data need not be exclusively in plan
• We live life on the ground, in perspective
• 3rd Dimension is often important (mixed use,
  condos, etc.)
• Sometimes, physically small things are important

7
From Aggregated to Individual

• Personal Planning
• Beyond simple route-finding
• Sharing experience opinions about great or
  terrible places
• Individual Firm Planning
• How can a small startup find
• A great location
• Talent

8
Major Challenges

• Where does bottom up socially-constructed
  content meet the world of real data?
• Why is existing GIS data not better used in
  mashups?
• Can an end-user correct an error in a base road
  map or terrain elevation model?
• Can mash-up cartography expand beyond pin maps
  and standardized base maps?
• Polygons are important!
• Can end users control symbology?
• Can we make basic analytic operations available
  useable by the general public?
• Example find highly-rated pizza shops near Mass
  Ave. within 5 minutes walk of my current location

9
Whats New Important

• Seamlessness
• Removal of arbitrary map tiles frames
• Streaming data
• Multiple LODs, swapped based on user position
• Direct Manipulation
• Interaction driving content presentation
• Indirect Manipulation
• Public APIs Web Services allow service
  chaining
• Construction of new geospatial environments
  bottom-up using social networking

10
Whats Old Important

• Robust links between graphics data
• Maintaining
• data beyond/behind what is currently visualized
• GIS attributes, CAD entity properties
• In traditional design systems, tightly coupled
  
• semantic information
• object characteristics in computer-readable
  formats
• Providing
• Ability to transform graphical attributes based
  on data
• Based on properties of object
• Based on geographic context/spatial relationships
• Interoperability the ability to reuse data in
  different contexts
• Previously mostly file-based methods
• Now APIs, web services protocols

11
Old Important (Continued)

• Georeferencing
• Many systems now support on-the-fly projection
  conversion
• However, proper georef requires knowledge of
  spatial referencing system used
• Lat/Lon alone not enough to get position within
  100m due to datum differences
• Spatial referencing system can be defined by
  protocol (i.e. always WGS84), or explicitly (UTM
  Zone 13N NAD27)
• If systems dont match metadata not recorded
• Subtle but serious errors
• Fixing requires human intervention
  detectivework (yuch!)
• Wonderful opensource GDAL libraries
• Provide most needed conversions

12
Old ImportantVisualization

• Human visual system has incredible pattern
  recognition abilities
• Can still be overwhelmed by visual clutter
• Clear thematic maps usually has 7/-2 ungrouped
  categories
• A human cognitive limit, not technical
• Web provides ability to progressively reveal data

13
Old ImportantMap Analysis vs. Inventory

• Simple graphical overlay
• Sufficient for human pattern recognition
• Map Analysis
• Required for computer to take intelligent action

14
Old ImportantMap Analysis Operations

• Thousands available in modern GIS
• Most used very rarely
• Question for mashupers
• What are the most important 3-5 ops?
• How can these be made end user-accessible?

15
Old ImportantMost Important Ops in GIS use

• Spatial Selection (i.e. point in polygon)
• Spatial Buffering (create new geom near/far from
  certain features)
• Spatial Overlay (i.e. find overlap between two
  layers)
• Reclassify (used to group/split elements,
  creating new layers or categories)

16
Available Steps Towards Simple Online Spatial
Manipulation

• Web Feature Service
• Can query layers and get individual features
  matching criteria
• Criteria can be traditional database-like
  attributes
• Can also be spatial (points in a polygon)
• Cant currently do buffering and spatial ops
  creating new geometries
• except with heavy servers such as ArcGIS server

17
FutureBoston Data

• Why these data?
• Types most often used for key business planning
  decisions
• Can support visualization (for example extruding
  building footprints by height)
• Relatively simple structure (flat attribute
  tables with only essential info)
• Common analysis factors pre-calculated as
  attributes (for example, distance to Harvard
  MIT)

18
FutureBostonGIS Data Sets

• COMPETITION 1 INNOVATION LANDSCAPE
• Data Set 1 National Landscape
• Data Set 2 Rich ZIP Code Landscape
• COMPETITION 2 SCIENCE CITY MODEL MAKER
• Data Set 3 Mass Ave City Parcel and Building
  Database

19
INNOVATION LANDSCAPEData Set 1 National
Landscape

• Purposes
• compare Massachusetts (MA) to other states in
  terms of factors influencing innovation
• compare different industries in MA relative to
  their national innovation context
• discover how MA has fared over time in a national
  perspective
• which industries within MA are the most
  influential for the state's growth.
• Contents
• RD investment per capita, 2000 - 2005
• Percent population with higher degrees, 2000 -
  2005
• Patents issued per capita, 2000 - 2005

20
INNOVATION LANDSCAPE Data Set 2 Rich ZIP Code
Landscape

• Purposes
• Examine biotech / pharmaceutical industry and
  it's ties to post-secondary education
• Identify visualize where this industry is
  located and why
• Data
• Summarized for each Zip Code geometry in MA
• Occupation by industry cluster per capita
• Percent change in occupation by cluster

21
SCIENCE CITY COMPETITIONData Set 3 Mass Ave
City Parcel and Building Database

• Purposes
• Create vision of life in Boston's "ScienceCity"
• Consider view relative to different audiences
  (new firms, students, employees, etc.)
• Combine given data with other data to show
  neighborhood characteristics
• 3D building elevations are included to help
  visualize physical landscape

22
SCIENCE CITY COMPETITIONMass Ave City Parcel and
Building Database

• Data Provided
• Building and parcel locations
• Average building elevation
• Average ground elevation
• Lot size
• Land use
• Street address and ZIP code
• Land and building values
• Tax paid
• Year built
• Number of buildings per parcel
• Number of residential units per parcel
• Number of commcerical units per parcel
• Distance Data

23
SCIENCE CITY COMPETITIONMass Ave City Parcel and
Building Database

• Distance Data Provided
• Distance to commercial land uses
• Distance to Harvard Yard
• Distance to Susan Hochfield's office (President
  of MIT)
• Distance to retail, restaurants and shopping
• Distance to nearest T stop

24
Data Formats

• CSV
• Comma-separated values in a text file
• Geometry abstracted to centerpoints
• ESRI-format Shape files
• Binary format
• Data Stored in multiple files
• Geometry lives in .shp
• Database attributes in .dbf (DBase4 format)
• Publicly-documented
• Open source shapelib can read/write
• Open source GDAL can read/write/convert

25
Data Distribution

• Data files
• available from FTP server
• Linked from main mobilab.mit.edu site
• Can download pre-process into whatever format
  most convenient
• Served live
• Using OpenGIS protocols
• WMS Web Mapping Service (images of maps)
• WFS Web Feature Service (vector features)
• Recommended approach
• More flexible, generalizable than file processing
• Same protocols can access other data from many
  servers
• For example...MassGIS

26
(No Transcript)
27
Examples of MassGIS Data Available via WMS/WFS
28
OpenGIS Protocol Options

• Use Directly
• Your application / script mashes up data
  received
• Use Indirectly
• Through clients which render/tile for you

29
OpenGIS ProtocolsWMS Web Mapping Service

• Purpose
• support creation and display of registered and
  superimposed map-like views of information that
  come simultaneously from multiple remote and
  heterogeneous sources
• Implementation
• Uses Simple HTTP Requests, Returns Graphics
• Not SOAP, REST, etc
• Typical return types available PNG, JPEG, PPM,
  and TIFF
• Sometimes SVG, depending on server
• Hundreds of existing servers
• Relatively mature and well-tested protocol

30
3 Basic WMS Requests

• Provide a map as a picture or set of features
  (GetMap)
• Provide information about the content of a map
  such as the value of a feature at a location
  (GetFeatureInfo)
• Provide information about what types of maps a
  server can deliver (GetCapabilities)

31
GetMap MassGIS Example

• http//maps.massgis.state.ma.us/mapaccess/main.jsp
  ?
• servicewms
• requestGetMap
• layersMHDRoads
• stylesClass
• srsEPSG26986
• bbox233500,900000,236500,902500
• width640height480
• formatimage/png

32
WMS Image Returned
33
MassGIS Example Dissected

• http//maps.massgis.state.ma.us/mapaccess/main.jsp
  ?
• Base URL (server/path)
• servicewms
• MassGIS-specific request
• requestGetMap
• WMS request type
• layersMHDRoads
• comma-sepd list of layer names ( for space in
  layer name)
• stylesClass
• Comma-separated list of symbology specification
  names (default)
• srsEPSG26986
• Spatial reference system (a code value for a
  projection/datum)
• bbox233500,900000,236500,902500
• Bounding box in West, South, East, North order in
  SRS coordinates
• width640height480
• Pixel width and height of graphic image requested
• formatimage/png
• MIME Type of image requested

34
How to Find Those Odd Parameters...

• Ask the Server Using GetCapabilities
• Inspect list of layers provided
• Inspect list of styles available for rendering
  these layers
• See which image formats are supported by the
  server
• Find semi-magical EPSG codes for some spatial
  reference systems
• Find spatial extents of layers provided

35
How to Find Those Odd Parameters (2)

• Stuff in some defaults...
• WGS84, Lat/Lon EPSG4326
• Can Use the world or layer bounds for starting
  extents on bounding boxes
• Default style polygon
• Default image type image/png or image/jpeg

36
WFS Web Feature Service

• Basic Idea
• Rather than requesting pre-rendered data as an
  image, get basic geometric/attribute data
  directly
• Advantage much better control of rendering,
  ability to download only what you need

37
WMS/WFS Protocol Exploration

• Use GeoServer Demo
• Substitute parameters as desired...

38
Basic WFS Requests

• GetCapabilities
• GetFeature

39
WFS GetCapabilities

• Returns GetCapabilities XML Document
• Fragment Example
• mitma_zip_landscape_region
• ma_zip_landscape_region_Type
• Generated from ma_zip_landscapet
• ma_zip_landscape ma_zip_landscape_region
  
• EPSG4326
• miny"41.237964" maxx"-69.928261"
  maxy"42.886589"/

40
WFS GetFeature Request

• http//mobilab.mit.edu8080/geoserver/wfs?request
  GetFeatureTypeNamemitma_zip_landscape_regionPr
  opertyNamemit20EMP_SQM,SALES_SQM,mitthe_geomB
  BOX71.15,42.30,71.05,42.40

41
FutureBoston Data/Protocols Conclusion

• Use data in whichever formats make most sense to
  you
• Using OpenGIS protocols directly takes some
  work...
• But gives access to many GIS layers beyond what
  was shown here
• In the end, rich mashups depend on rich data
• Multimedia and simple basemaps found elsewhere,
  but GIS data too rich to ignore!