Yahoo Pipes Modules

1
Yahoo! Pipes Modules
A tool for remixing the web
?
?
2
About Yahoo! Pipes

• Pipes is an interactive data aggregator and
  manipulator that lets you mashup your favorite
  online data sources
• Like Unix pipes simple commands can be
  combined together to create output that meets
  your needs

// To find the 10 most popular words in a
documenttr -c 'a-zA-Z' '\n' lt file grep .
sort uniq -c sort nr head -10
3
About Yahoo! Pipes

• Combine many feeds into one, then sort, filter
  and translate to create your ultimate custom feed
  
• Geocode your favorite feeds and browse the items
  on an interactive map
• Remix your favorite data sources and use the Pipe
  to power a new application
• Build custom vertical search pages that are
  impossible with ordinary search engines
• Power badges on your web site
• Extend your web site by accessing the JSON or RSS
  output from Pipes

4
About Yahoo! Pipes

• The frontend processing is done with
• JavaScript
• AJAX
• Canvas HTML 5 tag (with MSIE workaround)
• The backend processing is done with
• MySQL
• squid (for caching)
• PHP
• Perl (lots of CPAN modules)

5
Yahoo! Pipes ? User Guides

• James Maguires Yahoo! Pipes User Guide
• http//itmanagement.earthweb.com/netsys/article.ph
  p/3659226
• Mr. Speakers Introduction to Yahoo Pipes
• http//mrspeaker.webeisteddfod.com/2007/02/10/yaho
  o-pipes/

6
Yahoo! Pipes ? Video Tutorials

• Create a basic pipe, combine two feeds
• http//blip.tv/file/get/Jc174-YahooPipesBasics201.
  wmv
• Sorting, filtering, and debugging
• http//blip.tv/file/get/Jc174-YahooPipesSortingFil
  teringAndDebugging584.wmv
• The Yahoo Search module, the Union module and
  using other Peoples Pipes
• http//blip.tv/file/get/Jc174-YahooPipesTheYahooSe
  archModuleUnionModuleAndUsingOther879.wmv

7
Yahoo! Pipes ? Video Tutorials

• Use Feedity to get a feed from a site that
  doesn't have a feed (uses Truncate module)
• http//blip.tv/file/get/Jc174-YahooPipesUseFeeding
  ToCreateFeeds127.wmv
• Replace Yahoos most popular search terms with
  pictures from Flickr (uses For Each Replace
  module)
• http//blip.tv/file/get/Jc174-YahooPipesYahooBuzzT
  hroughFlickr985.wmv
• The Text Input module and the URL Builder module
• http//blip.tv/file/get/Jc174-YahooPipesTextInputM
  oduleAndUrlBuilder727.wmv

8
Yahoo! Pipes ? Video Tutorials

• The Location Input module and the String
  Concatenate module
• http//blip.tv/file/get/Jc174-YahooPipesLocationIn
  putModuleAndTheConcatenateModule167.wmv
• Use a custom module with For Each Replace
• http//blip.tv/file/get/Jc174-YahooPipesUseACustom
  ModuleWithForEachReplace223.wmv

9
YAHOO! PIPES MODULESDATES
10
Date Builder Module

• The Date Builder Module can create a date object
  from a string value.
• It understands most common date formats as well
  as relative date formats such as yesterday and 2
  days ago.

11
Date Builder Module
12
Date Formatter Module

• This module will format a DateTime value piped in
  from another module as a string.
• It supports PHP strftime conversion specifiers.

13
Date Formatter Module
14
Date Formatter Module
15
Date Input Module

• This module accepts user-defined date strings.
• These values can be piped in to date input fields
  in another module.
• This module can also be used to let users provide
  their own input on the Pipe's run page.

16
Date Input Module
17
YAHOO! PIPES MODULESLOCATIONS
18
Location Builder Module

• This module creates a location object from a
  string value.
• It understands most common address formats
  including zipcodes and many major airport codes.

19
Location Builder Module
20
YAHOO! PIPES MODULESNUMBERS
21
Simple Math Module

• This module performs basic arithmetic, such as
  addition and subtraction.

22
Simple Math Module
23
YAHOO! PIPES MODULESOPERATORS
24
BabelFish Module

• This module is replaced by the Translate Module.

25
Content Analysis Module

• This module is replaced by the Term Extraction
  module.

26
Count Module

• This module counts the number of items in a feed.
  
• The result is a typed number that you can pipe
  into a numeric input field within another module.

27
Count Module
28
Filter Module

• This module filters the input source, including
  or excluding fields, that match a set of defined
  rules.
• After connecting an input source, the Title
  pulldown menu dynamically updates to provide
  additional criteria.

29
Filter Module

• The text in the input box can be entered manually
  or piped in from a Text input module.
• To add additional rules click on the '' icon.

30
Filter Module
31
For Each Annotate ModuleThis module is replaced
by the Loop Module in "Assign" mode.

• This module constructs a loop that runs the
  sub-module, or pipe, once for each item in the
  feed.
• The output result will be appended to the
  original item using the defined attribute name.

32
For Each Annotate ModuleThis module is replaced
by the Loop Module in "Assign" mode.

• After connecting an input source any pulldown
  menus in a sub-module will dynamically update to
  provide additional criteria.
• You can choose which values to pass to the
  sub-module as parameters, or you can pipe them in
  from other modules.

33
For Each Annotate ModuleThis module is replaced
by the Loop Module in "Assign" mode.
34
For Each Replace ModuleThis module is replaced
by the Loop Module in "Emit" mode.

• This module will run an enclosed sub-module or
  pipe one time for each item in the feed.
• The output will replace the original item.

35
For Each Replace ModuleThis module is replaced
by the Loop Module in "Emit" mode.

• After connecting an input source, any pulldown
  menus in a sub-module will dynamically update to
  provide additional criteria.
• You can choose which values to pass to the
  sub-module as parameters, or you can pipe them in
  from other modules.

36
For Each Replace ModuleThis module is replaced
by the Loop Module in "Emit" mode.
37
Location Extractor Module

• This module analyzes text in each feed items
  title and description and attempts to identify
  addresses, location names or popular map service
  URLs.
• If the extractor finds location entities in the
  feed, it will annotate each item with a
  ylocation sub-element containing that item's
  latitude and longitude.

38
Location Extractor Module
39
Loop Module

• This module will run an enclosed sub-module or
  pipe one time for each item in the input feed.
• If set to 'emit results' the output will replace
  the content of the original item.
• If set to 'assign results to' the output will be
  inserted into a specified attribute.
• After connecting an input source, any pulldown
  menus in a sub-module will dynamically update to
  provide additional criteria.
• You can choose which values to pass to the
  sub-module as parameters, or you can pipe them in
  from other modules.

40
Loop Module
41
Regex Module
http//cheat-sheet.front.lv/wp-content/uploads/200
6/08/pcre-php.pdf

• Apply Perl-compatible regular expressions (PCREs)
  to item attributes.

42
Regex Module
43
Rename Module

• This module lets you rename or copy item
  attributes.

44
Rename Module
45
Reverse Module

• This module reverses the order of the items
  contained in the input feed.

46
Reverse Module
47
Sort Module

• This module sorts items by one or more item
  attribute fields.
• Sorts can be in either ascending or descending
  order.
• After connecting an input source, the Title
  pulldown menu dynamically updates to provide
  additional criteria.
• To add additional rules click on the icon.

48
Sort Module
49
Split Module

• This module splits a feed into two identical
  copies.

50
Split Module
51
Sub-element Module

• This module will emit the specified attribute or
  array for each item in the input feed.

52
Sub-element Module
53
Tail Module

• This module emits the last 'n' items from the
  input feed.

54
Tail Module
55
Truncate Module

• This module limits the number of items that
  passes through it.
• You can enter a number manually or pipe it in
  from a Number Input module.

56
Truncate Module
57
Union Module

• This module merges up to 5 sources together.

58
Union Module
59
Unique Module

• This module combines items containing identical
  strings.
• After connecting an input source, the Title
  pulldown menu dynamically updates to provide
  additional criteria.
• A count of duplicate items will be added as a
  yrepeatcount element.

60
Unique Module
61
Web Service Module

• This module POSTs the items in a pipe in JSON
  format to an external web service.
• This allows developers to extend the Pipes
  functionality to do whatever they need.
• The original items are replaced by the web
  services JSON or RSS response.

62
Web Service Module
63
YAHOO! PIPES MODULESSOURCES
64
Fetch CSV Module

• This module retrieves a source formatted in CSV
  (comma-separated values).
• Enter the URL that you want to retrieve and set
  how the columns in the source are separated.

65
Fetch CSV Module

• You can configure how the emitted element
  attributes are named by either
• setting them to the column names from a specified
  range of rows in the CSV source or
• custom naming them based on your own values.
• Optional configurations include the ability to
  skip emitting an initial 'x' number of rows.

66
Fetch CSV Module
67
Fetch Data Module

• This module retrieves any XML, JSON, iCal or KML
  file.
• It tries to extract a list of elements using the
  provided path parameter.

68
Fetch Data Module
69
Fetch Feed Module

• This module retrieves one or more RSS, Atom, RDF
  or iCal feeds from the URL(s) entered in the
  input box.
• To add additional sources click on the '' icon.

70
Fetch Feed Module
71
Flickr Module

• This module searches for images on Flickr using
  the supplied parameters.
• Users can refine the search by entering keywords,
  the number of desired results, and a location.
• The string in the input boxes can be entered
  manually or piped in from a User Input module.

72
Flickr Module
73
Google Base Module

• This module searches Google Base data using the
  supplied parameters.
• Users can refine the search by entering keywords,
  what they are looking for, and restricting the
  results to a geographic range.
• You can enter a string in the input box or pipe
  it in from a User Input module.

74
Google Base Module
75
Item Builder Module

• This module allows you to create a new single
  item data source by defining a set of key-value
  attributes.

76
Item Builder Module
77
Yahoo! Local Module

• This module searches Yahoo! Local using the
  supplied parameters.
• Users can refine the search by entering keywords
  and restricting the results to a geographic
  range.
• The string in the input boxes can be entered
  manually or piped in from a User Input module.

78
Yahoo! Local Module
79
Yahoo! Search Module

• This module searches the web using keywords.
• Users can refine the search by restricting it to
  specific URL sources.
• To add additional source restrictions click on
  the '' icon.
• The string in the input boxes can be entered
  manually or piped in from a User Input module.

80
Yahoo! Search Module
81
YAHOO! PIPES MODULESSTRINGS
82
Feed Auto-Discovery Module

• This module detects feed URLs that are embedded
  in a webpage using auto-discovery links.

83
Feed Auto-Discovery Module
84
String Builder Module

• This module creates a new string and allows
  multiple strings to be joined together.
• You can manually enter the text in the input
  boxes or pipe it in from another module.

85
String Builder Module
86
String Regex Module

• Apply Perl-like regular expressions to attributes
  in a string.

87
String Regex Module
88
String Replace Module

• This module lets you find and replace the first,
  last or all occurrences of a string.
• You can manually enter the text in the input
  boxes or pipe it in from an appropriate User
  Input module.
• To add additional rules click on the '' icon.

89
String Replace Module
90
Sub String Module

• This module cuts a range of consecutive
  characters from a string.
• Enter the numeric character starting position and
  the length of the characters you want to match.

91
Sub String Module
92
Term Extractor Module

• This module analyzes the text within each feed
  item.
• It will try to find significant words or phrases.
  
• If any are found it will add a sub-element
  containing the results of the analysis.

93
Term Extractor Module
94
Translate Module

• This module automatically translates a string
  from one language into another.
• Select the language combination you desire from
  the pulldown menu.

95
Translate Module
96
Yahoo! Shortcuts Module

• This module uses Yahoo! Shortcuts to analyze the
  input string for entities (person, place, etc).
• The module outputs a list of items containing
  additional information about every entity
  detected.

97
Yahoo! Shortcuts Module
98
YAHOO! PIPES MODULESURLs
99
URL Builder Module

• This module builds URLs in either traditional or
  Web 2.0 style query-string format from a series
  of input fields.
• If a URL is composed of a series of field-value
  pairs in the query string, just paste it into the
  Base input box.
• The module will automatically parse out the
  key/value pairs.

100
URL Builder Module

• To build a Web 2.0-style address, enter the Base
  URL followed by each directory path element on a
  new line.
• To retrieve the URL built up using the
  URLBuilder, connect its output terminal to the
  input box of a Fetch module.
• You can further configure the values for each
  element in the URLbuilder by wiring a User Input
  module to it, making the URL even further
  configurable.

101
URL Builder Module
102
YAHOO! PIPES MODULESUSER INPUT
103
Location Input Module

• This module accepts user-defined location
  strings.
• These values can be piped in to other modules or
  pipes that accept a location as input.
• This module can also be used to let users enter
  their own location data on a Pipes run page.

104
Location Input Module
105
Number Input Module

• This module accepts user-defined numeric strings.
  
• These values can be piped in to other modules or
  pipes that accept numeric input.
• This module can also be used to let users enter
  their own numeric data on a Pipes run page.

106
Number Input Module
107
Text Input Module

• This module accepts user defined text strings.
• These values can be piped in to other modules or
  pipes that accept text input.
• This module can also be used to let users enter
  their own text data on a Pipes run page.

108
Text Input Module
109
URL Input Module

• This module accepts user-defined URLs.
• These values can be piped in to other modules or
  pipes that accept URLs as input.
• This module can also be used to let users enter
  their own URLs on a Pipes run page.

110
URL Input Module