This week we have two updates to our site: A new module to fetch a pages source and a small fix for easy web paths when running a Pipe.

This weeks release incorporates a new module named “Fetch Page”. The new module will fetch the source of a given web site as a string. This data can then be converted into a RSS feed or merged with other data in your Pipe using the Regex module. The module is located under the “Sources” menu in the editor.

The module can fetch an entire page or you can specify a “starting” and “ending” point. We also offer support for specifying a delimiter to allow you to easily separate the text into strings.

Our team as also created a few examples:
1) Fetch Page Example #1 is our first example. The goal of this Pipe is to extract train schedules from a local train scheduled provided in the San Francisco Bay Area.
2) Fetch Page Example #2 is a simple example that utilizes the first example as a sub-Pipe to obtain addition train stops.

Additional information about this new module along with its limitations can be found in the documentation modules/sources section.

Our second change to the site this week is a minor fix/enhancement to allow for easy web paths of all Pipe.runs page. This will allow you to use the easy paths you created for your Pipes. For example, the Pipe above “Fetch Page Example #1″ would look like the following below (please note the “?” instead of the “&”):

Original name:
“http://pipes.yahoo.com/pipes/pipe.run?_id=1qoLq6ai3BGv06982R2EvQ&_render=rss&station=Palo+Alto”

Easy web address name:
“http://pipes.yahoo.com/pipes_team/fetch_page_example?_render=rss&station=Palo+Alto”

Four nice changes to Pipes this week.

First, ever wanted to share your Pipe but not your username, password or application ID? Now you can! Pipes now supports “private” fields in two new modules, the private input module and the private string module. Private fields are not visible to anyone but the Pipe creator, and are not copied if the Pipe is cloned. If a Pipe user doesn’t provide a value for a private input field, then any “default” value provided by the Pipe creator continues to be used - so your Pipes will run for anyone but only you can see the private values.

Secondly, in addition to the new private modules we’ve added a “source” view for HTML formatted data in the editor’s debugger. So now those filters and regexes get a lot easier to write when you can easily browse the source directly:

Thirdly, we’ve enabled direct editing for the field selection drop down controls in the editor. This doesn’t sounds like a big deal until you realize you can now add array offsets to dig down into sub-arrays in the data, in any module (not just regex), or work on fields that aren’t currently flowing through the Pipe. For example, want to get at the first link in your raw ATOM feed? Rather than referring to the field as item.link.href, just type item.link.0.href

Finally, the Google feed issues we started experiencing last week should now be thing of the past. No more captchas. Yay!

There are some exciting improvements in Pipes to announce including additional modules and improved documentation.

We’ve added a new module under Sources:

• The Fetch Site Feed module uses a web site’s auto-discovery links (link rel tags) to find an RSS, RDF or Atom feed. If multiple feeds are discovered, only the items in the first one discovered are fetched. This module is a simpler alternative to the Feed Auto-Discovery Module which returns feed URLs (not the content in them). Here’s an example that uses the Fetch Site Feed module to retrieve the first five items from a feed. The feed is auto-discovered, with the site’s URL provided by the user via a URL User Input module.

We’ve also added a new module under String:

• The String Tokenizer module splits a string into an array of strings. A delimiter string (often just a single character) tells the module where to split the input string. The delimiter string doesn’t appear in the output list. Here’s an example that connects a Text User Input module to the String Tokenizer to create an array of search terms. The term delimiter is a comma. These terms are passed into a Loop module containing a Yahoo! Search sub-module. The Search sub-module is site restricted to Wikipedia, and the loop is configured to only output the first result for each search. If you enter the string “frog,vanilla,pulp fiction” into the text user input, the resulting feed will have three items.

Finally, we’ve made some improvements to our Module Reference Documentation. Every Module in the Pipes Editor now has a enhanced description including details about what it does, the types of data it can input and output and links to examples.

The latest updates to Pipes includes a entity detection module and improvements to our existing keyword analysis capabilities.

There are 2 new modules under String:

• The Yahoo! Shortcuts module analyzes the input string for popular entities (for example: person, place, concept, organization, etc). If any entities are discovered additional information about each entity is returned (an example of the data returned could include geodata if the entity detected is a place). You can return the data either as a list of items or a string. Here’s an example.
  
• The Term Extractor module analyzes the content in the input string for significant words or phrases. If any terms are found it appends them as a sub-element under each item containing the results of the analysis. It’s functionally similar to the Content Analysis module but adds the ability to analyze any data element (not just title and description). You can return the data either as a list of items or a string. Here’s an example.
  

Some other updates worth noting:

• The Location Extractor module has been updated to support entity detection, and geocoding, using the Yahoo! Shortcuts API. If the title or description of your data source doesn’t contain one of the other popular geocode formats we look for now we’ll do keyword detection.can return the data either as a list of items or a string. Here’s an example.
• The Content Analysis module has been replaced by the Term Extractor module and can now be found under the list of deprecated modules. Deprecated modules will continue to work but we’re discouraging their use.

We’ve added a new Web Service module to Pipes that allows developers to power their own modules and extend the functionality Pipes offers.

In the Pipes Editor, the new Web Service module is listed under Operators:

• The Web Service module POSTs the items in a pipe in JSON format to an external web service. This allows developers to extend the functionality Pipes offers to do whatever they need. The original items are replaced by the web services JSON or RSS response. Here’s an example.
  

Note: We’ve documented the source code for the web service that we call in the example pipe.

We’ve made some major updates to Pipes. The big news is that we have 8 new modules to announce!

There are 2 new modules under Sources:

• The Feed Auto-Discovery module detects feed URLs that are embedded in a webpage using auto-discovery links (such as link rel tags). This example demonstrates how you could use the Feed Auto-Discovery module to find the RSS feed for a given webpage and then fetch its content.
  
• The Item Builder module allows you to create a new single item data source by defining a set of key-value attributes. This example demonstrates how you can use it to build up a new feed.
  

There are 4 new modules under Operators:

• The Loop module will run an enclosed sub-module or Pipe one time for each item in the input feed. Its output options can be set to ‘emit’ or ‘assign’ results. Emit results will replace the content of the original input item, here’s an example. Assign results will insert the content into a specified attribute, here’s an example. We introduced this module in order to support a wider range of modules, and data types, that can be looped over. You can use the Loop module with any module or Sub-Pipe except User Inputs, Operators, or Deprecated modules. Note: Old Skool Pipes users will notice that this module is similar to the For Each modules.
  
• The Reverse module reverses the order of the items contained in the input feed. This example demonstrates how it works.
  
• The Sub-element module will emit only the specified attribute or array for each item in the input feed. This example demonstrates how you could use it to take an RSS feed and emit only a specific item attribute for each item.
  
• The Tail module emits the last ‘n’ items from the input feed. This example demonstrates how you could use it to emit only the last 5 items in a feed.
  

There are 2 new modules under String:

• The Translate module automatically translates a string from one language into another. Select the language combination you desire from the pulldown menu. Note: Old Skool Pipes users will notice that this module is similar to the BabelFish module. Unlike the BabelFish module, the Translate module can be used within a Loop module and operate on any specified element attribute. This example demonstrates how you could use it to translate the Title and Description of a feed from one language into another.
  
• The String Regex module allows you to apply Perl-like regular expressions to attributes in a string. This example takes the delicious RSS feed and replaces the title for each item with its “dc:subject” element.
  

Some other updates worth noting:

• The String Concatenate module has been renamed the String Builder. It works exactly as before but if you’re new to it here’s an example that demonstrates how you can use it to build up strings!
• In a few cases several older modules have been Deprecated and replaced by what we consider to be improved modules. The For Each: Annotate and For Each: Replace modules have been replaced by the Loop module. And the BabelFish module has been replaced by the Translate module. The Deprecated modules will continue to work but we’re discouraging their use.
• In the Editor, we’ve updated how you can modify the properties for your Pipe. In the past, there was a button labeled Publish in its place there’s a new Properties button that brings up a dialog window that allows you to modify your Pipes name, description and tags. You now have the option to either make these changes so only you see them or to publish them along with your Pipe.
  
• And, as usual we also fixed miscellaneous bugs that have either been reported or that we stumbled upon. Thanks for continuing to share your suggestions!

The Pipes team is proud to announce two new features that we’ve been working on that enhance our support for GeoData.

• First, we added a interactive Yahoo! Map to the runpage of any Pipe containing GeoData. To see this feature in action run the Apartment Near Something Pipe. Here’s a still image of what the map looks like:
• Second, we added an additional output renderer which allows Pipes containing GeoData to be emitted as a KML file. If there’s GeoData in a Pipe you can find a link to the KML output at the bottom of a Pipes runpage listed in the Tools section. It’s possible to use the KML output in a variety of ways including in Google Earth.
  

We posted a followup entry that highlights some example Pipes that make use the new Geo features.

In addition to our enhanced geolocation support there are some other improvements to announce:

• You can now inline edit the Title and Description text listed on the runpage for your Pipes.
• We’ve added the ability to filter dates using the options “before” and “after” in the Filter Module. These parameters can accept dates in a variety of relative formats such as “n days ago”, “now” and “today”.

There are a bunch of updates to the Pipes Editor to tell you about including three new modules and many new and improved features.

Introducing the new Modules:

• The Location Module creates a location object from a string value and understands most common address formats including zipcodes and many major airport codes. Click on any of the previous links to see examples.
• The Date Builder Module can create a date object from a string value. It understands most common date formats as well as relative date formats such as yesterday and 2 days ago. Here’s an example Pipe that uses the Date Builder module to filter items in a feed by a specific date.
• We added a Simple Math Module that performs basic arithmetic, such as addition and subtraction. Here’s an example of the Simple Math module in action.

Here’s a list of the additional changes:

• We added a new status message area at the top of the Editor.
• Modules that contain date selection fields now include a date picker. We integrated the Yahoo! UI Calendar component.
  
• There’s a new look to the field selectors and drop downs.
  
• Most browsers should be faster when drawing lines.
• Finally, for users running Safari we’ve squashed a bug that made it difficult to connect inner terminal fields within modules.

The Pipes team pushed an update that introduces some basic error reporting in the Pipes Editor. Full error reporting is on the way. In the meantime we wanted to get something out quickly.

Error messages that are returned by a module are displayed in the Debugger. For example, the Fetch Feed Module now reports HTTP error codes so it’s possible to tell if a URL is failing to parse, etc.

Here’s how the Fetch Feed Module would return an error to a URL that’s not a valid feed:

We’ve introduced a new Fetch Data Module which allows you to use arbitrary XML and JSON data in your pipe just like you have previously been able to use RSS and Atom feeds.

With the introduction of this new module we’ve renamed the Fetch module to Fetch Feed.

To get you started here are a few examples demonstrating how you can use the Fetch Data Module:

• Create a custom RSS feed using the Yahoo! Shopping API and a combination of the Fetch Data and Rename Modules. Run the Pipe then see how the Pipe was built.
• Similarly, using the YouTube API you can turn your favorite tags into an RSS feed. Run the Pipe then see how the Pipe was built.
• Convert an OPML file into an RSS feed. Run the Pipe then see how the Pipe was built.
• Convert the JSON output from Metacarta into a RSS feed. Run the Pipe then see how the Pipe was built.