Since its v1 launch two years ago, Pipes has had over 24 feature releases. The Pipes system of today is quite different from its first version, with a sizable number of significant new features and fixes. Some of the things we've changed and added include 25 new modules, favorites, new output formats, search, private fields, a variety of badges, a new web site and so on.

Along the way, we've learned a lot and collected feedback from all of you. We're aware that there are currently some bugs and performance issues, and we're working hard to correct them. We understand your frustration and updating the underlying Pipes' data engine and infrastructure to address these issues is a priority for us.

Today, Pipes serves hundreds of millions requests monthly and its usage continues to grow. As part of the Yahoo! Open Strategy, we're excited about the ongoing potential for Pipes and can't wait to see what developers like you use it for next.

Thanks to everyone who uses Pipes. We value developer feedback greatly, so keep it constructive and keep it coming - ­ it helps us make Pipes better.

The Create RSS operator module makes it easy to convert an entire list of items into an RSS stream when the input data is not in RSS format, e.g., the fields are not named correctly for RSS display and output. Both the common required fields can be set, as well as the optional, but frequently used, media extensions. To rename non-RSS elements to a RSS structure, simply select an existing element name from the drop-down list provided.

This Create RSS example uses the YQL module to get the top music artists of the week. Because the data isn't in a RSS structure, we'll use the Create RSS Module to convert the data to RSS.

This is all done easily by mapping the item.element to the RSS Element you want. For example, I want item.name as my Title, item.ItemInfo.ChartPosition as my Description field, and item.website as my Link. Simply find the item in the dropdown next to the RSS Item you want it defined as.

If you want to further customize your fields, using the Regex Module is the easiest way. For example, in my description I want to prepend this text: "This weeks Chart Position: " in front the text currently being rendered as the description field. I first find the item.description field and I use "^" in the "replace" text area to signify that I want to start at the beginning of the string. In the "with" text area I enter "This weeks Chart Position: ". Check the debugger pane that it came out correctly and we're done!

With the RSS Item Builder module, you can create a single-item RSS data source by assigning values to one or more RSS attributes. RSS Item Builder can be used to create a single new RSS item from scratch, or reformat and restructure an existing item into an RSS structure. When the RSS Item Builder is used as a sub-module to the Loop module, the values from each original item can be converted to RSS. Unlike the CreateRSS module, each attribute within the RSS Item Builder can be "wired", so you can take values from other modules.

In this RSS Item Builder example, we'll use the RSS Item Builder to create a traffic RSS feed for San Francisco.

First we'll use the YQL module to get the traffic data for zip code 94123. We'll use the Loop module and embed a RSS Item Builder sub-module inside by drag and dropping it onto the target area.

We'll map item.Title to Title, item.Description to Description, we'll wire in an email address using the String Builder Module as my Author field, and map item.UpdateDate as my GUID.

We also want the media:content image URL to map to item.ImageUrl so we have that extra meta data that might be read by certain RSS readers.

Since I just want an RSS structure, I'll check off the "emit all results" radio button.

We also want to see the map of where the traffic is in our Description. Using the Regex module, select item.description in the drop down and in the "replace" textbox I use "$" which is regex for appending something to the end of the string and i'll use the String Builder again to create the image that will be appended.

Check the debugger pane and we're done!

In this blog post we'll go through the steps on how to use YQL Execute to power the Pipes Webservice Module - for the impatient, here's the example Pipe that we'll be creating.

In order to use YQL Execute, you'll need a website to host your XML file. The XML file is where YQL Execute allows you to manipulate data by using server side javascript. We can then post data to YQL via the Pipes Webservice module to allow further processing of our Pipes Data.

Some users on the Pipes message boards asked if we could add a "yindex" element that numbers each item in the output. This becomes trivial when using YQL Execute!

The first thing we will create is our XML file for YQL Execute. You can copy and paste the structure of the XML below, editing only whats inside the execute node.
The execute content is wrapped within a CDATA statement. Since we're dealing with json, we'll import Crockfords json parser and stringifier by using the y.include() method. First, we parse the json data using JSON.parse(), then loop through the json structure and add new elements by using pdata.items[i].new_element_here within the loop. "yindex" is the new item we want added to our Pipes data structure. We then assign a number to yindex within the loop and then use JSON.stringify() to bring it back together. Finally, we assign our "output" variable as our response.object. Then we want to host this XML file on a publicly accessible web host.

Now we'll set up our Pipe. We need to use the URL builder to setup the URL that the Webservice module will access.

In the URL Builder module, we'll use

• http://query.yahooapis.com as our base
• v1/public/yql as our Path element
• These name/values as our query params
  • q: use "http://paul.donnelly.org/yql/yindex.xml" as yindex; select * from yindex where data = @data
  • format: json
  • diagnostics: false

And simply wire the URL builder module into the url terminal in the Web Service module. Be sure to use query.results.result.items as the Path to item list.

Here is the example Pipe that takes the Pipes blog feed, uses YQL execute as a Webservice module to create an additional element in the Pipes data output: "yindex" which adds a number to each item in the Pipes results. Another example YQL Execute statement is: use "http://paul.donnelly.org/yql/random.xml" as ran; select * from ran where data = @data that randomizes a given feed.

Pipes continues to be one of the most popular mashup and feed processing tools around the web. We would like to thank everyone for using our services, reporting bugs and giving great suggestions!

Here is a short list of what happened since our last birthday.

• fixed international encoding bug
• released YQL module
• released badges
• released php output
• improved caching

This year we spent quite some time working on the YQL engine and we hope to migrate Pipes onto much of that infrastructure down the road.

We would also like to publicly acknowledge and thank community member hapdaniel for providing Pipes guidance and expertise on our message boards and around the internet, as well as other members such as Tony Loton and Dawn Foster (geekygirldawn) for being such great advocates.

As we posted on our message boards last week, we've tracked the issue down to Feedburner's recent migration onto Google's platforms (more details can be found on Feedburners status blog http://feedburnerstatus.blogspot.com/). During this migration, the IPs used by Feedburners to access feeds have changed and we were not made aware of this change before it happened. As a result, Pipes has been blocking some of their new IP's that came online. We have since added those IP's to our whitelist and we are in the process of unblocking them.

For now, If you do run your Pipes RSS through Feedburner please hold off on doing so until the IP whitelist propagates.

YQL complements Pipes by allowing you to fetch, aggregate and mash up data from many sources using a SQL-like SELECT syntax. To find out more about YQL click here.

To use YQL, simply enter a YQL statement (select * from feed where url='http://digg.com/rss/index.xml') into the textarea. To drill down further into the result set you can use either the Pipes Sub-element module or by using projection in a YQL statement. For example: select title from feed where url='http://digg.com/rss/index.xml' returns only the titles from the Digg RSS feed. Here is an example Pipe using YQL as a source.

The YQL module has 2 output modes: Results only or Diagnostics and results. Diagnostics provides additional data such as: count, language type and more.

You can test your query in the YQL console by clicking on the "Try in the console" link. The YQL console provides sample queries and shows what data tables are available to query against.

Here are some interesting queries to get you started:

• Fetch two rss feeds, Digg and Mixx and sort them by pubDate
  select * from rss where url in ('http://digg.com/rss/index.xml','http://feeds.mixx.com/MixxPopular') | sort(field="pubDate")
• Find Flickr photos that are tagged "fog" and are in San Francisco
  select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text="san francisco, ca") and tags = "fog")

We also listened to your suggestions and made it easier to access your Pipes RSS/JSON urls by putting it on the main Use this Pipe banner menu. It should speed up access to these widely used formats.

There are several ways you can achieve this. You can host your own - or even host it on Google App Engine. Sam Pullara has a great post on how to extend Pipes with GAE here.

One host that we've been playing around with is AppJet. Using AppJet as your host for your Pipes webservice module is easy to make, fast and effective. AppJet uses JavaScript as the server side language making it even easier to use for web developers.

For this example, I want to randomize my feed. First you'll need to create an appjet login, after that you can clone my example AppJet app. Appjet like Pipes lets you view source, which for developers is one of the best ways to learn and see how things work.

Some things to take note. You'll want to name your AppJet function name "post_<somename here>". This tells AppJet that you're posting data to this function. You'll also want to add page.setMode("plain"); and var data = JSON.parse(request.params.data); as your first and second line.

var data, gives you access to the pipes output data array. You can do whatever you want with it. For my example I just do a randomized sort on it. If you want to manipulate each item in the data array, its easy to loop through it and change the feed as necessary (see my commented code in the AppJet example). You can also add items to the data array by declaring something like: data.items[i].newitem = <whatever>;

AppJet makes it so easy and is a great way to extend Pipes functionality. Here is my example Pipe that will bring it all together.

You'll want to make sure that "Path to items list:" has a value of "items" or similar for this example to work.

We also fixed the pubDate on our output. We will display the 4 digits of the year (2008) vs. the 2 digits (08).

This is especially useful to Flash developers who use our yahooapis.com crossdomain policy file as the web2.0 URLs aren't supported in that domain.

To get the ID of your Pipe go to "More Options" on the Pipe info page and click "Get as RSS" (or as JSON etc). This will open a new window and you'll see the Pipe ID in the URL.

If you are a Flash developer, just replace pipes.yahoo.com with pipes.yahooapis.com as described in this previous post: http://blog.pipes.yahoo.net/2007/03/10/pipes-adds-flash-developer-support/

We have been in contact with Twitter trying to resolve this problem and supplied them several test cases. Based on various tests we've performed it appears that Twitter's servers are blocking requests from our outbound caches. At the moment they are still looking into it, but until they uncover the reason any Pipes using Twitter feeds will only work occasionally.

If there is any change in status regarding this situation, we will let you know.

Thanks - The Pipes team