Pipe into your netlife

Yahoo is doing a lot of interesting things these days. While Google gains a lot of the attention when it comes to search and web applications, Yahoo should not be ignored. del.icio.us and Flickr are of course two widely popular services, but they have a few useful utilities floating around their developer site.

There’s one tool in particular that I found tonight that has already proven useful. Pipes. Pipes allows a user to quickly put together a simple set of pipeline filters for turning various forms of data into an RSS feed and accompanying JSON file.

Such forms of data include user input (validated as a date, geographical location, number, text or URL) and web-based input (Flickr pictures, RSS2/Atom feeds, JSON/XML data, Google Base listings, Yahoo! Local searches, and Yahoo! search results). This data can be fed through several layers of pipes (including back into another Flickr pipeline and such as query input). The pipes can transform the data, walk through each feed item and modify or extract data, combine data together, sort, remove duplicates, apply regexes, translate languages, and so on.

This can be pretty powerful. While still a young project, many users have already published pipes, myself included. With the increase in API-enabled web services, I can only expect this to become more powerful, with work. It’s just a little tricky coming up with actual useful applications.

So I played around a bit and started to experiment with what could be done. I ended up with a couple of simple, but very useful pipes. One thing I have wanted for the longest time was a way to see feeds from several Planets in one listing (for Netvibes, since space is precious), without having to deal with duplicate entries. Pipes made this all too simple.

I’m feeding several feeds into a Unique operator, saying to filter based on the title. I then output that. That’s all it takes. You can see the results and even add the RSS feed.

I then took this one step further and decided to write a quick pipe for searching through the planets. Now, pipes are reusable, so I was able to incorporate the Unique Planets pipe into this. This was fed into a Filter, using a couple of text inputs (for a text string and a name) as parameters to the filter. The screenshot below will clarify this. The result is the ability to quickly search four planets by name or content.

You can play with the results. Go ahead, give it a try.

Pipes can be published for other people to use, or they can be used privately. Private pipes are great when you want to deal with data that can’t easily be queried, such as your Twitter feed or your own Flickr feed.

Pipes are also quite useful when you have a small web application that needs to deal with several other feeds, filtering results or combining data from multiple sources. Sure, you could write this all yourself, but it’s far easier to change and maintain a Pipe than a whole bunch of code.

If you want to play with pipes, I recommend just jumping in and playing. Also take a look at some other people’s pipes, and you may want to browse the tutorials. For some starter ideas, try making a pipe that searches your local area for sales or singles or something using Google Base and Craigslist, or one that searches all your favorite blogs for a certain keyword, or maybe something that keeps track of your friends’ blogs and Flickr posts.

Now, pipelines are hardly a new concept. Several programs offer them, including some development environments that rely solely on pipelines for development in order to quickly produce simple programs. What makes Yahoo!’s Pipes interesting is that they make it very easy for almost anybody to quickly build a pipe to modify or search all kinds of data on the web that people actually use. This makes them more immediately useful to many people, and of course Yahoo makes it dead simple to start out.

What would be useful in the future, aside from adding native support for more services, would be to output data in other formats or somehow easily lay out information onto a page from one or more feeds. The project seems pretty young though, so I’m sure in time, this will mature into a much more useful project, both to developers and (certain) end users.