Mateusz Stawecki

Thoughts on Mobile, Reactive & Serverless

Getting cross-site content with JavaScript is cool, since it usually requires a tiny bit of extra effort when designing the feeds or deploying a callback proxy. Thanks to Yahoo Pipes it has now become very easy to get any content, even a whole web page and crunch it in your JavaScript web app.
First go to pipes.yahoo.com and Create a new Pipe. You’re going to have to log in.
Drag “URL Input” from “User inputs” to your pipe diagram. Now get back to “Sources”. If you’re planning on fetching a website, drag “Fetch Page”. If it’s a JSON or XML feed “Fetch Data” (all data will be automatically transformed ).
The next step is linking the elements together. Drag a link from the “URL Input” into you source’s URL Attribute and from sources output to Pipe Output.
Your pipe should look something like this:

Click “Save” and “Run Pipe…”. You’ll be taken to the pipe information page, where you can enter the URL which you want to proxy. Submit with “Run Pipe”, after it has loaded, select “Get as JSON”.
You should see some nice JSON with the transformed content. The URL to such pipe looks like this:
http://pipes.yahoo.com/pipes/pipe.run?_id=d53d0d4793aa292d3e02885ba9b22cba&_render=json&fromurl=http://stawecki.com
Now guess how to add a callback 😉
http://pipes.yahoo.com/pipes/pipe.run?_id=d53d0d4793aa292d3e02885ba9b22cba&_render=json&fromurl=http://stawecki.com&_callback=test
Surprise 😉 There is an underscore before “callback” attribute name. (weirrrd) For anyone who is new to XSS, “callback” is basically the name of a JavaScript function that receives the content. To show how it works, here’s a simple script that downloads a website and returns the response’s character length:


	function test(response) {
		try {
			alert( 'Content size: '+
			response.value.items[0].content.length );
			} catch(err) { alert('Invalid response'+err); }
		}

	function getURL(urlstr) {
		var ka = document.createElement('script');
		ka.type = 'text/javascript';
		ka.src =
'http://pipes.yahoo.com/pipes/pipe.run?_id=d53d0d4793aa292d3e02885ba9b22cba&_render=json&fromurl='+
		escape(urlstr) + '&_callback=test';
		var ks = document.getElementsByTagName('script')[0];
		ks.parentNode.insertBefore(ka, ks);
	}

And here’s a live version to try:
Simple Pipes Callback

Here’s a slightly more complicated example. I’m getting HTML content, parsing it internally and listing all links on the website (NICE!):
Link Scraper Demo

These examples used “Fetch Page” in pipes. Here’s an example of reading an RSS feed using “Fetch Data”. This script alerts the most recent news item from an external RSS feed. Notice that Yahoo Pipes conveniently transforms any feed to JSON (or XML if you really really want).


	function test(response) {
		try {
			var newsItem = response.value.items[0].channel.item[0];
			alert( newsItem.pubDate +" - " + newsItem.title +" - " + newsItem.description );
			} catch(err) { alert('Invalid response'+err); }
	}

	function getURL(urlstr) {
		var ka = document.createElement('script');
		ka.type = 'text/javascript';
		ka.src =
		'http://pipes.yahoo.com/pipes/pipe.run?_id=4d625dfe6977e71acb45db4aa51726a6&_render=json&feedurl='+
		escape(urlstr) + '&_callback=test';
		var ks = document.getElementsByTagName('script')[0];
		ks.parentNode.insertBefore(ka, ks);
	}

And here’s a live version to try with BBC News RSS feed:
Simple RSS Callback

Now go and cross site the hell out of the internetz! Soon slightly more practical examples.

One Response to “Transfer any content Cross-site with Yahoo Pipes as JSON callback proxy”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Basic HTML is allowed. Your email address will not be published.

Subscribe to this comment feed via RSS

%d bloggers like this: