Mateusz Stawecki

UPDATE: This is an old piece, you probably want: phantom.js and casper.js

In this post I’ll show you how to remotely control your Google Chrome browser using JavaScript and scrape some data, even if it’s on an AJAX powered website or behind HTTPS authentication. Nice? I LOVE WebKit and now it got even sweeter. Very recently there was a small buzz about a new feature: WebKit Remote Debugging that allows to use Element Inspector remotely! (since it’s essentially just a web page and some javascript + websockets). What’s the real icing on the cake for me, is the ability to plug-in a different interface. I wrote a simple interface that can execute several pieces of JavaScript and return the values back to you. Here’s how to set it up. Open Terminal and find the Google Chrome executable. To use remote debugging, run it with a special parameter:

If you get:

Try symlinking Versions:

The browser should start normally. Now go to a different browser, e.g. Safari and check out: http://localhost:9222 Select a page and you should see the Inspector. That’s all nice and neat. But let’s see my remote script: http://gist.github.com/972742 – To connect to the debugger, we’re using WebSockets. Change the page number based on the link from http://localhost:9222 . Every “Tab” has a different “Page” number.

	// Set page number!
	var host = "ws://localhost:9222/devtools/page/5";
	socket = new WebSocket(host);

– To execute JavaScript I wrapped a JSON-RPC-like command into a method with callback. More protocol schema here.

function remoteEval(scriptString,callback) {
	seqCallback[seqNo] = callback;
	socket.send('{"seq":'+seqNo+',"domain":"Runtime",'+
	'"command":"evaluate","arguments":{"expression":"'+
	scriptString.replace(/["]/g,'\"')
	+'","objectGroup":"console","includeCommandLineAPI":false}}');
	seqNo++;
}

– And this is how a sample script works:

remoteOnLoad = function(result) {

// We might've ended up on the login page, so let's log in!
if (remoteURL.indexOf("ServiceLogin") > 0 )
remoteEval( " document.getElementById('Email').value = 'username'; "+
	" document.getElementById('Passwd').value = 'password'; "+
	" document.getElementsByTagName('form')[0].submit(); "
	, function(result) { alert(result); } );

	// We're home!
	if (remoteURL.indexOf("mail.google.com/mail/") > 0 )
	remoteEval( " try { document.getElementById('canvas_frame')"+
".contentWindow.document.getElementsByClassName( 'md' )[0].innerText } catch(e) { -1 }"
	, function(result) {  // Waiting for AJAX. Try again in 2 sec.
	if (result == -1) { setTimeout(remoteOnLoad,2000); }
				else { alert(result); } } );
				// This should return scraped information
				// about your data usage on gmail!
				// E.g. You're currently using 150MB out of 7000MB
			} ;

// This happenes first:
remoteEval(" location.href = 'http://gmail.com' ");
//Let's go to Gmail!

Don’t know how about you, but I just wanna wrap it in Node.js, run it somewhere on Linux with a dummy X11 server for Chrome and write lot’s of crazy tasks, so it does it all for me! Imagine e.g. that instead of that alert(result) you make it a WebHook or a service?