Lua Scripting - Submiting via JS

Hi,

I’ve been having trouble getting a lua script that calls something like:

splash:runjs("$(':submit')[0].click()")

to run. That is, splash doesn’t seem to actually render the result of the page after submission - as if the submit button had been clicked in the first place.

Any idea how to work around this or what I may be doing wrong?

Thanks, Alex

[kmike]

Aha, you’re right. There are two issues.

1. WebKit version Splash uses doesn’t provide .click() function for <a> elements - only buttons (<input> elements) work. This may be fixed by https://github.com/scrapinghub/splash/pull/222, but I haven’t checked it.
2. jQuery .click() method doesn’t allow to follow links - see e.g. http://stackoverflow.com/questions/1694595/can-i-call-jquery-click-to-follow-an-a-link-if-i-havent-bound-an-event-hand.

A workaround (until Splash gets a nice library to do these tasks) is to use window.location = document.getElementById('#link').href:

function main(splash)
  assert(splash:go(splash.args.url))
  splash:runjs("window.location = document.getElementsByTagName('a')[0].href")
  splash:wait(0.5)
  return {
    html = splash:html(),
    png = splash:png(),
    har = splash:har(),
  }
end

Another workaround is to use a library other than jQuery. For example, one can use CasperJS clientutils:

function setup_casperjs(splash)
  -- preload CasperJS client utils.
  -- __utils__ object is compatible with CasperJS
  splash:autoload("https://raw.githubusercontent.com/n1k0/casperjs/master/modules/clientutils.js")
  splash:autoload([[
    window.__utils__ = new ClientUtils({});
  ]])
end

function main(splash)
  setup_casperjs(splash)
  assert(splash:go(splash.args.url))
  assert(splash:runjs("__utils__.click('#link')"))
  splash:wait(0.5)
  return {
    html = splash:html(),
    png = splash:png(),
    har = splash:har(),
  }
end

As an example, I’m loading CasperJS ClientUtils from github; for production it is better to find a CDN URL for autoload or serve it from your own server.

[kmike]

There are now scraping helpers which allow to select elements, submit forms, etc. Fixed by #490 and #531.