Load URL from files?

Hi again

I have US state wise URLs like http://example.com/ca, http://example.com/al

I have created my scraper and saved it. Is it possible to assign URLs from file in load textfile instead of I run them manually?

Thanks

[schasins]

Internally, we already have the option to use parameters defined outside of the Helena program. I’ve been planning for a while now to add the option to provide them from the command line (for the Docker or the standard run scripts). I’ll go ahead and move that to the top of the to do list.

[schasins]

Here’s how you specify the start URL from the command line:

Load the Helena script you want to edit in the Chrome extension.

Open the “Advanced Options” menu.

You’ll see a section that starts with “You can enter a parameter name here…” In that section, enter a name (e.g., url_param) and default value (e.g., example.com). The default value will allow you to test the script from the Chrome extension.

Once you press the Add Parameter button, you can start using the parameter value anywhere in your program (e.g., in the load statement that loads the start URL). To find a node that uses the parameter, click on the “other” section in the toolbox, then pull out the pink block. You can browse the names in the block’s pulldown menu until you find the name of your new parameter.

If you run the program from the extension, it will now use the value you gave as the default value.

To run the program with different values from the command line, you use a docker command like this: docker run -t -p 5900:5900 -e VNC_SERVER_PASSWORD=password -e HELENA_PROGRAM_ID=4249 -e TIME_LIMIT_IN_HOURS=23 -e NUM_RUNS_ALLOWED_PER_WORKER=1 -e SCRIPT_PARAMS=“url=example2.com” --user apps --privileged schasins/helena:latest The key part there is: -e SCRIPT_PARAMS=“url=example2.com”