Need to extract data across a number of URLs?
From time to time, you may need to extract data based on a range of categories, multiple search terms, or simply from multiple pages. These values (categories, search terms, page numbers, etc) are often encoded as parameters in the URLs. By tweaking the parameters, you can create a list of URLs to run the Extractor on in order to get all the data you want.
Here are some example URLS:
However, URL generation can sometimes be cumbersome, so we have come up with a great little tool to help you.
Introducing the URL Generator
Under Extractor Settings inside your Dashboard, click the “Show URL Generator” button to reveal a simple interface where you can see all the parameters of the URL on which you trained your Extractor.
If the current value of a parameter is a "list of values", we allow you to specify a list of values (separated by commas).
On the other hand, if the value is a number, you can choose between specifying a range of numbers or a list of values. Choosing a range of numbers also allows you to specify step increments, to create a list such as 1, 11, 21 … 201.
As you tweak the different values, you will see the list of URLs in the preview box underneath. If you are happy with the generated URLs, just click the “Add to list” button to add them to the Extractor’s URL list, before running the Extractor.
Adding extra parameters
The list of parameters is based on the URL on which you trained your Extractor. You can add new parameters by highlighting the part of the URL that you want to make into a variable.
Simply click the X to the left of a parameter to remove it.
Editing the URL
You can change the URL by clicking "Edit" at any time.