Help Center

URL Generator: Create a list of URLs to run with your Extractor

Last Updated: Sep 05, 2016 01:46AM PDT

Need to extract data across a number of URLs?


From time to time, you may need to extract data based on a range of categories, multiple search terms, or simply from multiple pages. These values (categories, search terms, page numbers, etc) are often encoded as parameters in the URLs. By tweaking the parameters, you can create a list of URLs to run the Extractor on in order to get all the data you want.


Here are some example URLS:

Categories:
http://aviation.stackexchange.com/questions/tagged/engine
http://aviation.stackexchange.com/questions/tagged/weather

Search terms:
http://stackexchange.com/search?q=monkey
http://stackexchange.com/search?q=horse

Pages:
http://stackexchange.com/?page=2
http://stackexchange.com/?page=3
http://aviation.stackexchange.com/questions/tagged/engine?page=4

However, URL generation can sometimes be cumbersome, so we have come up with a great little tool to help you.


Introducing the URL Generator


Under Extractor Settings inside your Dashboard, click the “Show URL Generator” button to reveal a simple interface where you can see all the parameters of the URL on which you trained your Extractor.



If the current value of a parameter is a "list of values", we allow you to specify a list of values (separated by commas).



On the other hand, if the value is a number, you can choose between specifying a range of numbers or a list of values. Choosing a range of numbers also allows you to specify step increments, to create a list such as 1, 11, 21 … 201.



As you tweak the different values, you will see the list of URLs in the preview box underneath. If you are happy with the generated URLs, just click the “Add to list” button to add them to the Extractor’s URL list, before running the Extractor.




Adding extra parameters
The list of parameters is based on the URL on which you trained your Extractor. You can add new parameters by highlighting the part of the URL that you want to make into a variable. 


Removing parameters
Simply click the X to the left of a parameter to remove it. 


Editing the URL
You can change the URL by clicking "Edit" at any time. 





Get generating!
c2d12fc2f876f019701e1c3951e354bd@importio.desk-mail.com
http://assets0.desk.com/
false
desk
Loading
seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
about
false
Invalid characters found
/customer/en/portal/articles/autocomplete