In this featured article, our Resident Lead Engineer Dom Deren explains the JSON response.
Normal JSON looks like this:
While New Line delimited JSON looks like this:
We are aware that this kind of syntax can be confusing for developers, believe me we are using this syntax as well, and we have to add some additional processing around it, to support this format in our frontend applications.
But this format brings us also a few advantages, for which it is worth it to add some additional processing. The second format is really good for fast processing, list data. If we have stuff like list of URLs, or list of results from the execution done over time, it is much better to save those results, in the second format. It is much faster to save data in this format, because it is a simple append only operation for the file where the results are stored. It is also much faster to read, if you are not interested in the end of the list, before you start displaying the beginning of it. In other words, you don’t need to download the whole file, to start seeing meaningful results. This is really important with big data dumps, like our results. If your Extractor returned 400k rows, believe me you don’t really want to open this file in a usual text editor. Even Google Docs might have problem with processing such a file. But if the file is saved in the second format, you can easily have a look at the first 100 results, and later simply paginate through the rest, making it possible to easily display such results even on mobile devices.