Dashboard Order Developer FAQ
Login

Frequently asked questions

Keyword SERP scraping

Keywords and search operators

An introduction to keywords and search operators


Each of our jobs may contain up to 50.000 keywords, however it is recommended to use multiple smaller sized jobs instead.
A keyword may contain up to 250 characters, it's recommended to use shorter keywords if possible.


Especially when working with Google the keyword length and keyword complexity does matter, especially when scraping ten thousands of keywords.
Google prevents scraping through a self-learning system.

Our service is highly optimized in avoiding scraping detection, however it's good practice to use an organic type of keyword when possible.

The more organic (human-entered) your keywords looks, the less likely Google can detect the scraping activity.

The large majority of our clients is not affected by this, however when scraping very specific data from Google this can cause an issue.


When the detection system is triggered, keywords of similar structure might be rejected, in some cases we have seen temporary world-wide bans.
Especially when using "intitle:", "site:" and similar special search operators keywords can be detected by Google.

 

In many cases it's possible to reduce or avoid search operators:

Instead of searching for "best shoes site:ebay.com" you might just search for "ebay.com best shoes"

Searching for thousands of shoes using "site:ebay.com" is likely going to result in some skipped keywords from detection.

Searching without "site:" is likely not going to skip any keywords.

 

 


Bing is often a good alternative for pure result harvesting, similar search operators can be used with Bing just as with Google.

Google search operators: https://support.google.com/websearch/answer/2466433?hl=en
Bing search operators: https://msdn.microsoft.com/en-us/library/ff795662.aspx