Dashboard Order Developer FAQ
Login

 Introduction

 API call structure
 Scrape Google Search
 Scraping Bing Search
 Keyword Analytics
 Keyword rank tracker


 Source Code download

Basic API information

Minimal API link: https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_
API login: _USER@ACCOUNT_
API key: _CREDENTIALS_

Keyword Analytics and autocomplete spider - API

 

The keyword analytic module is a powerful scraping service.

If you are familiar with our Google Search of Bing Search scraping API, the keyword analytics API is nearly identical.

It allows to scrape jobs containing 1 to 50,000 keywords for a range of various analytical information.

 

The following information will be scraped

  • Monthly average search volume of the past 12 months
    • broad match average volume
    • exact match average volume
  • Search volume for each of the past 12 months
    • broad match volume
    • exact match volume
  • Search trends in percent of the past 12 months (very similar to Google Trends)
  • Country distribution
    • broad and exact match average monthly search volume for each country with relevance
  • Website density
    The website density is an indicator how many websites are available for this keyword
    • A level from 0 (barely any) to 6 (hundreds of millions) and a human readable form
  • Related keywords and phrases
    • trending keywords
      These keywords and phrases have been very relevant in organic searches for a long time
    • rising keywords
      These keywords are becoming more relevant in organic searches
  • Keyword suggestions
    Keyword suggestions are results from our Google autocomplete recursing spider, they contain up to 20,000 highly related
    keywords and long tail phrases from organic search.
    A very powerful research and idea generator.
    A default scrape contains up to 20 suggestions, it's possible to select other scraping modes to get up to 20,000.
    • Default mode: 20 most relevant suggestions from autocomplete (total max: 20 keyword ideas)
    • Recursing mode: The previous 20 results are each also checked for up to 20 suggestions each (total max: 420 keyword ideas)
    • Extended mode: In addition the first 20 keywords are researched for closely related suggestions (total max: 900 keyword ideas)
    • Trends mode: In addition gather results for all rising and trending keywords (total max: 1900 keyword ideas)
    • Recursive trends: In addition recurse into all trend results. (total max: 20000 keyword ideas)

 

How to use this API

It's as easy to use this scraping module as to use our Google or Bing scraper.

It can be done in just 4 lines of code!

  1. create_job()
    Create a new job, give it a unique identifier
  2. modify_job()
    Define scraping speed, keywords and autocomplete spidering mode
  3. start_job()
    Start the job, it will be scraped in background on our servers
  4. get_results()
    Download the results, either in CSV or JSON format

For a professional integration some more API functions will be useful:
get_jobs() and get_job() to find out whats going on and if a job is completed

Possibly also wallet_info() and license_info() 

General info:

The basic API documentation will help understanding how our API is structured.

The API always responds with JSON data which can easily be parsed into an array or object, our source code examples show how this is done.

Most API functions will not require complex data, they can be done through URI GET parameters.

To supply larger amount of data (large keyword lists) or arrays it's recommended to encode them into JSON and supply them as POST &json={..} parameters

 

Hint: When debugging the API in a browser it's recommended to use type=html instead of type=json, this will make the json human readable.

 

 

 

API Functions

get_jobs: 

https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_kw_analytics&type=json&task=get_jobs

This API function will return up to 50 jobs. All details about the jobs are included except the keywords,result data and detailed progress.

The API function "get_job" can be used to retrieve progress details about one specific job.

 

Example response: 

{
    "exceptions": [],
    "answer": {
        "jobs": [
            {
                "jobname": "QA_10\/07\/2017 01:24:13",
                "created": "2017-10-07 01:24:13",
                "job_type": "scrape_kw_analytics",
                "estimated_keywords": "17",
                "priority": "1",
                "finished": "2017-10-18 01:40:46",
                "credit_cost": "0.86",
                "start": "1",
                "progress": 100
            }
] } }

In this example a finished job is shown, if finished is null or progress below 100 the job is not finished.

If start is 0 the job was not started yet.
The progress variable will show either 0, 1 or 100. A more detailed progress is available through get_job()



get_job:

https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_kw_analytics&type=json&task=get_job&jobname=_JOBNAME_

This call is used to receive details about a specific job, especially once it has been started.

The API will respond with a live progress value as well as the number of scraped keywords as well as available pages.

 

Example response:

{
    "exceptions": [],
    "answer": {
        "job": {
            "jobname": "QA_10\/07\/2017 01:24:13",
            "created": "2017-10-07 01:24:13",
            "job_type": "scrape_kw_analytics",
            "estimated_keywords": "17",
            "priority": "1",
            "finished": "2017-10-18 01:40:46",
            "credit_cost": "0.86",
            "start": "1",
            "operation_mode": "mode_related_ideas",
            "progress": 100,
            "total_keywords": "17",
            "total_results": "17",
            "pages": 1,
            "total_pages": 1,
            "estimated_hours_left": "0.00",
            "estimated_hours_left_hr": "0h 0m"
        }
    }
}


The results are similar to get_jobs() but now they include time estimation in hours and human readbale format.

The progress variable is now showing intermediate steps

The pages variable tells how many result pages of collected keywords are available

The total_pages variable tells how many result pages will be available once the progress is finished.


get_keywords:

https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_kw_analytics&type=json&task=get_keywords&jobname=_JOBNAME_

This call is used to retrieve all keywords of a job, they will be separated into "finished" and "unfinished"

Available response types are: html, csv, json

 

Example response:

{
  "exceptions": [
    
  ],
  "answer": {
    "keywords": {
      "jobname": "QA_10\/07\/2017 01:24:13",
      "job_type": "scrape_kw_analytics",
      "keywords_finished": [
        "white wool blanket",
        "mexican recipes",
        "beds for sale",
        "nike tennis shoes"
      ],
      "keywords_unfinished": [
        
      ]
    }
  }
}


create_job:

https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_kw_analytics&type=json&task=create_job&jobnameTEST_JOB


The API will respond as usual with a specific json encoded object:

{
    "exceptions": [],
    "answer": {
        "job": {
            "jobname": "TEST_JOB",
            "job_type": "scrape_kw_analytics",
            "start": 0
        }
    }
}

remove_job:

https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_kw_analytics&type=json&task=remove_job&jobname=_JOBNAME_

This removes a selected job by it's unique identifier.

You may remove any Job that has not been started.

 

API Response:

{
"exceptions": [],
"answer":
[
true
]
}



modify_job:

https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_kw_analytics&type=json&task=modify_job

This function is recommended to use with the json POST parameter instead of GET parameters.

 

A job may be modified, extended or deleted until it has been started. 

Once a job was started it can not be modified in any way.

The supplied json object must include job_type and jobname, any other elements are optional.

When adding keywords to a Job the element keyword_operation may contain "replace" or "append".

  "replace" will remove any existing keywords from the selected Job before adding new keywords.

  "append" is default behaviour and will append any supplied keywords to the selected Job.

The variable priority defines the scraping speed (see priority), it's recommended to use a priority between 1 and 10.

The variable operation_mode defines the amount of data to scrape:

  "mode_only_searchvolume" : results without autocomplete spider, this setting should be used if autocomplete ideas are not relevant

  "mode_normal_analysis" : normal operation, 20 autocomplete results

  "mode_recurse_ideas" : recursion into autocomplete, up to 420 ideas

  "mode_extended_ideas": in addition searching for longer tail keywords, up to 900 ideas

  "mode_related_ideas" : in addition, recursion into trending related keyword, up to 1900 ideas
  "mode_recurse_trend_ideas": in addition recurse in all trending idea results, up to 20000 ideas

 

Example json object:

{
  "jobname": "QA_10\/07\/2017 01:24:14",
  "priority": 1,
  "job_type": "scrape_kw_analytics",
  "keyword_operation": "replace",
  "keywords": [
    "key word",
"another keyword"
], "operation_mode": "mode_normal_analysis" }

 

Example API response:

{
    "exceptions": [],
    "answer": {
        "job": {
            "jobname": "QA_10\/07\/2017 01:24:14",
            "priority": 1,
            "job_type": "scrape_kw_analytics",
            "estimated_keywords": 2,
            "credit_cost": 0.45,
            "progress": 0
        }
    }
}

start_job:

https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_kw_analytics&type=json&task=start_job&jobname=_JOBNAME_

Any created job that contains keywords may be started at any time.

Once started, the job is locked from further modification and will immediately deduct the credit cost from your wallet.

 

Once a job was started it can be monitored using the "get_job" API function to see how many keywords are already finished.

It is possible to download the keyword results using "get_results" while a job is running.

We currently do not guarantee to keep results for more than two months.

 

Example response:

{
"exceptions": [],
"answer":
{
"job":
{
"jobname": "demo_job",
"start": "1",
"job_type": "scrape_kw_analytics",
"credit_cost": "0.45"
}
}
}



get_results:

https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_kw_analytics&type=json&task=get_results


get_results() allows to export data, in comparison the Google and Bind search the function is a low more flexible.

 

Following parameters may be used to filter and fine tune results:

  • page_quantity
    Default is 25, max is 1000
    This is the number of keywords exported per api function call. 
    Used for pagination
  • page
    Default is 1
    This is the page of results, be default it will load the first (page=1) 25 (page_quantity=25) results.
  • jobname
    An optional jobname to limit results to this job only
  • jobnames
    An optional array of jobnames to limit results to these jobs only
  • keywords
    An optional array of keywords, up to 100 keywords max be added
    This is used to load the results of specific keywords
  • type
    csv, json or html
In addition you need to specify which sort of data to retrieve by adding additional parameters
  • get_all=1
    This setting includes all data, alternatively the sections can be fine tuned
  • get_search_volume=1
  • get_country_share=1
  • get_website_density=1
  • get_related_keywords=1
  • get_keyword_ideas=1


The results are always sorted with oldest results first.

It is possible to use no filter parameters at all, in this case all jobs and keywords can be exported page by page.


Example response (page_quantity=1, page=1):

{
    "exceptions": [],
    "answer": {
        "results": {
            "page": "15",
            "page_quantity": 1,
            "pages_available": 17,
            "total_results_available": "17",
            "job_type": "scrape_kw_analytics",
            "keyword_results": {
                "keyword analyzer": {
                    "keyword": "keyword analyzer",
                    "jobname": "QA_10\/07\/2017 01:24:13",
                    "accuracy": 350,
                    "search_volume": {
                        "monthly_average": 3480,
                        "volume": {
                            "10\/01\/2016": 3230,
                            "11\/01\/2016": 3230,
                            "12\/01\/2016": 2940,
                            "01\/01\/2017": 4110,
                            "02\/01\/2017": 3230,
                            "03\/01\/2017": 3820,
                            "04\/01\/2017": 3820,
                            "05\/01\/2017": 3530,
                            "06\/01\/2017": 3530,
                            "07\/01\/2017": 3530,
                            "08\/01\/2017": 3230,
                            "09\/01\/2017": 3820,
                            "10\/01\/2017": 3230
                        },
                        "trends": {
                            "10\/01\/2016": "0.79",
                            "11\/01\/2016": "0.79",
                            "12\/01\/2016": "0.71",
                            "01\/01\/2017": "1.00",
                            "02\/01\/2017": "0.79",
                            "03\/01\/2017": "0.93",
                            "04\/01\/2017": "0.93",
                            "05\/01\/2017": "0.86",
                            "06\/01\/2017": "0.86",
                            "07\/01\/2017": "0.86",
                            "08\/01\/2017": "0.79",
                            "09\/01\/2017": "0.93",
                            "10\/01\/2017": "0.79"
                        },
                        "monthly_average_exactmatch": 1580,
                        "volume_exactmatch": {
                            "10\/01\/2016": 1470,
                            "11\/01\/2016": 1470,
                            "12\/01\/2016": 1340,
                            "01\/01\/2017": 1870,
                            "02\/01\/2017": 1470,
                            "03\/01\/2017": 1740,
                            "04\/01\/2017": 1740,
                            "05\/01\/2017": 1600,
                            "06\/01\/2017": 1600,
                            "07\/01\/2017": 1600,
                            "08\/01\/2017": 1470,
                            "09\/01\/2017": 1740,
                            "10\/01\/2017": 1470
                        }
                    },
                    "countries": {
                        "US": "1.00",
                        "BE": "0.19",
                        "BR": "0.07"
                    },
                    "website_volume": {
                        "description": "MEDIUM",
                        "level": 3
                    },
                    "trending": {
                        "top": [
                            "keyword tool",
                            "keyword analyzer tool",
                            "free keyword analyzer",
                            "seo keyword analyzer",
                            "website keyword analyzer",
                            "google keyword tool",
                            "keyword analysis",
                            "adwords keyword analyzer",
                            "keyword analyzer software",
                            "elasticsearch keyword analyzer"
                        ],
                        "rising": [
                            "keyword analysis",
                            "adwords keyword analyzer",
                            "elasticsearch keyword analyzer",
                            "keyword tool",
                            "keyword analyzer tool",
                            "free keyword analyzer"
                        ]
                    },
                    "keyword_ideas": {
                        "0": "keyword analyzer elasticsearch",
                        "1": "keyword analyzer lucene",
                        "2": "keyword analyzer semrush",
                        "3": "keyword analyzer vs standardanalyzer",
                        "4": "keyword analyzer online",
                        "5": "keyword analyzer tool free",
                        "6": "keyword analyzer software",
                        "7": "keyword analyzer lucene example",
                        "8": "keyword analyzer pro v1.0.0",
                        "9": "keyword analyzer pro",
                        "10": "keyword lowercase analyzer",
                        "11": "keyword analyzer lucene.net",
                        "12": "keyword analyzer resume",
                        "13": "keyword analyzer download",
                        "14": "keyword analyzer plugin",
                        "15": "keyword analyzer script",
                        "16": "freshkey keyword analyzer",
                        "17": "elasticsearch keyword analyzer lowercase",
                        "18": "keyword analysis elasticsearch",
                        "20": "elasticsearch keyword analyzer example",
                        "21": "elasticsearch keyword analyzer vs not_analyzed",
                        "22": "elasticsearch keyword analyzer case sensitive",
                        "23": "elasticsearch keyword analyzer regex",
                        "26": "lucene keyword analyzer lowercase",
                        "27": "lucene keyword analyzer wildcard",
                        "28": "lucene keyword analyzer case insensitive",
                        "29": "lucene keyword analyzer vs standardanalyzer",
                        "30": "keyword analysis online tool",
                        "31": "online keyword analyzer tool",
                        "32": "keyword analysis tool free",
                        "33": "keyword competition analysis tool free",
                        "34": "competitor keyword analysis tool free",
                        "35": "free seo keyword analyzer tool",
                        "36": "free keyword density analyzer tool",
                        "37": "keyword density analyzer software",
                        "39": "keyword density and prominence analyzer",
                        "40": "elasticsearch keyword lowercase analyzer",
                        "41": "stealth keyword competition analyzer download",
                        "43": "adwords keyword analyzer",
                        "44": "amazon keyword analyzer",
                        "45": "best keyword analyzer",
                        "46": "best keyword analyzer tool",
                        "47": "keyword competition analyzer",
                        "48": "keyword competition analysis tool",
                        "49": "keyword analysis corpus",
                        "50": "keyword analysis competitor",
                        "51": "keyword analysis cost",
                        "52": "keyword analyzer code",
                        "55": "keyword competition analysis",
                        "57": "keyword cloud analysis",
                        "58": "keyword cluster analysis",
                        "59": "keyword content analysis",
                        "60": "keyword cipher analysis",
                        "61": "stealth keyword competition analyzer",
                        "63": "stealth keyword competition analyzer crack",
                        "64": "stealth keyword competition analyzer free download",
                        "65": "free keyword competition analyzer",
                        "66": "online keyword competition analyzer",
                        "67": "keyword analyzer density",
                        "69": "keyword density analyzer paste text",
                        "70": "keyword analysis definition",
                        "71": "keyword difficulty analysis",
                        "72": "keyword difficulty analysis tool",
                        "73": "keyword data analysis",
                        "74": "analytics keyword dimension",
                        "75": "google keyword density analyzer",
                        "76": "offline keyword density analyzer",
                        "79": "keyword analysis excel",
                        "80": "keyword analysis example",
                        "82": "lucene keyword analyzer example",
                        "83": "keyword extraction analysis",
                        "89": "keyword density analyzer español",
                        "90": "keyword analyzer free",
                        "91": "keyword frequency analyzer",
                        "92": "keyword analysis for seo",
                        "93": "keyword analytics free",
                        "94": "keyword analysis free tool",
                        "95": "keyword analytics facebook",
                        "96": "keyword analysis facebook",
                        "97": "google keyword analyzer free",
                        "98": "keyword density analyzer free",
                        "101": "keyword analysis google",
                        "102": "keyword google analytics not provided",
                        "103": "keyword gap analysis",
                        "104": "keyword google analytics api",
                        "105": "keyword analysis how to",
                        "106": "keyword analyzer in elasticsearch",
                        "107": "imdb keyword analyzer",
                        "109": "keyword analysis java",
                        "110": "keyword analysis journals",
                        "115": "keyword analysis live",
                        "117": "analytics keyword location",
                        "121": "keyword analysis meaning",
                        "122": "keyword analysis moz",
                        "123": "keyword analysis method",
                        "124": "meta keyword analyzer",
                        "125": "movie keyword analyzer",
                        "126": "keyword analytics not provided",
                        "128": "analytics keyword not set",
                        "129": "keyword network analysis",
                        "131": "npm keyword-analyzer",
                        "133": "keyword analyzer.org",
                        "134": "keyword analysis of competitors",
                        "140": "keyword page analyzer",
                        "141": "keyword analysis pdf",
                        "142": "keyword analysis process",
                        "143": "keyword analysis php",
                        "144": "keyword analysis php script",
                        "145": "keyword analysis python",
                        "146": "keyword analysis proposal",
                        "147": "keyword analysis planner",
                        "148": "keyword analysis ppc hero",
                        "150": "keyword performance analysis",
                        "151": "keyword position analytics",
                        "152": "keyword analysis qualitative research",
                        "154": "keyword results analyzer",
                        "155": "keyword analysis report",
                        "156": "keyword analysis report sample",
                        "157": "keyword analysis report template",
                        "158": "keyword analysis report example",
                        "160": "keyword ranking analysis",
                        "161": "keyword research analysis",
                        "162": "keyword ranking analytics",
                        "163": "analytics keyword report",
                        "164": "keyword research analysis tools",
                        "165": "analytics keyword research",
                        "166": "keyword ranking analysis tool",
                        "169": "keyword analyzer site",
                        "171": "keyword stuffing analyzer",
                        "172": "keyword analysis seo",
                        "173": "keyword analysis software",
                        "174": "keyword analysis step by step",
                        "175": "keyword analysis software free",
                        "176": "keyword analysis service",
                        "177": "keyword analysis spreadsheet",
                        "178": "keyword analysis social media",
                        "179": "keyword analysis search",
                        "180": "keyword analysis suggestions",
                        "182": "keyword sentiment analysis",
                        "183": "keyword search analysis tool",
                        "184": "keyword semantic analysis",
                        "185": "analytics keyword spam",
                        "186": "keyword search analysis free",
                        "187": "keyword analyzer tool",
                        "188": "keyword text analyzer",
                        "189": "keyword traffic analyzer",
                        "190": "keyword analysis tool",
                        "192": "keyword analysis tool online",
                        "193": "keyword analysis tutorial",
                        "194": "keyword analysis template",
                        "195": "keyword analysis twitter",
                        "196": "keyword analysis tips",
                        "197": "keyword analysis tableau",
                        "201": "keyword trend analysis",
                        "202": "keyword analysis url",
                        "206": "keyword volume analysis",
                        "207": "keyword analyzer website",
                        "208": "keyword analysis wiki",
                        "209": "keyword analysis with r",
                        "211": "website keyword analyzer free",
                        "212": "keyword analysis youtube",
                        "213": "youtube keyword analyzer",
                        "214": "stealth keyword competition analyzer zip",
                        "215": "keyword tool google",
                        "216": "keyword tool dominator",
                        "217": "keyword tool free",
                        "218": "keyword tool amazon",
                        "219": "keyword tool pro",
                        "220": "keyword tool for youtube",
                        "221": "keyword tool dominator review",
                        "222": "keyword tool google free",
                        "223": "keyword tool app",
                        "224": "keyword tool ebay",
                        "225": "keyword tool review",
                        "226": "keyword tool dominator amazon",
                        "227": "keyword tool alternative",
                        "228": "keyword tool etsy",
                        "229": "keyword tool extension",
                        "230": "keyword tool amazon free",
                        "231": "keyword tool analyzer",
                        "232": "keyword tool comparison",
                        "233": "keyword analytics tool",
                        "238": "free keyword analysis tool",
                        "239": "free keyword analytics",
                        "240": "free keyword analytics tool",
                        "241": "free keyword analysis software",
                        "242": "free keyword density analyzer",
                        "245": "seo keyword analysis",
                        "246": "seo keyword analysis report",
                        "247": "seo keyword analysis online",
                        "249": "website keyword analyzer tool",
                        "250": "website keyword analysis",
                        "251": "website keyword analysis google",
                        "252": "website keyword ranking analyzer",
                        "253": "google website keyword analyzer",
                        "254": "google keyword tool 2017",
                        "255": "google keyword tools youtube",
                        "256": "google keyword tool api",
                        "257": "google keyword tool without adwords",
                        "258": "google keyword tool alternative",
                        "259": "google keyword toolbox",
                        "260": "google keyword tools free",
                        "261": "google keyword tool ebay",
                        "262": "google keyword tool to find niche",
                        "263": "google keyword tool io",
                        "264": "google keyword tool english",
                        "265": "google keyword tool for seo",
                        "266": "google keyword tool là gì",
                        "267": "google keyword tool uk",
                        "268": "google keyword tool apk",
                        "269": "google keyword tool free alternative",
                        "270": "google keyword tool software",
                        "271": "google keyword tools planner",
                        "277": "keyword analysis r",
                        "278": "keyword analysis api",
                        "280": "keyword analysis in seo",
                        "282": "keyword analysis for website",
                        "285": "keyword analysis google adwords",
                        "286": "keyword analysis online",
                        "293": "elasticsearch keyword analyzer case insensitive"
                    },
                    "timestamp": "10\/18\/17 01:40"
                }
            }
        }
    }
}