Our Google scraping API is based on HTTP(s) POST or GET calls to our API server. All API requests are self contained and do not require to keep a session or cookies.
Each API call costs 0.01 Credits to prevent overusage (about $1 per 1000 calls) in addition to possible additional charges.
Our Google and Bing scraping APIs are very similar, you can use the same code for both types of scraping.
Not a developer?
We offer web based tools to scrape Google and export results without any API integration requirements.
Scrape google search by web-tool
We use cURL and URL examples during most of the documentation.
We also provide PHP and Python source code examples to help you get started quickly.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_
# This base URL contains the credentials of your account
URL: https://scraping.services/api
User: _USER@ACCOUNT_
Credentials: _CREDENTIALS_
Let's start by taking a look at a simple request: license_info
The URI is "https://scraping.services/api" with a set of essential GET parameters:
Some API calls require more complicated structures to be passed, for example large keyword arrays.
In these cases the additional variables are supplied as JSON object encoded POST parameter.
It is up to the developer when to use the "json" parameter or when to supply the parameters as GET through the URI.
To see the 'json' parameter being used take a look at modify_job or the source code examples.
The API response always contains two json objects: 'exceptions' and 'answer'.
The API response can be converted into a native object or array by almost any programming language.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=info&task=license_info&type=json&human_readable
curl -G https://scraping.services/api \
-d user=_USER@ACCOUNT_ \
-d credentials=_CREDENTIALS_ \
-d job_type=info \
-d task=license_info \
-d type=json \
-d human_readable
{ "exceptions": [], "answer": { "license": { "id": 1, "name": "1xsmall", "monthly_cost": "14.00", "monthly_credits": "100", "time_left": "386:31:46", "time_left_grace": "482:31:46" } } }
{ "exceptions": [ { "exception_text": "Wrong username or password", "level": 10000, "unixtime": 1567743178, "error_id": 109 } ], "answer": false }
When scraping Google for a collection of keywords you first create a 'job' with a limit of 200 unfinished jobs being held in your account.
The job contains all settings required to start the scraping run:
"job": { "jobname": "G09/06/2019 15:52", "created": "2019-09-06 15:53:03", "job_type": "scrape_google_search", "max_results": 200, "estimated_keywords": 1, "priority": 1, "finished": "2019-09-06 15:53:53", "credit_cost": "0.65", "start": 1, "progress": 100, "total_keywords": 1, "total_results": 1, "pages": 1, "total_pages": 1, "estimated_hours_left": "0.00", "estimated_hours_left_hr": "0h 0m", "country": "Default", "language": "English" }
This API call will return up to 50 latest jobs. All details about the jobs are included except the keywords,result data and current scraping progress.
To track the progress of a specific job in detail you need to use 'get_job' instead.
The right sided response example shows an account containing two jobs. The latest job was started and is finished, the second job was never started.
The 'progress' variable using get_jobs will only return 0(%) for unfinished or unstarted jobs or 100(%) for finished jobs.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=get_jobs&type=json&human_readable
{ "exceptions": [], "answer": { "jobs": [ { "jobname": "G09/06/2019 15:52", "created": "2019-09-06 15:53:03", "job_type": "scrape_google_search", "max_results": 200, "estimated_keywords": 1, "priority": 1, "finished": "2019-09-06 15:53:53", "credit_cost": "0.65", "start": 1, "progress": 100 }, { "jobname": "G04/21/2019 15:09", "created": "2019-04-21 15:09:26", "job_type": "scrape_google_search", "max_results": 1000, "estimated_keywords": 1, "priority": 1, "finished": "0000-00-00 00:00:00", "credit_cost": "0.65", "start": 0, "progress": 0 } ] } }
The API call get_job is used to retrieve details about one specific job.
This is used to track the current progress or the estimated finishing time.
In most use cases this call will not be required
The only difference to 'get_jobs' is that you specify the unique 'jobname'
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=get_job&jobname=_JOBNAME_&type=json&human_readable
curl -G https://scraping.services/api \
-d user=_USER@ACCOUNT_ \
-d credentials=_CREDENTIALS_ \
-d job_type=scrape_google_search \
-d task=get_job \
--data-urlencode jobname='G09/06/2019 15:52' \
-d type=json \
-d human_readable
"job": { "jobname": "G09/06/2019 15:52", "created": "2019-09-06 15:53:03", "job_type": "scrape_google_search", "max_results": 200, "estimated_keywords": 1, "priority": 1, "finished": "2019-09-06 15:53:53", "credit_cost": "0.65", "start": 1, "progress": 100, "total_keywords": 1, "total_results": 1, "pages": 1, "total_pages": 1, "estimated_hours_left": "0.00", "estimated_hours_left_hr": "0h 0m", "country": "Default", "language": "English" }
{ "exceptions": [ { "exception_text": "Job does not exist", "level": 10000, "unixtime": 1567786831, "error_id": 10013 } ], "answer": { "job": false } }
This call is used to retrieve all keywords of a particular job, they will be separated into "finished" and "unfinished"
get_keywords is used to retrieve keywords that have been left unfinished in an aborted job.
Available response types are: html, csv, json
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=get_keywords&jobname=_JOBNAME_&type=json&human_readable
{ "exceptions": [], "answer": { "keywords": { "jobname": "demo_job", "job_type": "scrape_google_search", "keywords_finished": ["keyword1","keyword2","keyword3"], "keywords_unfinished": ["keyword4","keyword5","keyword6"], } } }
To create jobs a JSON encoded array called 'json' needs to be added into the form data.
It is recommended to send all parameters as POST data instead of GET but both methods may be used.
The 'jobname' parameter is used to define a unique name for the new job, you may also use a timestamp.
The 'jobname' is used to access and modify the job or export results.
It is not recommended to use non default 'language' and 'country' parameters, Google makes scraping keywords in bulk much more difficult on non default languages.
default
Afghanistan
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Australia
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belgium
Belize
Benin
Bermuda
Bhutan
Bolivia
Bosnia and Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Canada
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Congo
Congo, the Democratic Republic of the
Cook Islands
Costa Rica
Cote D'ivoire
Croatia
Cuba
Cyprus
Czech Republic
Denmark
Djibouti
Dominica
Dominican Republic
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guinea
Guinea-Bissau
Guyana
Haiti
Heard Island and Mcdonald Islands
Holy See (Vatican City State)
Honduras
Hong Kong
Hungary
Iceland
India
Indonesia
Iran, Islamic Republic of
Iraq
Ireland
Israel
Italy
Jamaica
Japan
Jordan
Kazakhstan
Kenya
Kiribati
Korea, Democratic People's Republic of
Korea, Republic of
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Libyan Arab Jamahiriya
Liechtenstein
Lithuania
Luxembourg
Macao
Macedonia, the Former Yugosalv Republic of
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States of
Moldova, Republic of
Monaco
Mongolia
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Norway
Oman
Pakistan
Palau
Palestinian Territory, Occupied
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Helena
Saint Kitts and Nevis
Saint Lucia
Saint Pierre and Miquelon
Saint Vincent and the Grenadines
Samoa
San Marino
Sao Tome and Principe
Saudi Arabia
Senegal
Serbia and Montenegro
Seychelles
Sierra Leone
Singapore
Slovakia
Slovenia
Solomon Islands
Somalia
South Africa
South Georgia and the South Sandwich Islands
Spain
Sri Lanka
Sudan
Suriname
Svalbard and Jan Mayen
Swaziland
Sweden
Switzerland
Syrian Arab Republic
Taiwan, Province of China
Tajikistan
Tanzania, United Republic of
Thailand
Timor-Leste
Togo
Tokelau
Tonga
Trinidad and Tobago
Tunisia
Turkey
Turkmenistan
Turks and Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United Kingdom
United States
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Venezuela
Viet Nam
Virgin Islands, British
Virgin Islands, U.S.
Wallis and Futuna
Western Sahara
Yemen
Zambia
Afrikaans
Albanian
Amharic
Arabic
Armenian
Azerbaijani
Basque
Belarusian
Bengali
Bihari
Bosnian
Breton
Bulgarian
Cambodian
Catalan
Chinese (Simplified)
Chinese (Traditional)
Corsican
Croatian
Czech
Danish
Dutch
English
Esperanto
Estonian
Faroese
Filipino
Finnish
French
Frisian
Galician
Georgian
German
Greek
Guarani
Gujarati
Hausa
Hebrew
Hindi
Hungarian
Icelandic
Indonesian
Interlingua
Irish
Italian
Japanese
Javanese
Kannada
Kazakh
Kinyarwanda
Kirundi
Korean
Kurdish
Kyrgyz
Laothian
Latin
Latvian
Lingala
Lithuanian
Macedonian
Malagasy
Malay
Malayalam
Maltese
Maori
Marathi
Moldavian
Mongolian
Montenegrin
Nepali
Norwegian
Norwegian (Nynorsk)
Occitan
Oriya
Oromo
Pashto
Persian
Polish
Portuguese (Brazil)
Portuguese (Portugal)
Punjabi
Quechua
Romanian
Romansh
Russian
Scots Gaelic
Serbian
Serbo-Croatian
Sesotho
Shona
Sindhi
Sinhalese
Slovak
Slovenian
Somali
Spanish
Sundanese
Swahili
Swedish
Tajik
Tamil
Tatar
Telugu
Thai
Tigrinya
Tonga
Turkish
Turkmen
Twi
Uighur
Ukrainian
Urdu
Uzbek
Vietnamese
Welsh
Xhosa
Yiddish
Yoruba
Zulu
Feel free to contact customer support for more information regarding this topic.
The 'max_results' parameter defines the number of results to scrape, it's recommended to keep this parameter below 400 for best performance.
Possible values are 100, 200, .. 1000
The correct method for scraping large amounts of data on one keyword is to multiply the keyword with additional words.
The 'priority' parameter defines the resources our backend will spend on your job, in most cases priority 1 is recommended.
Priority 0 is a standby setting at significantly reduced Credit cost, jobs started with priority 0 are not guaranteed to finished or finish within any timeframe as they use only spare resources.
Priority higher than 1 is not recommended for jobs with less than 1000 keywords.
{
"job_type" : "scrape_google_search",
"language" : "English",
"country" : "default",
"max_results" : 100,
"jobname" : "demo_job",
"priority" : 1
}
curl -X POST https://scraping.services/api \
-d user=_USER@ACCOUNT_ \
-d credentials=_CREDENTIALS_ \
-d job_type=scrape_google_search \
-d task=create_job \
--data-urlencode json=' { "job_type" : "scrape_google_search", "language" : "English", "country" : "default", "max_results" : 100, "jobname" : "demo_job", "priority" : 1 }' \
-d type=json \
-d human_readable
{
"exceptions": [],
"answer": {
"job": {
"jobname": "demo_job",
"start": 0,
"job_type": "scrape_google_search"
}
}
}
The API call may be used to remove a job from your account, only jobs that have not been started can be removed.
Our support team can provide removal rights on finished jobs if required.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=remove_job&jobname=_JOBNAME_&type=json&human_readable
{
"exceptions": [],
"answer":
[
true
]
}
After create_job the job needs to be populated with keywords, modify_job is used to append or replace keywords within an existing job.
Similar to 'create_job', a JSON encoded object is appended including the keywords.
'keyword_operation' may be set to 'replace' or 'append'
A Job may be modified, extended or deleted until it has been started.
{
"job_type" : "scrape_google_search",
"language" : "English",
"country" : "default",
"max_results" : 200,
"jobname" : "demo_job",
"priority" : 2,
"keywords" :
[
"keyword 1",
"keyword 2",
"keyword 3"
],
"keyword_operation" : "replace"
}
curl -X POST https://scraping.services/api \
-d user=_USER@ACCOUNT_ \
-d credentials=_CREDENTIALS_ \
-d job_type=scrape_google_search \
-d task=modify_job \
--data-urlencode json=' { "job_type" : "scrape_google_search", "language" : "English", "country" : "default", "max_results" : 200, "jobname" : "demo_job", "priority" : 2, "keywords" : [ "keyword 1", "keyword 2", "keyword 3" ], "keyword_operation" : "replace" }' \
-d type=json \
-d human_readable
{
"exceptions": [],
"answer":
{
"job":
{
"jobname": "demo_job",
"max_results": 200,
"priority": 2,
"job_type": "scrape_google_search",
"estimated_keywords": 3,
"credit_cost": 0.9
}
}
}
Any created job that contains keywords may be started at any time.
Once started, the job is locked from further modification and will immediately deduct the credit cost from your wallet.
Such an action is non refundable and can not be interrupted anymore.
Once a job was started it can be monitored using the "get_job" API call to see how many keywords are already finished.
It is possible to download the keyword results using "get_results" while a job is running.
Scraped keyword results are stored for 14 days, once this timespan has passed the detail information will be removed.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=start_job&jobname=_JOBNAME_&type=json&human_readable
{
"exceptions": [],
"answer":
{
"job":
{
"jobname": "demo_job",
"start": "1",
"job_type": "scrape_google_search",
"credit_cost": "0.90"
}
}
}
This API call is deprecated since 2018, consider using the more flexible Export API
Using this call keyword results are downloaded in pages, each page contains 50 keyword results to prevent JSON parsing memory issues.
The estimated download size is between 1 and 10 MB per page.
The optional parameter 'result_per_keyword_quantity' is an method to limit the amount of results per keyword.
The optional parameter 'page_quantity' is an method to adjust the number of keyword results from 50 to a higher or lower value.
(50 keywords at 100+ results/keyword, 5000 kewords at 1 results/keyword). each 50 keywords are internally generating an API call (0.01 Credits charge)
The API response object contains an array "keyword_results" with up to 50 keyword results.
Each keyword_result contains information about the keyword as well as all organic results and advertisement results, again in form of two arrays.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=get_results&jobname=_JOBNAME_&type=json&human_readable
{
"results":
{
"exceptions": [],
"answer": {
"jobname": "demo_test_2",
"job_type": "scrape_google_search",
"keyword_results":
[
{
"count_organic": 101,
"count_creative": 0,
"language": "en",
"country": "Default",
"keyword": "A Chip on Your Shoulder",
"resultstats": {
"1": "799000",
"2": "650000"
},
"results_organic":
[
{
"url": "https:\/\/en.wikipedia.org\/wiki\/Chip_on_shoulder",
"title": "Chip on shoulder - Wikipedia",
"description": "To have a chip on one's shoulder refers to the act of holding a grudge or grievance that readily provokes disputation.",
"sitelinks": [],
"flags": {
"hacked": 0,
"harmful": 0
},
"page": 1
},
{
"url": "https:\/\/en.wikipedia.org\/wiki\/Chip_on_shoulder",
"title": "Chip on shoulder - Wikipedia",
"description": "",
"sitelinks": [],
"flags": {
"hacked": 0,
"harmful": 0
},
"page": 1
},
{
"url": "https:\/\/en.wikipedia.org\/wiki\/Chip_on_shoulder",
"title": "Chip on shoulder - Wikipedia",
"description": "To have a chip on one's shoulder refers to the act of holding a grudge or grievance that readily provokes disputation.",
"sitelinks": [],
"flags": {
"hacked": 0,
"harmful": 0
},
"page": 1
},
{
"url": "http:\/\/dictionary.cambridge.org\/us\/dictionary\/english\/have-a-chip-on-your-shoulder",
"title": "have a chip on your shoulder Definition in the Cambridge English ...",
"description": "have a chip on your shoulder definition, meaning, what is have a chip on your shoulder: to seem angry all the time because you think you have been treated ...",
"sitelinks": [],
"flags": {
"hacked": 0,
"harmful": 0
},
"page": 1
} /*,... additional results*/
],
"results_creative": []
} /*,... additional keywords*/
],
"page": 1,
"pages_expected": 1,
"pages_available": 1
}
}
}
Google Knowledge Graph results are part of the JSON Google result data.
"result_knowledge": {
"image_logo_src": "https:\/\/lh6.googleusercontent.com\/-2lJYGtfXKwQ\/AAAAAAAAAAI\/AAAAAAAB2HQ\/QSmIw0nQN_c\/s0-c-k-no-ns\/photo.jpg",
"header_text": "Barack Obama",
"header_sub_text": "44th U.S. President",
"times_open": [],
"description": "Barack Hussein Obama II is an American politician who served as the 44th President of the United States from 2009 to 2017. He was the first African American to serve as president, as well as the first born outside the contiguous United States. Wikipedia ",
"items_of_interest": [
"Barack Obama was the 44th and Donald Trump is the 45th President of the United States.",
"Hillary Clinton",
"Barack Obama and Michelle Obama have been married since 1992.",
"Ann Dunham was Barack Obama's mother.",
"Vladimir Putin"
],
"person_born_info": "August 4, 1961 (age 55 years), Kapiolani Medical Center for Women and Children, Honolulu, HI",
"person_education_info": "Harvard Law School (1988–1991), More"
}
"result_knowledge": {
"image_logo_src": "http:\/\/lh5.googleusercontent.com\/-SnTKdWZA8NI\/VWSg3sycQ6I\/AAAAAAAAACk\/37STRepzvJM5uuLfSs793X4Feo_CI1HJgCJkC\/s241-k-no\/",
"location_href": "https:\/\/www.google.com\/maps\/place\/MAG+Ferrari+In+Columbus,+Ohio\/@40.099688,-83.125521,15z\/data=!4m2!3m1!1s0x0:0x2ac116cff7ca4e58?sa=X&ved=0ahUKEwifqZeGy-XRAhUM7YMKHevBBQEQ_BII8wUwfA",
"location_img_src": "https:\/\/www.google.com\/maps\/vt\/data=RfCSdfNZ0LFPrHSm0ublXdzhdrDFhtmHhN1u-gM,oQRzStoSkLpDIbvDpw9QlO3VnR7-oeqWzIo7ofJbM-zACv_FTzt7ishssYouQq_AXBXkLbKUZbiHQ_7CzU4C1Dxr5Uzv4rs935AuzFkVvuhGcQ-EdBJhM_xRRlJ0QkCEHkn6P6Q",
"header_text": "Ferrari S.p.A.MAG Ferrari In Columbus, Ohio",
"header_sub_text": "Car manufacturer",
"website_href": "http:\/\/www.dublin.ferraridealers.com\/",
"address_text": "5035 Post Rd, Dublin, OH 43017",
"phone_text": "Phone:(614) 389-5557",
"times_open": [
{
"day": "Saturday",
"time": "9AM–6PM"
},
{
"day": "Sunday",
"time": "Closed"
},
{
"day": "Monday",
"time": "9AM–7PM"
},
{
"day": "Tuesday",
"time": "9AM–7PM"
},
{
"day": "Wednesday",
"time": "9AM–7PM"
},
{
"day": "Thursday",
"time": "9AM–7PM"
},
{
"day": "Friday",
"time": "9AM–6PM"
}
],
"description": "Ferrari S.p.A. is an Italian sports car manufacturer based in Maranello. Founded by Enzo Ferrari in 1939 as Auto Avio Costruzioni, the company built its first car in 1940. Wikipedia ",
"items_of_interest": [
"Lamborghini",
"Maserati",
"Tesla Motors",
"Bugatti Automobiles",
"Porsche"
],
"organization_founded_info": "September 13, 1939, Maranello, Italy"
}
Beta
For Google Seach we also provide Image results and Geolocation by city.
When using geolocation it is highly recommended to use the same location on each keyword within one job to prevent Google blocking your search.
These modes are activated by special parameters inside the keyword itself.
Refer to the FAQ for details:
https://scraping.services/?faq&topic=Keyword%20scraping&chapter=Country%20and%20City%20geolocation
https://scraping.services/?faq&topic=Keyword%20scraping&chapter=Google%20Image%20search