Our Google scraping API is based on HTTP(s) POST or GET calls to our API server. All API requests are self contained and do not require to keep a session or cookies.
Each API call costs 0.01 Credits to prevent overusage (about $1 per 1000 calls) in addition to possible additional charges.
Our Google and Bing scraping APIs are very similar, you can use the same code for both types of scraping.
Not a developer?
We offer web based tools to scrape Google and export results without any API integration requirements.
Scrape google search by web-tool
We use cURL and URL examples during most of the documentation.
We also provide PHP and Python source code examples to help you get started quickly.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_
# This base URL contains the credentials of your account
URL: https://scraping.services/api
User: _USER@ACCOUNT_
Credentials: _CREDENTIALS_
Let's start by taking a look at a simple request: license_info
The URI is "https://scraping.services/api" with a set of essential GET parameters:
Some API calls require more complicated structures to be passed, for example large keyword arrays.
In these cases the additional variables are supplied as JSON object encoded POST parameter.
It is up to the developer when to use the "json" parameter or when to supply the parameters as GET through the URI.
To see the 'json' parameter being used take a look at modify_job or the source code examples.
The API response always contains two json objects: 'exceptions' and 'answer'.
The API response can be converted into a native object or array by almost any programming language.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=info&task=license_info&type=json&human_readable
curl -G https://scraping.services/api \
-d user=_USER@ACCOUNT_ \
-d credentials=_CREDENTIALS_ \
-d job_type=info \
-d task=license_info \
-d type=json \
-d human_readable
{
"exceptions": [],
"answer":
{
"license":
{
"id": 1,
"name": "1xsmall",
"monthly_cost": "14.00",
"monthly_credits": "100",
"time_left": "386:31:46",
"time_left_grace": "482:31:46"
}
}
}
{
"exceptions": [
{
"exception_text": "Wrong username or password",
"level": 10000,
"unixtime": 1567743178,
"error_id": 109
}
],
"answer": false
}
When scraping Google for a collection of keywords you first create a 'job' with a limit of 200 unfinished jobs being held in your account.
The job contains all settings required to start the scraping run:
"job": {
"jobname": "G09/06/2019 15:52",
"created": "2019-09-06 15:53:03",
"job_type": "scrape_google_search",
"max_results": 200,
"estimated_keywords": 1,
"priority": 1,
"finished": "2019-09-06 15:53:53",
"credit_cost": "0.65",
"start": 1,
"progress": 100,
"total_keywords": 1,
"total_results": 1,
"pages": 1,
"total_pages": 1,
"estimated_hours_left": "0.00",
"estimated_hours_left_hr": "0h 0m",
"country": "Default",
"language": "English"
}
This API call will return up to 50 latest jobs. All details about the jobs are included except the keywords,result data and current scraping progress.
To track the progress of a specific job in detail you need to use 'get_job' instead.
The right sided response example shows an account containing two jobs. The latest job was started and is finished, the second job was never started.
The 'progress' variable using get_jobs will only return 0(%) for unfinished or unstarted jobs or 100(%) for finished jobs.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=get_jobs&type=json&human_readable
{
"exceptions": [],
"answer": {
"jobs": [
{
"jobname": "G09/06/2019 15:52",
"created": "2019-09-06 15:53:03",
"job_type": "scrape_google_search",
"max_results": 200,
"estimated_keywords": 1,
"priority": 1,
"finished": "2019-09-06 15:53:53",
"credit_cost": "0.65",
"start": 1,
"progress": 100
},
{
"jobname": "G04/21/2019 15:09",
"created": "2019-04-21 15:09:26",
"job_type": "scrape_google_search",
"max_results": 1000,
"estimated_keywords": 1,
"priority": 1,
"finished": "0000-00-00 00:00:00",
"credit_cost": "0.65",
"start": 0,
"progress": 0
}
]
}
}
The API call get_job is used to retrieve details about one specific job.
This is used to track the current progress or the estimated finishing time.
In most use cases this call will not be required
The only difference to 'get_jobs' is that you specify the unique 'jobname'
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=get_job&jobname=_JOBNAME_&type=json&human_readable
curl -G https://scraping.services/api \
-d user=_USER@ACCOUNT_ \
-d credentials=_CREDENTIALS_ \
-d job_type=scrape_google_search \
-d task=get_job \
--data-urlencode jobname='G09/06/2019 15:52' \
-d type=json \
-d human_readable
"job": {
"jobname": "G09/06/2019 15:52",
"created": "2019-09-06 15:53:03",
"job_type": "scrape_google_search",
"max_results": 200,
"estimated_keywords": 1,
"priority": 1,
"finished": "2019-09-06 15:53:53",
"credit_cost": "0.65",
"start": 1,
"progress": 100,
"total_keywords": 1,
"total_results": 1,
"pages": 1,
"total_pages": 1,
"estimated_hours_left": "0.00",
"estimated_hours_left_hr": "0h 0m",
"country": "Default",
"language": "English"
}
{
"exceptions": [
{
"exception_text": "Job does not exist",
"level": 10000,
"unixtime": 1567786831,
"error_id": 10013
}
],
"answer": {
"job": false
}
}
This call is used to retrieve all keywords of a particular job, they will be separated into "finished" and "unfinished"
get_keywords is used to retrieve keywords that have been left unfinished in an aborted job.
Available response types are: html, csv, json
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=get_keywords&jobname=_JOBNAME_&type=json&human_readable
{
"exceptions": [],
"answer":
{
"keywords":
{
"jobname": "demo_job",
"job_type": "scrape_google_search",
"keywords_finished": ["keyword1","keyword2","keyword3"],
"keywords_unfinished": ["keyword4","keyword5","keyword6"],
}
}
}
To create jobs a JSON encoded array called 'json' needs to be added into the form data.
It is recommended to send all parameters as POST data instead of GET but both methods may be used.
The 'jobname' parameter is used to define a unique name for the new job, you may also use a timestamp.
The 'jobname' is used to access and modify the job or export results.
It is not recommended to use non default 'language' and 'country' parameters, Google makes scraping keywords in bulk much more difficult on non default languages.
defaultAfghanistanAlbaniaAlgeriaAmerican SamoaAndorraAngolaAnguillaAntarcticaAntigua and BarbudaArgentinaArmeniaArubaAustraliaAustriaAzerbaijanBahamasBahrainBangladeshBarbadosBelarusBelgiumBelizeBeninBermudaBhutanBoliviaBosnia and HerzegovinaBotswanaBouvet IslandBrazilBritish Indian Ocean TerritoryBrunei DarussalamBulgariaBurkina FasoBurundiCambodiaCameroonCanadaCape VerdeCayman IslandsCentral African RepublicChadChileChinaChristmas IslandCocos (Keeling) IslandsColombiaComorosCongoCongo, the Democratic Republic of theCook IslandsCosta RicaCote D'ivoireCroatiaCubaCyprusCzech RepublicDenmarkDjiboutiDominicaDominican RepublicEcuadorEgyptEl SalvadorEquatorial GuineaEritreaEstoniaEthiopiaFalkland Islands (Malvinas)Faroe IslandsFijiFinlandFranceFrench GuianaFrench PolynesiaFrench Southern TerritoriesGabonGambiaGeorgiaGermanyGhanaGibraltarGreeceGreenlandGrenadaGuadeloupeGuamGuatemalaGuineaGuinea-BissauGuyanaHaitiHeard Island and Mcdonald IslandsHoly See (Vatican City State)HondurasHong KongHungaryIcelandIndiaIndonesiaIran, Islamic Republic ofIraqIrelandIsraelItalyJamaicaJapanJordanKazakhstanKenyaKiribatiKorea, Democratic People's Republic ofKorea, Republic ofKuwaitKyrgyzstanLao People's Democratic RepublicLatviaLebanonLesothoLiberiaLibyan Arab JamahiriyaLiechtensteinLithuaniaLuxembourgMacaoMacedonia, the Former Yugosalv Republic ofMadagascarMalawiMalaysiaMaldivesMaliMaltaMarshall IslandsMartiniqueMauritaniaMauritiusMayotteMexicoMicronesia, Federated States ofMoldova, Republic ofMonacoMongoliaMontserratMoroccoMozambiqueMyanmarNamibiaNauruNepalNetherlandsNetherlands AntillesNew CaledoniaNew ZealandNicaraguaNigerNigeriaNiueNorfolk IslandNorthern Mariana IslandsNorwayOmanPakistanPalauPalestinian Territory, OccupiedPanamaPapua New GuineaParaguayPeruPhilippinesPitcairnPolandPortugalPuerto RicoQatarReunionRomaniaRussian FederationRwandaSaint HelenaSaint Kitts and NevisSaint LuciaSaint Pierre and MiquelonSaint Vincent and the GrenadinesSamoaSan MarinoSao Tome and PrincipeSaudi ArabiaSenegalSerbia and MontenegroSeychellesSierra LeoneSingaporeSlovakiaSloveniaSolomon IslandsSomaliaSouth AfricaSouth Georgia and the South Sandwich IslandsSpainSri LankaSudanSurinameSvalbard and Jan MayenSwazilandSwedenSwitzerlandSyrian Arab RepublicTaiwan, Province of ChinaTajikistanTanzania, United Republic ofThailandTimor-LesteTogoTokelauTongaTrinidad and TobagoTunisiaTurkeyTurkmenistanTurks and Caicos IslandsTuvaluUgandaUkraineUnited Arab EmiratesUnited KingdomUnited StatesUnited States Minor Outlying IslandsUruguayUzbekistanVanuatuVenezuelaViet NamVirgin Islands, BritishVirgin Islands, U.S.Wallis and FutunaWestern SaharaYemenZambia
AfrikaansAlbanianAmharicArabicArmenianAzerbaijaniBasqueBelarusianBengaliBihariBosnianBretonBulgarianCambodianCatalanChinese (Simplified)Chinese (Traditional)CorsicanCroatianCzechDanishDutchEnglishEsperantoEstonianFaroeseFilipinoFinnishFrenchFrisianGalicianGeorgianGermanGreekGuaraniGujaratiHausaHebrewHindiHungarianIcelandicIndonesianInterlinguaIrishItalianJapaneseJavaneseKannadaKazakhKinyarwandaKirundiKoreanKurdishKyrgyzLaothianLatinLatvianLingalaLithuanianMacedonianMalagasyMalayMalayalamMalteseMaoriMarathiMoldavianMongolianMontenegrinNepaliNorwegianNorwegian (Nynorsk)OccitanOriyaOromoPashtoPersianPolishPortuguese (Brazil)Portuguese (Portugal)PunjabiQuechuaRomanianRomanshRussianScots GaelicSerbianSerbo-CroatianSesothoShonaSindhiSinhaleseSlovakSlovenianSomaliSpanishSundaneseSwahiliSwedishTajikTamilTatarTeluguThaiTigrinyaTongaTurkishTurkmenTwiUighurUkrainianUrduUzbekVietnameseWelshXhosaYiddishYorubaZulu
Feel free to contact customer support for more information regarding this topic.
The 'max_results' parameter defines the number of results to scrape, it's recommended to keep this parameter below 400 for best performance.
Possible values are 100, 200, .. 1000
The correct method for scraping large amounts of data on one keyword is to multiply the keyword with additional words.
The 'priority' parameter defines the resources our backend will spend on your job, in most cases priority 1 is recommended.
Priority 0 is a standby setting at significantly reduced Credit cost, jobs started with priority 0 are not guaranteed to finished or finish within any timeframe as they use only spare resources.
Priority higher than 1 is not recommended for jobs with less than 1000 keywords.
{
"job_type" : "scrape_google_search",
"language" : "English",
"country" : "default",
"max_results" : 100,
"jobname" : "demo_job",
"priority" : 1
}
curl -X POST https://scraping.services/api \
-d user=_USER@ACCOUNT_ \
-d credentials=_CREDENTIALS_ \
-d job_type=scrape_google_search \
-d task=create_job \
--data-urlencode json='
{
"job_type" : "scrape_google_search",
"language" : "English",
"country" : "default",
"max_results" : 100,
"jobname" : "demo_job",
"priority" : 1
}' \
-d type=json \
-d human_readable
{
"exceptions": [],
"answer": {
"job": {
"jobname": "demo_job",
"start": 0,
"job_type": "scrape_google_search"
}
}
}
The API call may be used to remove a job from your account, only jobs that have not been started can be removed.
Our support team can provide removal rights on finished jobs if required.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=remove_job&jobname=_JOBNAME_&type=json&human_readable
{
"exceptions": [],
"answer":
[
true
]
}
After create_job the job needs to be populated with keywords, modify_job is used to append or replace keywords within an existing job.
Similar to 'create_job', a JSON encoded object is appended including the keywords.
'keyword_operation' may be set to 'replace' or 'append'
A Job may be modified, extended or deleted until it has been started.
{
"job_type" : "scrape_google_search",
"language" : "English",
"country" : "default",
"max_results" : 200,
"jobname" : "demo_job",
"priority" : 2,
"keywords" :
[
"keyword 1",
"keyword 2",
"keyword 3"
],
"keyword_operation" : "replace"
}
curl -X POST https://scraping.services/api \
-d user=_USER@ACCOUNT_ \
-d credentials=_CREDENTIALS_ \
-d job_type=scrape_google_search \
-d task=modify_job \
--data-urlencode json='
{
"job_type" : "scrape_google_search",
"language" : "English",
"country" : "default",
"max_results" : 200,
"jobname" : "demo_job",
"priority" : 2,
"keywords" :
[
"keyword 1",
"keyword 2",
"keyword 3"
],
"keyword_operation" : "replace"
}' \
-d type=json \
-d human_readable
{
"exceptions": [],
"answer":
{
"job":
{
"jobname": "demo_job",
"max_results": 200,
"priority": 2,
"job_type": "scrape_google_search",
"estimated_keywords": 3,
"credit_cost": 0.9
}
}
}
Any created job that contains keywords may be started at any time.
Once started, the job is locked from further modification and will immediately deduct the credit cost from your wallet.
Such an action is non refundable and can not be interrupted anymore.
Once a job was started it can be monitored using the "get_job" API call to see how many keywords are already finished.
It is possible to download the keyword results using "get_results" while a job is running.
Scraped keyword results are stored for 14 days, once this timespan has passed the detail information will be removed.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=start_job&jobname=_JOBNAME_&type=json&human_readable
{
"exceptions": [],
"answer":
{
"job":
{
"jobname": "demo_job",
"start": "1",
"job_type": "scrape_google_search",
"credit_cost": "0.90"
}
}
}
This API call is deprecated since 2018, consider using the more flexible Export API
Using this call keyword results are downloaded in pages, each page contains 50 keyword results to prevent JSON parsing memory issues.
The estimated download size is between 1 and 10 MB per page.
The optional parameter 'result_per_keyword_quantity' is an method to limit the amount of results per keyword.
The optional parameter 'page_quantity' is an method to adjust the number of keyword results from 50 to a higher or lower value.
(50 keywords at 100+ results/keyword, 5000 kewords at 1 results/keyword). each 50 keywords are internally generating an API call (0.01 Credits charge)
The API response object contains an array "keyword_results" with up to 50 keyword results.
Each keyword_result contains information about the keyword as well as all organic results and advertisement results, again in form of two arrays.
https://scraping.services/api?user=_USER@ACCOUNT_&credentials=_CREDENTIALS_&job_type=scrape_google_search&task=get_results&jobname=_JOBNAME_&type=json&human_readable
{
"results":
{
"exceptions": [],
"answer": {
"jobname": "demo_test_2",
"job_type": "scrape_google_search",
"keyword_results":
[
{
"count_organic": 101,
"count_creative": 0,
"language": "en",
"country": "Default",
"keyword": "A Chip on Your Shoulder",
"resultstats": {
"1": "799000",
"2": "650000"
},
"results_organic":
[
{
"url": "https:\/\/en.wikipedia.org\/wiki\/Chip_on_shoulder",
"title": "Chip on shoulder - Wikipedia",
"description": "To have a chip on one's shoulder refers to the act of holding a grudge or grievance that readily provokes disputation.",
"sitelinks": [],
"flags": {
"hacked": 0,
"harmful": 0
},
"page": 1
},
{
"url": "https:\/\/en.wikipedia.org\/wiki\/Chip_on_shoulder",
"title": "Chip on shoulder - Wikipedia",
"description": "",
"sitelinks": [],
"flags": {
"hacked": 0,
"harmful": 0
},
"page": 1
},
{
"url": "https:\/\/en.wikipedia.org\/wiki\/Chip_on_shoulder",
"title": "Chip on shoulder - Wikipedia",
"description": "To have a chip on one's shoulder refers to the act of holding a grudge or grievance that readily provokes disputation.",
"sitelinks": [],
"flags": {
"hacked": 0,
"harmful": 0
},
"page": 1
},
{
"url": "http:\/\/dictionary.cambridge.org\/us\/dictionary\/english\/have-a-chip-on-your-shoulder",
"title": "have a chip on your shoulder Definition in the Cambridge English ...",
"description": "have a chip on your shoulder definition, meaning, what is have a chip on your shoulder: to seem angry all the time because you think you have been treated ...",
"sitelinks": [],
"flags": {
"hacked": 0,
"harmful": 0
},
"page": 1
} /*,... additional results*/
],
"results_creative": []
} /*,... additional keywords*/
],
"page": 1,
"pages_expected": 1,
"pages_available": 1
}
}
}
Google Knowledge Graph results are part of the JSON Google result data.
"result_knowledge": {
"image_logo_src": "https:\/\/lh6.googleusercontent.com\/-2lJYGtfXKwQ\/AAAAAAAAAAI\/AAAAAAAB2HQ\/QSmIw0nQN_c\/s0-c-k-no-ns\/photo.jpg",
"header_text": "Barack Obama",
"header_sub_text": "44th U.S. President",
"times_open": [],
"description": "Barack Hussein Obama II is an American politician who served as the 44th President of the United States from 2009 to 2017. He was the first African American to serve as president, as well as the first born outside the contiguous United States. Wikipedia ",
"items_of_interest": [
"Barack Obama was the 44th and Donald Trump is the 45th President of the United States.",
"Hillary Clinton",
"Barack Obama and Michelle Obama have been married since 1992.",
"Ann Dunham was Barack Obama's mother.",
"Vladimir Putin"
],
"person_born_info": "August 4, 1961 (age 55 years), Kapiolani Medical Center for Women and Children, Honolulu, HI",
"person_education_info": "Harvard Law School (1988–1991), More"
}
"result_knowledge": {
"image_logo_src": "http:\/\/lh5.googleusercontent.com\/-SnTKdWZA8NI\/VWSg3sycQ6I\/AAAAAAAAACk\/37STRepzvJM5uuLfSs793X4Feo_CI1HJgCJkC\/s241-k-no\/",
"location_href": "https:\/\/www.google.com\/maps\/place\/MAG+Ferrari+In+Columbus,+Ohio\/@40.099688,-83.125521,15z\/data=!4m2!3m1!1s0x0:0x2ac116cff7ca4e58?sa=X&ved=0ahUKEwifqZeGy-XRAhUM7YMKHevBBQEQ_BII8wUwfA",
"location_img_src": "https:\/\/www.google.com\/maps\/vt\/data=RfCSdfNZ0LFPrHSm0ublXdzhdrDFhtmHhN1u-gM,oQRzStoSkLpDIbvDpw9QlO3VnR7-oeqWzIo7ofJbM-zACv_FTzt7ishssYouQq_AXBXkLbKUZbiHQ_7CzU4C1Dxr5Uzv4rs935AuzFkVvuhGcQ-EdBJhM_xRRlJ0QkCEHkn6P6Q",
"header_text": "Ferrari S.p.A.MAG Ferrari In Columbus, Ohio",
"header_sub_text": "Car manufacturer",
"website_href": "http:\/\/www.dublin.ferraridealers.com\/",
"address_text": "5035 Post Rd, Dublin, OH 43017",
"phone_text": "Phone:(614) 389-5557",
"times_open": [
{
"day": "Saturday",
"time": "9AM–6PM"
},
{
"day": "Sunday",
"time": "Closed"
},
{
"day": "Monday",
"time": "9AM–7PM"
},
{
"day": "Tuesday",
"time": "9AM–7PM"
},
{
"day": "Wednesday",
"time": "9AM–7PM"
},
{
"day": "Thursday",
"time": "9AM–7PM"
},
{
"day": "Friday",
"time": "9AM–6PM"
}
],
"description": "Ferrari S.p.A. is an Italian sports car manufacturer based in Maranello. Founded by Enzo Ferrari in 1939 as Auto Avio Costruzioni, the company built its first car in 1940. Wikipedia ",
"items_of_interest": [
"Lamborghini",
"Maserati",
"Tesla Motors",
"Bugatti Automobiles",
"Porsche"
],
"organization_founded_info": "September 13, 1939, Maranello, Italy"
}
Beta
For Google Seach we also provide Image results and Geolocation by city.
When using geolocation it is highly recommended to use the same location on each keyword within one job to prevent Google blocking your search.
These modes are activated by special parameters inside the keyword itself.
Refer to the FAQ for details:
https://scraping.services/?faq&topic=Keyword%20scraping&chapter=Country%20and%20City%20geolocation
https://scraping.services/?faq&topic=Keyword%20scraping&chapter=Google%20Image%20search