Google Search Results via SERP API pip Python Package

Overview

Google Search Results in Python

Package Build

This Python package is meant to scrape and parse search results from Google, Bing, Baidu, Yandex, Yahoo, Home depot, Ebay and more.. using SerpApi.

The following services are provided:

SerpApi provides a script builder to get you started quickly.

Installation

Python 3.7+

pip install google-search-results

Link to the python package page

Quick start

from serpapi import GoogleSearch
search = GoogleSearch({
    "q": "coffee", 
    "location": "Austin,Texas",
    "api_key": "<your secret api key>"
  })
result = search.get_dict()

This example runs a search about "coffee" using your secret api key.

The SerpApi service (backend)

  • searches on Google using the search: q = "coffee"
  • parses the messy HTML responses
  • return a standardizes JSON response The GoogleSearch class
  • Format the request
  • Execute GET http request against SerpApi service
  • Parse JSON response into a dictionary Et voila..

Alternatively, you can search:

  • Bing using BingSearch class
  • Baidu using BaiduSearch class
  • Yahoo using YahooSearch class
  • duckduckgo using DuckDuckGoSearch class
  • Ebay using EbaySearch class
  • Yandex using YandexSearch class
  • HomeDepot using HomeDepotSearch class
  • GoogleScholar using GoogleScholarSearch class
  • Youtube using YoutubeSearch class
  • Walmart using WalmartSearch
  • Apple App Store using AppleAppStoreSearch class
  • Naver using NaverSearch class

See the playground to generate your code.

Summary

Google Search API capability

Source code.

params = {
  "q": "coffee",
  "location": "Location Requested", 
  "device": "desktop|mobile|tablet",
  "hl": "Google UI Language",
  "gl": "Google Country",
  "safe": "Safe Search Flag",
  "num": "Number of Results",
  "start": "Pagination Offset",
  "api_key": "Your SERP API Key", 
  # To be match
  "tbm": "nws|isch|shop", 
  # To be search
  "tbs": "custom to be search criteria",
  # allow async request
  "async": "true|false",
  # output format
  "output": "json|html"
}

# define the search search
search = GoogleSearch(params)
# override an existing parameter
search.params_dict["location"] = "Portland"
# search format return as raw html
html_results = search.get_html()
# parse results
#  as python Dictionary
dict_results = search.get_dict()
#  as JSON using json package
json_results = search.get_json()
#  as dynamic Python object
object_result = search.get_object()

Link to the full documentation

see below for more hands on examples.

How to set SERP API key

You can get an API key here if you don't already have one: https://serpapi.com/users/sign_up

The SerpApi api_key can be set globally:

GoogleSearch.SERP_API_KEY = "Your Private Key"

The SerpApi api_key can be provided for each search:

query = GoogleSearch({"q": "coffee", "serp_api_key": "Your Private Key"})

Example by specification

We love true open source, continuous integration and Test Drive Development (TDD). We are using RSpec to test our infrastructure around the clock to achieve the best QoS (Quality Of Service).

The directory test/ includes specification/examples.

Set your api key.

export API_KEY="your secret key"

Run test

make test

Location API

from serpapi import GoogleSearch
search = GoogleSearch({})
location_list = search.get_location("Austin", 3)
print(location_list)

it prints the first 3 location matching Austin (Texas, Texas, Rochester)

[   {   'canonical_name': 'Austin,TX,Texas,United States',
        'country_code': 'US',
        'google_id': 200635,
        'google_parent_id': 21176,
        'gps': [-97.7430608, 30.267153],
        'id': '585069bdee19ad271e9bc072',
        'keys': ['austin', 'tx', 'texas', 'united', 'states'],
        'name': 'Austin, TX',
        'reach': 5560000,
        'target_type': 'DMA Region'},
        ...]

Search Archive API

The search result are stored in temporary cached. The previous search can be retrieve from the the cache for free.

from serpapi import GoogleSearch
search = GoogleSearch({"q": "Coffee", "location": "Austin,Texas"})
search_result = search.get_dictionary()
assert search_result.get("error") == None
search_id = search_result.get("search_metadata").get("id")
print(search_id)

Now let retrieve the previous search from the archive.

archived_search_result = GoogleSearch({}).get_search_archive(search_id, 'json')
print(archived_search_result.get("search_metadata").get("id"))

it prints the search result from the archive.

Account API

from serpapi import GoogleSearch
search = GoogleSearch({})
account = search.get_account()

it prints your account information.

Search Bing

from serpapi import BingSearch
search = BingSearch({"q": "Coffee", "location": "Austin,Texas"})
data = search.get_dict()

this code prints baidu search results for coffee as a Dictionary.

https://serpapi.com/bing-search-api

Search Baidu

from serpapi import BaiduSearch
search = BaiduSearch({"q": "Coffee"})
data = search.get_dict()

this code prints baidu search results for coffee as a Dictionary. https://serpapi.com/baidu-search-api

Search Yandex

from serpapi import YandexSearch
search = YandexSearch({"text": "Coffee"})
data = search.get_dict()

this code prints yandex search results for coffee as a Dictionary.

https://serpapi.com/yandex-search-api

Search Yahoo

from serpapi import YahooSearch
search = YahooSearch({"p": "Coffee"})
data = search.get_dict()

this code prints yahoo search results for coffee as a Dictionary.

https://serpapi.com/yahoo-search-api

Search Ebay

from serpapi import EbaySearch
search = EbaySearch({"_nkw": "Coffee"})
data = search.get_dict()

this code prints ebay search results for coffee as a Dictionary.

https://serpapi.com/ebay-search-api

Search Home depot

from serpapi import HomeDepotSearch
search = HomeDepotSearch({"q": "chair"})
data = search.get_dict()

this code prints home depot search results for chair as Dictionary.

https://serpapi.com/home-depot-search-api

Search Youtube

from serpapi import HomeDepotSearch
search = YoutubeSearch({"q": "chair"})
data = search.get_dict()

this code prints youtube search results for chair as Dictionary.

https://serpapi.com/youtube-search-api

Search Google Scholar

from serpapi import GoogleScholarSearch
search = GoogleScholarSearch({"q": "Coffee"})
data = search.get_dict()

this code prints Google Scholar search results.

Search Walmart

from serpapi import WalmartSearch
search = WalmartSearch({"query": "chair"})
data = search.get_dict()

this code prints Google Scholar search results.

Search Youtube

from serpapi import YoutubeSearch
search = YoutubeSearch({"search_query": "chair"})
data = search.get_dict()

this code prints Google Scholar search results.

Search Apple Store

from serpapi import AppleAppStoreSearch
search = AppleAppStoreSearch({"term": "Coffee"})
data = search.get_dict()

this code prints Google Scholar search results.

Search Naver

from serpapi import NaverSearch
search = NaverSearch({"query": "chair"})
data = search.get_dict()

this code prints Google Scholar search results.

Generic search with SerpApiClient

from serpapi import SerpApiClient
query = {"q": "Coffee", "location": "Austin,Texas", "engine": "google"}
search = SerpApiClient(query)
data = search.get_dict()

This class enables to interact with any search engine supported by SerpApi.com

Search Google Images

from serpapi import GoogleSearch
search = GoogleSearch({"q": "coffe", "tbm": "isch"})
for image_result in search.get_dict()['images_results']:
    link = image_result["original"]
    try:
        print("link: " + link)
        # wget.download(link, '.')
    except:
        pass

this code prints all the images links, and download image if you un-comment the line with wget (linux/osx tool to download image).

This tutorial covers more ground on this topic. https://github.com/serpapi/showcase-serpapi-tensorflow-keras-image-training

Search Google News

from serpapi import GoogleSearch
search = GoogleSearch({
    "q": "coffe",   # search search
    "tbm": "nws",  # news
    "tbs": "qdr:d", # last 24h
    "num": 10
})
for offset in [0,1,2]:
    search.params_dict["start"] = offset * 10
    data = search.get_dict()
    for news_result in data['news_results']:
        print(str(news_result['position'] + offset * 10) + " - " + news_result['title'])

this script prints the first 3 pages of the news title for the last 24h.

Search Google Shopping

from serpapi import GoogleSearch
search = GoogleSearch({
    "q": "coffe",   # search search
    "tbm": "shop",  # news
    "tbs": "p_ord:rv", # last 24h
    "num": 100
})
data = search.get_dict()
for shopping_result in data['shopping_results']:
    print(shopping_result['position']) + " - " + shopping_result['title'])

this script prints all the shopping results order by review order.

Google Search By Location

With SerpApi, we can build Google search from anywhere in the world. This code is looking for the best coffee shop per city.

from serpapi import GoogleSearch
for city in ["new york", "paris", "berlin"]:
  location = GoogleSearch({}).get_location(city, 1)[0]["canonical_name"]
  search = GoogleSearch({
      "q": "best coffee shop",   # search search
      "location": location,
      "num": 1,
      "start": 0
  })
  data = search.get_dict()
  top_result = data["organic_results"][0]["title"]

Batch Asynchronous Searches

We do offer two ways to boost your searches thanks to async parameter.

  • Blocking - async=false - it's more compute intensive because the search would need to hold many connections. (default)
  • Non-blocking - async=true - it's way to go for large amount of query submitted by batch (recommended)
# Operating system
import os

# regular expression library
import re

# safe queue (named Queue in python2)
from queue import Queue

# Time utility
import time

# SerpApi search
from serpapi import GoogleSearch

# store searches
search_queue = Queue()

# SerpApi search
search = GoogleSearch({
    "location": "Austin,Texas",
    "async": True,
    "api_key": os.getenv("API_KEY")
})

# loop through a list of companies
for company in ['amd', 'nvidia', 'intel']:
    print("execute async search: q = " + company)
    search.params_dict["q"] = company
    result = search.get_dict()
    if "error" in result:
        print("oops error: ", result["error"])
        continue
    print("add search to the queue where id: ", result['search_metadata'])
    # add search to the search_queue
    search_queue.put(result)

print("wait until all search statuses are cached or success")

# Create regular search
while not search_queue.empty():
    result = search_queue.get()
    search_id = result['search_metadata']['id']

    # retrieve search from the archive - blocker
    print(search_id + ": get search from archive")
    search_archived = search.get_search_archive(search_id)
    print(search_id + ": status = " +
          search_archived['search_metadata']['status'])

    # check status
    if re.search('Cached|Success',
                 search_archived['search_metadata']['status']):
        print(search_id + ": search done with q = " +
              search_archived['search_parameters']['q'])
    else:
        # requeue search_queue
        print(search_id + ": requeue search")
        search_queue.put(result)

        # wait 1s
        time.sleep(1)

print('all searches completed')

This code shows how to run searches asynchronously. The search parameters must have {async: True}. This indicates that the client shouldn't wait for the search to be completed. The current thread that executes the search is now non-blocking which allows to execute thousand of searches in seconds. The SerpApi backend will do the processing work. The actual search result is defer to a later call from the search archive using get_search_archive(search_id). In this example the non-blocking searches are persisted in a queue: search_queue. A loop through the search_queue allows to fetch individual search result. This process can be easily multithreaded to allow a large number of concurrent search requests. To keep thing simple, this example does only explore search result one at a time (single threaded).

See example.

Python object as a result

The search results can be automatically wrapped in dynamically generated Python object. This solution offers a more dynamic solution fully Oriented Object Programming approach over the regular Dictionary / JSON data structure.

from serpapi import GoogleSearch
search = GoogleSearch({"q": "Coffee", "location": "Austin,Texas"})
r = search.get_object()
assert type(r.organic_results), list
assert r.organic_results[0].title
assert r.search_metadata.id
assert r.search_metadata.google_url
assert r.search_parameters.q, "Coffee"
assert r.search_parameters.engine, "google"

Pagination using iterator

Let's collect links accross multiple search result pages.

# to get 2 pages
start = 0
end = 40
page_size = 10

# basic search parameters
parameter = {
  "q": "coca cola",
  "tbm": "nws",
  "api_key": os.getenv("API_KEY"),
  # optional pagination parameter
  #  the pagination method can take argument directly
  "start": start,
  "end": end,
  "num": page_size
}

# as proof of concept 
# urls collects
urls = []

# initialize a search
search = GoogleSearch(parameter)

# create a python generator using parameter
pages = search.pagination()
# or set custom parameter
pages = search.pagination(start, end, page_size)

# fetch one search result per iteration 
# using a basic python for loop 
# which invokes python iterator under the hood.
for page in pages:
  print(f"Current page: {page['serpapi_pagination']['current']}")
  for news_result in page["news_results"]:
    print(f"Title: {news_result['title']}\nLink: {news_result['link']}\n")
    urls.append(news_result['link'])
  
# check if the total number pages is as expected
# note: the exact number if variable depending on the search engine backend
if len(urls) == (end - start):
  print("all search results count match!")
if len(urls) == len(set(urls)):
  print("all search results are unique!")

Examples to fetch links with pagination: test file, online IDE

Error management

SerpAPI keeps error mangement very basic.

  • backend service error or search fail
  • client error

If it's a backend error, a simple message error is returned as string in the server response.

from serpapi import GoogleSearch
search = GoogleSearch({"q": "Coffee", "location": "Austin,Texas", "api_key": "<secret_key>"})
data = search.get_json()
assert data["error"] == None

In some case, there is more details availabel in the data object.

If it's client error, then a SerpApiClientException is raised.

Change log

2021-12-22 @ 2.4.1

  • add more search engine
    • youtube
    • walmart
    • apple_app_store
    • naver
  • raise SerpApiClientException instead of raw string in order to follow Python guideline 3.5+
  • add more unit error tests for serp_api_client

2021-07-26 @ 2.4.0

  • add page size support using num parameter
  • add youtube search engine

2021-06-05 @ 2.3.0

  • add pagination support

2021-04-28 @ 2.2.0

  • add get_response method to provide raw requests.Response object

2021-04-04 @ 2.1.0

  • Add home depot search engine
  • get_object() returns dynamic Python object

2020-10-26 @ 2.0.0

  • Reduce class name to Search
  • Add get_raw_json

2020-06-30 @ 1.8.3

  • simplify import
  • improve package for python 3.5+
  • add support for python 3.5 and 3.6

2020-03-25 @ 1.8

  • add support for Yandex, Yahoo, Ebay
  • clean-up test

2019-11-10 @ 1.7.1

  • increase engine parameter priority over engine value set in the class

2019-09-12 @ 1.7

  • Change namespace "from lib." instead: "from serpapi import GoogleSearch"
  • Support for Bing and Baidu

2019-06-25 @ 1.6

  • New search engine supported: Baidu and Bing

Conclusion

SerpApi supports all the major search engines. Google has the more advance support with all the major services available: Images, News, Shopping and more.. To enable a type of search, the field tbm (to be matched) must be set to:

  • isch: Google Images API.
  • nws: Google News API.
  • shop: Google Shopping API.
  • any other Google service should work out of the box.
  • (no tbm parameter): regular Google search.

The field tbs allows to customize the search even more.

The full documentation is available here.

Owner
SerpApi
API to get search engine results with ease.
SerpApi
GBSLocalLauncher - A script to compose ENV file for Local Compose

GBSLocalLauncher This is a script to compose ENV file for Local Compose. It crea

2 Jan 27, 2022
Código que verifica se o grafo é Hamiltoniano (Em Python)

Código para encontrar um ciclo de Hamilton em um dado grafo e a partir daí verificar se o grafo é hamiltoniano. Um ciclo hamiltoniano é um ciclo gerad

Hemili Beatriz 1 Jan 08, 2022
0-1 knapsack with an additional constraint of maximum number of items used

extended_knapsack 0-1 knapsack with an additional constraint of maximum number of items used. Usage pip3 install extended_knapsack from extended_knaps

Lion Ralfs 1 Jan 06, 2022
Quack-SMS-BOMBER - Quack Toolkit By IkigaiHack

Quack Toolkit By IkigaiHack About Quack Toolkit Quack Toolkit is a set of tools

Marcel 2 Aug 19, 2022
Twitter-bot - A Simple Twitter bot with python

twitterbot To use this bot, You will require API Key and Access Key. Signup at h

Bentil Shadrack 8 Nov 18, 2022
聚合空间测绘搜索(Fofa,Zoomeye,Quake,Shodan,Censys,BinaryEdge)

#Search-Tools Search-Tools集合比较常见的网络空间探测引擎 Fofa,Zoomeye,Quake,Shodan,Censys,BinaryEdge 简单说明 ICO搜索目前只有Fofa,Shodan,Quake支持 代理设置是防止在API请求过于频繁,或者在实战中,好多红队打

311 Dec 16, 2022
Fully Dockerized cryptocurrencies Trading Bot, based on Freqtrade engine. Multi instances.

Cryptocurrencies Trading Bot - Freqtrade Manager This automated Trading Bot is based on the amazing Freqtrade one. It allows you to manage many Freqtr

Cédric Dugat 47 Dec 06, 2022
Declarative assertions for AWS

AWSsert AWSsert is a Python library providing declarative assertions about AWS resources to your tests. Installation Use the package manager pip to in

19 Jan 04, 2022
GG Dorking is a tool to generate GitHub and Google dorking for pentesters and bug bounty hunters.

GG-Dorking GG Dorking is a python tool to generate GitHub and Google dorking links for pentesters and bug bounty hunters. It will help you to find imp

Eslam Akl 80 Nov 24, 2022
WILSON Cloud Respwnder is a Web Interaction Logger Sending Out Notifications with the ability to serve custom content in order to appropriately respond to client-issued requests.

WILSON Cloud Respwnder What is this? WILSON Cloud Respwnder is a Web Interaction Logger Sending Out Notifications (WILSON) with the ability to serve c

48 Oct 31, 2022
Cookies is a project inspired by python cookiecutter but used for terraform generation.

Introduction Cookies is a project inspired by python cookiecutter but used for terraform generation. How to run your terraform After you download your

Digger 6 Mar 14, 2022
Neko is An Anime themed advance Telegram group management bot.

NekoRobot A modular telegram Python bot running on python3 with an sqlalchemy, mongodb database. ╒═══「 Status 」 Maintained Support Group Included Free

Lovely Boy 22 Jan 05, 2023
A way to export your saved reddit posts to a Notion table.

reddit-saved-to-notion A way to export your saved reddit posts and comments to a Notion table.Uses notion-sdk-py and praw for interacting with Notion

19 Sep 12, 2022
A small package to markdownify Notion blocks.

markdownify-notion A small package to markdownify notion blocks. Installation Install this library using pip: $ pip install markdownify-notion Usage

Sergio Sánchez Zavala 2 Oct 29, 2022
universal messaging & notifications api

Pronounced "boat-shahft" What is botschaft? Botschaft is unified messaging & notifications appliance. Want to text yourself when a long-running task c

Tyler M. Kontra 25 Aug 16, 2022
Deepl - DeepL Free API For Python

DeepL DeepL Free API Notice Since I don't want to make my AuthKey public, if you

Vincent Young 4 Apr 11, 2022
Manage Proxmox KVM Virtual Machines via Slack bot.

proxmox-slack-bot Create KVM Virtual Machines on Proxmox, the easy way. Not much works works here yet... Setup dev environment Setup fully editable st

Plenus Pyramis 3 Mar 20, 2022
Block Telegram's new

Telegram Channel Blocker Bot Channel go away! This bot is used to delete and ban message sent by channel How this appears? The reason this appears ple

16 Feb 15, 2022
A Sublime Text plugin that displays inline images for single-line comments formatted like `// ![](example.png)`.

Inline Images Sometimes ASCII art is not enough. Sometimes an image says more than a thousand words. This Sublime Text plugin can display images inlin

Andreas Haferburg 8 Jul 01, 2022