How to Scrape Yahoo Finance - 2024 Guide (2024)

TL:DR: Let’s learn how to build a Yahoo Finance scraper for extracting stock data to perform financial analysis for trading and investing.

This tutorial will cover:

• Why scrape financial data from the Web?
• Finance scraping libraries and tools
• Scraping stock data from Yahoo Finance with Selenium

Why Scrape Financial Data From the Web?

Scraping finance data from the Web offers valuable insights that come in handy in various scenarios, including:

• Automated Trading: By gathering real-time or historical market data, such as stock prices and volume, developers can build automated trading strategies.
• Technical Analysis: Historical market data and indicators are extremely important for technical analysts. These allow them to identify patterns and trends, assisting their investment decision-making.
• Financial Modeling: Researchers and analysts can gather relevant data like financial statements and economic indicators to build complex models for evaluating company performance, forecasting earnings, and assessing investment opportunities.
• Market Research: Financial data provide a great deal of information about stocks, market indices, and commodities. Analyzing this data helps researchers understand market trends, sentiment, and industry health to make informed investment decisions.

When it comes to monitoring the market, Yahoo Finance is one of the popular finance websites. It provides a wide range of information and tools to investors and traders, such as real-time and historical data on stocks, bonds, mutual funds, commodities, currencies, and market indices. Plus, it offers news articles, financial statements, analyst estimates, charts, and other valuable resources.

By scraping Yahoo Finance, you can access a wealth of information to support your financial analysis, research, and decision-making processes.

Finance Scraping Libraries and Tools

Python is considered one of the best languages for scraping thanks to its syntax, ease of use, and rich ecosystem of libraries. Check out our guide on web scraping with Python.

To choose the right scraping libraries out of the many available, explore Yahoo Finance in your browser. You will notice that most of the data on the site gets updated in real-time or changes after an interaction. This means that the site heavily on AJAX to load and update data dynamically without requiring page reloads. In other words, you need a tool that is able to run JavaScript.

Selenium makes it possible to scrape dynamic websites in Python. It renders site in web browsers, programmatically performing operations on them even if they use JavaScript for rendering or retrieving data.

Thanks to Selenium, you will be able to scrape the target site with Python. Let’s learn how!

Scraping Stock Data From Yahoo Finance With Selenium

Follow this step-by-step tutorial and see how to build a Yahoo Finance web scraping Python script.

Step 1: Setup

Before diving into finance scraping, make sure to meet these prerequisites:

• Python 3+ installed on your machine: Download the installer, double-click on it, and follows the installation wizard.
• A Python IDE of your choice: PyCharm Community Edition or Visual Studio Code with the Python extension will do.

Next, use the commands below to set up a Python project with a virtual environment:

mkdir yahoo-finance-scrapercd yahoo-finance-scraperpython -m venv env

These will initialize the yahoo-finance-scraper project folder. Inside it, add a scraper.py file as below:

print('Hello, World!')

You will add the logic to scrape Yahoo Finance here. Right now, it is a sample script that only prints “Hello, World!”

Launch it to verify that it works with:

python scraper.py

In the terminal, you should see:

Hello, World!

Great, you now have a Python project for your finance scraper. It only remains to add the project’s dependencies. Install Selenium and the Webdriver Manager with the following terminal command:

pip install selenium webdriver-manager

This might take a while, so be patient.

webdriver-manager is not strictly required. However, it is highly recommended as it makes managing web drivers in Selenium way easier. Thanks to it, you do not have to manually download, configure, and import the web driver.

Update scraper.py

from selenium import webdriverfrom selenium.webdriver.chrome.service import Service as ChromeServicefrom webdriver_manager.chrome import ChromeDriverManager# initialize a web driver instance to control a Chrome windowdriver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))# scraping logic...# close the browser and free up the resourcesdriver.quit()

This script simply instantiates an instance of ChromeWebDriver. You will use that soon to implement the data extraction logic.

Step 2: Connect to the target web page

This is what the URL of a Yahoo Finance stock page looks like:

https://finance.yahoo.com/quote/AMZN

As you can see, it is a dynamic URL that changes based on the ticker symbol. If you are not familiar with the concept, that is a string abbreviation used to uniquely identify shares traded in the stock market. For example, “AMZN” is the ticker symbol of the Amazon stock.

Let’s modify the script to make read the ticker from a command line argument.

import sys# if there are no CLI parametersif len(sys.argv) <= 1: print('Ticker symbol CLI argument missing!') sys.exit(2)# read the ticker from the CLI argumentticker_symbol = sys.argv[1]# build the URL of the target pageurl = f'https://finance.yahoo.com/quote/{ticker_symbol}'

sys is a Python standard library that provides access to the command-line arguments. Do not forget that the argument with index 0 is the name of your script. Thus, you have to target the argument with index 1.

After reading the ticker from the CLI, it is used in an f-string to produce the target URL to scrape.

For example, assume to launch the scraper with the Tesla ticker “TSLA:”

python scraper.py TSLA

url will contain:

https://finance.yahoo.com/quote/TSLA

If you forget the ticker symbol in the CLI, the program will fail with the error below:

Ticker symbol CLI argument missing!

Before opening any page in Selenium, it is recommended to set the window size to ensure that every element is visible:

driver.set_window_size(1920, 1080)

You can now use Selenium to connect to the target page with:

driver.get(url)

The get() function instructs the browser to visit the desired page.

This is what your Yahoo Finance scraping script looks like so far:

from selenium import webdriverfrom selenium.webdriver.chrome.service import Service as ChromeServicefrom webdriver_manager.chrome import ChromeDriverManagerimport sys# if there are no CLI parametersif len(sys.argv) <= 1: print('Ticker symbol CLI argument missing!') sys.exit(2)# read the ticker from the CLI argumentticker_symbol = sys.argv[1]# build the URL of the target pageurl = f'https://finance.yahoo.com/quote/{ticker_symbol}'# initialize a web driver instance to control a Chrome windowdriver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))# set up the window size of the controlled browserdriver.set_window_size(1920, 1080)# visit the target pagedriver.get(url)# scraping logic...# close the browser and free up the resourcesdriver.quit()

If you launch it, it will open this window for a fraction of a second before terminating:

Starting the browser with the UI is useful for debugging by monitoring what the scraper is doing on the web page. At the same time, it takes a lot of resources. To avoid that, configure Chrome to run in headless mode with:

from selenium.webdriver.chrome.options import Options# ...options = Options()options.add_argument('--headless=new')driver = webdriver.Chrome( service=ChromeService(ChromeDriverManager().install()), options=options)

The controlled browser will now be launched behind the scene, with no UI.

Step 3: Inspect the target page

If you want to structure an effective data mining strategy, you must first analyze the target web page. Open your browser and visit the Yahoo stock page.

If you are based in Europe, you will first see a modal asking you to accept the cookies:

To close it and keep visiting the desired page, you must click “Accept all” or “Reject all.” Right-click on the first button and select the “Inspect” option to open the DevTools of your browser:

Here, you will notice that you can select that button with the following CSS selector:

.consent-overlay .accept-all

Use these lines of ice to deal with the consent modal in Selenium:

try: # wait up to 3 seconds for the consent modal to show up consent_overlay = WebDriverWait(driver, 3).until( EC.presence_of_element_located((By.CSS_SELECTOR, '.consent-overlay'))) # click the "Accept all" button accept_all_button = consent_overlay.find_element(By.CSS_SELECTOR, '.accept-all') accept_all_button.click()except TimeoutException: print('Cookie consent overlay missing')

WebDriverWait allows you to wait for an expected condition to occur on the page. If nothing happens in the specified timeout, it raises a TimeoutException. Since the cookie overlay shows up only when your exit IP is European, you can handle the exception with a try-catch instruction. This way, the script will keep running when the consent modal is not present.

To make the script works, you will need to add the following imports:

from selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.common.by import Byfrom selenium.common import TimeoutException

Now, keep inspecting the target site in the DevTools and familiarize yourself with its DOM structure.

Step 4: Extract the stock data

As you should have noticed in the previous step, some of the most interesting information is in this section:

Inspect the HTML price indicator element:

Note that CSS classes are not useful for defining proper selectors in Yahoo Finance. They seem to follow a special syntax for a styling framework. Instead, focus on the other HTML attributes. For example, you can get the stock price with the CSS selector below:

[data-symbol="TSLA"][data-field="regularMarketPrice"]

Following a similar approach, extract all stock data from the price indicators with:

regular_market_price = driver\ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="regularMarketPrice"]')\ .textregular_market_change = driver\ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="regularMarketChange"]')\ .textregular_market_change_percent = driver\ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="regularMarketChangePercent"]')\ .text\ .replace('(', '').replace(')', '') post_market_price = driver\ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="postMarketPrice"]')\ .textpost_market_change = driver\ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="postMarketChange"]')\ .textpost_market_change_percent = driver\ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="postMarketChangePercent"]')\ .text\ .replace('(', '').replace(')', '')

After selecting an HTML element through the specific CSS selector strategy, you can extract its content with the text field. Since the percent fields involve round parentheses, these are removed with replace().

Add them to a stock dictionary and print it to verify that the process of scraping financial data works as expected:

# initialize the dictionarystock = {}# stock price scraping logic omitted for brevity...# add the scraped data to the dictionarystock['regular_market_price'] = regular_market_pricestock['regular_market_change'] = regular_market_changestock['regular_market_change_percent'] = regular_market_change_percentstock['post_market_price'] = post_market_pricestock['post_market_change'] = post_market_changestock['post_market_change_percent'] = post_market_change_percentprint(stock)

Run the script on the security you want to scrape and you should see something like:

{'regular_market_price': '193.17', 'regular_market_change': '+8.70', 'regular_market_change_percent': '+4.72%', 'post_market_price': '194.00', 'post_market_change': '+0.83', 'post_market_change_percent': '+0.43%'}

You can find other useful info in the #quote-summary table:

In this case, you can extract each data field thanks to the data-test attribute as in the CSS selector below:

#quote-summary [data-test="PREV_CLOSE-value"]

Scrape them all with:

previous_close = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="PREV_CLOSE-value"]').textopen_value = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="OPEN-value"]').textbid = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="BID-value"]').textask = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="ASK-value"]').textdays_range = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="DAYS_RANGE-value"]').textweek_range = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="FIFTY_TWO_WK_RANGE-value"]').textvolume = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="TD_VOLUME-value"]').textavg_volume = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="AVERAGE_VOLUME_3MONTH-value"]').textmarket_cap = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="MARKET_CAP-value"]').textbeta = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="BETA_5Y-value"]').textpe_ratio = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="PE_RATIO-value"]').texteps = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="EPS_RATIO-value"]').textearnings_date = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="EARNINGS_DATE-value"]').textdividend_yield = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="DIVIDEND_AND_YIELD-value"]').textex_dividend_date = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="EX_DIVIDEND_DATE-value"]').textyear_target_est = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="ONE_YEAR_TARGET_PRICE-value"]').text

Then, add them to stock:

stock['previous_close'] = previous_closestock['open_value'] = open_valuestock['bid'] = bidstock['ask'] = askstock['days_range'] = days_rangestock['week_range'] = week_rangestock['volume'] = volumestock['avg_volume'] = avg_volumestock['market_cap'] = market_capstock['beta'] = betastock['pe_ratio'] = pe_ratiostock['eps'] = epsstock['earnings_date'] = earnings_datestock['dividend_yield'] = dividend_yieldstock['ex_dividend_date'] = ex_dividend_datestock['year_target_est'] = year_target_est

Fantastic! You just performed financial web scraping with Python!

Step 5: Scrape several stocks

A diversified investment portfolio consists of more than one security. To retrieve data for all of them, you need to extend your script to scrape multiple tickers.

First, encapsulate the scraping logic in a function:

def scrape_stock(driver, ticker_symbol): url = f'https://finance.yahoo.com/quote/{ticker_symbol}' driver.get(url) # deal with the consent modal... # initialize the stock dictionary with the # ticker symbol stock = { 'ticker': ticker_symbol } # scraping the desired data and populate # the stock dictionary... return stock

Then, iterate over the CLI ticker arguments and apply the scraping function:

if len(sys.argv) <= 1: print('Ticker symbol CLI arguments missing!') sys.exit(2)# initialize a Chrome instance with the right# configsoptions = Options()options.add_argument('--headless=new')driver = webdriver.Chrome( service=ChromeService(ChromeDriverManager().install()), options=options)driver.set_window_size(1150, 1000)# the array containing all scraped datastocks = []# scraping all market securitiesfor ticker_symbol in sys.argv[1:]: stocks.append(scrape_stock(driver, ticker_symbol))

At the end of the for cycle, the list of Python dictionaries stocks will contain all stock market data.

Step 6: Export scraped data to CSV

You can export the collected data to CSV with just a few lines of code:

import csv# ...# extract the name of the dictionary fields# to use it as the header of the output CSV filecsv_header = stocks[0].keys()# export the scraped data to CSVwith open('stocks.csv', 'w', newline='') as output_file: dict_writer = csv.DictWriter(output_file, csv_header) dict_writer.writeheader() dict_writer.writerows(stocks)

This snippet creates a stocks.csv file with open(), initializes with a header row, and populates it. Specifically, DictWriter.writerows() converts each dictionary into a CSV record and appends it to the output file.

Since csv comes from Python Standard Library, you do not even need to install an extra dependency to achieve the desired goal.

You started from raw data contained in a webpage and have semi-structured data stored in a CSV file. It is time to take a look at the entire Yahoo Finance scraper.

Step 7: Put it all together

Here is the complete scraper.py file:

from selenium import webdriverfrom selenium.webdriver.chrome.service import Service as ChromeServicefrom webdriver_manager.chrome import ChromeDriverManagerfrom selenium.webdriver.chrome.options import Optionsfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.common.by import Byfrom selenium.common import TimeoutExceptionimport sysimport csvdef scrape_stock(driver, ticker_symbol): # build the URL of the target page url = f'https://finance.yahoo.com/quote/{ticker_symbol}' # visit the target page driver.get(url) try: # wait up to 3 seconds for the consent modal to show up consent_overlay = WebDriverWait(driver, 3).until( EC.presence_of_element_located((By.CSS_SELECTOR, '.consent-overlay'))) # click the 'Accept all' button accept_all_button = consent_overlay.find_element(By.CSS_SELECTOR, '.accept-all') accept_all_button.click() except TimeoutException: print('Cookie consent overlay missing') # initialize the dictionary that will contain # the data collected from the target page stock = { 'ticker': ticker_symbol } # scraping the stock data from the price indicators regular_market_price = driver \ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="regularMarketPrice"]') \ .text regular_market_change = driver \ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="regularMarketChange"]') \ .text regular_market_change_percent = driver \ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="regularMarketChangePercent"]') \ .text \ .replace('(', '').replace(')', '') post_market_price = driver \ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="postMarketPrice"]') \ .text post_market_change = driver \ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="postMarketChange"]') \ .text post_market_change_percent = driver \ .find_element(By.CSS_SELECTOR, f'[data-symbol="{ticker_symbol}"][data-field="postMarketChangePercent"]') \ .text \ .replace('(', '').replace(')', '') stock['regular_market_price'] = regular_market_price stock['regular_market_change'] = regular_market_change stock['regular_market_change_percent'] = regular_market_change_percent stock['post_market_price'] = post_market_price stock['post_market_change'] = post_market_change stock['post_market_change_percent'] = post_market_change_percent # scraping the stock data from the "Summary" table previous_close = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="PREV_CLOSE-value"]').text open_value = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="OPEN-value"]').text bid = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="BID-value"]').text ask = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="ASK-value"]').text days_range = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="DAYS_RANGE-value"]').text week_range = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="FIFTY_TWO_WK_RANGE-value"]').text volume = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="TD_VOLUME-value"]').text avg_volume = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="AVERAGE_VOLUME_3MONTH-value"]').text market_cap = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="MARKET_CAP-value"]').text beta = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="BETA_5Y-value"]').text pe_ratio = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="PE_RATIO-value"]').text eps = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="EPS_RATIO-value"]').text earnings_date = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="EARNINGS_DATE-value"]').text dividend_yield = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="DIVIDEND_AND_YIELD-value"]').text ex_dividend_date = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="EX_DIVIDEND_DATE-value"]').text year_target_est = driver.find_element(By.CSS_SELECTOR, '#quote-summary [data-test="ONE_YEAR_TARGET_PRICE-value"]').text stock['previous_close'] = previous_close stock['open_value'] = open_value stock['bid'] = bid stock['ask'] = ask stock['days_range'] = days_range stock['week_range'] = week_range stock['volume'] = volume stock['avg_volume'] = avg_volume stock['market_cap'] = market_cap stock['beta'] = beta stock['pe_ratio'] = pe_ratio stock['eps'] = eps stock['earnings_date'] = earnings_date stock['dividend_yield'] = dividend_yield stock['ex_dividend_date'] = ex_dividend_date stock['year_target_est'] = year_target_est return stock# if there are no CLI parametersif len(sys.argv) <= 1: print('Ticker symbol CLI argument missing!') sys.exit(2)options = Options()options.add_argument('--headless=new')# initialize a web driver instance to control a Chrome windowdriver = webdriver.Chrome( service=ChromeService(ChromeDriverManager().install()), options=options)# set up the window size of the controlled browserdriver.set_window_size(1150, 1000)# the array containing all scraped datastocks = []# scraping all market securitiesfor ticker_symbol in sys.argv[1:]: stocks.append(scrape_stock(driver, ticker_symbol))# close the browser and free up the resourcesdriver.quit()# extract the name of the dictionary fields# to use it as the header of the output CSV filecsv_header = stocks[0].keys()# export the scraped data to CSVwith open('stocks.csv', 'w', newline='') as output_file: dict_writer = csv.DictWriter(output_file, csv_header) dict_writer.writeheader() dict_writer.writerows(stocks)

In less than 150 lines of code, you built a full-featured web scraper to retrieve data from Yahoo Finance.

Launch it against your target stocks as in the example below:

python scraper.py TSLA AMZN AAPL META NFLX GOOG

At the end of the scraping process, this stocks.csv file will appear in the root folder of your project:

Conclusion

In this tutorial, you understood why Yahoo Finance is one the best financial portal on the web and how to extract data from it. In particular, you saw how to build a Python scraper that can retrieve stock data from it. As shown here, it is not complex and takes only a few lines of code.

However, Yahoo Finance is a dynamic site that relies heavily on JavaScript and implements advanced data protection technologies. For seamless data extraction from such sites, consider using our Yahoo Finance Scraper API. This API handles the complexities of scraping, including managing CAPTCHAs, handling fingerprinting, and performing automated retries, allowing you to get structured financial data with ease. Get started with our Yahoo Finance Scraper API today to streamline your data collection process.

No credit card required

Don’t want to deal with web scraping at all but are interested in financial data? Get a Yahoo Finance dataset.

Note: This guide was thoroughly tested by our team at the time of writing, but as websites frequently update their code and structure, some steps may no longer work as expected.