How to Scrape Competitor Prices

What you are really capturing when you scrape a price

The sticker price is the easy part. The reason price scraping is worth doing well is that a product page carries several things that drive a pricing decision, and most of them are not a single clean number.

The current price is what the shopper pays right now. Next to it sits the list price, the crossed-out 'was' price the store shows to make the deal look good. Amazon calls it the list price, Walmart the was price, Shopify the compare-at price, Etsy the original price. The gap between the two is the discount a shopper sees, and here is the first trap: that discount is almost never its own field. Most scrapers hand you the current price and the reference price and leave you to work out the percentage, and the deeper discounts, an Amazon coupon or Lightning Deal, a Shopify automatic or coupon-gated price, a bundle app, often do not show up in the data at all. They only appear in the rendered page or at checkout, so the reference price can quietly overstate what the customer actually pays.

Then there is stock. Almost every site gives you in stock or out of stock, not a real count. Shopify deliberately hides the number, eBay does not expose it, and Amazon's 'Only 3 left' is a scarcity label, not a reliable figure. So you track stock as a state that changes over time, in stock today, gone tomorrow, back on Friday, rather than a quantity you can trust.

Then the question most people forget: whose price is this? On a marketplace one listing has many sellers, and the price belongs to whichever offer is winning the buy box right now. The price can move because a different seller took the buy box, not because anyone actually cut a price. If you do not capture the seller next to the price, you cannot tell a real price cut from a seller swap. The per-unit price, per ounce or per 100ml, is the last piece, useful for comparing a big pack against a small one, though only Walmart reliably hands it to you.

The same product, a different price

Scrape the same product twice from two locations and you can get two different prices. This catches people out constantly.

On Amazon, Walmart, Google Shopping and grocery sites like Instacart, price and availability are tied to a location. Walmart prices are set by the store behind a ZIP code. Amazon shows a different price, a different buy-box winner and a different delivery estimate depending on the country, the state, even the ZIP. If you do not pin every run to the locations you actually sell against, you get a default that no real shopper in your market sees, and your day-over-day tracking drifts for no reason. A price change that is really just your scraper hopping to a different exit location looks like a competitor move, and you end up reacting to noise.

Currency is the same story one level up. Shopify stores running Markets, and Etsy with its two dozen-odd regional storefronts, serve the same item in different currencies by region. Pull one feed without thinking about it and you can be comparing dollars against pounds.

So two rules fall out of this. Pin the location on every run and keep it consistent, and store that location on every row, so a price always means this product, at this place, at this time. An average across mixed locations is worse than no number, because it looks precise and is wrong.

How the big platforms differ

The fields rhyme across sites, a title, an ID you track it by, a price, a seller, a stock flag, but each platform has its own shape and its own thing that breaks.

Amazon. You key everything on the ASIN and pull the current buy-box price, the list price, availability text like 'Only 5 left', the buy-box seller and who ships it, the rating and review count, and the Best Sellers Rank. The catch is that price and buy box move with location and with whichever seller is winning, and Amazon fights bots hard, so a lazy scraper can save a CAPTCHA page as if it were a $0 price. Every row has to be validated before it is trusted.

Walmart. You track the item ID and get the current price, the was price, the per-unit price, the Rollback or Clearance badge, the seller (Walmart itself or a third-party Marketplace seller), and the fulfillment options. The live price often is not in the page's embedded JSON at all, so you have to render the page or hit the pricing call, and a blocked request frequently comes back as a normal 200 with the price silently missing. Walmart is one of the harder retail sites to collect from cleanly.

Shopify stores. Because every Shopify store runs on the same platform, a lot of them expose a public products feed with the title, variants, SKU, price and the compare-at price for each variant. It is one of the friendlier catalogs, until you need real stock (the feed only says true or false, never a count), or you hit a store that turned the feed off, or you need the localized price a shopper in a specific country actually sees.

Etsy. You get the listing, the shop, the price and the original price (which is blank unless the item is genuinely on sale, so treat a blank as 'not discounted', not an error), plus favorites and review counts as demand signals. The headache is that there is no standard Etsy data shape, every tool names and types its fields differently, and the same listing comes back in a different currency across Etsy's regional sites.

Google Shopping. This one is less a store and more a price-comparison view, one product sold by many merchants, each at its own price. You pull the product, the merchant, the price (often with a strikethrough original), the shipping, and a cheapest-offer flag, which makes it the fastest way to see where you rank against everyone selling the same item. Prices shift by country and ZIP, availability is often blank, and Google blocks datacenter traffic aggressively.

eBay. You track the item ID, but price means two different things here. A Buy It Now price is fixed, an auction price is a live bid that keeps climbing. And a live listing only shows the asking price, not what anything sold for, so for real market value you need sold-listings data, which eBay keeps for only about 90 days. Always split auctions from fixed-price before you compare anything.

A one-off scrape is not price monitoring

A single pull tells you today's prices. That is a report, not monitoring, and price is the field that changes fastest. The value shows up when you keep the same list of products and re-check them on a schedule, because then you can see a trend, catch a drop the hour it happens, and prove a competitor's pattern instead of guessing at it.

Every scrape is a snapshot with a timestamp. None of these sites hand you a price history, you build it by storing each run and diffing it against the last. Keep the price, the was price and the stock flag every time, compare to the previous run, and emit the events that actually matter: price went up, price went down, went out of stock, came back, a new was price appeared, which usually means a promo just started. Those events are what feed an alert or a dashboard, and they are far more useful than a nightly dump of everything.

Match the cadence to the signal. Descriptions and images barely change, so crawl them slowly. Price and stock move fast, so check them often. For one reseller we run a monitor over around 3,000 products on a 10-minute cycle that fires an instant alert the moment a price drops or an item restocks, so they act in seconds instead of refreshing pages by hand. On a fast-moving category, that speed is the difference between catching the deal and reading about it afterwards.

Price scraping software, a DIY scraper, or a delivered feed

There are three ways to get this data, and they fail in different places.

Off-the-shelf price scraping software and self-serve APIs are the quickest start. You paste in some URLs and get prices back. They work well on easy sites and short lists, and they are a fair place to begin. Where they slip is the hard targets and the upkeep: they return thin or blank data on the sites that fight back, they rarely pin location the way you need, and when a site changes its layout you wait for the vendor to catch up or quietly collect bad data in the meantime.

Building your own price scraper gives you full control, and for a handful of friendly sites a scheduled script will do the job. The cost is not the first version, it is the second year: location pinning, residential proxies, anti-bot handling, validating every row so a block does not land as a $0 price, matching variants across runs, and the steady drip of fixes every time a site shifts. Each one is small. Together they turn a weekend project into something a person has to own every week.

A delivered feed is the third option. You say which products and which sites, and someone else builds the scraper, pins the locations, handles the blocking, validates the data, and keeps it running as the sites change. You get a clean price feed on your schedule and none of the maintenance. That is the trade: you give up some hands-on control, and you get your time back and data you can actually trust.

Have the price feed delivered

This is exactly what we do at DataScrape. You send the products and the sites you want watched, and we deliver a clean, structured price feed on your schedule and keep it running as those sites change. You work directly with the people building it, not a ticket queue.

The hard sites are the point, not a problem. We already run collection through heavy anti-bot on sites like Booking.com and Expedia, and we run retail price and stock monitoring at scale, around 3,000 products on a 10-minute cycle with instant price-drop and restock alerts for one reseller. Amazon, Walmart, a fleet of Shopify stores, Google Shopping, whatever you price against, we expect the site to fight back and we plan for it.

The easiest way to tell if the data is any good is to look at it. Send us a few products or pages you want priced and we will scrape a real sample and send it back, no signup and no commitment. If it is useful, we turn it into a feed. If it is not, you have lost nothing.