Web Crawling · Scraping
Web Crawling / Scraping
What is web crawling?
- Automatically collects data across the web
- Used for data collection and monitoring
- Excessive crawling creates load and legal risk
What crawling and scraping really change is how often you look at your data. When a staff member opens each site, checks it by eye, and types it into a spreadsheet, once a day is the limit. With automated collection, the same job can run once an hour. Prices, stock levels, and reputation shift from 'research findings' to 'real-time metrics'. An organization that used to receive a market research report once a quarter moves to a model where it sees what changed this morning and responds in the afternoon.
In practice, you need to tell three cases apart. First, if the site offers an official API, use that. It does not break when the page layout changes, and there is no terms-of-service dispute. Second, if there is no API and the values sit directly in the HTML, simple scraping is enough. Third, if the values are drawn later by JavaScript, you need a tool that actually launches a browser, and it runs dozens of times slower. On top of that come the scope allowed by robots.txt and the request interval. Typically you wait 1 to 2 seconds between requests, and as a rule you do not touch members-only data behind a login.
Maintenance, not collection itself, is the real work. Page structures change without notice, and when one marketplace tweaks its layout, values stop coming in from that day. So from the start, log the collection success rate and set up an alert for failures. It is also a different thing from RPA, which automates actions on internal screens. RPA clicks through authorized internal systems on your behalf, while crawling is sending requests to someone else’s server, so you have to factor in the other party’s circumstances too.
The most common misconception is that anything shown publicly is free to take. Public is not the same as free to use. In Korea, there have been disputes where mass-scraping a competitor’s site was treated as a violation of a database producer’s rights, and flooding a target server with dozens of requests per second to slow it down can escalate into an obstruction-of-business issue. Once personally identifiable values such as names and contact details get mixed in, you enter the territory of the Personal Information Protection Act. There are quieter failures too. When a page’s structure changes, the script silently brings back blanks or zeros with no error, and the dashboard plots those numbers as they are, which can even lead to cutting prices based on wrong figures.
Jaryong KooMarketing PhD and authority in AI, data analytics, and personal branding
John KimAuthor of B2B Marketing Design and CEO of Nanum ConsultingIt began with the crawlers search engines use to collect pages automatically and index the web, and grew into a range of scraping techniques as demand for data increased.
Let’s look at a hypothetical case. Company A makes and sells household goods, with 180 employees and KRW 90 billion (≈ US$67,335,000) in annual revenue, 40% of it from open marketplaces. It lists 300 SKUs across four marketplaces, and three merchandisers on the e-commerce team spend two hours every morning checking competitor prices by eye and copying them into a spreadsheet. Again and again, they learned they had lost the lowest price only half a day later.
- Kim, the e-commerce team lead, called spending two hours every morning copying data by hand a waste and proposed automation. Of the four marketplaces, the two that offer an official API would use the API, and only the other two would be scraped.
- Lee, the legal manager, went through each site's terms of use and robots.txt, limited the scope to public product pages only, and excluded reviews and buyer information behind a login from the scraping list from the start.
- Park, who leads DX, wrote a Python script that ran across 300 SKUs three times a day, at 9 a.m., 2 p.m., and 6 p.m. A 2-second interval between requests kept each collection run to 15 minutes.
- In week 2, one marketplace redesigned its pages and every price came in as KRW 0 (≈ US$1). Park, the assistant manager, added a rule so that an alert goes to the company chat if the scraping failure rate exceeds 5%.
- Three months later, the morning task had dropped from two hours to 10 minutes, and the time it took to notice a lost lowest-price position fell from an average of 6 hours to 40 minutes. Faster price responses on the 15 core SKUs lifted monthly sales for those items by 12%.