Big Data
Big Data
What is big data?
- Large, fast, varied data
- What matters is analyzing data to extract value
- Needs analytics infrastructure
What big data changes is not the size of the data but the basis for decision-making. It used to be that a few thousand samples compiled at month-end were used to explain why something happened. Now full datasets are accumulated and used to ask what will happen next. Because the record captures each customer's click sequence, each machine's vibration readings, and each store's hourly payments, decisions can be made at the individual level, not on averages. 'In my experience' used to carry the day in meetings; now 'according to the last 14 months of logs' is taking its place.
Beyond the 3Vs, the two that really trip up practitioners are the ones that follow: veracity and value. A large share of corporate data is unstructured, such as email, call recordings, and images, and can't be analyzed as is. Storage choices also diverge. A data warehouse, where data is cleaned and given a schema, suits regular reports, while a data lake, where raw data is poured in as is, suits exploratory analysis. Processing cycles differ too, batch (once a day) versus streaming (second by second), and only work where a one-second delay makes the result useless, such as card fraud detection, needs streaming. You don't need to handle all data in real time.
The most common failure in the field is the data swamp, where data is stockpiled and never used. A company installs a platform costing hundreds of millions of won and loads three years of logs. Then the owner changes, nobody knows what each table means, and nobody opens it again. We see this often. Analysis run on data collected without input rules produces plausible numbers that are wrong, and when orders or staffing are changed based on those numbers, the losses are real. Reading correlation as causation is also risky: ice cream sales and drownings rise together, but neither causes the other. It is far better to first pick one question that needs an answer and properly manage only the data that feeds it.
Jaryong KooMarketing PhD and authority in AI, data analytics, and personal branding
John KimAuthor of B2B Marketing Design and CEO of Nanum ConsultingA concept that emerged with the data explosion of the 2000s. Defined by the 3 Vs (volume, velocity, variety), it became practical as distributed processing technologies such as Hadoop matured.
Consider a hypothetical case. Company A makes household goods, with 380 employees and annual sales of KRW 240 billion (≈ US$179,560,100). Seasonal orders had been left to the sales division's gut feeling, so KRW 900 million (≈ US$673,400) of inventory was written off every year, while best sellers ran out of stock every peak season. Executives said, 'Let's try solving this with data,' and launched an eight-month demand forecasting project.
- When the logistics manager pulled three years of shipment data, the top 80 of 1,200 SKUs accounted for 62% of sales, yet half of the write-offs came from the bottom 400.
- When a data manager joined temperature, portal search volume, and promotion schedules to ERP sales results, it showed that dehumidifier demand responds to search volume two weeks before it responds to temperature.
- The head of sales pushed back, saying 'field instinct still counts,' so the teams agreed to compare errors over three months, placing the forecast next to the sales team’s orders for just 40 items.
- The comparison showed the forecast model's error was 11 percentage points lower on average than the sales team's orders. The division head said, 'I'll give you this one,' and began using the forecast as the baseline starting with the fall orders.
- Along the way, the team found that return-reason codes from dealers were entered differently by each person. They merged 12 codes into 5 and made the field mandatory, and accuracy improved again.
- Eight months later, annual inventory write-offs had fallen from KRW 900 million (≈ US$673,400) to KRW 530 (≈ US$1) million, and sales lost to stockouts had dropped from around 40 a month to 12.