Edge AI · On-Device AI
Edge AI / On-Device AI
What is Edge AI?
- Running AI on the device itself
- Fast and privacy-friendly
- Requires model compression
Adopting edge AI isn’t about swapping a model. It is redrawing the boundaries of your system. The moment you decide which data stays on the device and where it starts to leave, response time, network cost, security review scope, and failure-response scenarios all shift together. It looks like a technology choice, but it is really a decision that infrastructure, quality, and legal teams have to settle together.
In the field, it is usually viewed in three layers: on-device, running directly on the NPU inside a smartphone or camera; the edge server, where an industrial PC or gateway on the factory floor handles several devices together; and the cloud, which handles retraining and large-scale analytics. To fit a model on a device, you must quantize it to INT8 or shrink it through pruning and distillation. Size drops to a fraction, but accuracy typically loses 1–3 percentage points. Cutting a decision with 200–300 ms of round-trip latency down to a few dozen ms is why that loss is worth accepting.
The most common misconception is 'running it on the device settles the privacy question.' If a device is lost or stolen, all the data and the model inside go with it, and some attackers try to pull apart the model file to reverse-engineer the training data. The second misconception is leaving everything after deployment out of the calculation. In the cloud, fixing one server is enough, but with 3,000 devices in the field, without an OTA update process and version control, outdated models repeat the same wrong calls for months. It is also common for devices to throttle their own performance when they overheat, so a call that was right at dawn is wrong at midday. Before choosing between edge and cloud, it is better to decide who will manage these devices, and for how many years.
John KimInstructor John Kim, author of "B2B Marketing Design" and CEO of Nanum Consulting
Sungil KimGlobal pitching expert from KAIST and Samsung Electronics who coached the winning teams at the UN Tourism and Select USA IR pitch competitionsIt emerged as a way to run inference on the device itself, driven by the spread of IoT and mobile devices and advances in lightweight AI techniques.
Let’s look at a hypothetical case. Company A makes auto parts, with 320 employees and annual revenue of about KRW 90 billion (≈ US$67,335,000). To catch paint-surface defects, it sent line-camera images to the cloud for a pass/fail decision, but round-trip latency exceeded 0.4 seconds, so results came back only after the conveyor had already moved on to the next process. Defects missed over three months returned as rework, costing more than KRW 20 million (≈ US$15,000) a month.
- When Team Lead Kim of the production engineering team pulled the line logs, 78% of the delay was network round-trip time. The conclusion: 'We don't need a bigger model; we need to move it next to the camera.'
- The AI lead quantized the inspection model to INT8, cutting its size to a quarter; accuracy fell from 96.4% to 95.1%. The quality manager said, 'I’ll accept more false rejects as long as missed defects don’t increase.'
- The team chose an edge-server setup, with one industrial PC on each of three lines handling four cameras between them. At KRW 4.8 million (≈ US$3,590) per unit, the total came to KRW 14.4 (≈ US$1) million, and cloud data transfer fell from 9 TB to 0.4 TB a month.
- When the information security lead asked, 'What if a device gets compromised?', the team documented model file encryption and a quarterly OTA deployment procedure, and added a rule to company policy requiring a factory reset for any device taken off-site.
- Eight weeks later, decision latency dropped from 420 ms to 35 ms, and the escape rate of undetected defects fell from 1.8% to 0.6%. Monthly rework costs came down from the KRW 20 million (≈ US$15,000) range to the KRW 7 million (≈ US$5,240) range, and the investment paid for itself in four months.