Inference Comes Down to the Device

The 2026 Silicon Valley chip roadmaps all repeat the same word: NPU. Cores dedicated exclusively to neural network computation have become standard in mobile SoCs and PC chips. Qualcomm, Apple, AMD, and Intel tout tens of trillions of operations per second in every product announcement.

On the surface, it looks like yet another spec race — faster chips, smarter phones. Read it that way, and you miss the point.

More powerful NPUs mean inference is leaving the data center. Until now, nearly all AI responses were generated on cloud servers. Whether editing a photo or summarizing a paragraph, data made a round trip through someone else's GPU before coming back.

That round trip is disappearing.

Why is Big Tech suddenly pushing inference down to the device? There are two pressures. One: the cost of GPU server inference explodes in proportion to user count. The other: every piece of data that passes through the cloud becomes a target for regulation and litigation.

This is where the camps diverge. OpenAI and Anthropic keep large models in the cloud and sell access via API. Apple and Qualcomm embed smaller models inside the chip so data never leaves the device. Both use the word AI, but the layer each is trying to control is entirely different.

The cloud camp holds the brain — the model. The on-device camp holds the user's data and the point of contact. Which side sets the standard will determine the billing architecture of the next decade.

Capital has already moved. Venture funding is dispersing away from single bets on massive foundation models and toward inference efficiency, model compression, and edge deployment stacks. Money is flowing to companies that can deliver the same performance cheaper and closer to the user. The calculation is clear: inference cost is the survival line.

This is where the story spills into data sovereignty. If inference ends at the device, sensitive data never crosses a border in the first place. The data localization that Europe has been trying to enforce through regulation is being solved physically by the chip. Cloud dependency loosens — but which chip your data lives on becomes the new locus of power.

So where do Korean companies stand in this restructuring? As suppliers, customers, or architects of standards?

Samsung Electronics and SK Hynix are laying the foundation of this shift through memory and foundry. On-device inference consumes high-bandwidth memory and low-power DRAM. It is a good position — but it is precisely a supplier's position. They provide the chips and memory, but the model and data standards running on top are set by the United States.

A counterargument is possible. Component supply is a large enough business on its own, and the view holds that memory market share translates directly into negotiating leverage even without controlling the standard. That is true. But a supplier's margins are always set by whoever holds the knife of standardization. If U.S. chip companies own the NPU instruction architecture and on-device model formats, Korea remains at the upper tier of subcontracting — selling excellent parts, but subcontracting nonetheless.

In Busan, it may be tempting to see this as someone else's distant fight. It is not. The industrial terminals, inspection equipment, and vehicle terminals that Busan's manufacturing and logistics sectors will deploy in the years ahead will all run on these on-device inference chips. Who sets the standard for those chips will set the cost structure for Busan's factories.

That is why the cost of watching from the sidelines must be reckoned with. If NPU announcements are dismissed as product news today, Korea will find itself in three years being notified of the inference standards for its own devices through U.S. chip company roadmaps. Silicon Valley's chip announcements are not someone else's story. They are the first draft of the cost sheet Korean companies will pay next. Ignore it now, and you will read it later — as an invoice.