Matériel pour l'IA en périphérie et quand privilégier le calcul local ou cloud

The phrase edge AI hardware can describe a sensor running a tiny neural network, a laptop with an NPU, an industrial computer, a robot carrying an embedded GPU, or a server sitting inside a factory.

Those machines have almost nothing in common except location.

They run AI close to where data is created or where the result is needed, rather than sending every input to a remote data center first.

That distinction matters because AI does not always belong in the cloud.

A machine detecting a dangerous condition may need to respond even when its internet connection disappears. A camera can analyze video locally instead of uploading every frame. A laptop can transcribe speech without continuously sending audio elsewhere. A robot cannot wait for a round trip across the internet before deciding how to move.

But local execution creates its own constraints.

The hardware has less power and memory. Models may need to be smaller or quantized. Cooling can be limited. Updating thousands of deployed devices is harder than updating one cloud service. And a model that fits today's edge hardware may not fit the one you want to use next year.

The useful decision is therefore not edge or cloud.

It is:

Which parts of this AI workload need to run locally, and which parts are better somewhere else?

Edge Impulse describes edge computing as processing and storing data at the periphery of a network and notes that edge AI can reduce bandwidth use, network dependence, and latency while keeping more data local. It also makes an important point: edge AI complements cloud computing rather than simply replacing it. (Edge Impulse)

Edge AI hardware at a glance

There is no single edge AI processor.

‍

‍

An edge deployment can use several rows at once.

A factory camera might perform object detection on a small local accelerator, send unusual events to an edge server, and use cloud GPUs for model retraining.

That is often a better architecture than forcing the whole system into one location.

What is edge AI?

Edge AI is machine-learning inference performed close to the data source or user rather than exclusively in a remote cloud environment.

“Close” is deliberately vague.

For a smartwatch, the edge is the watch.

For an autonomous machine, it may be an embedded computer inside the machine.

For a factory, the edge could be a server inside the building receiving data from hundreds of sensors.

For a retail business, it might be a computer in each store.

This means edge AI and on-device AI overlap without being identical.

On-device AI runs directly on the end device.

Edge AI can also run on a nearby gateway, local server, or other infrastructure positioned between the device and a distant cloud.

That distinction becomes useful once workloads exceed what a small device can handle.

Edge describes location, not processor architecture

A common mistake is treating an NPU as an “edge processor,” a GPU as a “cloud processor,” and the CPU as something in between.

Hardware does not divide that neatly.

You can find CPUs, GPUs, NPUs, FPGAs, and purpose-built ASICs at the edge.

You can also find GPUs, NPUs, TPUs, CPUs, and other accelerators in data centers.

The processor determines how the computation runs.

The edge determines where it runs.

Our practical guide to AI accelerators explains the processor categories themselves. Edge hardware selection adds another layer: physical space, power, network availability, sensors, deployment lifetime, environmental conditions, and the cost of maintaining equipment outside a central data center.

Why run AI at the edge?

There are several good reasons, and “AI PCs are fashionable” is not one of them.

You need to respond without waiting for the network

A remotely served model adds network time to the inference path.

For many applications, that delay is acceptable.

For others, it is not.

A voice assistant may feel sluggish if every small interaction requires a distant server. A robot, industrial control system, or safety application may have much tighter response requirements.

Running inference locally removes the remote-network round trip from the critical path.

That does not guarantee lower total latency. A powerful cloud GPU can sometimes finish the model so much faster that it compensates for network delay.

The comparison needs to be:

local processing time versus network + remote processing time.

Measure the complete path rather than assuming that “local” automatically means faster.

You need the system to work offline

This is one of the clearest edge cases.

An application may operate:

• underground;
• inside a vehicle;
• in remote infrastructure;
• on unreliable mobile connectivity;
• aboard a machine moving between networks;
• somewhere that simply cannot depend on continuous internet access.

If the application must keep working when the connection goes down, the essential model needs to live locally.

Arm's current edge AI guidance explicitly treats offline operation as one of the reasons to deploy inference on-device, alongside low latency and tight power constraints. (Arm edge AI)

The cloud can still synchronize data, distribute models, perform training, or handle more demanding requests when connectivity returns.

Offline capability does not require cloud isolation.

Uploading the raw data would be wasteful

Cameras are an obvious example.

A network of cameras produces enormous amounts of data, most of which may be uninteresting.

Suppose the actual application only needs to answer:

• Is someone entering this restricted area?
• Is this product defective?
• Is a vehicle occupying this space?
• Did this machine start vibrating abnormally?

Sending every frame or sensor reading to a distant service just to receive a small answer can be wasteful.

Local inference can reduce the stream before anything leaves the site.

The edge device might send an event, metadata, a cropped image, or an anomaly rather than hours of raw input.

This changes the bandwidth problem from move everything and decide later to decide locally what is worth moving.

You want to reduce how much raw data leaves the device

Local processing can also reduce data movement.

A speech model can extract text without uploading the original audio.

A camera can detect an object without transmitting the entire video stream.

A local document model can extract information without sending the document to a third-party endpoint.

That can simplify some privacy and data-governance decisions because fewer raw inputs leave the device or site.

It does not make the system secure by default.

Edge devices still need secure storage, software updates, access control, encryption where appropriate, credential management, and protection against physical or remote compromise.

“Local” describes where computation happened.

It is not a security certification.

Power and heat may limit everything

A cloud server can use hundreds or thousands of watts across its processors and cooling infrastructure.

A wearable cannot.

Neither can many cameras, industrial controllers, mobile machines, or passively cooled devices.

This constraint has driven much of the interest in dedicated NPUs and edge accelerators.

A neural processor can devote more hardware to the small set of operations it expects to run and avoid keeping a larger general-purpose processor active for continuous inference.

That is where the NPU becomes especially useful.

But power budgets vary enormously even within the edge category.

One edge device may operate on milliwatts.

Another may be a 100-watt industrial computer.

A third may be a rack server sitting ten meters from the machines it controls.

There is no meaningful “edge power budget.”

You have to define yours.

Edge AI hardware ranges from tiny to surprisingly large

The word “edge” can make the hardware sound small.

Sometimes it is.

Microcontrollers

At the low end, AI can run on microcontrollers with extremely constrained memory and power.

These systems are suited to narrow tasks such as:

• wake-word detection;
• vibration classification;
• simple anomaly detection;
• sensor classification;
• compact vision workloads.

STMicroelectronics' current edge AI portfolio, for example, includes STM32 microcontrollers with dedicated neural acceleration as well as larger microprocessor systems. Its STM32N6 combines a microcontroller with the company's Neural-ART Accelerator for local AI workloads. (ST Edge AI)

The models here are nothing like the large language models people usually associate with generative AI.

That is fine.

A model does not need billions of parameters to tell you that a motor sounds wrong.

Integrated CPUs, GPUs, and NPUs

Move up another level and the edge computer begins to resemble an ordinary PC.

AMD's current Ryzen AI Embedded X100 family combines x86 CPU cores, an integrated GPU, NPU acceleration, and unified memory in one embedded platform. AMD positions the architecture for workloads such as industrial systems and autonomous machines where general computation, graphics, sensor processing, and AI may all need to happen locally.

That combination matters because an edge application rarely consists solely of inference.

A robot may also need:

• sensor processing;
• navigation;
• control logic;
• networking;
• visualization;
• logging;
• application software.

The NPU handles AI that fits it.

The GPU handles more parallel or flexible work.

The CPU runs the rest.

This is the same heterogeneous design we explored in CPU vs GPU vs NPU.

Embedded GPUs

Some edge applications need much more AI performance.

Robotics and machine vision are good examples.

NVIDIA's Jetson platform packages GPU compute, CPU resources, memory, sensor support, and an embedded software stack into systems intended for robotics and edge AI.

At the high end, current Jetson Thor modules reach up to 128 GB of memory, and NVIDIA specifies configurable power profiles reaching 130 W on Jetson AGX Thor-class hardware. Those are vendor specifications rather than independent performance comparisons, but they show how far “edge hardware” now extends beyond tiny low-power inference devices. (NVIDIA Jetson Thor)

A machine carrying that hardware can run models locally that would have been impractical on an embedded system only a few years ago.

It is still an edge device because of where the computation happens.

FPGAs and specialist accelerators

Some edge systems require custom data paths, strict timing, unusual sensor interfaces, or very predictable latency.

FPGAs and adaptive SoCs can be attractive there.

Their programmable hardware lets engineers create pipelines around the application instead of accepting a fixed processor design.

Dedicated AI accelerator cards provide another route. A gateway or industrial computer can keep its existing CPU while adding an NPU, ASIC, FPGA, or GPU for inference.

We cover those hardware categories in the AI accelerator guide.

The practical issue at the edge is usually less about the label and more about what the complete board or system can support:

• memory;
• power;
• cooling;
• connectivity;
• sensors;
• physical dimensions;
• supported operating system;
• inference runtime;
• environmental tolerance;
• lifecycle.

Local GPU systems blur the boundary again

Suppose an architecture firm runs an RTX workstation in its office.

Is that edge AI?

It can be, depending on the system architecture.

If that workstation performs AI locally for nearby users instead of sending requests to a distant service, it is functioning as local or edge compute.

The same applies to a server installed inside a factory, hospital, laboratory, or retail site.

That creates a useful middle ground between on-device AI and public cloud infrastructure.

You gain far more compute than a tiny device can provide without sending every workload outside the site.

You also inherit responsibility for the machine.

Someone has to buy it, secure it, cool it, update it, monitor it, replace failed parts, and plan what happens when the next model no longer fits.

Can you run an LLM at the edge?

Yes.

The more useful answer is: which LLM, on which edge?

A small quantized language model can run on a laptop or compact edge system.

Larger embedded GPU systems can run much larger models.

A local workstation may have enough GPU memory for serious generative AI.

Un microcontrôleur, lui, ne le fait clairement pas.

La taille du modèle n'est que la première contrainte.

Vous devez également prendre en compte :

• la précision numérique ;
• la RAM ou VRAM disponible ;
• la longueur du contexte ;
• la croissance du cache KV ;
• le nombre de jetons par seconde ;
• le temps d'attente acceptable pour le premier jeton ;
• la consommation électrique ;
• le moteur d'exécution du modèle ;
• les opérateurs pris en charge ;
• la concurrence des applications.

C'est pourquoi les chiffres bruts de performance en IA peuvent être trompeurs.

Un processeur peut afficher des TOPS impressionnants tout en manquant de la mémoire requise par le modèle.

Notre prochain article sur les TOPS par rapport aux FLOPS et les mesures de performance en IA aborde directement ce problème.

La quantification rend les LLM en périphérie beaucoup plus pratiques

La compression de modèle change la donne pour l'informatique en périphérie.

Convertir les poids d'un modèle de FP16 vers INT8, INT4, FP4 ou une autre représentation compacte réduit la quantité de mémoire nécessaire pour les stocker.

Cela peut permettre à un modèle trop volumineux de tenir dans la mémoire disponible.

Cela peut également réduire le volume de données devant transiter par la mémoire lors de l'inférence.

Le gain de vitesse dépend de la présence, au niveau du matériel et du moteur d'exécution, de noyaux optimisés pour le format choisi.

Cet aspect est d'autant plus crucial en périphérie (edge), où la mémoire constitue souvent l'une des contraintes les plus fortes.

Un déploiement dans le cloud permet de passer à un GPU doté de plus de VRAM.

Ce n'est pas le cas pour un appareil embarqué déjà fabriqué.

La planification matérielle doit donc prévoir une marge de manœuvre pour les futurs modèles, plutôt que de se limiter aux besoins des benchmarks actuels.

L'inférence en périphérie est généralement plus adaptée que l'entraînement en périphérie

La plupart des systèmes en périphérie sont principalement conçus pour l'inférence.

L'entraînement exige généralement beaucoup plus de puissance de calcul, de mémoire, d'accès aux données et de flexibilité. De plus, le développement des modèles évolue rapidement, ce qui rend le matériel déployé aux ressources limitées peu adapté à cet usage.

Un flux de travail courant consiste à :

1. collecter ou préparer les données ;
2. entraîner ou affiner le modèle sur une infrastructure plus puissante ;
3. l'optimiser ou le quantifier ;
4. déployer le modèle résultant sur le matériel en périphérie ;
5. collecter de nouvelles données ou erreurs ;
6. entraîner la version suivante ailleurs ;
7. redéployez-le.

Cette séparation permet au système déployé de rester concentré sur la tâche qu'il doit accomplir.

Il existe des exceptions, notamment pour l'apprentissage en ligne et les tâches d'adaptation locale plus modestes, mais l'entraînement de modèles substantiels nécessite généralement une infrastructure plus puissante.

Notre matériel d'entraînement vs matériel d'inférence article examine pourquoi ces deux étapes imposent des exigences différentes en matière de processeurs, de mémoire, de précision et d'interconnexions.

NPU vs GPU en périphérie

Les NPU et les GPU présentent tous deux des cas d'utilisation pertinents en périphérie.

Un NPU devient intéressant lorsque :

• la consommation électrique est strictement limitée ;
• le modèle est pris en charge ;
• la charge de travail est stable ;
• l'inférence s'exécute en continu ;
• le refroidissement de l'appareil est limité ;
• le calcul généraliste par GPU n'est pas nécessaire.

Un GPU devient intéressant lorsque :

• le modèle est plus volumineux ;
• la flexibilité de l'architecture est importante ;
• plusieurs charges de travail IA doivent partager l'appareil ;
• des noyaux personnalisés sont requis ;
• le modèle change fréquemment ;
• la vision par ordinateur, les graphismes, la simulation ou tout autre calcul parallèle s'exécutent également localement.

Certains systèmes incluent les deux.

Notre comparatif NPU vs GPU examine plus en détail la prise en charge des modèles, la mémoire, la portabilité logicielle, l'entraînement, l'inférence et la consommation d'énergie.

L'inférence sur CPU a toujours sa place en périphérie

Un accélérateur d'IA dédié n'est pas obligatoire.

De nombreux petits modèles fonctionnent parfaitement sur CPU.

Si la tâche est peu fréquente, présente des exigences de latence modérées ou s'exécute déjà sur un appareil disposant d'une capacité CPU inutilisée, l'ajout d'un processeur supplémentaire peut créer une complexité inutile.

C'est particulièrement vrai pour :

• les petits classificateurs ;
• la détection d'anomalies légère ;
• le prétraitement des données ;
• les charges de travail de vision simples ;
• l'apprentissage automatique classique ;
• l'inférence locale à faible volume.

Notre guide sur les charges de travail vCPU et quand vous n'avez pas besoin de GPU défend le même argument pour l'infrastructure cloud : utilisez du matériel spécialisé lorsque la charge de travail tire profit de cette spécialisation.Hivenet)

Le matériel en périphérie mérite la même rigueur.

La latence réseau n'est qu'une composante de la latence globale

Dire que « la périphérie est plus rapide parce qu'il n'y a pas de réseau » est trop simpliste.

Imaginons qu'un appareil local nécessite 600 millisecondes pour effectuer une inférence.

Un GPU dans le cloud nécessite 80 millisecondes, plus un aller-retour de 30 millisecondes.

Le cloud reste gagnant.

Supposons maintenant que la connexion prenne parfois 300 millisecondes ou disparaisse complètement.

Le résultat change à nouveau.

Vous devez mesurer :

capteur/entrée → prétraitement → inférence → post-traitement → réseau si utilisé → réponse de l'application

C'est là le véritable chemin de latence.

Pour les LLM interactifs, des métriques telles que le temps jusqu'au premier jeton et le nombre de jetons par seconde comptent également. La méthodologie de benchmark de Hivenet distingue la latence, le débit, la concurrence, la forme du prompt et la longueur de sortie précisément pour cette raison.

Un petit processeur local et un GPU dans le cloud peuvent tous deux être « rapides » tout en optimisant des aspects totalement différents d'un problème.

La bande passante peut permettre de prendre une décision en périphérie avant que la latence n'intervienne.

Un simple capteur de température produit très peu de données.

Ce n'est pas le cas d'une caméra.

Il en va de même pour le lidar, l'audio haute résolution, l'imagerie médicale, les réseaux de capteurs industriels ou les flux vidéo synchronisés multiples.

Lorsque les données d'entrée sont volumineuses et que le résultat utile est restreint, le traitement à proximité de la source devient intéressant.

Un système local peut transformer :

des gigaoctets de données brutes

en :

une classification, un événement, un vecteur, une alerte ou un échantillon sélectionné.

Le cloud peut ensuite prendre en charge le sous-ensemble nécessitant un traitement plus approfondi.

Ce modèle réduit la bande passante et peut rendre la partie distante du système d'IA beaucoup moins coûteuse.

Le traitement local ne résout pas automatiquement les problèmes de confidentialité.

L'IA en périphérie peut réduire l'exposition, car les données brutes n'ont pas toujours besoin de quitter l'appareil ou le site.

C'est un atout précieux.

Cependant, un déploiement en périphérie mal sécurisé peut engendrer ses propres problèmes.

Des milliers d'appareils physiques créent autant d'endroits où :

• des identifiants peuvent fuiter ;
• des logiciels peuvent devenir obsolètes ;
• du stockage peut être dérobé ;
• les ports de débogage peuvent être exposés ;
• les modèles peuvent être extraits ;
• la configuration peut dériver.

Un service cloud centralisé peut en réalité être plus facile à corriger et à surveiller de manière cohérente.

La confidentialité et la sécurité nécessitent donc des questions distinctes :

Les données brutes doivent-elles quitter le site ?

et :

Pouvons-nous exploiter les appareils locaux en toute sécurité ?

Elles sont liées, mais ne sont pas identiques.

Pour les charges de travail sensibles nécessitant une infrastructure distante, l'approche IA privée de Hivenet est conçue pour permettre de choisir le modèle, le chemin des données, la région de déploiement et l'infrastructure en fonction de la charge de travail, plutôt que d'envoyer systématiquement tout vers une API publique générique.

Le coût caché de l'IA en périphérie est la gestion de parc

Un serveur cloud peut être mis à jour une seule fois.

Dix mille appareils en périphérie doivent être mis à jour dix mille fois.

Cela change la nature du problème d'ingénierie.

Les systèmes en périphérie destinés à la production doivent pouvoir gérer :

• le versionnage des modèles ;
• les mises à jour logicielles ;
• le retour à une version antérieure ;
• télémétrie ;
• défaillances matérielles ;
• identité de l'appareil ;
• certificats et identifiants ;
• débogage à distance ;
• journaux ;
• dérive de configuration ;
• différentes révisions matérielles.

Un excellent modèle exécuté sur des appareils non gérés devient un handicap opérationnel.

C'est l'une des raisons pour lesquelles le premier prototype ne doit pas immédiatement déterminer le matériel de production.

Edge Impulse recommande un processus de sélection matérielle itératif basé sur les contraintes réelles d'utilisation et de conception, plutôt que de choisir une carte avant d'avoir compris les exigences de l'application. (Sélection de matériel Edge Impulse)

Ce conseil s'applique bien au-delà des appareils embarqués.

Les coûts du matériel en périphérie sont concentrés au départ

Les systèmes cloud et en périphérie exposent les coûts différemment.

Achetez un appareil en périphérie et une grande partie des dépenses survient avant la première inférence.

Vous payez pour :

• le processeur ;
• la mémoire ;
• carte ;
• stockage ;
• boîtier ;
• réseautage ;
• alimentation ;
• capteurs ;
• déploiement ;
• installation ;
• inventaire de remplacement.

Vient ensuite la maintenance.

L'infrastructure cloud déplace une plus grande part des coûts vers l'usage.

Vous louez la puissance de calcul nécessaire et pouvez modifier le matériel en fonction de l'évolution de la charge de travail.

Aucun des deux modèles n'est systématiquement moins cher.

Une caméra effectuant la même inférence en continu pendant cinq ans peut largement justifier l'utilisation de matériel local dédié.

Une équipe évaluant six LLM différents sur trois mois ne devrait probablement pas concevoir un appareil de périphérie sur mesure en premier lieu.

Le taux d'utilisation et la stabilité de la charge de travail déterminent en grande partie la viabilité économique.

C'est le même schéma que celui observé lors de la comparaison entre les accélérateurs d'IA spécialisés et les GPU pour l'inférence.

IA en périphérie vs IA dans le cloud

Cette comparaison utile semble moins spectaculaire une fois les contraintes mises par écrit.

‍

‍

Rien n'oblige chaque ligne à s'appuyer sur la même architecture.

L'IA hybride, combinant edge et cloud, est souvent la solution la plus pragmatique.

Certains des systèmes les plus robustes répartissent délibérément la charge de travail.

Modèle 1 : détection locale, escalade vers le cloud

Un petit modèle local traite les entrées courantes.

Seuls les cas incertains, inhabituels ou importants sont transmis à un modèle distant plus puissant.

Une caméra peut reconnaître des objets ordinaires localement et envoyer les images ambiguës à un modèle de vision plus performant.

Cela permet de maintenir une exécution rapide en fonctionnement normal tout en conservant l'accès à des modèles plus vastes en cas de besoin.

Modèle 2 : perception locale, raisonnement distant

Un robot peut traiter les données de ses capteurs localement, car la perception et le contrôle exigent des réponses immédiates.

La planification plus lente, l'analyse de flotte, l'optimisation à long terme ou les mises à jour de modèles peuvent être effectuées ailleurs.

Le cloud s'occupe de ce qui peut attendre.

L'edge s'occupe de ce qui ne peut pas attendre.

Modèle 3 : entraînement dans le cloud, inférence en edge

C'est probablement le flux de travail en IA edge le plus courant.

Une infrastructure GPU puissante entraîne ou affine le modèle.

Le modèle final est ensuite compressé ou quantifié.

The optimized version is then deployed onto edge hardware.

When enough new data has accumulated, training runs again.

Pattern 4: small local model, larger cloud model

A local language model can handle simple or private requests.

More difficult prompts can be routed to a larger remote model.

The application decides which capability is required rather than forcing every request through the same model.

Pattern 5: edge cache with cloud fallback

A device may run local inference when connectivity is poor, then use remote inference when a stronger connection is available.

This is useful where offline reliability matters but local hardware cannot match the best remote model.

Pattern 6: edge preprocessing, cloud inference

Sometimes the model itself does not need to move to the edge.

Preprocessing does.

A device might resize images, extract regions of interest, filter events, compress inputs, or remove unnecessary data before sending the result to cloud inference.

This can reduce bandwidth and improve privacy while retaining the larger cloud model.

When cloud GPUs make more sense

There are several cases where trying to force AI onto edge hardware creates more problems than it solves.

Your model is still changing

If you have not settled on the model, do not freeze the hardware around it.

You may need more memory next month.

A new architecture may use unsupported operators.

A better quantization scheme may arrive.

The model may gain vision.

During experimentation, programmable GPU infrastructure gives you room to move.

The model is too large

Memory is unforgiving.

A device with 16 GB of shared memory does not become a 32 GB device because the model is important.

Cloud infrastructure lets you select larger GPUs or several GPUs when the workload grows.

Hivenet's current Compute path provides GPU and CPU instances for model experiments, inference, fine-tuning, notebooks, and other programmable workloads.

The principle matters more than any particular GPU: remote infrastructure can change without replacing a physical fleet.

Traffic is unpredictable

Edge hardware is capacity you already bought.

If demand is low, it sits idle.

If demand exceeds its capacity, you need another device.

Cloud capacity is easier to change around variable demand.

This is particularly useful for batch jobs, temporary projects, experiments, and workloads that spike.

Several users need the same model

A personal AI feature can run on one user's device.

A company-wide model may be easier to operate centrally.

Centralized inference can share a larger accelerator among users and concentrate updates, monitoring, access control, and model management.

Whether that central service sits inside the company's own building or in cloud infrastructure is a separate decision.

You need training or fine-tuning

Large training and fine-tuning workloads usually demand more memory, throughput, and flexibility than edge devices were designed to provide.

Use the edge for the deployed model.

Use larger compute for the work required to create it.

The model is part of a larger cloud application

If your data already lives in cloud databases and storage and most application logic runs there, moving inference to thousands of edge devices may add network paths rather than remove them.

Architecture should follow where the input and users actually are.

When local edge hardware clearly makes more sense

The strongest local cases are equally easy to recognize.

Use edge hardware seriously when:

• losing network access cannot stop the application;
• raw data should not routinely leave the device or site;
• the application has hard local-response requirements;
• continuously uploading sensor data would be expensive or wasteful;
• the model is stable;
• the workload runs often enough to justify dedicated hardware;
• power and physical constraints favor a specialized processor;
• the device already contains hardware capable of running the model;
• integration with sensors or physical control requires local execution.

These are architecture requirements, not marketing preferences.

Edge AI hardware decision matrix

‍

‍

Do not select the processor first

A surprising amount of edge AI development starts with someone buying a board.

Then the team tries to fit the application onto it.

Reverse the order.

1. Define the decision the model must make

What exactly is the output?

Classification?

Detection?

Generated text?

Control signal?

Anomaly score?

2. Define the response requirement

How quickly does the answer need to arrive?

And what happens if it arrives late?

3. Decide whether the application must survive network loss

This can settle the local-versus-remote question immediately.

4. Measure the input volume

How much data would you send remotely?

A tiny sensor sample and twelve 4K camera streams create very different network problems.

5. Select the model

Do not size hardware around an imaginary model.

Use the architecture you actually expect to deploy.

6. Measure memory

Include model weights and runtime state.

For LLMs, include the KV cache and realistic context length.

7. Check the software path

Does your inference runtime support the target processor?

Are all important operators available?

What happens when they are not?

8. Define the power and thermal envelope

Ten watts and 100 watts create different hardware categories.

9. Plan updates before deployment

How will the model change?

How will you roll back a bad release?

How will you know whether the fleet is healthy?

10. Benchmark locally and remotely

Use the same model and realistic input.

Measure:

• end-to-end latency;
• model quality;
• memory;
• power if relevant;
• network traffic;
• throughput;
• operational cost.

Then choose the architecture.

Where Hivenet fits into an edge architecture

Hivenet does not sell embedded AI boards or NPUs.

That makes its role in this decision fairly clear.

If a workload genuinely belongs on a sensor, robot, laptop, or industrial controller, it should run there.

Remote infrastructure becomes useful for the parts that do not.

Compute with Hivenet provides programmable GPU and CPU infrastructure for development, experiments, model testing, fine-tuning, larger inference workloads, and the backend of hybrid AI systems. Hivenet currently exposes self-managed compute separately from its managed inference path.

The Hivenet Inference API is the managed alternative when the application needs a model endpoint rather than direct control over the server and accelerator. Current endpoints use an OpenAI-compatible interface and regional deployment paths.

For workloads involving sensitive data, Private AI with Hivenet adds a guided path around model choice, data handling, region, infrastructure, and deployment rather than assuming a public endpoint is always appropriate.

A hybrid design could therefore look like:

sensor → local edge model → selected request → Hivenet inference or Compute → result

or:

Hivenet GPU training → optimized model → edge deployment

or:

local model for ordinary requests → remote model for difficult requests.

Hivenet does not need to own the entire AI path to be useful in it.

Edge AI is where hardware decisions become physical

Cloud infrastructure makes hardware feel abstract.

If a GPU is too small, you choose another one.

If the model changes, you change the environment.

If utilization falls, you stop the instance.

Edge systems do not have that luxury.

Once hardware is attached to a robot, mounted above a production line, installed in 2,000 shops, or sealed inside a product, processor decisions acquire a physical lifetime.

That makes flexibility, memory headroom, software support, and maintainability more important than chasing the best benchmark available on purchase day.

It also makes specialization more valuable when the problem is stable.

A tiny device that performs one inference task for seven years has little reason to carry the flexibility of a large GPU.

A robotics platform expected to adopt larger multimodal models every year probably does.

This is the same specialization-versus-flexibility trade-off that runs through the rest of this series, but edge deployment gives the trade-off a screwdriver and a power cable.

FAQ about edge AI hardware

What is edge AI hardware?

Edge AI hardware is computing hardware used to run machine-learning workloads close to where data is generated or used. It can include CPUs, GPUs, NPUs, FPGAs, microcontrollers, embedded systems, accelerator cards, local workstations, and edge servers.

What is an edge AI device?

An edge AI device is a device that performs some AI processing locally or near the data source rather than depending entirely on a remote cloud service. Examples range from wearables and cameras to robots, industrial computers, laptops, and local servers.

What is edge inference?

Edge inference means running a trained machine-learning model on edge hardware near the user, sensor, application, or physical system. Training is usually performed elsewhere and the resulting model is deployed to the edge.

What hardware is best for edge AI?

There is no universal best hardware. Microcontrollers suit tiny, low-power models. NPUs suit supported neural-network workloads under constrained power. Embedded GPUs provide more model flexibility and compute. FPGAs and ASICs can suit highly specialized applications. Local GPU servers support larger models.

Is an NPU an edge AI accelerator?

It can be. NPUs are commonly used for edge and on-device AI because they specialize in neural-network execution, but NPUs also exist in larger systems. See what an NPU is for the distinction.

Can GPUs be used for edge AI?

Yes. Embedded GPU platforms such as NVIDIA Jetson are explicitly designed for robotics and edge AI, and local workstations or servers can also provide GPU inference at the edge.

Can edge AI run without the internet?

Yes, if the model, runtime, and required data are stored locally. Offline operation is one of the main reasons to deploy inference at the edge.

Is edge AI faster than cloud AI?

Sometimes. Edge AI avoids remote network round trips, but local hardware may execute the model more slowly than a cloud accelerator. Compare complete end-to-end latency rather than network latency alone.

Is edge AI more private?

It can reduce the amount of raw data sent off the device or site. That can improve data minimization. It does not automatically make a system secure or compliant; local devices still need appropriate security and operational controls.

Is edge AI cheaper than cloud AI?

It can be for stable, continuously used workloads because dedicated hardware can be amortized over time. Cloud compute can be cheaper for variable, experimental, or bursty workloads because capacity does not need to be purchased and maintained locally.

Can I run an LLM at the edge?

Yes, if the hardware has enough memory, the runtime supports the model, and the resulting latency is acceptable. Small quantized models can run on client and embedded devices, while larger edge GPU systems can support considerably larger models.

Should AI training happen at the edge?

Usually not for substantial models. Training tends to require more memory, throughput, software flexibility, and data access than deployed edge hardware provides. A common architecture trains or fine-tunes on larger infrastructure and deploys an optimized model for local inference.

What is the difference between edge AI and on-device AI?

On-device AI runs directly on the end device. Edge AI is broader and can include nearby gateways, industrial computers, or local servers positioned close to the data source.

Will edge AI replace cloud AI?

Probably not. The two solve different constraints and can work together. Local inference handles tasks that benefit from proximity, while cloud or data-center infrastructure handles larger models, training, centralized services, heavy compute, and variable demand.

The best edge architecture sends only the work that needs to travel

The strongest argument for edge AI is not that everything should become local.

It is that sending everything somewhere else can be wasteful.

Some decisions need to happen beside the sensor.

Some data should stay where it was created.

Some systems need to survive a broken connection.

And some small models can run on dedicated hardware far more sensibly than on a distant GPU.

Other workloads belong elsewhere.

Large models need memory.

Training needs throughput.

Experiments need flexibility.

Shared services need centralized capacity.

Variable demand benefits from hardware that does not sit idle after you have paid for it.

The useful architecture draws that boundary deliberately.

Run locally what benefits from being local.

Move remotely what benefits from larger, more flexible infrastructure.

And when the workload spans both worlds, design the edge and cloud as parts of the same system rather than competing answers.

Continue with NPU vs GPU for AI workloads, what an NPU is, the practical guide to AI accelerators, CPU vs GPU vs NPU, AI accelerators vs GPUs for inference, TOPS vs FLOPS, et matériel d'entraînement vs matériel d'inférence.

‍