Machine Learning at the edge: What's actually feasible on small devices?

What is machine learning at the edge?

Machine learning at the edge means running a trained model on a device, gateway or nearby edge system instead of sending every inference request and every piece of raw data to a centralized cloud service.

Local inference can reduce response time, limit bandwidth use, keep sensitive raw data closer to the source and allow a product to continue making decisions when connectivity is slow, intermittent or unavailable.

The practical question is not whether AI can run on a small device. It is which model and workload can run reliably within the device's RAM, storage, compute, latency and power constraints while leaving enough headroom for the rest of the product.

Machine learning can run surprisingly well on small edge devices, but only when the workload is matched to the hardware. Sensor classification, anomaly detection, gesture recognition, keyword spotting and even limited computer vision are practical on modern microcontrollers. Large generative models and high-resolution real-time vision are not.

The engineering question is therefore not simply, “Can this device run AI?” It is: can the complete inference pipeline fit within the available RAM, flash, compute, latency and power budget while leaving enough resources for the rest of the product?

This distinction is central to edge AI deployment. A model that runs successfully in a notebook or on a development board may still be unsuitable for a battery-powered production device once preprocessing, connectivity, security, firmware and over-the-air updates are included.

Key takeaways

• Small machine-learning models can run directly on 32-bit microcontrollers without an operating system or cloud connection.
• RAM is often the first hard constraint because model activations, tensor buffers, sensor windows and application firmware all compete for memory.
• Int8 quantization can dramatically reduce model size and make optimized integer inference practical on embedded processors.
• Time-series sensing and audio are usually easier to fit than computer vision because image inputs create much larger memory and compute requirements.
• A dedicated NPU or neural accelerator changes what is feasible, especially for vision and always-on inference.
• The correct decision is made by benchmarking the complete pipeline on the target hardware, not by comparing model file sizes alone.

What counts as a small edge device?

“Edge device” covers a wide spectrum. At one end are deeply constrained microcontrollers with tens or hundreds of kilobytes of RAM. At the other are embedded Linux systems with multicore processors, gigabytes of memory and dedicated AI accelerators. Treating them as one hardware category leads to unrealistic expectations.

For truly constrained systems, Google’s LiteRT for Microcontrollers is designed for 32-bit platforms with very small memory footprints and does not require an operating system or dynamic memory allocation. That makes embedded inference possible far below the resource level normally associated with AI applications.

What can small devices actually handle?

Small devices are best suited to narrow, repeatable inference tasks with compact inputs and clearly defined outputs. Commonly feasible workloads include:

• Classification of sensor or signal patterns
• Simple regression and estimation
• Anomaly detection and predictive-maintenance signals
• Event detection and state recognition
• Gesture and activity recognition
• Keyword spotting and compact audio classification
• Low-resolution image classification on capable hardware
• Other lightweight inference tasks matched to the available compute, memory and energy budget

What is TinyML?

TinyML refers to machine-learning applications engineered for highly constrained embedded devices, particularly microcontrollers with limited RAM, flash storage, processing performance and energy budgets.

Unlike cloud AI, where additional compute can often be provisioned, TinyML requires the model, preprocessing pipeline and runtime to fit a fixed hardware envelope. Model efficiency therefore becomes a first-class product requirement rather than a later optimization exercise.

What is actually feasible on a microcontroller?

The most practical TinyML workloads are narrow, repetitive problems where the input is small and the output is simple. The model does not need to “understand” the world. It needs to distinguish a limited number of states reliably enough to trigger a useful action.

Sensor classification and anomaly detection

Accelerometers, vibration sensors, current sensors, temperature sensors and other time-series inputs are excellent candidates for on-device ML. Feature extraction can reduce raw signals into a compact representation, after which a small classifier or anomaly detector can identify events such as unusual vibration, motion patterns or operating states.

Gesture and activity recognition

IMU-based gesture recognition can be extremely compact because the input consists of a short window of accelerometer or gyroscope samples rather than a large image. This makes it well suited to wearables, appliances, industrial tools and human-machine interfaces.

Keyword spotting and simple audio classification

Always-on wake words and a small vocabulary of commands are feasible on many Cortex-M-class processors. The audio is typically transformed into features such as MFCCs before inference, allowing a compact neural network to classify a short audio window.

Low-resolution computer vision

Vision is feasible, but the limits appear much sooner. Small grayscale images and highly constrained classification problems can run on capable MCUs. Increasing resolution, RGB channels, object count or detection complexity quickly increases the activation memory, model size and multiply-accumulate workload.

Representative TinyML resource requirements

Published embedded benchmarks illustrate how different workloads change the resource budget. The figures below are representative Edge Impulse benchmarks using 8-bit quantized models and should not be treated as fixed requirements for every implementation.

Source: Edge Impulse inference performance benchmarks. Their published figures exclude some application overhead such as boot code, peripheral drivers and the raw input buffer, which is exactly why production sizing must include more than the inference engine itself.

What limits machine learning on small devices?

Edge-ML feasibility is governed by the complete product resource budget, not only by the size of the model file.

• RAM: model activations, tensor buffers, sensor windows, stacks and application data must fit alongside the rest of the firmware.
• Storage: the model competes with firmware, bootloaders, security assets, logs and OTA update space in flash or other non-volatile storage.
• Compute: inference and preprocessing must complete within the required response time without starving real-time product functions.
• Power: continuous sensing and inference can materially affect battery life even when the model technically fits.
• Connectivity: devices may need to operate through network loss, which can make local inference or graceful fallback essential.

The real constraint is often memory, not just compute

Engineers often focus on processor frequency or TOPS, but a model can fail before compute becomes the problem. The device needs memory for model weights, intermediate activations, the tensor arena, sensor buffers, RTOS objects or stacks, communications, security libraries, application state and update logic.

A 200 KB model file does not mean the device needs only 200 KB of memory. Model weights may live in flash while activations consume RAM, and some layers can create peak working-memory requirements that are much larger than the average. Camera frame buffers can dominate RAM before inference begins.

This is why hardware and ML architecture should be evaluated together as part of the embedded hardware and firmware design, not as independent decisions.

How are models optimized for the edge?

The goal of edge-model optimization is to reduce memory, storage, latency and energy requirements while preserving enough accuracy for the product use case.

• Quantization: reduces numerical precision, commonly from floating point to 8-bit integers, to lower storage and compute requirements.
• Pruning: removes parameters or connections that contribute little to the model's output.
• Compression: reduces the model's storage footprint and, depending on the approach, can also simplify deployment.
• Knowledge distillation: trains a smaller student model to reproduce useful behavior learned by a larger teacher model.
• Efficient architectures: chooses networks and operators designed for constrained inference instead of shrinking an unsuitable large model after training.
• Hardware-aware optimization: selects operators and kernels that map efficiently to the target MCU, DSP, SIMD engine or NPU.

These techniques should be evaluated together with measured accuracy. A smaller model is only useful if it still meets the product's detection, classification or prediction requirements.

Why int8 quantization changes the feasibility equation

A neural network trained with 32-bit floating-point weights can often be converted to 8-bit integer representation for deployment. In simple storage terms, int8 weights use one quarter of the bytes required by float32 weights, although the total application memory reduction depends on the architecture and runtime buffers.

Quantization also matters because embedded inference libraries can use optimized integer kernels. Arm’s CMSIS-NN, for example, provides neural-network kernels optimized for Cortex-M processors and supports int8/int16 quantization workflows.

The trade-off is accuracy. Post-training quantization may be sufficient for one model and unacceptable for another. Quantization-aware training can recover accuracy when the model is sensitive to reduced numerical precision.

What becomes difficult very quickly?

• High-resolution vision: larger images create larger input and activation buffers and substantially more convolutional work.
• Continuous object detection: detection is generally more demanding than simple image classification because the model must identify both classes and locations.
• Large recurrent or transformer models: parameter count, activation memory and attention-related workloads can exceed MCU capabilities rapidly.
• General-purpose LLM inference: modern large language models require a different class of memory and compute than tiny microcontrollers provide.
• On-device training: training requires gradients, optimizer state and substantially more memory and compute than inference. On very small devices, inference is the realistic target.

When a workload crosses these limits, the design should move up the hardware ladder to an accelerator-equipped MCU, application processor, gateway or embedded Linux system—or move part of the workload to the cloud.

Power can invalidate an otherwise successful model

A model can fit in memory and meet latency requirements but still be unsuitable for a battery-powered product. Inference frequency matters as much as inference cost. A model that runs for 20 milliseconds once every minute has a very different energy profile from one that runs continuously.

The complete power budget should include sensor sampling, preprocessing, inference, wireless communication and sleep behavior. Often the most efficient design uses a low-power trigger or simple first-stage detector to wake a more expensive model only when needed.

Edge, cloud or hybrid: where should inference run?

Use edge inference when latency, intermittent connectivity, privacy or bandwidth makes local decisions valuable. Keep compute-intensive training, fleet-wide analytics, historical analysis and centralized model management in the cloud. In many connected products, a hybrid architecture provides the best balance.

In many connected products, the best architecture is hybrid. The device detects a local event and takes an immediate action, while selected telemetry is sent to the IoT cloud for fleet analytics, model monitoring, retraining and lifecycle management.

A practical feasibility workflow

1. Define the decision. Specify exactly what the device must classify, predict or detect and how quickly the result is needed.
2. Characterize the input. Measure sample rate, sensor channels, audio window, image resolution and preprocessing requirements.
3. Establish the hardware budget. Reserve RAM, flash, CPU time and power for the existing firmware before allocating resources to ML.
4. Train the smallest useful baseline. Start with a compact model and only add complexity when measured accuracy requires it.
5. Quantize and optimize. Evaluate int8 quantization, operator support, optimized kernels, pruning or architecture changes.
6. Benchmark on the target MCU. Measure peak RAM, final binary size, inference latency and CPU duty cycle using production-like firmware.
7. Measure energy. Test the actual inference schedule, sensor duty cycle and radio activity on the target board.
8. Plan the lifecycle. Include secure model packaging, versioning, OTA deployment, rollback and telemetry for detecting model degradation.

How do you validate an edge ML model?

An edge model should be validated on the actual target hardware and with representative real-world data. Desktop accuracy alone is not a production-readiness metric.

• Measure accuracy, precision/recall or other task-specific metrics on representative device data.
• Benchmark preprocessing and inference time on the target processor or accelerator.
• Measure peak RAM usage, final firmware size and model storage requirements.
• Measure power consumption and battery impact under realistic duty cycles.
• Test behavior during connectivity loss and degraded network conditions.
• Validate secure model updates, versioning, rollback and recovery procedures.
• Monitor deployed model performance so drift, false positives and changing field conditions can be detected.

Do not design the model before designing the product constraints

Edge ML works best when data scientists, embedded engineers and product architects make the feasibility decisions together. A model team optimizing only for accuracy may produce an architecture that cannot fit on the device. An embedded team optimizing only for BOM cost may select hardware that leaves no headroom for future model updates.

A better approach is hardware-software co-design: set an explicit model budget for RAM, flash, latency and power, evaluate representative data early, and keep enough headroom for firmware growth and future revisions.

From prototype to production edge AI

Getting one inference to run is only the prototype milestone. Production deployment must also handle sensor quality, calibration, false positives, environmental variation, secure boot, device identity, model authenticity, OTA updates, rollback, fleet monitoring and long-term support.

Thinxtream’s capabilities span IoT devices, firmware and embedded applications, IoT machine learning, cloud integration and IoT security. This makes it possible to evaluate edge-ML feasibility as part of the complete product architecture rather than as an isolated model-development exercise.

For broader product programs, Thinxtream also provides product development services covering embedded systems, cloud, applications, AI/machine learning and QA/testing across the product lifecycle.