Neural Processing Units (NPUs)

An NPU is a matrix engine surrounded by memory. The arithmetic core is an array of multiply-accumulate units, often arranged as a systolic structure where operands flow between neighbours so each value is read once and reused many times. Throughput depends on keeping that array fed: weights and activations must arrive in the shape and cadence the dataflow expects.

Local scratchpads and tiling are the mechanism. A tensor too large for on-chip memory is split into tiles; each tile is loaded, used for as many accumulations as possible, then discarded. The compiler chooses tile sizes, and a poor choice multiplies traffic to off-chip memory. This is why bandwidth, not arithmetic, is usually the limit on real networks, and why a smaller tensor with fewer reuse opportunities can be slower than a larger one.

Precision support defines what ran without conversion. int8 is the universal baseline; many engines add int16 or fp16 for accuracy-sensitive layers, and newer designs add block-scaled 4-bit formats for language models. A model in an unsupported precision is either converted — with a quality and speed consequence — or executed elsewhere, which fragments the graph.

Operator support is a hardware property expressed through the compiler. Convolution, depthwise convolution, fully connected and common activations are almost always native. Elementwise chains, unusual reshaping, dynamic shapes, top-k, non-maximum suppression and custom pre-processing frequently are not. Each unsupported node becomes a partition boundary with associated memory copies and synchronisation.

Sparsity and structured pruning are supported unevenly. An engine with 2:4 structured sparsity hardware can deliver a real speedup; an engine without it treats sparse weights as dense and gains nothing. Before designing a pruning strategy, confirm that the target hardware and compiler exploit the pattern you intend to create.

The compiler is where portability ends. A framework graph is imported, validated, fused, quantised, tiled and lowered to a hardware binary. If a node is unsupported, the compiler either rejects the graph or partitions it, delegating unsupported regions back to CPU. Partitioning is functional but expensive: every boundary requires a layout conversion, a memory copy and a synchronisation point.

Graph design should therefore target the engine from the start. Prefer operators the compiler fuses; avoid dynamic shapes and data-dependent control flow; keep post-processing such as non-maximum suppression on the CPU deliberately, as a clearly measured cost rather than an accidental partition in the middle of the network. Where possible, move preprocessing — resize, normalisation, colour conversion — to the accelerator or to a fused pre-processing stage.

Version dependency is a real operational constraint. Vendor compilers, runtimes and device drivers are versioned independently, and a binary compiled for one combination may fail or silently fall back on another. Pin the toolchain per release, record the exact versions in the model artefact, and validate on the device firmware that customers actually run.

Measurement must attribute time correctly. End-to-end latency includes input transfer, inference, output transfer and any CPU post-processing. Profilers often report only the inference kernel, which flatters the result. Instrument boundaries, measure on device under realistic thermal conditions, and report the sum rather than the engine number.

Finally, treat fallback as a defect class rather than a detail. Track the number of CPU-executed nodes per model version, alert when it increases, and include it in release criteria. A single unsupported activation introduced during model experimentation can erase the entire benefit of the accelerator.

Real devices have several engines, and the best partition is not always the accelerator. Prefill of a language model is compute-heavy and suits the NPU; decode is bandwidth-bound and may run as well or better on the GPU. Vision pipelines may split between NPU for the backbone and GPU for post-processing, while audio front ends run on a DSP.

Scheduling across engines requires a cost model and measurement. Assign by measured per-stage cost on the target, account for queueing when several workloads share an engine, and avoid ping-ponging intermediate tensors between memory domains. Where the platform supports it, use asynchronous execution with explicit fences so engines overlap rather than serialise.

Sustained performance is the number that matters. A phone NPU can burst to peak for a short period before thermal management reduces clocks; a fanless laptop sustains more but still derates under long inference. Measure after minutes of realistic load, inside the product enclosure, and design the experience around the sustained figure.

Energy per inference should be measured alongside latency. Accelerators win on energy only when they are kept busy; an engine that idles between small jobs while the CPU prepares data can consume more than a single efficient CPU path. Batch work where latency permits, and keep the accelerator fed.

Portfolio-level thinking completes the picture. A product family will span several SoCs with different engines. Maintaining a small ladder of models and a per-target conversion pipeline is more sustainable than betting on one accelerator, and it makes it possible to fall back when a vendor changes support.

The Roofline model separates compute-limited and memory-traffic-limited operating regions. Define arithmetic intensity using an explicit convention for operations and measured bytes transferred at a named memory level. The crossover intensity is peak achievable operations per second divided by sustained bandwidth in bytes per second. It is hardware- and workload-specific, not a universal 50-operations-per-byte threshold. For the illustrative 4-TOPS, 50-GB/s configuration below, the nominal crossover is 80 operations per byte. Fusion, tiling and on-chip reuse change DRAM traffic, so assess the compiled graph rather than counting source-level tensor accesses.

Consider a hypothetical NPU with a 4 TOPS peak performance and 50 GB/s of LPDDR5 bandwidth. If an activation layer processes a feature map requiring 100 million operations and transferring 50 megabytes of data, the system achieves an intensity of 2 operations per byte. Even if the NPU could compute at 4 TOPS, it is capped at 100 GOPS by the memory throughput limit (50 GB/s * 2 ops/byte). Engineers must treat these limits as hard physical laws, not soft suggestions, when planning the pipeline.

DRAM traffic is further exacerbated by weight loading. In a large model, weights cannot reside entirely in on-chip SRAM, requiring constant swapping. If the model architecture does not permit high-locality reuse, the NPU becomes a data-shuttling engine rather than a compute engine. The cost of this movement is not just the latency, but the energetic penalty of moving electrons across the board to the DRAM controller, which often dominates the power budget.

Measuring the operational intensity requires granular profiling of the graph. You must instrument the hardware's data-transfer controllers to see how many bytes are actually crossing the memory interface versus how many are reused via local scratchpads. If the profiler reports high DRAM bus utilization alongside low MAC array utilization, the model is definitively memory-bound. Strategies to address this include increasing batch sizes to improve reuse, or model compression techniques that increase the operational intensity.

A common methodological limitation occurs when developers ignore the non-deterministic nature of modern interconnects. When the NPU shares memory bandwidth with the display controller or CPU, the actual available bandwidth fluctuates. Troubleshooting this requires isolating the NPU load during a controlled stress test to establish the baseline roofline. If performance is inconsistent, verify if the operating system is dynamically throttling clocks due to thermal load, which shifts the entire roofline curve downward.

Finally, expect to encounter cache effects that blur the theoretical limits. Small tensors might fit entirely within an L3 cache or the NPU's global buffer, leading to transient spikes in performance. Do not use these bursts to estimate throughput for larger production inputs; always baseline against the worst-case tensor size that the application will realistically encounter in the field.

SRAM tiling is the process of partitioning a large tensor into smaller sub-volumes that fit within the NPU's local scratchpad memory. Because on-chip SRAM is significantly faster and more energy-efficient than DRAM, the compiler's primary optimization target is maximizing the time a tile resides in the scratchpad. A 'halo' or 'overlap' region occurs during convolution where the sliding window requires input pixels that reside on an adjacent tile. Efficient architectures use hardware-managed DMA engines to prefetch these halos while the current tile is being processed to hide latency.

Effective tiling requires a precise understanding of the NPU's memory hierarchy. If the NPU has 4MB of local SRAM, you cannot simply fill it with 4MB of data; space must be reserved for weight buffers, output accumulations, and double-buffering structures. A hypothetical NPU with a 4MB buffer might only allow 2.5MB for input feature maps to ensure sufficient space for ping-pong buffering, where the DMA engine loads the next tile into one half of the memory while the MAC array computes on the other.

The halo reuse problem is critical for performance. In a 3x3 convolution with a stride of 1, the halo represents the boundary elements required by the kernel's spatial extent. If tiling is performed poorly, these halo elements are redundantly fetched from DRAM, doubling or tripling the bandwidth requirements. Sophisticated compilers optimize this by generating 'ghost zones' in local memory, where the boundary of one tile is replicated to provide context for the next, reducing the need for subsequent DRAM fetches.

Troubleshooting tiling efficiency involves analyzing the ratio of compute-to-DMA time. If the NPU shows significant idle cycles, the DMA engine is likely struggling to keep up with the compute array. Check the memory layout: tensors formatted in non-contiguous memory layouts—such as channels-last (NHWC)—often enable better tiling efficiency than NCHW because they minimize strided access patterns, which are inherently slower for DMA controllers.

A common methodological limitation is the assumption that tile sizes are static. In reality, compiler heuristics often adjust tile sizes based on the specific kernel shape and padding requirements. When diagnosing performance dips, inspect the generated assembly code or internal compiler graph to see if the tiling strategy changed unexpectedly. Some compilers force a 'safe' sub-optimal tile size for certain operators, causing a sharp drop in performance relative to others.

Data residency is the goal of a well-designed graph. The objective is to ensure the output of layer N is consumed by layer N+1 without being flushed to DRAM. If the hardware supports it, design your model such that intermediate tensors are sized to remain in the scratchpad. If a tensor is too large, the NPU forces an intermediate write-back, effectively turning the SRAM into an expensive buffer for the DRAM, which destroys the performance benefit of local memory.

Operator legality is a binary condition dictated by the hardware's fixed-function logic. An NPU supports specific mathematical structures; for example, a 3x3 convolution with a stride of 1 is usually native, while a 3x3 convolution with a dilation factor of 4 might force a fallback to the CPU. Each unsupported operator acts as a 'break' in the dataflow. The cost is not just the latency of the CPU execution, but the penalty of transitioning from the NPU's memory layout to the CPU's memory layout, which often involves a full reordering or copy of the tensor.

Quantization parameter contracts are equally vital. Most NPUs rely on integer arithmetic, typically int8. The contract stipulates how floating-point numbers are mapped to these integers: specifically, the 'scale' and 'zero-point' parameters. If the model's quantization parameters do not align with the NPU's supported formats—such as needing asymmetric quantization when the hardware only supports symmetric—the compiler will insert a 'dequantize-requantize' node. This node is a silent performance killer that adds significant latency.

For a numerical example, assume an NPU uses a symmetric int8 format with a scale factor of 0.05. If the model output requires a scale of 0.02, the compiler must perform a division and cast operation at the boundary. Doing this for a 224x224x64 feature map requires 3.2 million operations of overhead just for the scaling step, which can take several milliseconds on the scalar CPU, even if the surrounding layers are fast.

To test operator legality, create a 'canary' graph containing only the specific operator in question. If the toolchain reports that the operator was partitioned to the CPU, you have a hard stop. Do not assume that a later version of the compiler will support it. Always check the vendor-provided support matrix, but supplement this with empirical evidence from a dry-run of your specific layer parameters.

Methodological limitations arise when developers treat quantization as a post-training afterthought. If you quantize your model without considering the NPU's specific calibration requirements, you may incur severe accuracy loss or, worse, performance fragmentation. Use quantization-aware training to ensure the model naturally respects the hardware's parameter contracts, avoiding the need for expensive conversion layers during execution.

Troubleshooting performance degradation at graph nodes requires logging the compiler's fusion pass. Many compilers perform 'operator fusion,' where multiple operations like 'Convolution + ReLU + Addition' are combined into one hardware instruction. If an unsupported operator breaks this fusion chain, the performance cost is the sum of the overhead of all fragmented parts. Always aim for maximum fusion; if a node breaks the chain, evaluate if it can be replaced with a functionally equivalent sequence that the hardware supports natively.

Partitioning occurs when the compiler determines that a sub-graph cannot execute on the NPU and routes it to the CPU or GPU. This boundary is the most frequent source of latency jitter in production models. At this interface, the system must perform a 'sync,' which involves flushing the NPU's internal write buffers, synchronizing cache states, and moving the tensor memory into a format the CPU can read. This entire process is a high-latency event that can take 50 to 500 microseconds depending on the bus protocol.

The latency crossover point is the threshold where the cost of the copy exceeds the speed benefit of using a specific hardware unit. For instance, if you have a small 1x1 convolution, it might be faster to compute it on the CPU than to move the data to the NPU and back, even if the NPU is technically capable of executing the operation. Understanding this crossover requires building a performance model that accounts for the latency of the system bus and memory copy overheads.

Consider an example where moving 1MB of activation data across the SoC interconnect takes 100 microseconds. If the NPU executes the layer in 50 microseconds, but the CPU executes it in 150 microseconds, the total time for the NPU path is 150 microseconds (copy + compute) versus 150 microseconds for the CPU path. At this point, the NPU is not helping. If the data transfer increases, the NPU path becomes strictly worse.

Lab procedures for measuring this involve using high-resolution hardware timers to profile the execution of the entire pipeline. Focus on the time spent in 'wait' states or 'synchronization' blocks. If you notice the total inference time is significantly higher than the sum of the kernel execution times reported by the vendor profiler, the remaining time is almost certainly spent in these boundary transitions.

Methodological limitations are common when using aggregate timers. A system-level timer might show a 10ms inference, but individual layer profiling is needed to see that 3ms of that time is spent exclusively in data transfers between the NPU and CPU. You cannot optimize the kernel if the bottleneck is the bus transfer. The troubleshooting step is to minimize the number of partition boundaries, even if it requires re-writing parts of the graph or accepting slight changes in mathematical precision.

Minimizing copies requires optimizing the memory layout to match the NPU's preference. If the NPU requires NC/32HW (a packed channel layout) and the CPU expects NCHW, you are guaranteed a copy at every switch. If the entire pipeline can be kept in the NPU's native format—even for CPU tasks if the CPU supports it—you eliminate the overhead. Always treat the memory layout as a contract that must be maintained across the entire graph to avoid unnecessary transformations.

Stall diagnosis is the process of identifying why the MAC array is not achieving its maximum throughput. Modern NPUs provide performance counters that track stalls caused by various factors, such as memory bus arbitration, DRAM refresh cycles, scratchpad bank conflicts, or waiting for synchronization fences. By querying these counters after an inference, you can mathematically prove where the performance is bleeding. A 'DRAM_STALL' counter incrementing rapidly is a clear indicator that your model is bandwidth-constrained, while an 'ISSUE_STALL' counter suggests a bottleneck in the controller or instruction sequencer.

Fair benchmarking requires a standardized methodology that accounts for realistic thermal and system conditions. A common pitfall is benchmarking on a cold device with high-performance power profiles, which gives a 'hero number' that is impossible to sustain in a real-world scenario. Instead, run the inference in a continuous loop for at least 10 minutes, with the screen brightness and background tasks active, to observe the steady-state performance. This is the only figure that matters for an end-user experience.

Consider a hypothetical lab procedure: run 1,000 inference cycles and record the time of each. The variance in your data is as important as the mean. If the standard deviation is high, you have non-deterministic performance, likely due to background task interference or thermal throttling. The benchmarking protocol must report the p99 latency, not just the average, to guarantee a consistent user experience. If the p99 is significantly higher than the mean, your model is not yet ready for production.

Methodological limitations arise when comparing NPU performance across different SoCs using different compilers. Since the compiler generates the binary, you are benchmarking the entire toolchain, not just the silicon. To perform a fair comparison, you must pin the compilers to similar optimization levels and ensure that the graph fusion passes are equally aggressive for both platforms. If one platform supports more operators, it has an inherent, unfair advantage; normalize the graph to the lowest common denominator for a pure silicon comparison.

Troubleshooting stall data requires looking at the ratio of computation to stall cycles. If compute utilization is below 50% and DRAM stalls are high, you have a software architectural issue: the model simply isn't using the NPU's internal parallelism correctly. If compute utilization is high but latency is still high, the model is simply too large for the hardware. You cannot optimize a 10-billion parameter model into a 100-millisecond window on a low-power NPU; you must either scale the model or accept higher latency.

When collecting data, distinguish between 'busy' cycles and 'productive' cycles. A cycle is busy if the hardware clock is running, but it is only productive if it performs useful arithmetic. Many NPUs clock-gate to save power, so simple busy-cycle counters can be deceptive. Always correlate counter data with the total energy consumed per inference, which is a better measure of actual workload intensity and hardware efficiency than clock-based utilization.

Contention budgets manage the reality that an NPU is rarely the only client on a memory bus or an execution fabric. In a modern smartphone or laptop, the NPU competes with the ISP, GPU, and display engines for access to shared LPDDR bandwidth and internal crossbars. If your model uses 90% of the available bandwidth, it will starve the display controller, causing frame drops, or interfere with camera processing, leading to dropped frames or preview latency. You must establish a contention budget, which is a predefined share of the total system resources that the NPU is allowed to consume.

To define this budget, you must first profile the entire SoC's traffic. Use system-level tools to observe the bandwidth consumption of the camera, display, and GPU during normal operation. If the total peak demand exceeds the memory subsystem's capability, the hardware arbiter will start delaying requests. Since the NPU is often latency-sensitive, these delays cascade through the neural network, potentially causing a performance drop that exceeds the actual reduction in available bandwidth.

Example calculation: A system has 80 GB/s peak LPDDR5 bandwidth. The display requires 10 GB/s, the ISP requires 20 GB/s, and the GPU consumes 20 GB/s. This leaves 30 GB/s for the NPU and CPU. If your model requires 40 GB/s, you have a contention problem. You must either reduce the model's throughput requirements—perhaps through aggressive weight pruning—or implement an execution strategy where the NPU operates at a lower clock frequency to throttle its own bandwidth demand, effectively trading latency for system stability.

Lab procedures should involve 'load-injection' testing. Run the inference while concurrently stressing the GPU with a 3D workload and the camera with a 4K frame stream. Measure the performance delta. If the NPU latency increases by 20% in this stressed environment, you have not adequately budgeted for contention. Troubleshooting requires adjusting the hardware task priorities, if the platform allows, or modifying the model's execution cadence to avoid simultaneous peak activity with other high-bandwidth clients.

Methodological limitations arise when ignoring the system's scheduling priority. Many SoCs have a quality-of-service (QoS) mechanism where certain hardware blocks are prioritized over others. If the ISP is marked as a critical real-time path, it will always preempt the NPU, regardless of your bandwidth calculations. You must understand the platform's QoS hierarchy to correctly predict which of your tasks will be 'demoted' during periods of high contention.

Finally, communicate these budgets early in the model development cycle. It is much easier to design a model that uses 20 GB/s of bandwidth from the start than to retrofit a model that uses 50 GB/s. Use the contention budget as a hard constraint for the model designers. A model that runs fast in isolation but fails under system-level load is effectively broken for production deployment.

Quantify fallback nodes and their cost

Compile a detection model for a target NPU, list unsupported nodes, measure the latency penalty of each partition boundary and propose graph changes to eliminate them.

Identify whether latency is compute or bandwidth limited

Capture a profiler trace showing engine utilisation, memory traffic and idle gaps, then explain the dominant limit and one change that would improve it.

Establish the product’s real throughput

Run continuous inference for ten minutes on target hardware inside the enclosure, plot latency and throughput over time, and document the sustained figure the product should advertise.

Challenge: Running vision inference at the edge at low power and low cost, without a GPU and without cloud connectivity.

Engineering response: Used a purpose-built int8 accelerator with a constrained operator set and a compiler that requires full model quantisation to map the graph onto the device.

Pilot outcomes to validate, not verified company results: Enabled real-time vision inference on low-power devices and clarified the engineering trade: strict quantisation requirements in exchange for efficiency

Transferable lesson: Constrained operator sets are a feature, not a defect — they enforce models the hardware can execute efficiently