SK hynix and TetraMem collaborate on experimental chip to bolster energy efficiency for edge AI devices — memristor-based in-memory SoC research leaves performance questions up in the air

SK hynix, TetraMem, and researchers from the University of Southern California have developed a memristor-based in-memory computing (IMC) system-on-chip (SoC) for AI edge devices. The device is designed to accelerate neural network inference in lightweight AI models while consuming a fraction of the power that higher-end GPUs or NPUs would. To a large degree, the SoC is a proof-of-concept chip, as its performance would peak at around 2.54 TOPS in a theoretical best-case scenario, which is 16X below Microsoft‘s Copilot+ requirements.

A DWC-optimized IMC architecture

Memristor-based in-memory computing (IMC) accelerates neural networks by performing analog computations directly inside memory arrays, which reduces data movement and power consumption. However, depthwise convolution (DWC) — a core operation in lightweight networks such as MobileNet — performs independent per-channel filtering with limited data reuse and therefore maps poorly onto conventional crossbar arrays. To address this limitation, researchers from SK hynix, TetraMem, and USC developed an SoC that features both conventional IMC crossbars and a memristor-based IMC architecture specifically optimized for DWC.

The jointly developed SoC is based on an embedded RISC-V processor that schedules workloads and features 10 neural processing units (NPUs). One NPU out of 10 is dedicated to depthwise convolution, while the remaining nine execute pointwise and dense operations. Nine out of 10 NPU include a 256 × 256 memristor crossbar that performs the analog vector-matrix multiplication (VMM), 256 8-bit DACs that convert digital activations into analog voltages, 256 8-bit ADCs that convert the analog outputs back into digital values, and additional peripheral circuitry for reading, writing, programming, and controlling the crossbar.

Great efficiency, low performance overall

To demonstrate the architecture, the researchers deployed a customized MobileNetV1Small neural network for the Visual Wake Words benchmark. The network contains approximately 36,000 parameters; all depthwise layers were mapped to the dedicated NPU, and pointwise layers were mapped to the remaining NPUs.

Because the memristor-based IMC hardware natively performs unsigned analog vector-matrix multiplication, inputs and weights are quantized to unsigned 8-bit values before execution. Since each memristor device can be programmed with only slightly more than 2 bits of effective precision, the design uses a two-subarray compensation technique that boosts effective weight precision to roughly 4 bits.

Conceptually, the approach is somewhat analogous to Nvidia’s NVFP4 philosophy, in that both seek to achieve higher effective precision from low-precision hardware. However, the implementations are fundamentally different: NVFP4 relies on a digital floating-point representation and scaling factors, whereas the memristor SoC improves precision by compensating for analog programming errors using two programmed subarrays.

When it comes to accuracy, the SoC achieved an end-to-end inference accuracy of 80.36%, which matches the corresponding 4-bit software model. As for performance, the SoC delivers a peak throughput of 0.254 TOPS per NPU and reaches an energy efficiency of 21.3 TOPS/W at 100 MHz and 11.9 TOPS/W at 400 MHz. According to the authors, this compares favorably with published SRAM-based compute-in-memory accelerators despite being manufactured on an older 65 nm process. The SoC also exceeds Nvidia’s A100 INT8 energy efficiency by an order of magnitude, the joint paper claims. Yet, these claims are largely unsubstantiated.

First up, the MobileNet demonstration does not even use all 10 NPUs. It uses one dedicated DWC NPU, five standard NPUs for pointwise layers, and leaves four standard NPUs idle. The demonstration thereby does not reveal total SoC throughput (TOPS), sustained throughput running a real network, and throughput with all 10 NPUs simultaneously saturated. In fact, the paper does not even reveal whether all 10 NPUs can be used at the same time. To that end, the 2.54 TOPS figure we mentioned earlier in the story is highly theoretical.

Validated approach

SK hynix, TetraMem, and researchers from the University of Southern California have developed a memristor-based IMC SoC featuring a novel depthwise convolution accelerator that improves crossbar utilization for lightweight AI workloads. The partners have managed to fabricate it using an outdated 65nm process technology and make it work, achieving a 21.3 TOPS/W energy efficiency and inference accuracy comparable to a 4-bit software model despite the fact that memristors can be programmed with a circa 2-bit accuracy. While the architecture validates that the approach works, the paper does not disclose the full performance of the SoC, and it is not clear whether the chip’s 10 NPUs can be saturated at all.