New Memory Architecture For SoCs And Multi-chip Systems

As Numem rolls out NuRAM chips in packages, the memory hierarchy diagram may need to be further updated.Before diving into the current state of chips and multi-chip systems, this article will briefly review the current industry context. Artificial intelligence (AI) and machine learning (ML) are becoming ubiquitous and becoming the core drivers of technological development. Modern system designs rely on a variety of processing units, including CPUs, GPUs, NPUs, TPUs, and other hardware accelerators. Today, ASICs (application-specific integrated circuits), ASSPs (application-specific standard products), and SoCs (system-on-a-chips) are typically designed by purchasing intellectual property (IP) modules for commonly used functions from trusted third-party vendors. These IP modules may be processors, memory controllers, high-speed interfaces, etc. In addition, companies develop their own "customized" IP modules to differentiate their products in the market. These IP blocks are referred to as "soft IPs" because they are represented in an abstract form of a register transfer level (RTL) and described by a hardware description language such as Verilog or VHDL. These IP blocks are then integrated and synthesized into gate- and register-level netlists, which are ultimately fabricated on silicon chips. Of course, the above description is a high-simplification of the complex process. Some of the biggest companies, such as AMD, Intel (and its newly spun off Altera), and Nvidia, have the ability to integrate multiple silicon chips (i.e., chiplets) onto the same silicon substrate, resulting in multi-chip systems. And other smaller companies dream of having this capability. In the future, even small companies may be able to develop their own "custom" chiplets by purchasing chiplets in the form of hard IP, leveraging existing ASIC/ASSP/SoC design tools and technologies, and assembling these chiplets on the same substrate and packaging them into a single module. This vision is gradually becoming a reality. According to the 2025 Chiplet Summit, tools, technologies, and ecosystems with chiplets at their core are rapidly emerging. Next, let's turn our attention to the memory space. In the past, the classification of memory was relatively straightforward. In the field of semiconductor memory (excluding the early mercury delay line and magnetic core memory), there are mainly ROM (read-only memory) and RAM (random access memory), the former is non-volatile (i.e., persistent memory), and the latter is volatile (i.e., temporary memory). In RAM, there are two main types: dynamic RAM (DRAM) and static RAM (SRAM). From a per-memory cell perspective, DRAM is less costly, has a smaller footprint, and consumes less power, but at a slower rate; SRAM, on the other hand, is faster, but at a higher cost, has a larger footprint, and consumes more power. When it comes to mass storage, it used to rely heavily on hard disk drives (HDDs). However, as technology evolves, the classification of memory becomes more and more complex. Today, the hierarchy of memory has become difficult to summarize. Here's a simplified diagram of the memory hierarchy:

0010-13927 Pedestal Lift Assembly Preclean

At the top of the memory hierarchy (source: Max Maxfield) are registers embedded in processors (e.g., CPU, GPU, etc.) and can typically be accessed within a single clock cycle. Next is the L1, L2, and L3 caches, as well as any embedded SRAM (eSRAM) and system-level caches (SLCs), which are typically implemented by SRAM. Registers, cache, eSRAM, and SLC are all integrated on the chip. Historically, the main memory, such as DDR devices, was located outside the chip and mounted on a printed circuit board (PCB). Since around 2015, high-end ASICs, ASSPs, and SoCs have been integrating high-bandwidth memory (HBM), a type of DRAM stacked chip connected via through-silicon via (TSV) and connected to the main chip via the appropriate interface. While HBMs are not generally considered chiplets, they essentially fall into this category. Recently, DDR chiplets have also started to appear (DDR is another form of DRAM). In addition to DRAM and SRAM, there are other types of memory, each with its own advantages and disadvantages. These include flash memory (NAND and NOR), MRAM (magnetoresistive random access memory), ReRAM (resistive random access memory), FRAM (ferroelectric random access memory), and PCM (phase change memory). MRAM is attracting attention because it is non-volatile, consumes less power than DRAM, and is faster. Although MRAM consumes much less power than SRAM, it has been significantly faster than SRAM – until recently. Recently, Numem introduced a memory technology called NuRAM, which is based on a standard MRAM process, but its unique MRAM array architecture and SmartMem subsystem bring its performance close to SRAM. According to Numem, NuRAM delivers superior power consumption, performance, and reliability, with an area 2.5 times smaller and leakage power consumption 85 to 2,000 times lower than traditional SRAM. Combined with the SmartMem SoC subsystem, NuRAM enables SRAM-like performance and supports comprehensive adaptive memory management as well as optional SoC memory computing capabilities. Numem initially sells its MRAM and SmartMem technologies as IP modules for use by ASIC, ASSP, and SoC designers. Now, the company is planning to launch NuRAM chips in package form and further develop NuRAM technology in chiplet form. That's why Numem is attending the Chiplet Summit 2025 – they want to work with ecosystem partners to provide reference designs for other vendors. Numem has obtained the test chip and provides the following two charts based on the evaluation results:Bandwidth comparison of AI memory modules. Source: Numem

0010-13774 Assy, Wafer Lift Cooldown/passthru

Standby Power Comparison AI Memory Modules (Source: Numem)

Based on these advances, the memory hierarchy can be further updated. Below is a schematic diagram of the latest memory hierarchy, with red boxes indicating areas where Numem technology is likely to be used and red stars indicating persistent storage capabilities implemented by NuRAM.

Potential applications of high-performance MRAM in memory hierarchies. Source: Max Maxfield

As Numem rolls out NuRAM chips in packages, the memory hierarchy diagram may need to be further updated. However, this can be left for discussion in the future.