Processing-in-memory puts small compute units inside memory chips so some work happens where data already sits. Samsung is among the firms researching it, aiming to reduce the cost of moving data between memory and processors.
Key takeaways
- Processing-in-memory, usually shortened to PIM, is an approach that places arithmetic units inside or immediately beside memory chips rather than only in a separate processor.
- The problem PIM targets is data movement: shifting numbers between memory and a processor often costs more time and energy than the calculation itself.
- Samsung is one of several memory manufacturers that has publicly researched and demonstrated PIM concepts, alongside work by other DRAM makers and academic groups.
- The workloads most often cited as a good fit are memory-bandwidth-bound ones, including parts of large language model inference and some database and scientific operations.
- PIM has not displaced conventional accelerators, and its adoption depends as much on software, standards and economics as on the silicon itself.
What is processing-in-memory?
In a conventional computer, memory stores data and a processor operates on it. Every calculation therefore requires a round trip: values are read out of DRAM, travel across a bus or interposer into the processor, are worked on, and results travel back. That journey is the subject of a long-standing observation in computer architecture, often called the memory wall — processor speed has improved faster than the speed and bandwidth of the link to main memory, so the link increasingly determines how fast a system runs.
Processing-in-memory attacks the problem from the other side. Instead of making the link faster, it reduces how much has to cross it. Simple compute units are integrated into the memory device itself, close to the arrays where bits are stored. When an operation can be performed there, the data never leaves the chip; only a smaller result does. The internal bandwidth available inside a DRAM die is far larger than what can be exposed through its external pins, so in principle a computation placed inside the memory has access to more data per unit of time than one sitting outside it.
The term covers a family of designs rather than a single product. Some proposals put logic on the DRAM die itself. Others place it on a separate logic layer beneath stacked memory, an arrangement sometimes described as processing-near-memory. Others still add compute to the memory controller or to a module. The engineering trade-offs differ in each case, but the motivation is the same.
Why is this being discussed now?
Attention to memory architecture has risen sharply with the growth of machine learning workloads. Generating text with a large language model involves repeatedly reading very large sets of weights out of memory to perform comparatively simple arithmetic on them. In that regime, the accelerator is frequently waiting on memory rather than saturating its arithmetic units. Anything that reduces the volume or cost of memory traffic becomes commercially interesting.
That has put memory manufacturers, Samsung among them, in an unusually prominent position. High-bandwidth memory has become a constrained and closely watched component, and ideas that were previously discussed mainly in academic architecture papers now attract broader attention. PIM is one of those ideas. Renewed interest in a research direction is not the same as a shipping product in wide use, and the specifics of any current commercial programme — which parts are in production, at what volume, in which systems — are not something that can be established from a discussion thread alone.
The background a newcomer needs
The idea is not new. Architects have proposed computing inside or beside memory for decades, under a series of names, and prototypes have appeared repeatedly. The reasons it has not become mainstream are instructive.
DRAM manufacturing is optimised for density and cost per bit, not for logic. The process used to make memory cells is not the same as the process used to make fast transistors, so logic built on a DRAM die tends to be slower and less area-efficient than logic on a dedicated chip. That constrains how sophisticated an in-memory compute unit can be. It generally ends up capable of a limited set of operations rather than being a general-purpose core.
There is also a software problem. A processor and its compiler present a familiar model to programmers. A memory device that can compute does not, and code must be written or transformed to know which operations can be dispatched into memory and which cannot. Without library and framework support, the hardware is difficult to use even where it would help. Standardisation matters here too: memory is a commodity sold into an ecosystem of controllers, modules and interfaces, and a feature that only one supplier offers is harder for system builders to adopt than one defined in a common specification.
Who would be affected, and how?
The most directly affected parties are operators of large-scale inference and analytics infrastructure, where memory bandwidth and the electricity consumed moving data are significant costs. For them, a technique that reduces traffic has a measurable value that can be weighed against integration effort.
Memory manufacturers are affected differently. DRAM is a cyclical commodity business with thin differentiation. A capability that turns memory into something closer to a specialised component changes that dynamic, which is part of why suppliers invest in the research even when near-term volumes are uncertain.
Chip designers and system integrators sit in between. Adding compute to memory shifts responsibilities across a boundary that is currently well defined, raising questions about who schedules work, how coherence is maintained and how errors are handled.
Most end users would see nothing directly. The effects, if they arrive, would appear as capacity or efficiency changes upstream rather than as a feature.
Where informed people disagree
There is genuine disagreement about how much of a real workload PIM can absorb. Sceptics argue that the fraction of operations suited to simple in-memory units is small, and that the remaining work still requires a conventional accelerator — so the memory-side hardware adds cost and complexity for a bounded gain. Supporters counter that the bounded gain lands precisely on the bottleneck that limits current systems, and that this is where improvement is most valuable.
A second disagreement concerns where the logic should live. Putting it on the DRAM die maximises proximity but is constrained by the manufacturing process. Putting it on a separate logic die in a stack allows better transistors at some distance and cost. Moving it further out, to a controller or an interconnect-attached device, makes it easier to build and standardise but recovers less of the benefit.
A third is about competition from other directions. Faster memory interfaces, larger on-chip caches, disaggregated memory over emerging interconnects, and model-side techniques such as quantisation and sparsity all reduce the same pressure. Whether PIM is the most economical answer, or one that is overtaken by more incremental changes, is unsettled.
What are the practical implications?
For anyone evaluating claims in this area, a few distinctions are useful. Benchmark figures for PIM are usually reported for a specific kernel rather than an end-to-end application, and speed-ups on the memory-bound portion do not translate directly into whole-system gains. The relevant comparison is against a well-optimised conventional system, not an unoptimised one.
It is also worth separating research demonstrations, standardisation activity and shipping products. Papers, conference presentations and prototype modules can all be real without implying that a customer can buy the technology at volume. Where a claim concerns a particular vendor’s roadmap, the reliable sources are that company’s own technical disclosures and the standards bodies that define memory interfaces.
What to watch next
The clearest signal would be movement in standards: whether compute-capable memory features appear in publicly agreed specifications rather than as single-vendor extensions. A second is software — support in widely used inference and database frameworks, since without it the hardware remains a research artefact. A third is packaging and stacking, because much of the practical near-memory work depends on how logic and memory dies are assembled together. A fourth is whether cloud operators disclose deployments, which would indicate the economics work at scale rather than in a laboratory.
None of these has an announced timetable that can be stated with confidence, and the pace of memory architecture change is typically measured in generations rather than months.
Frequently asked questions
What does processing-in-memory actually mean?
It means placing computation inside or immediately next to memory devices, so that some operations can be carried out where data is stored rather than after transferring it to a processor. The compute units involved are usually simple and limited to specific arithmetic patterns. The aim is to cut the time and energy spent moving data across the interface between memory and processor, which in many workloads dominates the cost of the calculation itself.
Why is memory a bottleneck for AI workloads?
Generating output from a large model requires reading very large quantities of parameters from memory to perform relatively simple arithmetic. The arithmetic units in an accelerator can often complete their work faster than memory can supply the next batch of values, so the system spends time waiting. This is described as being memory-bandwidth-bound. It is why memory capacity and bandwidth have become central constraints in machine learning infrastructure.
Is Samsung the only company working on this?
No. Processing-in-memory is an active area across the memory industry and in academic computer architecture, and multiple manufacturers and research groups have published designs or prototypes. Samsung is one of the more visible participants because it is a large DRAM producer that has presented work publicly. The specific state of any individual company’s commercial programme is not something that can be confirmed from general discussion.
Is processing-in-memory a new idea?
No. Architects have proposed computing inside memory for decades, and prototypes have been built repeatedly under various names. What has changed is the workload mix: modern machine learning inference is unusually memory-bound, which strengthens the economic case. The historical obstacles — manufacturing constraints on logic built in a memory process, and the difficulty of programming such devices — have not disappeared, and they still shape what is practical.
Would this replace GPUs or AI accelerators?
The designs discussed publicly are generally complements rather than replacements. In-memory compute units handle a narrow set of operations; the wider variety of work in a model still requires a general accelerator. The proposition is that offloading the most bandwidth-hungry steps relieves the bottleneck, not that the memory takes over the whole computation. How much of a real workload can be offloaded is disputed.
How would software use compute-capable memory?
It would need explicit support. Either the programmer, a compiler, or a runtime library must identify which operations can be dispatched into memory and issue them accordingly. In practice this means changes in frameworks and drivers rather than in application code. The absence of such support in widely used software is one of the main reasons in-memory compute has historically remained confined to prototypes and research systems.
Sources and further reading
- Peer-reviewed computer architecture conferences and journals, which have published work on processing-in-memory and near-memory computing over many years.
- Memory manufacturers’ own technical presentations and white papers, which describe proposed architectures and reported benchmark results.
- JEDEC, the standards organisation responsible for memory interface specifications, for the state of any agreed features.
- Technical trade press covering the semiconductor and memory industries, for context on manufacturing and market conditions.
Surfaced from the hackernews signal “memory chip compute architecture”. AI-assisted draft, editorially reviewed.
