Our inference platform uses high-capacity LPDDR, a RISC-V-based XPU and an architecture designed to keep the compute working while the memory catches up.

TOPS are easy to print on a slide, easy to compare and a useful indication of how much work an accelerator can do. 

But at Semidynamics we think that number is becoming less useful on its own as a way to understand real rack performance. It does not matter how many tensor operations a chip can theoretically perform if the memory system cannot supply the data needed to perform them.

We don't think TOPS are the most important piece in the game, although rest assured, we've got plenty of TOPS. But more importantly, it's the memory architecture that matters. 

At ISC in Hamburg, we showed how that argument is moving beyond processor IP and into a complete rack-scale inference product. Our rack concept contains our own XPU, a RISC-V-based processor combining general-purpose, vector and tensor compute, together with considerably more memory per accelerator than is typical in current AI systems.

The key part is not simply that we want to attach more memory. We have chosen LPDDR rather than HBM and designed the architecture around the consequences of that decision.

HBM provides huge bandwidth, but it is also expensive, supply-constrained and difficult to package. LPDDR is cheaper, widely available and can provide the capacity needed for large models and their expanding KV caches. It also has higher latency and a different bandwidth profile.

We are not pretending those differences do not exist. Our approach is to accept them and then build a processor that can live with them.

At the centre of that approach is Gazillion Misses, a technology built deep into the CPU and its data-cache architecture. It allows the processor to keep many memory transactions in flight at the same time, hiding much of the delay involved in fetching data from external memory.

While one request is waiting, other work can continue. The objective is to stop the tensor units sitting idle simply because a particular piece of data has not arrived yet.

That makes it possible to use high-capacity LPDDR without letting its latency dictate the performance of the whole system. It also means that the tensor hardware has a better chance of doing what the headline TOPS number assumes it will do: continuously crunching through multiplications rather than waiting for the memory subsystem.

The technology is proprietary, but it does not require developers to manage it directly. Software issues normal memory instructions and the processor takes care of scheduling the outstanding transactions underneath. We are solving the memory problem in the architecture rather than passing it up to the programmer.

That approach also explains why we describe our processor as an XPU rather than simply a GPU.

The chip is based around the RISC-V instruction set and is designed to cover several types of processing inside one architecture. Its general-purpose CPU is compliant with the RVA23 profile and can boot Linux. It supports the ratified RISC-V Vector Extension 1.0, while we have also developed our own tensor capability ahead of a final standard RISC-V tensor extension.

Rather than treating the CPU, vector engine and tensor accelerator as unrelated devices with separate programming environments, we are building around a common RISC-V programming model. That should make the platform easier to use than AI systems that require developers to work across several distinct processor architectures and proprietary software stacks. The hardware may change character depending on the workload, but it remains part of the same underlying system.

The rack concept shown at ISC is where these pieces come together.

We are not proposing another server that integrates accelerators from Nvidia or AMD. The compute element inside the rack is our own XPU, designed from the outset around high memory capacity and LPDDR.

That distinction matters because the memory decision affects the economics of the entire rack.

Putting more memory on each XPU means models can be held closer to the compute that is using them. It reduces the amount of communication required between accelerators and can reduce the number of devices needed to serve a model. This will result in more users being served by each rack and a better cost per delivered token.

There is a trade-off. LPDDR does not provide the raw bandwidth of HBM, and simply replacing HBM with LPDDR in an existing accelerator design would almost certainly hurt performance.

That is not what we are doing. Our processor, memory system and Gazillion Misses architecture have been developed together. We started with the assumption that LPDDR would be the memory and worked backwards from there to determine how the compute units could be kept busy.

That does not rule out HBM completely. The architecture could support it in the future if pricing, supply or customer requirements changed. Gazillion Misses would still provide value by allowing the processor to tolerate latency and manage more outstanding work. But HBM is not required to make the current platform function.

The rack also gives us a European infrastructure story. Our XPU is being developed in Europe and can be paired with a European Arm-based CPU from French processor company SiPearl. That creates the option of a rack in which the principal compute components, rather than just the enclosure and integration work, come from European suppliers.

For customers that do not require a fully European configuration, we will also support x86 CPU trays. The choice between Arm and x86 is a customer decision, with the software environment supported on both. 

This makes European sovereignty a configuration of the platform rather than a restriction placed on it. Customers interested in regional technology independence can choose the European CPU and XPU combination. Others can deploy the same accelerator architecture alongside more conventional x86 hosts.

We are making a bold bet. HBM has become closely associated with high-performance AI because bandwidth matters, and the industry has spent years building systems and software around that assumption. Our counterargument is not that bandwidth has stopped mattering. It is that memory capacity, latency tolerance, communication overhead and rack-level economics matter as well.

A chip with a very large TOPS number can look impressive while still spending too much time waiting. An accelerator with less exotic memory can look slower on paper while producing a more useful system if the architecture keeps the compute fed and allows more of the model to remain close to it.

That is the real test for our platform. Not whether LPDDR can beat HBM in a bandwidth comparison, because it cannot, but whether a processor designed around LPDDR can deliver more useful inference capacity at the rack level. The answer will ultimately be measured in users served, tokens delivered and what those tokens cost.