Inside the Semidynamics approach to AI inference
Semidynamics is developing AI accelerator racks for inference, bringing CPU, vector and tensor compute together in a single architecture. CTO Iakovos Stamoulis explains how that integration simplifies programming, how Gazillion Misses keeps computation supplied with data, and how the company is building its software stack around familiar AI frameworks.
For someone coming to Semidynamics for the first time, what are you developing?
Stamoulis: Semidynamics is developing AI accelerator racks aimed at inference. We are primarily targeting LLMs at this stage. However, the architecture could be used for other kinds of workloads, and the data types we support are also suitable for imaging. Because the system includes full CPU capability, it can also run general-purpose code.
In addition, our systems support FP64, which is very useful for high-performance computing and scientific workloads.
You have CPU, vector and tensor compute in the same architecture. What does integrating those pieces give you?
Stamoulis: The great thing about integrating our CPU, vector unit and tensor compute in a single processing unit is that, with a single programmable instruction flow, we can address all the different units and utilise the different components in our system.
That makes porting our software much easier and simpler, and it also makes it easier for third parties to develop software for our system. There are no exotic compute units with their own programming model to learn. With a single instruction stream, and using standard compilers such as GCC and LLVM, you can get the most out of our systems.
How does that help when a workload moves between different types of computation?
Stamoulis: The CPU and the different compute units use shared memory and a shared instruction stream. Exchanging data between different stages of computation is easy and transparent, without having to move things around with DMAs or other mechanisms that would require deeper architectural knowledge of our system.
What role does RISC-V play in your development approach?
Stamoulis: Using the RISC-V instruction set gives us access to the RISC-V ecosystem in terms of compilers, debuggers and existing free software that can run on our system.
It’s an R&D acceleration choice. Using RISC-V means we can focus on putting in our secret sauce and optimising for real AI workloads, as opposed to developing generic technologies or support technologies that we need anyway.
Keeping the compute units supplied with data is a major challenge in AI. What does Gazillion Misses do?
Stamoulis: It’s hiding latency while keeping the compute units busy. This is what the Gazillion unit does with multiple outstanding memory reads. The compute element can continue processing while we’re still requesting data from memory.
This can be done in a continuous pipeline fashion, where both the compute units and the memory are fully utilised.
What happens as models and working sets get larger?
Stamoulis: It keeps feeding the units even from bigger amounts of memory. Its effectiveness doesn’t depend on memory size, so it works just as well with very large memories.
Does the software developer have to do anything to take advantage of it?
Stamoulis: It is completely transparent to the developer. From a software point of view, they don’t even need to know that it exists. It handles everything in the background. They don’t need to write or customise software kernels or other routines specifically for this. They can have their normal or generic code, and our technology will take care of it.
What does the software stack look like for someone bringing an existing AI model to the system?
Stamoulis: All the software for running AI is taken care of by us, starting with our drivers. There are at least three layers.
There are the low-level kernel drivers and our SDK. Then there are libraries for things like ONNX Runtime, so we can parse ONNX and feed our system, but also a backend for PyTorch.
At an even higher level, we also have a customised vLLM, which orchestrates PyTorch so it can execute on our system.
Our goal is to have a plug-and-play system based around vLLM orchestration. As long as the deployment uses vLLM to manage the software, everything is taken care of by our libraries.
How is the engineering team organised to build both the hardware and software?
Stamoulis: The engineering team is split between the silicon team, the team that designs systems, and the software team.
A substantial part of our engineering team works on software, with the rest working on silicon and hardware development.
As we’ve moved from mostly CPU IP towards AI acceleration and more complete systems, the split has remained roughly the same percentage-wise, although I expect our software team will need to grow a lot. As we move on, it will need to support more use cases, more frameworks and more different kinds of compute environments.
Looking ahead two years, what would success look like from an engineering perspective?
Stamoulis: We should have proved the value of our technology, and we should be working on our next-generation products based on real workloads and the current trends in the industry. That will be our definition of a win.