An Engineering Deep Dive on SiMa.ai MLSoC Modalix

On September 9, 2024, we announced the release of our second generation MLSoC: SiMa.ai MLSoC Modalix. We are excited to unlock new realms of possibility for existing and future customers. This blog shares perspective on the engineering journey behind the development of MLSoC Modalix  - a long and intentional process defined by front line customer feedback, specialized experience, perseverance and a commitment to SiMa.ai’s founding principles.

 

What was the genesis for MLSoC Modalix? How does it differ from MLSoC, our first generation?

 

When we designed the first generation MLSoC, the primary goal was to demonstrate our ability to build an impactful, valuable platform primarily focused on video applications for the embedded edge market. Through the process of bringing Gen 1 to market, we began to quickly identify the many use cases best served at the edge, working with customers across a number industries to build a deep understanding of what matters most to them. Along the way, we learned that customers require not just visual, but also audio and text integrated into embedded edge applications. Inbuilding, delivering and working with customers on productizing MLSoC for their applications, our engineering team developed a thorough understanding of the requirements to address the needs of all edge applications – hence the kernel behind ONE Platform for all edge AI was born. 

 

The new MLSoC Modalix product family is about incorporating all of those learnings to ultimately deliver a best-in-class product that represents a range of cost-effective and power-efficient solutions covering not just video and image, but also speech and text-based applications for the embedded edge market - truly a single platform to suit any Edge AI need. This gives customers increased coverage, from CNNs to GenAI and everything in between, relative to power, performance, cost and size (device and form factor). 

 

Our first gen MLSoC continues to be ideal for CNN-based video applications across a variety of use cases including industrial inspection, retail, aerospace and defense, security and surveillance and smart vision. With MLSoC Modalix we’ve broadened the scope of capabilities the SiMa.ai ONE Platform for Edge AI can deliver beyond CNNs to different applications like Natural Language Processing (NLP) and Generative AI. While Modalix can handle computer vision/CNN applications very well, it has been significantly enhanced to support multi-modal GenAI models i.e. LLMs, LMMs and LVMs. As ML becomes ubiquitous, the complete MLSoC product family enables a feature-rich, highly performant ML experience to customers at a fraction of the cost and with significantly better performance than the cloud.

 

A Glimpse into Designing MLSoC Modalix 

 

We maintained a laser focus on power and performance throughout the entire design and engineering process for MLSoC Modalix. We carefully assessed which factors could be cost optimized in a scenario where customers needed to deploy MLSoC Modalix, for example, not only on straightforward CNN models but across a variety of multi-modal GenAI models as well. To enable a cost effective solution, our team added core capabilities covering back-end/physical design, package/substrate design, test engineering, quality and reliability engineering and worked with best-in-class partners to deliver Modalix.

 

Our focus is software-first, and all of the different capabilities in the MLSoC Modalix family can be addressed and customized by software to make it more efficient still. We leveraged the same intellectual property cores, allowing customers to leverage our Palette software for compatibility across the entire MLSoC product line - MLSoC and MLSoC Modalix. Continuous enhancements to Palette enable easy, painless upgrades and management as customers mix and match from the entire MLSoC product family based on their unique needs.

 

Key innovations in MLSoC Modalix include: 

• Improved accuracy for large language models; 
• State of the art vision transformer models; 
• The ability to seamlessly handle multi-modality, and; 
• Targeted improvements to memory and caching subsystems to enable efficient handling of multi-billion parameter models. 

 

This is combined with upgrades on the compute & DSP subsystem to handle more complex customer applications and next generation IO to handle significantly increased data rates. All key innovations were guided by real world experience gained by working on end user applications with (first generation) MLSoC.

 

On the silicon side, we brought on board significant expertise around physical design, packaging, test engineering, quality and reliability to further optimize a cost-effective, feature-rich platform for our customers. We added several new interfaces, such as MIPI, to expand video sensor options for our customers.

 

With our team’s deep expertise and knowledge of those workloads, we were able to stress test for specific requirements and capabilities with well known customer pain points in mind. Our own software team leveraged their expertise in the ecosystem to make their own asks and requests from the hardware team, nearly all of which have been incorporated into the MLSoC Modalix family. 

 

Our experience running the SiMa.ai MLSoC in production environments has allowed us to better understand how our comprehensive system-on-chip works with different workloads, and we’ve made significant changes in our architecture and system interconnect to optimize for each. Interconnectivity has been greatly improved, fine tuned to handle disparate workloads, while the physical design and packaging have been enhanced as well. 

 

And to drive even better power efficiency, we’ve designed multiple voltage islands within the chip, allowing customers to run specific workloads on a lower voltage supply depending on the requirements of that workload. 

 

What can MLSoC Modalix do for customers? 

 

MLSoC Modalix offers a purpose built, cost effective and power efficient platform that enables customers to seamlessly integrate AI with any sensor – text, image, audio or visual – into their applications. Included Palette software SDK enables customers to quickly and easily develop and deploy applications on MLSoC-enabled platforms.

 

As the use of ML proliferates, models are becoming significantly larger (billions of parameters and growing) and more complex. Competing solutions use significantly more compute, more expensive ad exotic memory technologies, and expensive heat management and cooling solutions, which result in very high cost of hardware and higher power consumption. Almost all models rely on traditional compute technologies developed for other use cases to address ML applications, which leads to designs that are highly inefficient in both cost and power.

 

Our purpose built MLSoC solutions accelerate 100% of ML applications. We use low-cost, high-volume, off-the-shelf memories and heatsinks, and enable fanless designs which not only reduce cost but improve reliability for our customers.

 

We designed MLSoC Modalix to allow customers to truly accelerate all of their ML operations. This differs from alternative vendors, which require customers to start with a specific accelerator for a specific function. We accelerate 100% of ML operations for our customers, improving performance without corresponding increases in power. This also makes Modalix extraordinarily cost effective — having a purpose-built product that targets specific applications allows us to optimize for ML instead of a GPU-based architecture, delivering a more specialized experience. Unlike General Purpose Processors, SiMa.ai only delivers the AI capabilities you need, allowing us to deliver performance-per-watt metrics that competitors can’t match. 

 

Our team represents an incredibly diverse and unique set of skills within ML — people who understand ML inherently, people who have spent their entire careers working to develop embedded systems and chips, and people who understand how to develop software for extremely complex, highly-efficient SoCs. We take great pride in melding these different specialties and experiences to build the best possible product that enables embedded developers to integrate ML into their applications. 

 

At SiMa.ai, our mission is to accelerate the entire pipeline for the entire application with ONE Platform for Edge AI. Our MLSoC Modalix product family delivers on that promise, and we’re just getting started!