Everyone wants to be a superstar with autonomous agents and robots, but initially, everything that we're missing for our beautiful digital beasts - brains and arms... maybe claws, if you're building a crab (shotout to carcinization) but nevertheless: perception.

If you'd check the currently last (9th) season of Rick & Morty, the sci-fi series about overcoming challenge soberness, there is one episode about body without a head, and head without a body. Both configurations doesn't satisfy humanity that much and today we'll focus on delivering your robotics platform an additional layer of ~virtualisation (?) useful when you trying to maintain predictable latency and quality of service for workloads such as perception, sensor fusion, motion planning, and safety monitoring on Jetson Thor and still want to have some fun with inferencing, rendering, visualization, and general NVIDIA CUDA workloads, but at the same time.

Everyone wants to abuse, be abused and have best of both sides at the same time. If your agent is constantly trying to break a body, nothing gonna work. This blog will help you separate streams for critical workflows on edge-device hardware.

TL;DR

Use MIG with 2g.0gb+gfx (id:83) and 1g.0gb+me (id:78), assign processes accordingly:

OpenShell (NemoClaw) to 83 or (bravo) solution from Marco
cap-y (workloads) to 78

No random stalls/fails anymore. 12SM's for one, 8SM's for the second. You're very welcome. Don't understand what does it mean? Was relatable. Enjoy the read!

Meaning

If you building just an autonomous car (looser): check this.

Many days ago, people loved to have a discussion on brain composition and structure. Some agreed some inner core is a bit more focused on some tasks, others discussed the memory storage in the lobe area. What we could state today:

"I don't know how my brain works and my brain haven't told me about it, yet"

What else could we state?

Neumann architecture is quite common for calculators and digital machines. And we understand how it works.

Saying so, we need to imagine, how do we divide cognition towards logical semi/fully-autonomous blocks (as in reference). Logical blocks are easy to replicate within available devices and if we do so:

How can we preserve secure and versatile environment around wild VLM (or MIMO-LM) with extended environment (since physical is also included) with some preventive measures? At least, let's cover overflow and buffering + preservation of some other piece of the system from crashing every time something goes off-road.

Under the request, VLM's occupy a lot of resources, specifically on the edge devices and it doesn't help with control loop. What helps is allocated non-shared bandwidth-bounded resources for safety/balance/core pipelines.

The simplest use-case for me - parallel streams for Nvidia Halos and Nvidia cuMotion.

Sources

This blog is going to be a short read from multiple sources and patches, you can always refer to original sources in the end of the text (if still available in this burdened Web 2.0)