I'm curious to hear what bottlenecks you encountered in the traditional path. Of all the compute and data shuffling involved in LLM inference, I would have thought shuffling the raw input/output around would have been a trivial part of the overall cost, and thus not a big optimization target?
I addressed this a little bit in the comment below, but the cycles add up. I'm doing some pretty crazy things higher up in the stack, that I'm not quite ready to release yet. But even micro optimizations here add up at the scale I'm working at to allow me the headroom I need. I'm relying on a high-frequency recursive agentic loop that chokes a real-time guarantee without every optimization I can give it.
I started by removing the IPC overhead from a weaviate db connection, and doing all my vector math in house with a lightweight sqlite db. This became the next obvious target for optimization once I saw how much that saved me doing things in-house.
What are the benefits of this. I get the impression it improves speed of…something? Most of the time when using AI comes from the LLM, so I’m curious what this improves?
To start with, it eliminates the IPC and Sidecar overhead of using something like Ollama. It might seem trivial when comparing the speed of a REST request to that of inferencing, but the ms add up at scale. And for what I'm doing higher up the stack, every nanosecond I save here, gives me headroom to do cooler things.
It also allows you to make lots of smaller round trips to your AI generating layer. Even if you're just operating REST on the loopback layer, there is overhead and kernel-level context switching that adds up. I can go further into this if you'd like!
Once you make the decision to forgo IPC and do things in house, you typically would go the JNI route. Project Panama in JDK 22+ however opens us up to direct memory access at the lowest of levels to shave even more cycles.
We get direct, off-heap memory access inside the same process space.
This allows us to do some cool things like creating a memory and GPU governor with more direct access to the hardware its controlling, while completely bypassing the Java GC and its overhead!
It also allows us to do direct memory copying without ever going through java heap allocations nor primitive array copying.
With this code, I mapped the native C-structs directly to Java objects to bridge the ABI gap cleanly.
The naive way to do so, without writing the argus c layer, and interfacing directly with llama.cpp. By having a static, immutable ABI, you save yourself the headache of having to rewrite your entire brittle Panama bindings every time upstream changes a single variable.
Then I went a step further and provided the Java API for you directly to interface with!
Idk about you, but if you've ever had to write a JNI wrapper, I much prefer an easy to use clean Java API ready-made.
Then I package everything inside one nice friendly JAR, allowing jvm developers to have access to multi-modal inference with one import.
Thank you for the detailed and technical response. I’m not doubting it providers benefits at the technical level, and I’m keen to dive into it deeper, but I was hoping to understand what benefits this translates to at a user level. I’m imagining something like it can lead to faster switching between between agent calls, or even at a developer level, it allows you to run agents/llms in a more constrained environment, therefore you can run more agent containers in parallel?
I’m familiar with enough with Java that I can squint and follow some of it, but I’m hoping to understand more at a macro level before committing to diving into the technical details.
Again, thank you for the information already provided.
Your intuitions are right, faster switching between agent calls, and tighter packing of agents in the same amount of compute space.
There's a second thing besides just efficiency that I provide here however.
At a developer-user level, it automagically provides JVM developers (not just java) a frictionless way to add AI into their stack with a single dependency.
It makes using AI in java as easy as it is in python.
I'm providing the [what I've found to be missing] layer for AI in java in the most efficient way I could.
I'm not sure if by user you meant developer (user of this code) or user as in end-user.
Let me know if I still haven't answered your question fully yet!
Great. I feel like this is something that I’m going to dig into more. At day job I work with a platform that’s built on OSGI and JCR. I’m keen to explore if this could enable any interesting interactions there (they might not necessarily be good interactions though haha).
I'm curious to hear what bottlenecks you encountered in the traditional path. Of all the compute and data shuffling involved in LLM inference, I would have thought shuffling the raw input/output around would have been a trivial part of the overall cost, and thus not a big optimization target?
I addressed this a little bit in the comment below, but the cycles add up. I'm doing some pretty crazy things higher up in the stack, that I'm not quite ready to release yet. But even micro optimizations here add up at the scale I'm working at to allow me the headroom I need. I'm relying on a high-frequency recursive agentic loop that chokes a real-time guarantee without every optimization I can give it.
I started by removing the IPC overhead from a weaviate db connection, and doing all my vector math in house with a lightweight sqlite db. This became the next obvious target for optimization once I saw how much that saved me doing things in-house.
What are the benefits of this. I get the impression it improves speed of…something? Most of the time when using AI comes from the LLM, so I’m curious what this improves?
To start with, it eliminates the IPC and Sidecar overhead of using something like Ollama. It might seem trivial when comparing the speed of a REST request to that of inferencing, but the ms add up at scale. And for what I'm doing higher up the stack, every nanosecond I save here, gives me headroom to do cooler things. It also allows you to make lots of smaller round trips to your AI generating layer. Even if you're just operating REST on the loopback layer, there is overhead and kernel-level context switching that adds up. I can go further into this if you'd like!
Once you make the decision to forgo IPC and do things in house, you typically would go the JNI route. Project Panama in JDK 22+ however opens us up to direct memory access at the lowest of levels to shave even more cycles. We get direct, off-heap memory access inside the same process space. This allows us to do some cool things like creating a memory and GPU governor with more direct access to the hardware its controlling, while completely bypassing the Java GC and its overhead! It also allows us to do direct memory copying without ever going through java heap allocations nor primitive array copying.
With this code, I mapped the native C-structs directly to Java objects to bridge the ABI gap cleanly. The naive way to do so, without writing the argus c layer, and interfacing directly with llama.cpp. By having a static, immutable ABI, you save yourself the headache of having to rewrite your entire brittle Panama bindings every time upstream changes a single variable. Then I went a step further and provided the Java API for you directly to interface with!
Idk about you, but if you've ever had to write a JNI wrapper, I much prefer an easy to use clean Java API ready-made.
Then I package everything inside one nice friendly JAR, allowing jvm developers to have access to multi-modal inference with one import.
Thank you for the detailed and technical response. I’m not doubting it providers benefits at the technical level, and I’m keen to dive into it deeper, but I was hoping to understand what benefits this translates to at a user level. I’m imagining something like it can lead to faster switching between between agent calls, or even at a developer level, it allows you to run agents/llms in a more constrained environment, therefore you can run more agent containers in parallel?
I’m familiar with enough with Java that I can squint and follow some of it, but I’m hoping to understand more at a macro level before committing to diving into the technical details.
Again, thank you for the information already provided.
You're welcome, I appreciate the engagement!
Your intuitions are right, faster switching between agent calls, and tighter packing of agents in the same amount of compute space.
There's a second thing besides just efficiency that I provide here however. At a developer-user level, it automagically provides JVM developers (not just java) a frictionless way to add AI into their stack with a single dependency.
It makes using AI in java as easy as it is in python.
I'm providing the [what I've found to be missing] layer for AI in java in the most efficient way I could.
I'm not sure if by user you meant developer (user of this code) or user as in end-user. Let me know if I still haven't answered your question fully yet!
Great. I feel like this is something that I’m going to dig into more. At day job I work with a platform that’s built on OSGI and JCR. I’m keen to explore if this could enable any interesting interactions there (they might not necessarily be good interactions though haha).
Glad to hear! Let me know if you run into any issues!
This is pretty impressive.
Thanks! It was a fun challenge getting an AI agent to pair with me while adhering to strict c-struct padding and memory alignment.
Let me know if anything sticks out you'd like to discuss deeper!