TLDR
Since the code for the backend of “Anytwo” (name TBD) was made available a couple of days ago, I’ve been playing with it and setting up my usual sandboxing around the new tooling, and making it work with a local model and GPU. If you are interested on Anytwo or AI inside Anytype in general, this might be a fun read.
Anytype AI before Anytwo
For context, over the last few months I’ve developed my own version of an AI agent for Anytype, which consists of a single Go process that is put together as a heavily modified Picoclaw that bundles a new custom backend for Anytype, and the anytype-heart project which, as the name implies, is the heart of every Anytype version that runs on desktops, mobiles, or headless in servers. So this Go process runs sandboxed in a machine with a GPU right next to it as an independent Anytype user, and that user has access to some of my spaces inside Anytype.
Some people wonder about why I might even want something like this, so here are a few concrete examples of things that I’ve been doing over the past months by talking to this agent:
- Summarize bookmarks and videos and append it to the content of the page.
- Keep track of interesting content and reply to questions about it.
- Extract the transcription of a specific video and organize the information inside a page.
- Create and support a list of GitHub PR issues, summaries, label management, questioning, etc.
- Take a series of screenshots from a bad website using Chrome, append them to a collection, extract the text in a specific way, and build a page with details of the content extracted.
- Book parking space for a trip.
- Maintain a set of howtos for itself, so it can learn and remember how to do things.
- and so forth…
As a fun example, here is the custom agent attempting to book parking a few months ago:
Native AI in Anytwo
Well, the first part that is really nice is that I don’t have to develop it alone. More seriously, though, the underlying foundation that the team is putting in place is very interesting, and I’m actually excited to migrate over my workflow and my code to the new base. I won’t get into the internal details here (see references at the end), but here are a few highlights that you might have sympathy for.
any
any is the new anytype-heart, but unlike the old one, this is clearly designed as a comprehensive tool instead of a library, and as such exposes a complete API for integrating and manipulating the data, the internals have also been significantly improved, so have a look if you are into this kind of thing.
anyrt
anyrt is the agent runtime: the home of Bao. There are several interesting aspects of the architecture that the team is putting in place. For example, instead of doing a tool call every time the agent needs to manipulate content or perform an action, the mechanism is designed so that the agent is writing code, and the code is run in a sandbox. If that’s unclear, imagine you want to do a simple change in 10 documents - in the typical way this would require a lot of back and forth with the agent, which is slow and expensive, but instead the agent can just write a small program and do all changes at once with the model not even looking at the actual data. If you want a more comprehensive read on the approach and the differences, Cloudflare has a nice article on it.
There are also other interesting choices on the design of anyrt that are worth looking into. Just to steer some interest, anyrt is actually a platform for running sandboxed applications in general, AI-related or not, and those applications are Python programs that live as normal content inside a space. The execution of these applications is strictly controlled, and their side effects are recorded. The implementation of Bao itself is, in fact, mostly a sandboxed application. So not only Bao runs sandboxed applications, but Bao is a sandboxed application. The sandboxing is done using industry-leading projects, instead of custom.
Deploying a local private Bao
With the above, it’s relatively easy to put in place a completely local and private deployment of Bao using a local GPU. To my surprise, this is possible without changes to the in-development software even with my unusual setup requirements: a headless setup, with sandboxed deployments, that use a local GPU with a local LLM model, so that I can leverage it privately from any of my devices. At a high-level, these are the steps to get this running with the new stack:
- Setup
anyand initialize the database to create your account, if necessary. Unlike my classic Anytype agent setup, this is the same personal account used on other devices. - Run
any runso the headless device becomes active and exposes the API endpoint. - Setup
anyrtand point it to the local API endpoint fromany. - Change the setup of
anyrtso that it uses a local LLM. That part is under-documented. You’ll need some extra settings in youranybao.toml:
[config]
"llm.tier.codegen" = { provider = "openai-compat", model = "llama", base_url = "http://localhost:<port>", api_key_ref = "llm.key.llamacpp", backend = "llamacpp" }
"llm.tier.classify" = { provider = "openai-compat", model = "llama", base_url = "http://localhost:<port>", api_key_ref = "llm.key.llamacpp", backend = "llamacpp" }
"llm.tier.vision" = { provider = "openai-compat", model = "llama", base_url = "http://localhost:<port>", api_key_ref = "llm.key.llamacpp", backend = "llamacpp" }
- Run
anyrt serveso that the runtime becomes active. - If you are running an Anytwo Desktop, it’s probably been elected to run Bao itself, so you’ll need to kick the local device via the API to override it:
curl -s -X POST http://127.0.0.1:7001/v1/devices/activate -d '{"app": "bao"}'
And with that, you should have your own private Bao running in your own local hardware. Similar to my own custom agent, you can run interactions from any of device you own. Here is Bao with his default sarcastic personality, running with a Qwen 3.8 27B local model:
Disclaimer
Two important things about the information provided:
- The software is in heavy development, and many of these details may change soon, and in particular some of the manual steps I outline above are going to become simple settings in a UI near you.
- I’ve started looking into this over the last weekend, so my understanding of these details is obviously shallow.
References
Much of what I briefly mentioned here has proper documentation at:
The Cloudflare article I mentioned regarding the MCP approach being adopted lives at:
If you haven’t heard of Anytwo at all:
Congratulations and thanks
… to the whole Any org team, for developing and supporting Anytype in the open all these years, and specially now that there is investment in a relevant modern stack, for continuing to do so as open source and as an open process.
Also, thanks to Tolya for the interesting technical discussions about the present and future of the platform. Much appreciated.
