Extending Doly with Local AI, Memory and Learned Behaviours

TonyNorriss

Hi Everyone,

I have just ordered a Doly which I should be getting in a month or so. I love the philosophy around this project.

There are obviously other alternatives such as Loona etc. But, they tend to be baked in with their behaviour a lot more than Doly seems to be.

One of the things I find unappealing about some of those robots that connect to LLMs is that they just seem to end up as a lower quality LLM compared to what I can get from my Chatgpt subscription. Here is what I am interested in doing. I am not a programmer so would need to use AI to assist with this, and would take building up from simple ideas to start with.

My idea is to run Gemma 4 locally on another computer on the same Wi-Fi network, and have Doly call it when higher-level reasoning, vision or language understanding is needed.

What I’d really like is for Doly itself to retain persistent learning locally. For example:

memories of people, objects and places
learned vocabulary
person-specific preferences
a simple reward system (“stars”) that changes the probability of future behaviours
newly created behaviour routines that remain available even when Gemma is offline
eventually, perhaps, AI-generated behaviour scripts that are tested and then saved locally

Ideally, Doly would continue using its normal local functions for movement, sensors, charging, safety and existing behaviours, while Gemma acts more like a higher-level reasoning/learning assistant.

Has anyone experimented with this sort of architecture?

In particular, I’d be interested to know:

How much of Doly’s existing behaviour can run alongside a custom Python service?
Can the stock Doly service and a custom AI/learning service share access to sensors, camera and motors, or does one need to replace the other?
Is there a recommended way to store persistent custom memories/behaviour data locally?
Has anyone connected Doly to a local LLM running elsewhere on their LAN?
Are there any existing community projects around reinforcement learning, persistent memory, or dynamically generated behaviours?

I’m not trying to replace Doly’s existing personality so much as gradually extend it so that its individual behaviour can develop from experience.

Levent

TonyNorriss Hi Tony, welcome to the Doly family! 😊

As the creator of Doly, I just wanted to jump in and say a few things about your project because a lot of what you described is actually very interesting.

First, I’m currently working on an official project that will bring online AI/LLM features to Doly. I know many people have been asking for more possibilities with LLMs, vision, smarter conversations, memory, etc. I’m already working on this and hopefully I’ll be able to start some beta testing soon.

About your idea, technically a lot of it should be possible. The main thing to know is that currently you can’t really connect another custom service directly into the stock Doly software and simply expand its existing features while everything continues working as normal.

But luckily, you can create your own version using the Doly SDK.

The SDK gives you quite a lot of flexibility to build your own app and features. And even if the SDK doesn’t currently cover something you need, Doly is basically running on Raspberry Pi + Linux, and the hardware is open source too. So you’re not really locked into anything.

There are tons of Raspberry Pi, Python, computer vision, AI, memory/database and robotics libraries out there that you can experiment with. And these days, even without a strong programming background, AI coding tools can help a lot with building things step by step. Please start with simple things and don’t expect AI to build a fully working application from a single prompt. I strongly recommend building it step by step, module by module. This will make it much easier to test each part, understand what is working, and fix problems as you go.

I actually like your idea of keeping Doly responsible for the more basic and local functions while letting another computer on the same network handle the heavier AI processing. However, expanding or integrating directly with Doly’s current software isn’t possible at the moment.

Also, like I mentioned, I’m working in this direction myself and I’m planning to share an open-source example project that can give people a starting point for integrating custom AI features with Doly.

So definitely keep me updated with what you’re building. I’d love to hear more about how you’re planning to structure it, especially the persistent memory and learning part.

And don’t hesitate to ask questions. I’ll try to help as much as I can. 👍

TonyNorriss

Hi Levent,
Thanks for the encouraging and supportive reply.

I think it is definitely worth waiting for your update prior to launching into anything myself.

I am motivated by my background in psychology I think. I don't find it that interesting when a robot becomes essentially a less effective interface to an LLM than say a PC. But, I am much more interested in seeing if it is possible to extend the interactions that can be had directly with the robot itself, and whether it is possible for the robot to grow in its own knowledge and experience of the world.

Very interesting, and one of the main reasons I decided on Doly because I think most of what else that is out there doesn't have the flexibility to allow this sort of development.

TonyNorriss

Hi Levent,

This is the guy who I got the idea from. I think you would find this really interesting. He goes into making a self-programing robot that develops its own personality and intelligence. And with much simpler gear than what you have in Doly. So, it should be possible to achieve more with Doly in this respect.

https://www.youtube.com/watch?v=NAz847JrGU4

JRTEIT

This is a bot built by Naz Louis. I have built this bot and bulk of the programs are running on your PC not the bot itself. Pixel, the bot, runs only the front end and your PC runs the backend, which is where the intelligence is created. If your PC is not running the backend then this Pixel robot is basically a simple bot with limited capabilities. Pixel uses the ESP32-S3, which means the bot is the robot head and your PC is the brain. The ESP32-S3 only contains the display, WIFI, touch screen, firmware and WebSocket Client. Doly is a self contained bot with the brains of the bot onboard and does not need your PC to run it. I know from experience that creating this backend to run this bot is not a simple task in order to get it to work. I believe most users of Doly would be hard pressed to get this Pixel backend and frontend to function properly in order to have it work as Naz Louis is demonstrating.

TonyNorriss

JRTEIT Thanks for that JRTEIT. Great to hear from someone who has actually done this. Did it behave in the way that Naz was showing on his videos? I am interested to see what Levant is going to come up with so far as the LLM interactivity is concerned. I am running Gemma 4 as a local LLM on my system, so I thought that might be able to get Doly to store its own memories locally. Anyway, fascinating stuff.

JRTEIT

Hello Tony, I have built two of Naz's projects. Pixel the walking robot and this one is simply a bot that will respond to simple commands like walk, guard, time of day, turn around etc. The challenge was getting the programs and all the electronics and servos to work in order to make it function properly. The other project Omnibot is the one Naz uses his PC to act as the brains for this bot. I have the backend working just fine and the firmware built and flashed into the bot but have not connected it to my PC yet because I screwed up the camera hardware and had to order a new one before continuing on. I am sure it will work as he shows in his video but time will tell. Without the help of the VSCode AI agent there is no way I would have been able to get this far. But it is challenging and fun no less.

TonyNorriss

JRTEIT Thanks for that. From what you are saying I am thinking it probably would be too hard to muck around with trying to make alterations within Doly itself. Probably what is do-able is to have Doly as an embodiment for the local LLM to explore the world, and build up its own memories etc on my Mac Studio. I think that achieves the end I was aiming for.

I am teaching myself Python at the moment through "Brilliant.org" which is an amazing learning package. I want to know enough so I can start coding with a coding LLM as well.