AI-powered Furby

Make a Furby talk using an ESP32 microcontroller and locally hosted AI chatbot models.

Tech Stack

Frontend: C++, PlatformIO
Backend: Python, Whisper (faster-whisper), Llama (llama.cpp + phi-2), Text-to-Speech, LangChain, Faiss

Backstory

When home electronics are broken, I try to repair them as much as possible. When new products have much better energy efficiency or improved functions, I would rather buy them than repair them. Although sometimes I choose not to repair them, I often disassemble them to get ideas on how they work. I have repaired home electronics such as a computer mouse, an air purifier, an electric toothbrush, and a washing machine. I spent some time choosing what I bought and I want to use it up. Some people might see me as tight-fisted, but nowadays people perceive me as a person who contributes to sustainability.

While repairing products, I have come to realize most products have the main machinery and are controlled by microcontroller units. From time to time, I yearned to deepen my knowledge of electronics, and I even bought an Arduino kit. However, despite my enthusiasm, as I prioritized ongoing projects, I left the kit in a closet and never opened it. After completing the projects, I finally opened the kit and started learning Arduino. In the meantime, ChatGPT was thriving and I enjoyed using it.

A while back, the news "ChatGPT-powered Furby reveals toy's 'plan to take over the world'" caught my eye, and the memory of it came back to me. That was brilliant! I remembered how iconic and adorable Furby was. I wanted to have one too. Since I have been learning electronics, I wanted to use the components Furby already had. I wanted to tweak more freely and tried not to rely on online services.

Most Challenging Parts

After purchasing a Furby, I explored its original functions. I found them utterly fascinating. Considering its release date in 1998, the capabilities of the Furby completely blew my mind. I started to look for more information. I found its patent and parts of the code. It was written in the assembly language. Reading the code was both challenging and amazing.

I started to work on the project. My first goal was the transmission of sound data between the MCU and the server. Although I used a microphone module and an amplifier module instead of building circuits from scratch, it was quite challenging. It was hard for me to detect if the program was wrong or if the modules were already broken. After countless attempts, I managed to work it out. I integrated AI models into the server program, and I got a device to have conversations.

Next, I needed to control the movement of Furby. The easiest way was running its motor while the sound was playing. Furby can be controlled more precisely with a gear, an infrared LED, and a phototransistor. It is the same way to measure the movement of the gear as how the wheels of some computer mice do. Since motors can not be stopped immediately, the timing of the stop signals was crucial for precise control. Even though I wrote a program to measure the motor drifts, I found that adjusting the timing was more challenging than I initially anticipated.