A robot that holds a conversation

In July 2024 a design professor emailed me about his robot. We had crossed paths in the spring through a request that reached me as "set up a private ChatGPT," and I built a small chat site for that in two days. What he actually wanted was the AI wired into NAO, a two-foot humanoid his lab uses, holding a real conversation. The next afternoon he walked the robot to my office and left it with me.

Young yet brave NAO.

Out of the box, NAO converses the way a phone tree does: a keyword recognizer wired to scripted replies. A fancy hairless Furby, if you will. The natural enhancement was AI, and due to the privacy-first needs of research, the project was constrained to the university's own AI infrastructure through our Azure contract. I had five to ten hours a week and a flight to Mexico a week out.

The first job was getting it online, and a campus network makes you earn that. Nothing gets past DNS until its hardware address is registered, and the only way to get the robot's was to stick a wire into the back of its head and plug it into my Mac. Register it, press the chest button for an IP, ssh in (the credentials ship in the docs). One cat /etc/os-release later: Gentoo¹. I had spent years daily-driving Arch Linux. I was home. The afternoon went to exploring: poke at something odd, hunt down the docs, repeat (the SDK documentation is barely indexed, sometimes in French, sometimes gone). Then I found the qi command line and the order of victories set itself: speech first (the robot will say anything you type, which my coworkers and I verified thoroughly), then motion, then the audio APIs. By the end of day one it could talk and walk on command.

NAOqi, the robot's operating layer, publishes every part of the machine as a network service: microphones, speakers, motors, the gesture library². So the robot keeps doing what it is good at, being a body, and a laptop on the same Wi-Fi does the thinking. Between them, a few hundred bytes of text per turn.

The conversation program is about 150 lines of Python. Connect, load a persona, then loop: listen, transcribe, ask the model, speak the answer with gestures.

session = qi.Session()
session.connect(f"tcp://{ip}:9559")

history = [{"role": "system", "content": open("prompt.txt").read().strip()}]

while True:
    user_input = listen_and_transcribe()          # the laptop listens
    history.append({"role": "user", "content": user_input})
    response = get_gpt_response(history)
    history.append({"role": "assistant", "content": response})
    nao_speak_with_animations(session, response)  # the robot answers

The speaking half is the part that makes people grin. NAOqi's animated-speech mode picks gestures to match what the robot is saying, and it takes stage directions inline with the words:

nao_speak_with_animations(
    session,
    "^start(animations/Stand/Gestures/Hey_1) Hello! I am NAO, and I'm "
    "ready for a conversation. ^wait(animations/Stand/Gestures/Hey_1)",
)

NAO does not balance itself out of the box: one careless "walk" and you are lunging to catch a very expensive toddler. Those inline stage directions mean certain words can trip a gesture. Python versioning on a fourteen-year-old SDK is archaeology. Listening is the hard one: knowing when a person has finished talking. Cut the mic too early and you interrupt; too late and the robot feels broken. I leaned on a speech SDK with that judgment built in, listening through the laptop while the robot carried the voice and the body. Doing it through the robot's own four head mics has been rebuilt twice and still is not finished.

The first real conversation, I asked the robot for the longest word in the English language. NAO tracks your face while you talk, head tracking is one of its built-in background processes, and with a few gestures on top it can really feel alive and responsive. Easier shown:

The professor was thrilled, and what he wanted next was to run it himself, which is what the five days were for. The deliverable was a setup script that builds the whole environment on a fresh Mac and tests the AI connection and the robot connection separately, so he could tell which half was unhappy without me in the room. By mid-September he was running it from his own desktop and writing personas for it, and he kept it running for the next ten months. The work fed into a federal grant proposal the lab submitted that winter.

Current project status.

The project is paused but ongoing. The robot's qi library now speaks Python 3, so the next round simplifies the chain and explores more integrated capabilities: tying the built-in behaviors into the model's orchestration, so the robot knows when to be a Furby and when to improvise.