Conversational Robot

Problem

A NAO is a two-foot humanoid robot. This one could only recite scripted replies. The goal, written before any code: "NAO can be naturally induced into a conversation in which it answers to any statement appropriately in natural time without relying on a laptop."

The hard part is turn-taking. The robot must detect when a person stops speaking, even while its own voice is on the microphone.

Approach

Rules handle turn-taking, filtering and session limits. The language model only writes the reply.

flowchart LR
  P([person and the robot's own voice]) -->|sound| MIC[front mic 16 kHz]
  MIC --> VAD{onset 150 ms, endpoint 600 ms}
  VAD --> STT[speech to text]
  STT --> G{blank, echo or goodbye?}
  G -->|text| LLM[chat, streamed]
  LLM -->|sentence by sentence| TTS[robot speaks and gestures]
  TTS --> P
  VAD -->|onset while speaking| STOP[stop speech, cancel stream]
  STOP --> P

The first version, in 2024, ran on a laptop. The key constraint was the robot's limited processing: I used a MacBook to run the script and process speech, freeing the robot's thread for rapid speech production through the NAO SDK. The 2026 rebuild runs on the robot.

Solution

Voice activity detection compares microphone loudness to a noise floor that adapts to the room. While the robot speaks, the threshold rises above its own voice.

# whose turn is it? energy against a floor that tracks the room
start = rms > max(floor * 1.7, 300)            # sustained 150 ms: onset
if speaking:
    start = rms > max(base, echo_level * 1.2)  # louder than its own voice, 250 ms
end = quiet_for_ms(max(floor * 1.3, 150)) >= 600

Rules discard utterances under 300 ms, blank transcripts, echoes and goodbyes. The robot speaks each streamed sentence as it arrives.

flowchart LR
  A([armed]) -->|both feet within 1.5 s| I[idle]
  I -->|onset| C[capture]
  C -->|under 300 ms| I
  C -->|600 ms of silence| T[think]
  T -->|first sentence| S[speak]
  S -->|barge-in or head touch| C
  S -->|done| I
  I -->|goodbye, 30 s idle or a cap| A

An interruption stops the speech and cancels the stream.

# barge-in: stop, cancel, and record only what was actually said
def interrupt(self):
    self.gen += 1                 # any late write from the old turn is ignored
    self.stream.cancel()
    self.tts.stop_all()
    self.history.append(("robot", self.spoken_so_far + " [interrupted]"))

Each session stops after 4 empty turns, 600 seconds or 60 transcription calls. These limits were added after two runaway sessions on 2026-10-02.

SSH access revealed it ran Gentoo with about 8GB to spare. The robot came with a GUI application and simulator, but I stuck with the terminal for speed to a POC. Digging through the binaries led to NAOqi, for which I found deeply nested documentation. NAOqi only loads in Python 2.7, and there is no root access. A Python 2.7 bridge process therefore owns all robot calls.

flowchart LR
  subgraph Robot[NAO, Gentoo, no root]
    AL[NAOqi autoload] --> BR[bridge, Python 2.7 with NAOqi]
    BR <-->|local socket, 5-byte frames| TK[talk, Python 3.5, standard library]
    BR --> SVC[audio, speech, eyes, touch]
  end
  TK -->|HTTPS| GW[model gateway]

The NAOqi developer guide documents the speech and motion services.

Outcome

Latency over seven turns in three runs:

After a reboot, the robot was ready in one minute without a laptop. One failure is still open: the robot once heard its own voice and replied to itself.

Applicability

The design fits any voice interface that takes turns with a person, such as kiosks, phone lines and assistive devices. The bridge pattern fits any SDK that only runs on an old runtime.