Skip to content

App UI

What is worth implementing in a mobile client, in priority order, and what happens if you skip an item. Frame mechanics are in Protocol v1.

Required minimum

A live WebSocket connection, a message feed and text input.

  • Do: connect with hello, a feed of responses (message), an input field → text with a constant id, a connection indicator, reconnect with a pause.
  • If you don't: the app does not work as a chat. Without reconnection, any network failure (mobile internet, device sleep) "hangs" forever — the feed stops responding and the user thinks the agent has broken.

Card buttons

The prompt frame carries text and options[{id,label,style}]. For exec_approval the server additionally sends the fields command (the exact command) and description — show the command next to the buttons so the person sees what they are approving. A press → one prompt frame with prompt_id and option_id; the server confirms with prompt_done (or prompt_done{expired:true}). The card can be closed by only the device it is shown to: a response from another connection receives forbidden.

Card types:

  • exec_approval — approval of a dangerous command. options — once / session / always / deny. Show command and description next to the buttons.
  • slash_confirm — slash-command confirmation: once / always / cancel.
  • clarify — a clarifying question: c0, c1, … plus other ("custom answer", together with text).

Styling: style — primary (primary button), danger (dangerous), empty — ordinary.

  • If you don't: the agent is forced to ask via text, and the user has to type /approve and the like by hand. Dangerous actions either don't go through or go through less deliberately; clarifications have to be entered manually. Do not duplicate the response to a card with a regular text — otherwise it will go to the agent as an extra message.

Server-side validation

Do not plug in option_id at random: the server itself will replace an invalid value with a safe one (deny / cancel) and return the actual choice in prompt_done.option_id.

Media in both directions

Sending (up). sendImage / sendDocument / sendVoice / sendMedia. media frames, kind ∈ image/video/document/audio. A file ≤ 8 MiB — in one frame; larger — in chunks of ~3 MiB with a constant id (the SDK does this itself).

Receiving (down). media frames from the agent (image/voice/video/document/animation): collect data by id, display/save on final: true; if there is a url — download by link.

  • If you don't: the user cannot send a photo/file/PDF to the agent, and images, reports and voice messages generated by the agent simply disappear. If you don't assemble chunks by id, you will get broken files for everything larger than 3 MiB.

Streaming (edit by id)

If streaming is enabled on the agent side, the response arrives in full and is then edited by message frames with the same id and the edit: true field. The app must store messages by id and, on edit: true, redraw the existing one rather than adding a new one.

{"t":"message","id":"r1","text":"Calculating","edit":true,"chunk":{"index":0,"total":1},"final":false}
{"t":"message","id":"r1","text":"Calculating the answer…","edit":true,"chunk":{"index":0,"total":1},"final":true}
  • If you don't: each edit frame will show up as a new message — duplicates will appear in the feed ("Calculating", "Calculating the answer…" as separate bubbles). The text is not lost, though: if edit_message fails, the gateway sends the remainder as a regular message.

Avatar

The avatar is an HTML page in a WebView implementing the window.hermesAvatar contract:

window.hermesAvatar = {
  setState:  (state) => {},  // 'idle' | 'thinking' | 'speaking'
  speak:     (text)  => {},  // request to voice/show the agent's line
  setVolume: (level) => {},  // microphone level while recording, 0..1
};

The page URL arrives in the ready frame (avatar_url; set by MOBILE_AVATAR_URL). If the URL is empty — use your own page (in examples/react-native/avatarHtml.ts there is a minimal one with the same contract). Take states from the avatar frames (idle/thinking/speaking).

  • If you don't: there is no "liveliness" — the user does not see that the agent is thinking/speaking. This is an optional item: without an avatar the protocol works fully, but feedback during a long response is lost (the typing indicator partially covers it).

Voice

Record WAV (16 kHz, mono, PCM 16-bit) → an audio frame with format: "wav" and data in base64. If the app has its own recognition — fill in text, then the agent gets both the text and the audio; otherwise Hermes recognizes the speech itself (with STT configured on the agent machine). Voice from the agent arrives as a media frame with kind: "voice".

  • If you don't: there is no voice input and no voicing of responses. Do not use uncompressed formats and a frequency above 16 kHz: recognition expects WAV 16 kHz — otherwise recognition quality drops and the recording "eats" the 8 MiB limit faster.

Local history

Cache the feed on the device so that on launch and on a network drop the user sees the latest messages, and the ids of already sent messages are preserved.

  • If you don't: an empty screen on every launch, and duplicates after a resend after a drop. The source of truth for history is still Hermes (the dialogue is not lost on the server), but a local cache is needed for pleasant UX and for idempotency (a repeat with the same id is ignored by the server).

Summary: what happens if you skip it

Item Without it
Reconnection the connection is not restored after a network failure
Card buttons approval/clarify/confirmation has to be typed as text
Media in both directions files and images "get lost"; large ones — broken
Edit by id duplicates in the feed during streaming
Avatar no feedback during a long response (non-critical)
Voice no voice input and no voicing
Local history empty screen on launch, duplicates after a drop