In our first month of live calls, 31% of voice-agent conversations ended the same way: the receptionist said 'one moment', music started, and our agent, hearing nothing it recognised as speech, politely said goodbye and hung up. The retrieval was marked failed, a human called back, and the practice remembered us as the robot that hangs up.
Hold is not silence. It is music, recorded messages, ringback tone, and sometimes a live person breathing near an unmuted handset. We built a hold classifier on 40 hours of labelled audio from our own calls: a small model that runs on the audio stream and emits one of four states, speech, hold, ringback or dead air. It is deliberately conservative: when unsure it assumes 'hold' and waits.
The second problem was state. When a human comes back from hold they say 'okay, sorry about that, what did you need?' and the agent has to pick up exactly where it left off, not restart. We made the conversation plan explicit: a small state machine with named steps (identify, verify authorisation, state request, agree channel, confirm turnaround) and a 'resume' transition that summarises where we were in one sentence.
IVRs were the third. 'Press 3 for medical records' is easy; 'say the reason for your call' is not, and a surprising number of clinics route to voicemail after 5pm. The agent now navigates DTMF menus, recognises voicemail greetings and leaves a structured message with a callback number, and schedules a retry within business hours from the provider's memory profile.
Results after eight weeks: abandoned calls fell from 31% to 9%. Median call time rose by 90 seconds, entirely time spent on hold. Retrieval specialists report that the 'robot that hangs up' reputation has, at least at Riverside Family Practice, been replaced with 'the polite one that waits'.