How Does an AI Receptionist Work? One Call, Step by Step
What happens between the first ring and the message on your phone: how the audio travels, how speech becomes text, how the model decides and uses tools, how it talks back and hands off, and what to test at each step.

An AI receptionist is a relay of five parts working on every call: the phone network delivers the caller's audio to software, speech recognition turns it into text, a language model reads that text with your instructions and decides what to say or which tool to use, text-to-speech turns the reply into a voice, and the phone system transfers the call when a person is needed. This guide follows one after-hours call through each step, with links to the technical documentation behind each part and the question each part should prompt you to ask a vendor. If you want the plain-English definition and whether you need one, start with our guide to what an AI receptionist is.
Key takeaways
- Phone audio is thin. Twilio, a widely used telephony platform, streams call audio to developers as mu-law at a sample rate of 8,000 Hz, and Google offers a speech-recognition model built for 8 kHz phone audio. Recognition on phone calls is harder than on a studio recording.
- Knowing when the caller has finished is a setting. Deepgram's endpointing defaults to 10 milliseconds of silence and can be set higher. Too short cuts callers off; too long feels slow.
- The language model predicts; it doesn't look things up on its own. NIST calls confident wrong answers "confabulation," so prices and open times should come from your data through tools, not from the model's memory.
- Tools are how it acts. The model returns a structured request (check the calendar, look up a price, send a text), and the vendor's software runs it.
- Interruptions and hand-offs are engineering choices. Ask whether it stops talking when the caller does, and how a transfer reaches a person.
What happens on one call, step by step?
Here is the whole relay in one table. The sections after it take each step in turn.
| Step | What happens | The technology | What can go wrong | What to ask the vendor |
|---|---|---|---|---|
| 1. Ring and route | The call reaches the AI by forwarding or a ported number | Carrier plus a telephony platform | Forwarding switched off; calls to the old voicemail | How does my number reach you, and what happens if that fails? |
| 2. Listen | Speech becomes text as the caller talks | Streaming speech recognition | Misheard addresses, numbers and names | Which model handles phone audio, and how are key details confirmed? |
| 3. Decide | The model reads the conversation and your rules | A large language model | Confident wrong answers | Where do prices and facts come from? |
| 4. Act | The model asks for a tool: calendar, price book, text | Tool calls to your systems | Double bookings, messages not sent | Which actions can it take without asking? |
| 5. Speak | The reply becomes audio | Text-to-speech | Mispronounced names, talking over the caller | Can I fix pronunciations? Does it stop when interrupted? |
| 6. Hand off | The call goes to a person when needed | Call transfer | Dropped transfers, no one answers | What does the person hear, and what's the fallback? |
| 7. Wrap up | A summary and the details reach you | Messages, logs, recordings | Missing details, late delivery | Show me exactly what lands on my phone |
To make it concrete, picture an example call (illustrative, not a recording): at 11:40 p.m., a homeowner calls a plumbing shop and says, "Water's coming through my kitchen ceiling." Each step below says what happens to that call.
Step 1: How does the call reach the AI?
The shop has either forwarded its number to the AI provider or ported the number over. Forwarding keeps your existing carrier; porting moves the number itself, and the FCC says you can keep your number when you switch providers as long as you stay in the same geographic area. Our guide to keeping your number covers both.
Once the call arrives, a telephony platform hands the audio to the AI software. Twilio's documentation describes how its Media Streams feature "provides access to the raw audio from a Programmable Voice call by streaming it over WebSockets to a destination you specify." The format is fixed: Twilio's message reference says the encoding is "always audio/x-mulaw" and the sample rate is "always 8000" hertz (Twilio). Google's speech-to-text documentation describes mu-law as an "8-bit PCM encoding", and pulse code modulation of telephone voice is the subject of the ITU's G.711 standard, "Pulse code modulation (PCM) of voice frequencies," whose in-force edition is dated November 1988. In practice, phone audio carries much less detail than a studio microphone, and every later step has to cope with that.
What to test: call your own number from a cell phone after hours, with forwarding set the way it will be in real life.
Step 2: How does it turn speech into text?
As the homeowner talks, streaming speech recognition turns the audio into text, word by word. Vendors pick models built for phone audio; Google's documentation, for example, describes a "telephony" model to "use for audio that originates from an audio phone call, typically recorded at an 8 kHz sampling rate." Modern recognizers learn from very large, varied collections of recordings. OpenAI's Whisper paper on arXiv reports models trained on 680,000 hours of multilingual and multitask supervision, which is part of why today's systems cope with accents and noise better than older phone menus did.
The system also has to decide when the caller has finished talking. Deepgram's documentation explains that endpointing relies on a voice activity detector that "triggers when a sufficiently long pause is detected." By default it returns transcripts after 10 milliseconds of silence, and developers can set it to wait longer, such as 500 milliseconds. Set it too short and the AI jumps in while a stressed caller is still thinking; set it too long and the line goes quiet.
What gets misheard most is the information you need most: street names, house numbers, phone numbers and spelled names. A well-built receptionist reads those back ("That's 1412 Linden Avenue, right?") before relying on them. Our guide to accents, noise and bad reception covers the hard cases.
Be skeptical of any single accuracy percentage, ours included. TheKeyBot's own FAQ takes the same line: "Rather than quote an accuracy figure, we would rather you test it on your own line and your own callers."
What to test: call from a car or a noisy room, give a street name that's easy to mishear, and check whether it reads the address back.
Step 3: How does it decide what to say?
The transcript goes to a large language model along with the vendor's instructions and your business details: your greeting, hours, service area, services, safety rules, escalation rules and, if the plan quotes, your price book. The model writes the next thing to say.
It's worth understanding what that model is doing. NIST's Generative AI Profile (NIST AI 600-1, July 2024) defines the main risk plainly:
"Confabulation: The production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications')."
The same document says confabulations are "a natural result of the way generative models are designed," because the models generate likely-sounding text. For a receptionist, that means facts such as prices, open time slots and whether you serve an address should come from your own data at the moment of the call, not from the model's general knowledge. When the data has no answer, the right reply is "a technician will confirm that," never a guess.
For our ceiling-leak caller, the instructions decide the first move, and safety rules belong ahead of booking. Some come straight from regulators: Pennsylvania's utility commission, for example, tells anyone who smells gas indoors to leave the building and call 911 and the gas utility from a safe distance, so a plumbing or HVAC receptionist should say exactly that before anything else. For a leak, your rules might have the AI ask whether water is near light fixtures, whether the caller can shut off the main, and then take the address. Those rules are yours to write.
What to test: ask for a job that isn't in your price book, and ask about an address just outside your area.
Step 4: How does it check the calendar, look up a price or send a text?
The model can't touch your systems directly. Instead, it asks for a tool. Anthropic's documentation for Claude explains the pattern: tool use, "also called function calling," lets the model call functions you define, and "it then returns a structured call that your application executes." Other model providers work the same way.
On our example call, the tools might run like this:
- Look up the service area for the caller's ZIP code. In area, so continue.
- Check the on-call schedule or calendar for the next available plumber. Calendar systems answer that with free/busy checks; Google's Calendar API, for example, returns free/busy information for a set of calendars.
- Look up the after-hours rate in your price book, if your plan quotes.
- Create the job or the message, and text the on-call tech.
- Text the caller a confirmation, or a payment link if you take deposits.
Each tool result goes back to the model, which turns it into plain speech: "Our on-call plumber is Dan, and he can be there between 12:30 and 1:00." What a product can do depends entirely on which tools it has and which you've switched on; message-takers stop after step 4. Our guides to integrations and to booking versus dispatch go deeper.
What to test: book a slot that's already taken, and check your calendar afterward rather than trusting what the AI said.
Step 5: How does it talk back, and what if the caller interrupts?
Text-to-speech turns the reply into audio. Google's documentation describes its service as an API that lets you "create natural-sounding, synthetic human speech as playable audio", from plain text or SSML. SSML, the W3C's Speech Synthesis Markup Language (version 1.1, 2010), is "a standard way to control aspects of speech such as pronunciation, volume, pitch, rate, etc." That is how a system learns to say your business name, a street name or a phone number the way locals do.
Callers interrupt, especially upset ones. A well-built system stops talking when the caller starts. On Twilio's platform, for instance, the software sends a "clear" message to interrupt audio it has already queued, which "empties all buffered audio." Whether a product does this well is easy to hear on a test call.
What to test: interrupt the AI mid-sentence with a correction, and listen to how it says your business name and two local street names.
Step 6: How does it hand the call to a person?
Some calls need a human, and the phone system does the hand-off. On Twilio, the Dial verb is used "to connect the current caller to another party." In the SIP standards that business phone systems use, the REFER method (RFC 3515, 2003) is the mechanism behind many transfers; the standard says it "can be used to enable many applications, including call transfer."
The technology is the easy part. The hard part is the rules: which calls transfer, to whom, what the person hears before the caller is connected, and what happens when nobody answers at midnight. Our guides to warm and blind transfers and what an AI receptionist can't do cover those choices.
What to test: at 2 a.m., say "I need to talk to a real person," and see where the call goes.
Step 7: What happens after the call?
When the homeowner hangs up, the system writes up the call: who called, from where, what they need, what was promised, plus a recording and transcript if the product keeps them. Then it delivers that summary to you by text, app, email or your scheduling software. Recording has legal rules of its own; see our guide to AI receptionist privacy. Listening to a handful of recordings each week is the best way to catch problems early; our call recording review guide shows how.
What to test: check whether you could act on the summary without calling the customer back to ask something basic.
Where does each step usually fail?
| Step | Failure to watch for | A test call that catches it |
|---|---|---|
| Ring and route | Calls still going to the old voicemail | Call your number after hours from a phone the vendor doesn't know |
| Listen | Misheard street or phone number | Call from a noisy place and give a tricky street name |
| Decide | An invented price or a wrong yes | Ask for a job that isn't in your price book |
| Act | A double booking or a text that never arrives | Ask for a time that's already taken; request a payment link |
| Speak | Talking over the caller | Interrupt mid-sentence with a correction |
| Hand off | A transfer that rings out | Ask for a person at 2 a.m. |
| Wrap up | A summary missing the address or callback number | Read what lands on your phone after each test |
Run the set on every finalist; our trial test-call script has 20 calls, and our annotated call transcript shows what a good call looks like line by line.
How does TheKeyBot fit this picture?
Disclosure: TheKeyBot, which publishes this guide, makes an AI receptionist. Its features page says the AI receptionist "automatically detects and responds in English or Spanish. No manual switching required." KeyBot Lite ($149 a month) runs the relay through step 7 as a message-taker: per the Lite page, it sends each call to your phone as a structured Telegram message, alerts you when a caller hangs up early, and texts callers who hang up before leaving details. It doesn't quote or book; Core ($500 a month) adds scheduling and GPS dispatch. Test it the same way you'd test anyone else's.
What should you do this week?
- Write your rules before you shop: safety calls, what to quote, what to book, who gets transfers.
- List the details you need from every call (name, callback number, address, job, urgency) so you can check every summary against it.
- Collect your hardest real calls, such as noisy lines, accents and long addresses, and use them as test scripts.
- Run the seven test calls above on each finalist and read what arrives.
- Ask each vendor the "what to ask" column from the first table, in writing.
Frequently Asked Questions
How does an AI receptionist understand callers?
It streams the call audio to a speech-recognition model that turns speech into text as the caller talks, often a model built for 8 kHz phone audio. A language model then reads that text along with your instructions and business details and decides what to say next.
Does an AI receptionist make things up?
It can, if it's allowed to answer from memory. NIST calls confidently stated false content "confabulation," so a well-built receptionist takes prices, open times and service areas from your data through tools, and says a technician will confirm anything it can't look up.
How does an AI receptionist book appointments or check prices?
Through tools. The language model returns a structured request, such as "check the calendar" or "look up this job in the price book," and the vendor's software runs it against your systems and hands back the result.
Can callers interrupt an AI receptionist?
On a well-built system, yes. The software can stop the audio it has queued when the caller starts talking, so test it by interrupting mid-sentence during a trial.
How does an AI receptionist transfer a call to a person?
The phone system connects the caller to another number, using a transfer feature like Twilio's Dial verb or the SIP REFER method. What matters most is your rules: which calls transfer, what the person hears first and what happens if nobody answers.
How accurate is an AI receptionist's speech recognition?
There's no single honest number, because accuracy depends on the audio, the caller and the model. Test it on your own line with your own callers, including noisy calls and hard street names, and check whether it reads key details back.
Sources
- Twilio Docs, "Media Streams": https://www.twilio.com/docs/voice/media-streams
- Twilio Docs, "Media Streams WebSocket messages": https://www.twilio.com/docs/voice/media-streams/websocket-messages
- ITU-T Recommendation G.711, "Pulse code modulation (PCM) of voice frequencies": https://www.itu.int/rec/T-REC-G.711
- Google Cloud, Speech-to-Text, "Introduction to audio encoding": https://docs.cloud.google.com/speech-to-text/docs/encoding
- Pennsylvania Public Utility Commission, "Gas Emergencies": https://www.puc.pa.gov/pipeline-safety/gas-emergencies/
- Google Cloud, Speech-to-Text, "Select a transcription model": https://docs.cloud.google.com/speech-to-text/docs/transcription-model
- Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision," arXiv:2212.04356: https://arxiv.org/abs/2212.04356
- Deepgram Docs, "Endpointing": https://developers.deepgram.com/docs/endpointing
- NIST AI 600-1, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" (July 2024): https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
- Anthropic, Claude Platform Docs, "Tool use with Claude": https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview
- Google Calendar API, "Freebusy: query": https://developers.google.com/workspace/calendar/api/v3/reference/freebusy/query
- Google Cloud, Text-to-Speech, "Text-to-Speech basics": https://docs.cloud.google.com/text-to-speech/docs/basics
- W3C, "Speech Synthesis Markup Language (SSML) Version 1.1" (September 7, 2010): https://www.w3.org/TR/speech-synthesis11/
- Twilio Docs, TwiML "Dial" verb: https://www.twilio.com/docs/voice/twiml/dial
- IETF, RFC 3515, "The Session Initiation Protocol (SIP) Refer Method": https://datatracker.ietf.org/doc/html/rfc3515
- Federal Communications Commission, "Porting: Keeping Your Phone Number When You Change Providers": https://www.fcc.gov/consumers/guides/porting-keeping-your-phone-number-when-you-change-providers
- TheKeyBot FAQ: https://www.thekeybot.com/faq
- TheKeyBot features: https://www.thekeybot.com/features
- TheKeyBot, KeyBot Lite: https://www.thekeybot.com/lite
About the Author
TheKeyBot Team is dedicated to helping locksmiths grow their businesses through AI automation and smart technology. With years of experience in the locksmith industry, our team provides actionable insights and proven strategies.
