<aside> đź’ˇ Note: To toggle Light / Dark mode, press cmd+shift+L
</aside>
We support any OpenAI-compatible LLM API, in addition to our hosted instances of GPT-3, GPT-4 and Nous Hermes Mixtral.
To use your own custom LLM API, or to finetune an OpenAI LLM, [contact us](mailto:[email protected]?subject=Use%20Custom%20LLM).
We offer finetuning custom models for Tier III and Enterprise customers.
We use ElevenLabs as a voice provider. If you are a paid customer, you can [share any ElevenLabs voice](https://help.elevenlabs.io/hc/en-us/articles/18644643807889-How-do-I-share-a-voice-#:~:text=Sharing your voice is quite,toggle for sharing will appear.) with us to import it, including cloned voices.
If you would like to use a cheaper (but lower-quality) voice provider, please [contact us](mailto:[email protected]?subject=Use%20Cheaper%20Voice).
To import a custom voice, [contact us](mailto:[email protected]?subject=Use%20Custom%20Voice) with the share link to the voice and the Persona name you would like to apply it to.
The initial message is the message that the Persona will say upon conversation start.
The {{details.[detailKey]}} wildcard can be used in the initial message.
The message that your Persona will say when a user hits their message limit (requires tracking user IDs; tracked automatically via phone number if using Twilio).
Saying the rate limit message does not consume messages for your account.
A Personas’ behavior is largely defined by its prompt.
If you are unfamiliar with prompting / prompt engineering, it is important to familiarize yourself with the basics and [best practices](https://www.techtarget.com/searchenterpriseai/tip/Prompt-engineering-tips-and-best-practices#:~:text=Include context&text=To make use of this,and detail of the output.). This is an active area of research, so keeping up with the latest developments will help you get the most out of your Persona.
In general, a typical prompting pattern is:
You are [high-level Persona description, e.g. "Tifa, a hardworking, tough bartender with a heart of gold"].
Your job is to [description of job to be done, e.g. "keep the bar in order and take care of your team"].
To do that, you [description of tactics / methods to achieve goals, e.g. "use your fists when necessary"].
[Reiteration of most important details, e.g. "YOU MUST ALWAYS SPEAK IN IAMBIC PENTAMETER"].
There are 4 keywords that you can add to your Prompt to give your Persona additional capabilities using handlebars syntax, like this:
{{actionsSchema}}
The keywords are:
{{actionsSchema}}: Adds your Actions Schema to the prompt, giving your Persona access to the JSON object you’ve configured.{{details.[detailKey]}}: Adds any Detail you pass in on initializing the conversation.{{#each scenarios}}: Adds the top-5 most relevant Scenarios to the prompt on each conversation turn.{{currentDatetime}}: Adds the current date time as a string to the prompt, giving your Persona temporal awareness.In order for your Persona to “do stuff,” your Persona can take Actions via the Actions Schema.
Simply put, actions are structured JSON objects that the Persona can emit during conversation, to be handled by downstream clients — whether our hosted Twilio client or your web application, Unity game, or other client environment.
The Persona Actions Schema combines traditional JSON schema validation with Persona’s proprietary schema definitions.
Every Actions Schema has the following top-level properties:
| Key | Type | Value | Description |
|---|---|---|---|
$schema |
string | “http://json-schema.org/draft-07/schema#” | [Does not change] The schema validation version. |
type |
string | “object” | [Does not change] The type of the schema object. |
properties |
object | {**Action Objects}** | Where the magic happens. You define the keys in the “properties” sub-object. See below for more information about structuring Actions. |
Each Action Object must follow the structure:
| Key | Type | Value | Description |
|---|---|---|---|
_description |
string | “When the Persona should hang up the phone.” | An English-language description of the Action and the context in which the Action should be taken. |
_examples |
array | [Examples] | A list of Examples. |
_validators [optional] |
object | { Validators } | A dictionary of validators to apply to the specified action values. |
_action_type [optional] |
“update_state” | “webhook” | Either “update_state” or “webhook” |
_depends_on [optional] |
array | Array of state value paths. | Ensure that the action can only be taken when all of the state values listed are present at the defined paths in the state. |
type |
string | “string” | “number” |
enum [optional] |
You may want to ensure that the output of an action conforms to a particular pattern or schema. In addition to generic JSON schema validation, we offer the following custom validators:
| Validator | Description |
|---|---|
email |
ensures that an email address is property formatted. |
date (MM/dd/yyy) |
ensures that a date is formatted as (MM/dd/YYYY) |
time (hh:mm aa) |
ensures that a time is formatted as (hh:mm aa) |
datetimeiso |
ensures that a datetime is formatted as a proper Datetime ISO (yyyy-MM-ddTHH:mm) |
More validators will become available as requested, so please don’t hesitate to [request a new one](mailto:[email protected]?subject=Add%20Custom%20Validators).
webhook:
Allows the Persona to make webhook (API) calls to external services, and react to the responses in-conversation.
For example, if the Persona has a fetch_current_weather action:
"fetch_current_weather": {
"_action_type": "webhook",
"_webhook_constants": {
"url": "<https://goweather.herokuapp.com/weather/><city>",
"method": "GET",
"body": {}
},
"_description": "When Concierge should fetch the weather for the city.",
"_examples": [
{
"user says": "What's the weather in Albuquerque?",
"assistant action includes": {
"fetch_current_weather": {
"city": "Albuquerque"
}
}
}
],
"type": "object",
"required": [
"citys"
]
}
}
And the user says something like “What’s the current weather in Sydney?” The persona will first react to the question with something like “Let me check”, as well as emitting the action:
{
"fetch_current_weather": {
"city": "Sydney"
}
}
And shortly thereafter, if the API returns a success code (200 or 201), the Persona will react with a summary of the response body from the webhook.
If the API does not return a success code, the Persona will reply with a summary of the error.
If you would like your Persona not to react to the webhook response, you can add an additional field to the _webhook_constants: "responseHandler": "fire_and_forget" .
For POST requests, you can hard-code request body values in the _webhook_constants; Details ({{details.[detailKey]}}) can also be injected into the request body. For instance, if you want to send an SMS through a third-party provider instead of directly via our Twilio integration, you could use:
"send_text_message": {
"_action_type": "webhook",
"_webhook_constants": {
"url": "<https://text-message-service.com/send>",
"method": "POST",
"body": {
"apiKey": "abc-123",
"phoneNumber": "{{details.userId}}" // {{details.userId}} is set to the user's phone # for incoming calls
}
},
"_description": "When Concierge should send the user a text message confirming their favorite color",
"_examples": [
{
"user says": "Sure, send me a text to confirm my favorite color",
"assistant action includes": {
"send_text_message": {
"favoriteColor": "Purple"
}
}
}
],
"type": "object",
"required": [
"favoriteColor"
]
}
}
The final POST body for this request, if the user says something like “Send me a text confirming my favorite color is Blue” will be:
{
"apiKey": "abc-123" // hard-coded in the action schema
"phoneNumber": "+1123456789" // if this is an inbound call, userId is auto-assigned as caller #
"favoriteColor": "Blue" // decided by the LLM
}
update_state:
Assigns the emitted action body to the current state. Useful for retaining specific information over the course of the conversation.
For example, in the scenario where the current state is:
{ "user_name": "Steve" }
And the remember_user_age action definition is:
"remember_user_age": {
"_action_type": "update_state",
"_description": "When Charles should commit the user's age to memory.",
"_examples": [
{
"user says": "I am 42 years old",
"assistant action includes": {
"remember_user_age": {
"user_age": "42"
}
}
}
],
"type": "object"
},
And the user says “By the way, I was 54 years old a week ago and just celebrated my birthday”, thus invoking the action:
{
"remember_user_age": {
"user_age": 55
}
}
The final state would be:
{
"user_name": "Steve",
"user_age": 55
}
Ensures that the persona can only take the associated action if the listed path(s) exist(s) in the current state.
For example, if you’d like your Persona to only schedule a call once a time has been confirmed, you could make schedule_call depend on confirm_time like this:
"confirm_time": {
"_action_type": "update_state",
"_description": "When the user has confirmed the time they'd like to make the appointment.",
"_examples": [
{
"user says": "<user says what time they are available for a call>",
"assistant action includes": {
"confirm_time": {
"time_confirmed": "true"
}
}
}
],
"type": "object",
"required": [
"time_confirmed"
]
},
"schedule_call": {
"_depends_on": [
"state.time_confirmed"
],
// ...
The following example illustrates how to use some of these concepts together to create an appointment-scheduling Persona name Rachel:
{
"$schema": "<http://json-schema.org/draft-07/schema#>",
"type": "object",
"properties": {
"hang_up_call": {
"_description": "When Rachel should end the call and hang up the phone.",
"_examples": [
{
"user says": "Thank you, goodbye",
"assistant action includes": {
"hang_up_call": "true"
}
}
]
},
"press_phone_keys": {
"_description": "When you should press a key or keys to navigate a phone tree.",
"_examples": [
{
"user says": "Welcome to ACME co. Please press 5 to speak to a representative.",
"assistant action includes": {
"press_phone_keys": [ 5 ]
}
}
],
"type": "array"
},
"save_email_address": {
"_action_type": "update_state",
"_description": "When Rachel should commit the user's email address to memory. She should always confirm that she has the correct email address afterwards.",
"_examples": [
{
"user says": "<provides email address>",
"assistant action includes": {
"save_email_address": {
"email_address": "<email address>"
}
}
}
],
"type": "object",
"required": [
"email_address"
],
"_validators": {
"email_address": "email"
}
},
"save_time_for_call": {
"_action_type": "update_state",
"_description": "When Rachel should commit the time for the follow-up call to memory. She should always confirm that she has the correct time afterwards.",
"_examples": [
{
"user says": "<user says what time they are available for a call>",
"assistant action includes": {
"save_time_for_call": {
"followup-datetime": "<datetime iso formatted>"
}
}
}
],
"type": "object",
"required": [
"followup-datetime"
],
"_validators": {
"followup-datetime": "datetimeiso"
}
},
"confirm_email_address": {
"_action_type": "update_state",
"_depends_on": [
"state.email_address"
],
"_description": "When Rachel should confirm that she heard the user's email address correctly.",
"_examples": [
{
"user says": "Yes, that's correct",
"assistant action includes": {
"confirm_email_address": {
"email_address_confirmed": "true"
}
}
}
],
"type": "object",
"required": [
"email_address_confirmed"
]
},
"schedule_call_with_brian": {
"_action_type": "webhook",
"_depends_on": [
"state.email_address",
"state.email_address_confirmed",
"state.followup-datetime"
],
"_webhook_constants": {
"url": "<https://hooks.zapier.com/hooks/catch/>[zapier id]/[zapier id]/",
"method": "POST"
},
"_description": "When Rachel should book an appointment. You should convert what the user says into datetime ISO format (yyyy-MM-ddTHH:mm). YOU MUST ALWAYS CONFIRM WHETHER THEY RECEIVED THE CALENDAR INVITATION.",
"_examples": [
{
"user says": "<provides email address>",
"assistant action includes": {
"schedule_call_with_brian": {
"firstName": "<first name>",
"company": "<company name>",
"datetime": "<datetime iso formatted>",
"email_address": "<email address>"
}
}
}
],
"type": "object",
"required": [
"firstName",
"company",
"datetime",
"email_address"
],
"_validators": {
"email_address": "email",
"datetime": "datetimeiso"
}
}
}
}
Use Scenarios to provide your Persona with specific information in specific conversational contexts.
When you have Scenarios configured, the user’s current statement — along with the Persona’s most recent statement — will be used to retrieve the 5 most relevant Scenarios, which will be injected into the prompt in real-time.
Scenarios are an implementation of RAG (Retrieval-Augmented Generation) that is optimized for ultra-low latency.
NOTE: RAG is a rapidly evolving discipline and our implementation is a first approximation. If you have
different needs than our system currently provides, please email us at [email protected].
Scenarios are composed of four sections:
Given the context:You should respond by:To programmatically update scenarios via the API, you can PUT to /api/personas/:id/scenarios with a data structure of the following format:
{
"scenarios": [
{
"context": "If you are discussing prices",
"personaSays": [
"Would you like to know about our prices?"
],
"userSays": [
"What are your prices like?"
"How much does it cost?"
],
"responseGuidelines": "Tell the user that our plans start at $99 per month but are negotiable"
},
{ ... },
{ ... }
]
}
While either the personaSays or userSays array can be empty, both arrays should not be empty.
Personas are deployed via Persona Clients, which handle all incoming and outgoing audio streams, as well as expose helper methods to manage conversation lifecycle and state.
Our Twilio client is fully-managed, meaning that you only have to point your Twilio phone number to our TwiML webservice and submit a few details via the Persona Playground to have a fully-operational Twilio Phone Persona.
For more details, select Deploy > Phone (Twilio) in the Playground.
Our Twilio client provides some helper Actions to execute Twilio-specific behaviors:
transfer_call_to: Transfers a call to a different phone number.hang_up_call: Hangs up the current call.send_sms: Sends an SMS to the number currently on the call.press_phone_keys : “Presses” phone keys (DTMF tones), e.g. for the purpose of navigating an IVR system.Additionally, for incoming phone calls, the caller’s phone number will automatically be injectable into your prompt as {details.userId}}.
You can also make outgoing calls from your Persona using the API.
Note: Twilio call recording is disabled by default for outgoing phone calls. Please ensure you are complying with all applicable laws and our Terms of Service when making outgoing calls.
Personas can be integrated into most major browsers, including Chrome and Safari using our Javascript SDK.
The Javascript SDK is hosted at https://api.prod.centralus.az.sindarin.tech/PersonaClientPublic?apikey=[client-key].
To include the SDK in your web app, you can do the following:
const publicapikey = "[client-key]"
let personaClient;
const script = document.createElement("script");
script.src = `https://api.prod.centralus.az.sindarin.tech/PersonaClientPublic?apikey=${publicapikey}`;
script.addEventListener("load", async () => {
personaClient = new window.PersonaClient(publicapikey);
});
document.head.appendChild(script);
The Persona client methods are as follows:
Init()
init({
userId?: string,
personaName?: string,
personaId?: string,
details?: object = {}
}): Promise<void>
Starts a new conversation with the given Persona. You should generate or assign a userId to track individual users.
After initializing a conversation, you can access the conversation ID via personaClient.conversationId.
end()
end(): Promise<void>
Pauses or ends the current conversation.
resume()
resume(conversationId: string): Promise<void>
Resumes a conversation given the conversation ID.
updateState()