Answered

Real world experience and bot training

T
Rank
Badge +3

As per question, the article and help is all well and good but it is not really landing our bots conversion.

 

Our Design

  • We reviewed 200K cases and distilled answers down to a few hundred to cover core needs and built the first 100 (then adding 50 per cycle)
  • Custom bot paths all ending with a resolution bot answer
  • 150+ answers all with at least 10 variations of the question (trained)
  • Some answers having key words to limit where they apply
  • We are refining the each resobot answer as we go (we have a weekly review process)

 

Current Performance

  • We are seeing a small number of resobot answer sends on the end of the the custombot
  • Our product is such that a request to add something could look like a request to remove something - but neither of these answers is being sent
  • 200 of 4000 conversations covered which seems low for 150 answers that cover our core business

 

Would be interested in how y'all have approached this and what has moved the needle for your org.

icon

Best answer by kevin 4 September 2020, 16:22

View original

9 replies

Eric Fitz
Userlevel 1

Hey @craig​, I did a lot of work in helping our Support team roll out its own iteration of Resolution Bot (back when it first launched as Answer Bot), so I might be able to offer some insight here.

 

My first question is - within the variations of the question you're using to train Resolution Bot, are you clustering? "Clustering" refers to using multiple examples of each variation within your training.

 

For example, for a question on "How do I reset my password?", you'd want to include three examples of "How do I change my password?", three of "How do I get a new password?", three of "How do I reset my password?" etc.

 

Love the weekly review process, could you share some insight on that?

 

When you say you're seeing a "small number" of answers sending, what number are we talking here?

 

What kind of resolution rate are you seeing, on average, for your top-performing Answers?

T
Rank
Badge +3

Yeah so using variations of;

do, could, should, would etc

 

Where it gets tricky is words like "Channel" for us which is a partner you connect to and you could add, connect, map, remove etc and they all read very similar sans that key word (which we have also added for exact matches).

 

Small numbers as in less than 3 or never sent.

 

Res rate at about 30%, send rate below 5%

Eric Fitz
Userlevel 1

Right, sounds to me like upping the sending rate is the key to success here. If a sending rate is low, it means that Resolution Bot is having a tough time matching the answers you've shared to conversations from your customers. Do you use qualifiers when setting up Answers?

T
Rank
Badge +3

Correct but probably more work required there

K
Badge +1

Hey @craig​ 

 

Have you been able to optimize Resolution Bot? We tinkered with it at the start of 2019 but were getting similar results and having similar struggles, and we decided to stop it. I've been wondering about trying to train it up again, but our app is fairly complex. I'm suspecting this might be more useful for simpler user experiences, such as ecommerce websites.

T
Rank
Badge +3

Hey Kevin, not to the level I am happy with. I am struggling to get basic, let alone advanced answers land even when training the bots with 15-20+ variations of the question.

 

I find the following challenges;

  • Where complex scenarios require end user explanations (or streams of consciousness)
  • Where terminology is shared across common features / scenarios
  • Where the first answer, generally then paths towards other answers - currently you cannot link answers or trigger further answers and would need to point to a knowledge base / other content

 

I still feel with consistent use of bots across the journey, there is scale and value to be had but not until they actually send and land in the correct context.

 

Am currently trying to talk to some experts @ Intercom on this to see what we are doing wrong, or what is wrong with the bot(s).

K
Badge +1

That makes sense. Thank you for the update.

 

Terminology being shared across features / scenarios was a large issue for us too -- particularly since popular misnomers from our users carry across too! I think another challenge here is that if you have a great product team, common questions will increasingly be answered in the product, such that over time the bot will be relegated to answer the more obtuse questions that are challenging to train against.

 

If you ever do have an "ah ha" moment that significantly increases the effectiveness / accuracy of the bot, I hope you update this thread or group.

 

I suspect this feature will be useful to our team in the future as Intercom improves its learning and accuracy. I'll give it another whirl one day.

T
Rank
Badge +3

@eric f11​ shout out mate, who can you loop in to help us all build our own SKYNETS and help start the resobot revolution (also cannot believe there is no robot emoji)

Eric Fitz
Userlevel 1

Hey @craig​ @kevin​ - my own experience with Answer Bot is that it's best served answering basic, "binary" questions, like "How do I reset my password?" or "How do I change my payment details?"

 

When Resolution Bot first launched (or Answer Bot, as it was known then), I was heavily involved in helping our support team build out our first 100 answers. It was challenging - in truth, Intercom is its own worst customer when it comes to Reso Bot due to the complex nature of inbound queries we regularly get.

 

I wrote a blog post about the experience with my colleague Leanne Harte - I hope it's insightful, but happy to answer any other questions you might have!

Reply


Terms & ConditionsCookie settings