Getting the experience right is one of the most difficult things when building chatbots and voice assistants. Your backend may be very powerful, but if the experience is not engaging and compelling, users will just leave. The amazing tech behind it will be useless if the users are actually confused by the interaction. With Botsociety, is now very easy to perform user testing on your design before actually building the chatbot.

Do not feel like reading? Check out the video tutorial! Otherwise, keep reading

1. Design your chatbot or voice experience

The first step is to design the experience you would like your users to have. For the purpose of this tutorial, I’ve come up with a classic Acme Chatbot. It is an e-commerce chatbot that is capable of selling equipment to Wile the Coyote. You get the idea 🙂

I’ve started by designing the happy path and added the other scenarios (If you want to learn how to design a conversational experience, check out this blog post first). This is what my design looks like:

User testing on Messenger chatbots

While designing the experience, I took my time to craft not only the main flow but also the alternative paths — for example, when a user selects “Add to my wishlist” instead of “Buy”:

User testing on Slack chatbots

Now that I designed multiple scenarios, I’m ready to start the testing session

2. Create a user testing session

In order to create a testing session, simply click on the blue button on the top right corner that says “Test:

User testing on chatbots and voice

I can now set up a minimum tests goal. Optionally, I can also select an end date for this session

User testing on chatbots

You can now choose if you want to involve your own set of testers, or you would like to use the Botsociety testers. I’ll go on and select the second one:

Tutorial on user testing on chatbots

Botsociety testers are high-quality, confidential users that will try out your design and give you a more precise idea about how your future users will interact with your chatbot.

Now I confirm the purchase and just lay back for around 20 minutes. Results will show up automatically, and I will receive an email when that’s done. I can check partial results by clicking on the “Test” button.

User testing a conversational design

Once the tests are completed, I will be able to check how people have used my design. Turns out, 75% of the testers selected “I need 10 [gadgets]” as the first option

Checking the results of user testing

After that, 38% of the testers chose the “Purchase” option under “Tennis balls”

User testing tutorial part two

Only 25% of the people chose “100 [gadgets]” as the first option

User testing a GoogleHome design

After such choice, all the users decided to continue with 10 (the Acme chatbot will not support 100 gadgets when launched). Nobody decided to get notified, which is surprising. I guess the feature is not perceived as useful as I thought. Maybe it’s not that clear.

I can also check out the results in a flowchart format:

User testing results

Very interesting results. I now have lots of ideas about how I can improve my design.

3. Iterate you design and test the new one

Ok so now I will revise my design — the notification feature needs to be explained better; we should propose a different choice as first (10 or 100 gadgets) as people do not seem to care a lot.

So by clicking on the “Back” arrow, I will be back to design mode. I can now change the design and run another user test. I will see something like this:

Two different sets of user testing

I will be able to check the results of the second test with the new design. I will also be able to check my old user testing session (which will show the old design) and make a comparison. Hopefully, the users will respond better to the features I care.

Once the design is finished, I will be ready to develop it!

Hope you find this tutorial useful. Please let me know if you have any questions in the comments. Try out Botsociety here

Also published on Medium.


Vittorio Banfi

Co-founder and CEO

Related Posts


How to design a Facebook Carousel with Sketch

When designing a Messenger chatbot, one of the nicest message formats that you can send is a carousel. The carousel allows you to send multiple images, titles and multiple links for each element of a Read more…


How to create your first Slack bot mockup

In this tutorial, we are going to create a simple Slack bot mockup. Designing a Slack interaction before actually building it will help you save a lot of time. How? By previewing and prototyping your Read more…


How to design a Facebook Messenger mockup

In this brief tutorial, we will go through the basic steps of designing your first Facebook Messenger mockup. Designing this chatbot interaction is crucial to the success of your project. If you get it wrong, Read more…