The Turing test, created and developed by Alan Turing during the ’50, is the most famous test related to Artificial Intelligence capabilities. Its aim is to test the machine’s ability to express intelligent behavior indistinguishable from the one of a human. The test, as proposed by Alan Turing, can be synthesized as follows:

Turing test scheme

  • Participants
    • Actor A: A human evaluator
    • Actor B: Another human
    • Actor C: The machine you want to test
  • Setting
    • The participants are separated
    • The conversation has to take place on text-only channels, like screen and keyboard
    • The evaluator knows in advance that either (B) or (C) is a machine, but he does not know which one
  • The test
    • The evaluator (A) can have a conversation with (B) or (C)
    • The evaluator has to reliably tell the machine from the human. If he can’t, the machine has passed the test!

Sounds simple, right?

Since its inception, the Turing test has been highly influential. It is the most famous concept in the philosophy of artificial intelligence. You can find out more on Wikipedia.

Running a Turing test with Slack

Given that one of the prerequisites of the test is for it to happen through a text interface, Slack would be great to run it. So say you would like to run a Turing test on Slack: How would you do it? I came up with the following procedure to test it out. Please let me know what you think in the comments!

Don’t feel like reading? Check out the video below! Otherwise, keep scrolling

Note: The steps that follow assume that you already have a software to be tested. It does not need to be a Slack bot per se. If you are not sure on how to hook up your software with a Slack bot, check out this dedicated tutorial.

1. Set up the Slack workspace

You can create a Slack workspace or use one you already own. I suggest creating a new one with the name of the software you would like to test + the “Turing-test” postfix. It will make it easier to recruit human evaluators. So for example in my case, the Slackworkspace would be:

societybot-turing-test.slack.com

You can create a new Slack workspace from the Slack homepage here.

Once you have your workspace, it’s time to set it up. I like to have a room for evaluators to wait and chat with each other, and a secret one that is used by the evaluator during the test itself. The objective is to gather the evaluators ahead of time, and not have them bored out in an empty channel. At the same time, you want your evaluator not to be disturbed during the actual test. To achieve that:

  • Create a new channel called “Waiting room”
  • Delete the “random” channel
  • Create a new private channel called “Testing”
  • Create a new private channel called “Scientists”

Ok, now you just need to adjust your Workspace Settings so that users join the waiting room by default. Click on the name of your channel > Administration > Workspace settings. From this view, expand the default channels and select “Waiting room”:

2. Set up your human component

Since Slack clearly shows, on the left-hand side, if a user is a person or a bot, we will need to create a ‘fake bot’ that will actually be controlled by a human. In order to do that, we will use Zapier. Zapier allows you to visually connect API with one another. If you don’t have an account, create one it’s free. What we want to do is set Zapier up so that everything we say in the ‘scientists’ channel will be posted in the ‘testing’ channel as if a bot was saying that. At the same time, we want everything that is being said in the ‘testing’ channel to be posted in the ‘scientists’ channel. To do that we are going to create two Zaps.

To create the first Zap, select Slack as a trigger. Then select your Slack workspace. Zapier will ask you to authorize with Slack if you have not done it already. You can then choose the type of trigger you prefer. Look for the option ‘Posted message on a private channel’ and select it:

Turing test on Slack: how to set up

Then select ‘Scientists’ as a channel. Zapier will require you to test this step. To test it, write anything in the Scientists channel, any phrase.

You can complete the trigger. Now you can add another step to your Zap. Select ‘Slack’ again, and select again your workspace. Now choose “Send private channel message”. Select the ‘testing’ channel from the list. The message text should reflect the first step, so click on the icon on the top right corner, then select the field ‘text’ from the drop-down menu:

Turing test step two

As you can see in the screenshot, I’ve also changed the name of the bot to “Bot A”, and written :robot_face: as bot icon. This is important as you need to hide the fact that there is actually no “Bot A” to the evaluator. Now Zapier will ask you to test this step. You should get to a situation like this:

Turing test step three

The second Zap is the same but in reverse. So you want to select ‘tests’ as channel trigger, and ‘scientists’ as the step. It does not really matter how your bot will look, as only ‘scientists’ will take a look at this channel, not your testers.

3 Connect your software to Slack

Connecting an existing software to Slack is kind of simple. Generally speaking, Slack offers a bunch of options, from webhooks to the creation of a stand-alone chatbot user. There are a lot of tutorials out there about this topic, including one I’ve written that involves bot framework and Botsociety. Another option is to set up another Zap, put ‘Webhook’ as a trigger and ‘Slack’ as step. Then have your software call the Zapier webhook to send messages. In any case, feel free to connect your software as you prefer. It works either way.

Once you have connected your bot, you can invite him or remove him from the testing channel as you wish.

4 Invite the evaluators

This is the fun part. You can invite more people on your Slack workspace by clicking on the “+ Invite people” button on the bottom of the left-hand side panel. Once they are in the ‘Waiting room’, you can start the test.

5 Have fun

Invite one user at a time in the #testing room. Then invite either your bot or start talking in the ‘scientists’ channel. Have the two talk a bit. Then either kick your bot out or intervene in the testing room yourself and announce that the other ‘bot’ is joining. Once the two have finished talking, ‘announce’ again that the test is finished. Ask your evaluator to choose: Which one was the bot, and which was the human?


Is of course almost impossible, given the current technology, to pass a Turing test. Nonetheless, running them has been a meaningful, funny way of exploring the current state of the art.

That’s about it for today! We will run a Turing test with our Slack community in the next month. If you are curious, leave a comment on this post, and I will invite you to our Slack channel.

Comments


Vittorio Banfi

Co-founder at botsociety.io