Create a Slack ChatBot using the new IBM Watson Conversation service

IBM has just released a new Watson service called Conversation, available on IBM Bluemix.  Conversation is a service that allows you to add a Natural Language interfaces to your applications to simulate conversations with your end users using Watson's cognitive capabilities.  There are a lot of uses for this service, but one I thought would be fun to prototype is a chat bot integrated with an instant messaging client.  You can find out more about the new IBM Watson Conversation service here: https://www.ibm.com/watson/developercloud/conversation.html

For the chat client, I decided on Slack because it provides a very rich set of integration APIs.  Specifically, Slack has the built in ability to create a custom bot that can be used within your team channels.  It also has the ability to distribute that bot to other teams, but that is a more advanced topic I will not cover in this post.  Instead of describing all the technical aspects of the Slack APIs, I will instead demonstrate how to use them with a practical example of building a Slack bot that your team can converse with using the Watson Conversation service as the backend logic for conducting the conversation with the bot.

So what type of bot to create?  Well, there is no shortage of ideas out there that's for sure.  Even Taco Bell has gotten into the game by creating their own Slack bot for ordering their food (https://www.tacobell.com/feed/tacobot).  What a great idea for those developers so busy that they just don't have any time to stop coding to go order food :)  So in the spirit of this, I will create a very simple bot to simulate a conversation about ordering food to demonstrate one potential way of using the Conversation service.

Let's first start simple, and build a very rudimentary Slack bot that does only one thing by responding generically to any message that is sent to it.  Once we've laid out how to integrate a bot with Slack, then we can move on to integrating that bot into the Conversation service to simulate food ordering.

Our first step in creating our own team bot is to go to the Slack team's setting page and make sure you are logged into the correct team (especially if you belong to multiple domains), and then create a new bot user by going to this URL and selecting a name in all lowercase: https://my.slack.com/services/new/bot

This will result in an integration settings page where you can optionally customize more aspects to your bot, including giving the bot an icon, first and last name, etc.  Most notably though is the API Token that you will need to keep private and use later for integration.

Integration with Slack and giving the bot the ability to communicate with other users is accomplished through the Real Time Messaging (RTM) API (https://api.slack.com/rtm).  This WebSocket-based API is able to authenticate with Slack, receive events, and send messages to users in real time.  The previous link shows the many types of events that your bot can respond to.  There is some API gymnastics required if we coded this integration from scratch.  But since others have already done a lot of the hard work associated with the API integration, let's use that and focus our attention on the overall integrated example.  In this case, we will use a node.js based toolkit for integrating with the Slack APIs.  I choose Botkit (https://howdy.ai/botkit/) since it is very well done and complete with a lot of examples and good instructions (https://github.com/howdyai/botkit/blob/master/readme.md).

Let's first start by installing Botkit locally with the following command (note, if you don't already have node.js installed, then hop on over to http://nodejs.org to download a package that will install both node and npm):

$ npm install --save botkit

Now using Botkit, we need to create our bot logic in node.js.  Let's start with something extraordinarily simple, the classic hello world.  First, we need to add the necessary setup code by requiring the Botkit toolkit, and then we create a controller that we will use to interact with Slack:

var Botkit = require('botkit');
var controller = Botkit.slackbot({ debug: false });

Once we have the controller, we can spawn our bot and connect the bot to the stream of messages coming from Slack (via the Real Time Messaging API).  Replace the token with the token you generated earlier so that it can be authenticated with the RTM API:

var bot = controller.spawn({ token: '<your_token>' }).startRTM();

Now, we simply want the bot to listen for the word "hello" by a direct message and reply back "Hello World".  This is accomplished by registering a listener as follows:

controller.hears(
   'hello',
   ['direct_message'],
   function(bot, message) {
      bot.reply(message, 'Hello World');
});

Save this Javascript code to a file called "foodbot.js" and run it in node on the command line:

$ node foodbot.js

Assuming you do not get any errors, you should be able to navigate over to your Slack client and direct message your new bot (note, if you do have errors, you can enable debugging by changing the parameter passed to the controller to true):

In this example, we are instructing the controller to listen for the word "hello" and only respond to that text while ignoring other messages.  We can use any regular expression pattern we want for the listener, so this can be very useful in building a bot where all the business logic is contained in this one service.  However, since we will be connecting this bot to the Watson Conversation service, we would like to just provide a pass through of any messages.  Let's change the line to the following which will allow the bot to respond to any message sent to it:

controller.hears(
   '(.*)',
   ...

Now let's turn our attention to how we can use the Conversation service on IBM Bluemix.  Go ahead and log into your Bluemix (http://bluemix.net) account and navigate to the dashboard and filter down to Watson services.  Then select the Conversation service and create it with all the defaults:

Before we get started, let's talk a bit about what the Conversation service can do.  As I mentioned earlier, this service allows you to add natural language interfaces to your applications to automate interactions with end users, exactly what we want to do with our Slack food bot.  It does this through 3 very simple concepts:

1. Intents.  You first need to train Watson to understand your user's input with example utterances.  You do that by declaring an "Intent" and giving examples of that intent.  For example, in our food bot, an intent can be to "place an order".

2. Entities.  You then need to define what are the things that the user will be trying to interact with.  You do that by declaring an "Entity" and provide terms that may vary in your users' input.  For example, in your food bot, an entity can be "pizza" that you want to order.

3. Dialog.  Now that you have the basic building blocks of a conversation, you can use the provided dialog builder to build a flow of how the service will create responses to your user's input.

This is all best understood through an example, so let's go ahead and get started.  Of course a dialog flow could get pretty complicated, but let's just do something very simple to demonstrate the basics.  Let's say that a user can ask to order one of two food items for now, a "pizza" or a "sandwich".

The basic flow looks like this.  User sends the food bot the message "Place an order for food".  The bot responds to the user clarifying food options "You can order a pizza or a sandwich".  The user can now respond with either option, e.g. "Pizza" or "Sandwich".  If the user asks for something not available, then we want the response to again ask for either the pizza or sandwich.  Very simple, but enough to show the integration we intended.

Once you launch the Conversation tool in Bluemix, the first step is to create a Workspace.  This is a way to maintain separate intents, entities, and dialogs for each application.  For this example, I will call my Workspace "FoodBot".  Once the Workspace is created, there will be a "Get Started" button to launch the tool to start creating Intents.

Go ahead and create your first Intent, give it the name "#order_food".  Then it will ask you for some User examples.  These are example of how you expect users to interact with this intent.  For example, a user might say "I want to order food", or "Place an order for a pizza".  Let's add more examples to this intent:

A more typical intent would have a much broader amount of examples, but this is good enough for now to train Watson on the basic things a user might say.

Now, let's create our Entities.  Since we have just this simple example, where the user is only ordering food, we only need one.  Let's name it "@food" and give it the two options: Pizza and Sandwich.  For each option, we will add a couple synonyms, so that for example if the user asks for a Sandwich or a Sub, either will be recognized.

And lastly, let's create the dialog flow.  In this simple example, we first add the intent "#order_food".  If that isn't recognized, then "Anything else" will kick in and ask the user what they want.  Then we add a child dialog and make it recognize any food, and respond with confirmation that their food has been order repeating the food item.  If the food entity is not recognized, then we default to the "true" condition, and which instructs the user as to which food items can be ordered.  Lastly, since we want to give the user the option to say the whole order in one sentence, verses needing to prompt what type of food, we have to add the "Continue from..." condition as show in the dialog flow here:

And you can see from the "Try it out" panel on the right side, that we get the answers we were expecting based on the dialog flow we created.

Now that we have a Conversation instance set up and ready to take commands, let's turn our attention back to integrating our original Slack Food bot with this conversation instance.  The API documentation for the Watson Conversation service can be found here ( http://www.ibm.com/watson/developercloud/conversation/api/v1/ ) although note that we are using V1 of the API and that will obviously change sometime soon.

To make sure we can interact with the Conversation API, let's first start simple and use CURL ( https://en.wikipedia.org/wiki/CURL ) from the command line to send our API commands.  The first command we will make is simply to verify we get an appropriate response.  So we need to issue the following from the command line:

$ curl -X POST -u "<username>":"<password>" -H "Content-Type:application/json" -d "{}" "https://gateway.watsonplatform.net/conversation/api/v1/workspaces/<workspace_id>/message?version=2016-07-11"

A couple things to note.  You first have to replace the username and password highlighted above with the credentials you use for your Conversation instance.  These are not your Bluemix credentials.  Instead, go to your Conversation instance, and select "Service Credentials" from the left hand navigation as shown here:

Secondly, you need to replace the workspace_id with your actual workspace ID for your Conversation instance.  You can find that by by first launching the Conversation tool.  Then select the "..." ellipsis from the Workspace and select "View details"

Notice that in the CURL command you are issuing, there is empty brackets as input for the POST.  This is because the Conversation service expects a JSON formatted input that we will provide later.  For now issue this command, and you should get back a JSON response like this:

{"context":{"conversation_id":"<conversation_id>","system":{"dialog_stack":["root"],"dialog_turn_counter":1,"dialog_request_counter":1}},"intents":[],"entities":[],"input":{},"output":{"log_messages":[],"text":["Would you like to order a pizza or a sandwich?"],"nodes_visited":["<node_id>"]}}

There are a couple important things to note in this response.  First, the service responds with a "context" which has a conversation ID that you can use to continue the conversation if you desire, and a "system" object which provides information about the dialog.  Because we provided no input, the service just responds with our catch all phrase "Would you like to order a pizza or a sandwich?".

Ok, let's take this and see if we can hook together our node.js code integrated with the Slack API to make this request to the Conversation service.  So first to make this easier, let's use the third party node module "request" which was built to make it very simple to make HTTP(S) calls.  Install it on your command line with the following command to the package manager:

$ npm install request

Now add the following require code to your node.js file:

var request = require('request');

And replace the line we created earlier:

bot.reply(message, 'Hello World');

With these lines of code:

function(bot, message) {
  request({
    auth: {
      username: '<username>',
      password: '<password>'
    },
    method: 'post',
    json: true,
    url: '<service_URL>',
    headers: {
      'Content-Type': 'application/json'
    }
  },
  function (error, response, body) {
    if (!error && response.statusCode == 200) {
      console.log('BODY: ', body);
    }
  }).end('{}');
});

What this code does is use the request module to do the POST method to the Conversation API.  First we need to specify the options for our request, including the "username" and "password" from before, as well as the URL we used in the CURL command (which included your workspace ID).

First stop your node.js service and start it again (you will need to do this any time you make a code change).  Now, when you message your bot in Slack, you should see a response in your node command window from the Conversation service like we did when we used CURL directly.  For example:

$ BODY: {
    context: { 
      conversation_id: '<conversation_id>',
      system: {
        dialog_stack: [Object],
        dialog_turn_counter: 1,
        dialog_request_counter: 1 } },
    intents: [],
    entities: [],
    input: {},
    output: { log_messages: [],
     text: [ 'Would you like to order a pizza or a sandwich?' ],
     nodes_visited: [ '<node_id>' ] } }

Now we want to extract the text from the output so we can have our bot reply with that text response in Slack.  So let's replace the code snippet:

console.log('BODY: ', body);

With the following which takes the first text response from the output:

bot.reply(message, body.output.text[0]);

Go ahead and message your Slack bot (after restarting the node.js service), and it should respond with the appropriate message from the Watson Conversation service:

Looks good so far, but upon further inspection this is not exactly what we want.  Because our node code doesn't actually simulate a conversation, so you will get this response no matter what you type in.  What we really need is for the code to first call the Conversation service to get the generic response as well as provide us a conversation id, so that we can actually continue the dialog with the service.  All the details for the format of the request expected, and the response given,  by the Conversation API are detailed in the documentation link I provided earlier.  So for now I will just show you the code that we need to 1) start the dialog with Conversation service, and then 2) make a second request to the Conversation service with the input from the user as well as the conversation id and some other information regarding the request counter, etc.  Given the need to make 2 Conversation API calls now, this is an opportunity for me to modularize the code into reusable functions.  Here is the completed code example:

var Botkit = require('botkit');
var request = require('request');
var controller = Botkit.slackbot({ debug: false });
var bot = controller.spawn({ token: '<your_token>' }).startRTM();

var username = '<username>';
var password = '<password>';
var workspce = '<workspace_id>';

var url = 
  'https://gateway.watsonplatform.net/conversation/api/v1/workspaces/' + 
  workspce + 
  '/message?version=2016-07-11';

// Gets the first text from an array of potential responses.
function getResponseText(params) {
  for (i = 0; i < params.length; i++) {
    if (params[i]) return params[i];
  }
  return "";
}

// Calls the Watson Conversation service with provided request JSON.
// On response, calls the action function with response from Watson.
function callConversationService(json, action) {
  request({
    auth: {
      username: username,
      password: password
    },
    method: 'post',
    json: true,
    url: url,
      headers: {
        'Content-Type': 'application/json'
      }
    },
    function (error, response, body) {
      if (!error && response.statusCode == 200) {
        action(body);
      }
  }).end(json);
}

// Register to listen for any user communication with bot.
controller.hears(
  '(.*)',
  ['direct_message'],
  function(bot, message) {      
    callConversationService('{}', function (resp) { 
      
      var conv_id = resp.context.conversation_id;
      
      var req_json = 
        '{\"input\":{\"text\":\"' + 
        message.match[1] + 
        '\"},\"context\":{\"conversation_id\":\"' + 
        conv_id + 
        '\",\"system\":{\"dialog_stack\":[\"root\"],\"dialog_turn_counter\": 1,\"dialog_request_counter\": 1}}}';
            
      callConversationService(req_json, function (resp2) {
        
        var txt = getResponseText(resp2.output.text);
        bot.reply(message, txt);
      });
  });
});

Let's break this code down a bit and describe what it's doing.  I register to listen for any user communication though Slack to the Food bot.  Then the code calls the Conversation service with an empty "{}" JSON request just to start the dialog and get back a conversation ID.  Then a new JSON request is constructed, this time providing the message the user gave to the bot, as well as context information needed to pass to the Conversation service (see details in documentation for each parameter).  Then the Conversation service is called once again passing this JSON request, and as a result, the Conversation service reply is located in the response and then sent to Slack to present to the user.  Note, that the Conversation service returns an array of responses, some of which might be empty due to the nature of the conversation, thus a function was created to get the first non-empty textual response.

This is a pretty simple example, and if you wanted to have an example that had more dialog with the user and the Conversation service, than this code example would need to be enhanced to make that possible since this code simulates the conversation with the Conversation service in a single message to the bot, instead of saving state and allowing the user to go back and forth in a dialog with the Conversation service.  But I'll leave that to you as a homework assignment ;-)

Lastly, we can see our code in action by interacting with the Food bot in Slack as shown here:

IBM OpenWhisk

I recently attended an IBM developerWorks Open Tech Talk on a new technology called IBM OpenWhisk.  At the time of this blog post, IBM OpenWhisk is experimental, and while trying it for the first time I ran into some issues you'd expect with any new technology. I'll address those issues in this post.

This technology is a great new open source cloud-based platform for letting developers easily build apps that are triggered by an event, and then automatically kick off a set of actions to respond. It fits perfectly in the trend of micro-service architectures what are emerging as the preferred way of building scalable cloud-based software using small, loosely coupled services distributed throughout the internet.

If you are familiar with IFTTT (https://ifttt.com/) which stands for "If This Then That", then you will be have an easy time understanding OpenWhisk. Basically, it's a simple way of using cloud services to connect the apps you use in a sequence to produce some desired result.

For example, I personally use IFTTT to connect Dropbox events (e.g. a picture is uploaded from my mobile phone) to Flickr actions (e.g. upload picture to Flickr). Once set up, this sequence of events happens automatically without my intervention.

So back to OpenWhisk.  Events trigger an action (or sequence of actions) to produce a result.  In OpenWhisk lingo, there are several main concepts:

• Trigger: A class of events that can happen and are fired (activated) by using a dictionary of key-value pairs (the event).
• Action: An event handler (or composition of event handlers) - stateless code that runs in response to an an event that uses a dictionary of key-value pairs for input, as well as for output where the key is a string, and the value is JSON.
• Rule: An association between a trigger and an action.
• Feed: A piece of code that configures an external event source to fire trigger events.
• Package: A bundle of feeds and actions.

A source emits events (e.g. CRUD operations) as triggers, and a rule determines the association between that trigger and an action the developer provides that encapsulates some code to be executed.

Currently, OpenWhisk supports Actions that developers code in 2 languages, either NodeJS or Swift, and also has the ability to support other languages or native code through an Action that wrappers a Docker container.

There is a limited set of triggers today, but seeing as this is open source and growing, I'm sure that will grow quickly. As of today, the available triggers are Cloudant and GitHub, as well as scheduled actions (basically timed triggers). You can also directly invoke an action by using the OpenWhisk API, CLI or iOS SDK.

My point of this blog post was not to describe OpenWhisk in detail to you, but instead show you how to use OpenWhisk through an example and provide feedback as an early adopter. If you want to find out more, there is a lot of documentation on OpenWhisk at either IBM developerWorks (http://developer.ibm.com) or IBM Bluemix (http://bluemix.net) as soon as you are approved for your access to OpenWhisk Beta.

So for this post, I decided it would be fun to try and connect GitHub as a trigger (for notifications) to Slack as an action to post messages so that when a notification occurs on GitHub, it is posted as a message to a specific Slack channel.

Like I mentioned, at the time of writing this post, IBM OpenWhisk is still experimental, and so you need to register and sign up for early access and be approved first. You can do that by going to IBM Bluemix, creating an account if you have not already, and then going to the OpenWhisk page and requesting access: https://new-console.ng.bluemix.net/openwhisk/

In my case, I'm running on the Mac OS X operating system, so my experience will come from that perspective.  The first thing to do when getting started is to install the CLI (Command Line Interface). This is pretty simple, and they provide instructions, but it requires pip, or the package management system for installing and managing software written in Python. In my case, I did not have it installed on my machine already. But there is an easy fix to that, simply run the following command and use your password when prompted.

sudo easy_install pip

To get started with my experiment, I first needed to configure the provided GitHub package as a service to fire a trigger when there is activity in one of my Git repositories.  In order to do that, you need your GitHub username, repository and a personal access token from GitHub. In order to get a personal access token for this application, simply generate one from this link from your GitHub profile settings: https://github.com/settings/tokens

Select "repro" and "notifications" as shown above to get notifications for when source control changes (commits) happen.

Now create a package binding for GitHub with the following CLI command:

$ wsk package bind /whisk.system/github myGit --param username <username> --param repository <git_repository> --param accessToken <token_in_previous_step>

You should see the following result:

ok: created binding myGit

Now you need to create a trigger which will fire on a GitHub event and using your feed. You can choose from any of the Github webhooks (an HTTP POST callback) available here: https://developer.github.com/v3/activity/events/types/. For this example, I want to see when anything is pushed to a branch in the repository. So I will execute the following CLI command:

$ wsk trigger create myGitTrigger --feed myGit/webhook --param events push

With the following results:

ok: created trigger feed myGitTrigger

Also note, that it's easy to make mistakes and get erroneous triggers that you don't want.  In that case you can simply execute the following command to get the full list of triggers:

$ wsk trigger list

And then the following command to delete the ones you don't want:

$ wsk trigger delete <trigger_name>

Next, you have to set up Slack for accepting incoming webhooks to integrate with your action.  First, goto Slack and sign into your account.  Then select "Apps & Integrations" from the Profile dropdown menu.

Then type in "incoming" to the search filter.

In the search results, install WebHooks for your domain.

You will be prompted to select a channel.

And then you will be given the Webhooks URL, as well as a bunch of options on how to customize the incoming messages.

To test out this integration, you can use cURL to run a quick test by POSTing the message "Hello World" to your Slack channel: 

$ curl -X POST -H 'Content-type: application/json' --dat'{"text":"Hello World"}' https://hooks.slack.com/services/<your_url_details>

Once you verify you get the message in Slack, then you can take your test one step further by binding the OpenWhisk Slack package with your configuration.

$ wsk package bind /whisk.system/slack mySlack --param url 'https://hooks.slack.com/services/<your_url_details>' --param username '<username>' --param channel '#<channel_name>'

And then directly invoking the action.

$ wsk action invoke mySlack/post --blocking --result --param text 'Message from OpenWhisk'

Again, you should see the message posted to your Slack channel.

Now we want a trigger coming from GitHub to invoke the posting to Slack action. But this can't be a direct call through a rule, because we need to somehow pass through the commit message from GitHub to Slack. This needs to be done in an indirect way, which can be accomplished through a Javascript action. For example, save this simple Javascript to a file called "myGitToSlackAction.js"

function main(params) {

  var commits = param.commits;

  var parameters = {

    text : "The file '" + commits[0].modified + "' modified by '" + commits[0].author.name + "'."

  };

  whisk.invoke({

    name: "/<myNamespace>/mySlack/post",

    parameters: parameters,

    blocking: true,

    next: function(error, activation) {
      whisk.done(undefined, error);

    }

  });

  return whisk.async();

}

Note, that you need to explicitly provide your namespace in the Javascript file. This is currently a defect being addressed by the OpenWhisk team. Just go ahead and replace the <myNamespace> with your actual namespace which can be found using the following command.


$ wsk package list

This Javascript is very simple, but hopefully provides the baseline of how you can indirectly go through actions to call Slack passing your intended message (in this case the commit message created using the commits object). From this simple sample, you can do all sorts of interesting things to the message you post to Slack, like provide a more details of the Git commit, or change aspects of the message on Slack like change the icon or username doing the posting. And in this example, it only posts the first commit commits[0], and you could instead loop through and post them all.

Now you need to create an action from this Javascript file:

$ wsk action create myGitToSlackAction myGitToSlackAction.js

And lastly we need to tie together the trigger with these actions through a rule:

$ wsk rule create --enable myGitSlackRule myGitTrigger myGitToSlackAction

Now the pipeline of events should be complete. When you commit a change to Git, it will trigger an event as was defined by your myGitTrigger which in turn invokes the myGitToSlackAction action, and as part of that action, it calls from Javascript the mySlack package binding passing the commit message you created.

You can test this out pretty simply by making a change to a file and committing it to your Git repository, for example:

$ git add <somefile>
$ git commit -m "Testing the GitHub-to-Slack message event"
$ git push origin master

So far we've done everything related to OpenWhisk on the command line.  But you can also do a lot from the IBM Bluemix web site, including editing your Javascript or creating actions and rules.  It also has a very helpful dashboard which shows you the details regarding anything that happens so you can easily debug any issues you may have. In terms of this particular example, you can go to the Activity Log and expand the "myGitTrigger" entry to see specifically what parameters are passed from the Git commit message to further customize the message posted to Slack.  Here is a partial screen capture of some of those key/value pair parameters:

Hopefully that gets you moving in trying out your own IBM OpenWhisk application pipelines!