As I showed last time, you can use the Conversation service visual Dialog Builder to create your conversation flow using the Intents and Entities you define. But what if you have a very large conversation you are trying to model? Let's say hundreds, or maybe thousands of potential questions that the user can ask with various different ways to ask the same question. Trying to model that with the visual Dialog Builder without errors would be quite a challenge. So in this blog post, I'm going to show you how to automate the creation of your dialog flow programmatically without needing a person to create the flow visually.
In order to demonstrate this capability, I am going to use a currently undocumented method. The Conversation service definition (all your Intents, Entities, and Dialog flow in a Workspace) are exportable in JSON format. Similarly, you can also take a previously exported JSON Conversation definition and import it into a new Workspace to create an instance of that definition. While the import/export function is documented in the official APIs, the details of that JSON format are not (at least not yet, it is anticipated that it will be at some point). In this post, I am going to leverage this capability to show how you can programmatically create this JSON format of a Conversation Workspace to import and create your Conversation instance. Doing it this automated way will reduce time to create your Dialog flow, as well as reduce errors, and make it a repeatable process for testing purposes, especially when we are talking about scaling to hundreds or thousands of Dialog entries.
For this demo, I'm going to keep the Dialog flow relatively simple and just concentrate on an Conversation app that will mimic a Question and Answer forum where the Dialog flow is only limited to a depth of one. This means, in this case I don't have to concern myself with defining Entities, but instead create a series of Intents with sample user input that represent the questions, and map that directly to the expected answer. In Watson terms, this represents our ground truth which is the gold standard for the learning algorithm in what the answer should be for a set of questions.
In order to create the ground truth, I need to define a corpus (all the questions, and their resulting answer) in a way that it can be programmatic read to build our Conversation model. Keeping things simple for this demo, I will go ahead and define the corpus in a CSV (Comma Separate Value) file which for each row will contain in the first column the answer, and the remaining columns the various ways of asking the question. Doing it this way will allow me to enter this information using any one of my favorite spreadsheet editors (e.g. OpenOffice) and export the contents to a CSV which has the added benefit of automatically encoding special characters so they can be imported directly to create a valid JSON document. For example, the corpus would take the format:
A.1, Q.1-1, Q.1-2, Q.1-3, Q.1-4, Q.1-5
A.2, Q.2-1, Q.2-2, Q.2-3, Q.2-4, Q.2-5, Q.2-6, Q.2-7, Q.2-8
A.3, Q.3-1, Q.3-2, Q.3-3, Q.3-4, Q.3-5, Q.3-6
...
Where A.N is the answer to questions Q.N-M. For example:
A.1 = On target earnings is a term often seen in job advertisements. The typical pay structure may be composed...
Q.1-1 = What is on target earnings?
Q.1-2 = What is target earnings?
Q.1-3 = What are the elements of on target earnings?
Q.1-4 = How is on target earnings calculated?
Q.1-5 = Is on target earnings part of a reference salary?
This is just a very small sampling of questions, a more typical example would have many more to more accurately train Watson.
Once the corpus is defined, I need to build the JSON definition for the Conversation Workspace based on this corpus. So first, let's look at what this JSON format looks like. To obtain one, just go to a Conversation Workspace instance that you have previously created and select the menu item to export as shown:
Here is an example downloaded JSON format. Note that I've beautified the output using a simple online tool ( http://jsonprettyprint.com ) as well as color coded it to highlight 4 distinct parts of the JSON that I will explain below. I also removed some text that was repetitive in terms of the explanation and replace it with "..." to make the size of this example manageable. Lastly, I inserted the corpus elements (questions and answer) in how I intend to model this Q&A model and will explain that below. Note, if you are unfamiliar with some of the sections and terms in use here, please go back to the original blog post I mentioned above to first become familiar with the Conversation service.
{
"workspace_id": "fede7b10-f035-44d8-9fcd-80cddbcf08db"
"name": "Workspace Test Name",
"description": "Workspace Test Description",
"language": "en",
"metadata": null,
"created": "2016-08-16T02:10:04.048Z",
"created_by": null,
"modified": "2016-08-16T02:10:04.048Z",
"modified_by": null,
"intents": [
{
"intent": "my_intent_0",
"created": "2016-08-16T02:10:04.048Z",
"description": null,
"examples": [
{
"text": "Q.1-1",
"created": "2016-08-16T02:10:04.048Z"
},
{
"text": "Q.1-2",
"created": "2016-08-16T02:10:04.048Z"
},
{
...
}
]
},
{
"intent": "my_intent_1",
...
}
],
"entities": [
...
],
"dialog_nodes": [
{
"go_to": null,
"output": {
"text": "A-1",
},
"parent": null,
"context": null,
"created": "2016-08-16T02:10:04.048Z"
"metadata": null,
"conditions": "#my_intent_0",
"description": null,
"dialog_node": "node_1_1471313404048,
"previous_sibling": null
},
{
...
},
...
]
}
The first section (highlighted in green) is the general information regarding the Workspace definition. Keep in mind, depending on how you set up your instance, the values might be different. In this case, the fields that we are most interested in are "workspace_id", "name", "description", and "language". These are pretty self explanatory, where the "workspace_id" is a Universal Unique Identifier (UUID).
The second section (highlighted in blue) defines all the Intents for this Workspace. This section of the JSON contains an array containing each intent. For each intent, there is an intent "name", "created" date and "description" (that I didn't use). Then there is an array of "examples" for each intent. Remembering how I defined the corpus, I can map the questions the user might ask for a particular answer each as an "example" under the parent intent. In this example you can see I used the symbols (i.e. "Q.1-1") but in the real example this is the actual text the user would say.
The third section (highlighted in red) defines all the Entities for this Workspace. Since I previously said that this Q&A example was not going to model a back and forth dialog for this particular demo with synonyms for objects, I don't need to use entities and thus this section is blank. If you did use entities, it would be modeled very similar to how the intents are modeled except using the synonyms instead of examples. So we can ignore this section for the remainder of this demonstration.
The fourth section (highlighted in yellow) defines all the Dialog nodes that you would normally model in the visual Dialog Builder. There is an array that contains for each element a definition of a node. For this example, the most important attributes to worry about are "output", "conditions", "dialog_node" and "previous_sibling". The "output" attribute defines a child "text" attribute that contains the text output that Watson would say for this particular node. Going back to how we modeled our corpus, this would be an answer (i.e. A-1) for a set of questions. In order to map this answer to the original questions defined by the intents, I need to set the "conditions" attribute. In this case, our condition is simply our intent name that then maps our questions directly to the ground truth answer. Lastly, there are two attributes "dialog_node" and "previous_sibling" which need to be defined correctly. For "dialog_node" this is a name that starts with the prefix "node_" and then has a count number (1-N) for each node defined, followed by a time stamp (milliseconds from 1970). For "previous_sibling", this points back to the name of the node previously defined at this depth of the node tree branch. In the example above, it shows null because this is the first node at that branch, but this would point to the previous sibling node for any subsequent nodes.
So now that I have defined the Q&A corpus model, and explained how we intend to insert that information in the JSON definition for a Conversation Workspace instance, I will turn attention to how we code up this solution in a generic way. I've decided to use Node.js because it will allow me to quickly create a CLI (Command Line Interface) tool that I can publicly share on npmjs.org. This CLI will import the CSV corpus file, create the required Conversation JSON object based on that corpus, and export it to a JSON file that can be imported into a Conversation Workspace. I will also set up the project so that the source code is available on GitHub.
So I start by picking a Node.js npm name for this project (e.g. bluemix-watson-conversation). You can use the following command to test to see if it's already in use:
$ npm view bluemix-watson-conversation
npm ERR! 404 'bluemix-watson-conversation' is not in the npm registry.
$ mkdir bluemix-watson-converation
$ cd bluemix-watson-conversation
$ npm init
This will ask you a series of questions, most of which are self explanatory, or you can take the defaults for. If you will be managing your code in GitHub source control as I do, you will need to create a new Git repository for your project, and then provide the "git repository" information when prompted so it can be included in your project descriptor.
The result of this command and questions will be a new directory for your Node.js project containing the package.json descriptor for your project. Assuming you took the default index.js file for your main project code file, you will need to create this empty file in your directory first which will be where all the following code I demonstrate will go.
So let's think a bit again about our CSV corpus model and the resulting Conversation JSON format. The modeling of these two artifacts are different in how the Questions and Answers are exposed in each document. In the CSV, the resulting answer and the possible questions are all contained in each row of the spreadsheet. However, in the JSON document, these elements are broken apart across the two different sections "intents" (for the questions) and "dialog_nodes" (for the answers) with the dependency established using the "conditions" attribute in the "dialog_node" as intent name reference. With that in mind, I will structure the code so that we can build our JSON document in one pass through without having to create an in between in-memory model to build that linkage. Here is the outline of the code:
// REQUIRE STATEMENTS ...
module.exports = function() {
main();
};
// CONSTANTS ...
function buildDialog( ... ) { ... }
function buildWorkspace( ... ) { ... )
function promptUserForInput(action) { ... }
function main() {
promptUserForInput(function() {
buildWorkspace(FILE_CSV_INPUT);
});
}
In this incomplete code snippet, I define the basic structure for our Node.js code.
- First, I will list the Node.js module dependencies that will be used.
- Second, I export the function so that anyone downloading this module will be able to run the CLI tool.
- Third, I have a place holder section for constants I will define that will represent the default values for various inputs that I might want the user of the tool to customize.
- Fourth, I define 2 functions. The first function buildDialog will be used to build the sections of JSON responsible for defining our dialog nodes. The second function buildWorkspace will be used to build the overall JSON document, calling out to buildDialog where necessary to build those sections.
- Fifth, is a function promptUserForInput which will be used to allow the CLI to get input from the user to customize the generation of the output JSON.
- Lastly, is the main function that the module exports and will first prompt the user for input, and then call the building of the Workspace function.
It's always best to reuse code, rather then build from scratch, so I chose a couple existing Node.js modules ( http://npmjs.org ) to help me build this program. First, fs which is the standard Node.js module for interacting with the file system. Second, csv-parse which is a module that can be used to parse our input CSV to ultimately create our JSON output. Third, prompt which is a slick module for gathering input from the user for our CLI tool. So in order to create these dependencies, first we need to install them locally as well as add those dependencies to our project descriptor package.json with the following commands inside our project directory:
$ npm install fs --save
$ npm install csv-parse --save
$ npm install prompt --save
Now I need to add those dependencies to the very top of our code index.js file:
var fs = require('fs');
var csv = require('csv-parse');
var prompt = require('prompt');
Next I will define the constants that the code will use placing these right under the //CONSTANTS comment:
var FILE_CSV_INPUT = './corpus.csv';
var FILE_JSON_OUTPUT = './output.json';
var WORKSPACE_ID = 'fede7b10-f035-44d8-9fcd-80cddbcf08db';
var WORKSPACE_NAME = 'Workspace Test Name';
var WORKSPACE_DESC = 'Workspace Test Description';
var WORKSPACE_LANG = 'en';
var WORKSPACE_ANYTHING_ELSE = 'I do not understand what you are asking.';
var WORKSPACE_INTENT_NAME = 'my_intent';
var WORKSPACE_DATE = new Date();
var WORKSPACE_DATE_JSON = WORKSPACE_DATE.toJSON();
The constants are defined as follows:
- The first 2 constants starting with FILE_ are used to designate the input corpus CSV file to use, and the output JSON file to generate.
- The next 4 constants WORKSPACE_<ID, NAME, DESC, LANG> are pretty self explanatory, but note that the UUID used for the ID is arbitrary as it will be replaced on import.
- The constant WORKSPACE_ANYTHING_ELSE will be used to define a dialog node text to generically respond if none of the corpus questions are recognized and matched.
- The constant WORKSPACE_INTENT_NAME will be used as the prefix for our intent names in the JSON.
- Lastly, the two WORKSPACE_DATE constants are used to populate dates in the JSON document as we build it in the code.
Now I will write the code for how to modify these default constants to customize the CLI using the prompt module. It would be repetitive to do this for each constant shown above, so I will just show one here in the promptUserForInput function. For more details on how to use the prompt module, see it's accompanying documentation ( https://www.npmjs.com/package/prompt ).
function promptUserForInput(action) {
var promptSchema = {
properties: {
fileInput: {
description: 'Enter the File Input (CSV Corpus)',
pattern: /^[a-zA-Z0-9\_\.\/]+$/,
message: 'CSV file name must be a valid file name.',
required: true,
default: FILE_CSV_INPUT
},
...
}
};
prompt.start();
// Prompt user with defaults, then override with input from user.
prompt.get(promptSchema, function (err, result) {
FILE_CSV_INPUT = result.fileInput;
...
action();
});
}
First thing I do is create the schema for the prompts we want for each constant setting the default value for the prompt to what we already defined as the default value in the code. Notice that for each prompt you can provide validation using a regular expression which is very flexible and useful for providing fool proof user prompts.
After all prompts have been defined, I call the start method to start the module and the get method to prompt the user for input on the command line for each prompt defined in the schema.
Lastly, I set the constant variables to the input from the user and when all variables are set, I call the provided action function that was passed in to kick off the creation of the JSON object.
The next step is to build the heart of the code in the buildWorkspace function which is defined to take the CSV file name as input. This is a lot of code with string manipulation to create the JSON which I will describe in more detail below. It also uses a few helper functions that I leave out of this code snippet to reduce the size (the complete code can be downloaded at the links at the end of the article).
function buildWorkspace(csvFile) {
var parser = csv({delimiter: ','}, function(err, data) {
var intents = '[', dialogs = '[';
var nodeName = null, nodeNamePrev = null;
var i = 0;
for (; i < data.length; i++) {
var intentName = WORKSPACE_INTENT_NAME + '_' + i;
var intent =
'{"intent":"' + intentName + '",' +
'"created":"' + WORKSPACE_DATE_JSON + '",' +
'"description":null,';
var answer = '', questions = '[';
for (j = 0; j < data[i].length; j++) {
if (data[i][j]) {
if (j == 0) {
answer = data[i][j];
}
else {
questions +=
'{"text":"' + data[i][j] +
'","created":"' + WORKSPACE_DATE_JSON + '"},';
}}}
questions = removeComma(questions) + ']';
intent += '"examples":' + questions + '}';
intents += intent + ',';
nodeNamePrev = nodeName;
nodeName = createNodeName(i);
dialogs += buildDialog('#' + intentName, nodeName, nodeNamePrev, answer)
+ ',';
}
intents = removeComma(intents) + ']';
dialogs += buildDialog(
'anything_else', createNodeName(i), nodeName, WORKSPACE_ANYTHING_ELSE)
+ ']';
var workspace =
'{"name":"' + WORKSPACE_NAME + '",' +
'"created":"' + WORKSPACE_DATE_JSON + '",' +
'"intents":' + intents + ',' +
'"entities": [],' +
'"language":"' + WORKSPACE_LANG + '",' +
'"metadata":null,' +
'"modified":"' + WORKSPACE_DATE_JSON + '",' +
'"created_by":null,' +
'"description":"' + WORKSPACE_DESC + '",' +
'"modified_by":null,' +
'"dialog_nodes":' + dialogs + ',' +
'"workspace_id":"' + WORKSPACE_ID + '"}';
fs.writeFile(FILE_JSON_OUTPUT, workspace, function (err) {
if (err) {
return console.log(err);
}
console.log('Worksplace JSON file saved to "' + FILE_JSON_OUTPUT + '".');
});
});
fs.createReadStream(csvFile).pipe(parser);
}
Let me break this down a bit:
- First I use the csv Node.js module to parse the CSV resulting in a data object 2x2 array containing the columns and rows of the input spreadsheet.
- Next I use a for loop to go through each row of this CSV (remember a row represents the answer followed by a list of potential questions).
- Within each pass of this for loop is another loop to iterate through each column entry in the row until there are no more questions (there might be blank entries depending on the amount of questions, so we ignore those).
- For the inner loop I extract the questions from the CSV that will be used to create the child "examples" attribute that will be contained in the "intent" attribute of the JSON object.
- For the outer loop I create the "intents" JSON attribute which contains all the intents and their examples (questions). For each pass, I also build up the dialog nodes definitions that will be needed later. I use the helper function buildDialog passing the intent name (used as the "condition"), along with the node name, previous node name and the answer text.
- After the loops are completed, I do one more call to buildDialog to generate the "anything else" condition node that will be selected if Watson can't find a matching answer to a question asked.
- And lastly I build the overall Workspace JSON inserting the "intents" and "dialog_nodes" that I created in the for loop passes and save that resulting JSON object representing the Conversation Workspace to the local file.
Lastly, I define the creation of the dialog JSON representing a dialog node:
function buildDialog(conditions, nodeName, nodeNamePrev, textOutput) {
return dialogJSON =
'{"go_to":null,' +
'"output":{"text":"' + textOutput + '"},'+
'"parent":null,' +
'"context":null,' +
'"created":"' + WORKSPACE_DATE_JSON + '",' +
'"metadata":null,' +
'"conditions":"' + conditions + '",' +
'"description":null,' +
'"dialog_node":"' + nodeName + '",' +
'"previous_sibling":' +
(nodeNamePrev ? '"' + nodeNamePrev + '"' : null) + '}';
}
And that's it! This code will now take as input our CSV corpus file, ask the user a series of questions to customize the output, and then generate our Conversation Workspace JSON file that can be imported into a new, empty Workspace you create on Bluemix. Just select import as shown below:
You can run the CLI with the following command inside your project directory:
$ node
$ > var run = require('./index.js');
$ > run();
Before I finish this blog post, let me show you how to publish this new CLI tool as a module on http://npmjs.org as a module you can share, as well as check in the code to source control.
If you are a new user to npm, you first need to create your account:
$ npm adduser <your username>
Now we can check in the code and publish it with the following commands:
$ git add package.json index.js
$ git commit -m "Check in 1.0.0"
$ npm version 1.0.0
$ git push && git push --tags
$ npm publish
That's it, now all the world can see and download your work! You can download the full code for this project from my GitHub repository here:
https://github.com/mc30ski/bluemix-watson-conversation
And download the Node.js module here and include it in your own projects as a required module:
https://www.npmjs.com/package/bluemix-watson-conversation
To try it out locally:
$ node install bluemix-watson-conversation
$ node
$ > var run = require('bluemix-watson-conversation');
$ > run();
