We’ve changed our name! API.AI is now Dialogflow. Learn more here.

Cortana Integration

Overview

The Cortana integration feature allows you to export your agents in a Cortana compatible VCD format and use it for building apps.

The example code for an integration can be found here.

Below are instructions.

Preparation

Create a Universal Windows Project in Visual Studio.

Add a reference to the Dialogflow .NET library using Nuget.

Create a new application page and add a microphone/listen button. Then, add TextBlock control to display results.

In the Dialogflow developer console, go to Integrations from the left side menu and enable Microsoft Cortana integration.

Download a Cortana voice commands file from the Dialogflow developer console > Agent settings > Export and Import.

Unpack and copy the _W10.xml file to your project. Rename it VoiceCommands.xml. Set the Build Action of the file to Content.

Add a new field of AIService type to the App class.

C#

public AIService AIService { get; private set; }

Initialize the aiService field in the App constructor.

C#

var config = new AIConfiguration("CLIENT_ACCESS_TOKEN", SupportedLanguage.English);
AIService = AIService.CreateService(config);

There are two integration options in Cortana:

  • Register voice command patterns and Cortana will open your app and pass some parameters to it. Official Cortana documentation
  • Register voice command service to provide custom logic for request processing. Voice command service can provide answers without opening your app. Official Cortana documentation

Integration with Voice Commands

Add code to register VoiceCommands.xml at the end of OnLaunched method.

C#

try
{
    var storageFile = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///VoiceCommands.xml"));
    await AIService.InstallVoiceCommands(storageFile);
}
catch (Exception ex)
{
Debug.WriteLine(ex);
}

Implement OnActivated method to receive parameters from Cortana. The Dialogflow SDK has a pre-built method for processing application launch parameters (passed by Cortana) – AIService.ProcessOnActivatedAsyncfor processing. This method calls Dialogflow to get an action from Dialogflow based on speech / text input.

C#

protected async override void OnActivated(IActivatedEventArgs e)
{
    AIResponse aiResponse = null;
    try
    {
        aiResponse = await AIService.ProcessOnActivatedAsync(e);
    }
    catch (Exception)
    {
    // ignored
    }

NavigateToMain(aiResponse);

}

private void NavigateToMain(AIResponse aiResponse) { Frame rootFrame = Window.Current.Content as Frame;

// Do not repeat app initialization when the Window already has content,
// just ensure that the window is active
if (rootFrame == null)
{
    // Create a Frame to act as the navigation context and navigate to the first page
    rootFrame = new Frame();

    rootFrame.NavigationFailed += OnNavigationFailed;

    // Place the frame in the current Window
    Window.Current.Content = rootFrame;
}

rootFrame.Navigate(typeof(MainPage), aiResponse);

// Ensure the current window is active
Window.Current.Activate();

}

In the main application page, add processing for the AIResponse result. For example, output it to a TextBlock.

C#

protected override void OnNavigatedTo(NavigationEventArgs e)
{
    base.OnNavigatedTo(e);

    var response = e.Parameter as AIResponse;
    if (response != null)
    {
        var aiResponse = response;
        resultTextBlock.Text = JsonConvert.SerializeObject(aiResponse, Formatting.Indented);
    }
}

Integration with Voice Command Service

Add Windows Runtime Component project to the solution.

In the Extensions node of your application Package.appxmanifest add the following XML tags.

XML

<Extensions>
    <uap:Extension Category="windows.appService" 
        EntryPoint="DialogflowDemo.VoiceCommands.DialogflowVoiceCommandService">
      <uap:AppService Name="DialogflowVoiceCommandService" />
    </uap:Extension>
    <uap:Extension Category="windows.personalAssistantLaunch"/>
</Extensions>

Add class DialogflowVoiceCommandService to your Windows Runtime Component and add the following code to your VoiceCommandService class.

C#

public sealed class DialogflowVoiceCommandService : IBackgroundTask
{
    public async void Run(IBackgroundTaskInstance taskInstance)
    {
    }
}

Also, you will need BackgroundTaskDeferral to wait until the command is processed, and until VoiceCommandServiceConnection interacts with Cortana and Dialogflow interacts with the Dialogflow service.

C#

private BackgroundTaskDeferral serviceDeferral;
private VoiceCommandServiceConnection voiceServiceConnection;
private Dialogflow Dialogflow;

In the Run method we will process Cortana request using the following steps.

Store BackgroundTaskDeferral instance to wait until work will be completed.

C#

serviceDeferral = taskInstance.GetDeferral();
taskInstance.Canceled += OnTaskCanceled;

Initialize Dialogflow instance.

C#

var config = new AIConfiguration("YOUR_CLIENT_ACCESS_TOKEN",
                SupportedLanguage.English);

Dialogflow = new Dialogflow(config);

Get AppServiceTriggerDetails to get VoiceCommandServiceConnection.

C#

var triggerDetails = taskInstance.TriggerDetails as AppServiceTriggerDetails;

if (triggerDetails != null) { voiceServiceConnection = VoiceCommandServiceConnection.FromAppServiceTriggerDetails(triggerDetails); voiceServiceConnection.VoiceCommandCompleted += VoiceCommandCompleted; var voiceCommand = await voiceServiceConnection.GetVoiceCommandAsync(); ...

Use VoiceCommandServiceConnection to receive request text and command name.

C#

var recognizedText = voiceCommand.SpeechRecognitionResult?.Text;
var voiceCommandName = voiceCommand.CommandName;

Check the command name for different cases. For example you can:

  • Make requests to Dialogflow and launch your app with AIResponse from the Dialogflow.
  • Make request to Dialogflow and send response to Cortana with SendResponseToCortanaAsync method.
    (e.g. see different processing for "type" and "unknown" voice commands below)

C#

switch (voiceCommandName)
{
    case "type":
        var aiResponse = await Dialogflow.TextRequestAsync(recognizedText);
        await Dialogflow.LaunchAppInForegroundAsync(voiceServiceConnection, aiResponse);
        break;
    case "unknown":
        var aiResponse = await Dialogflow.TextRequestAsync(recognizedText);
        if (aiResponse != null)
        {
        await Dialogflow.SendResponseToCortanaAsync(voiceServiceConnection, aiResponse);
        }
        break;
}

Of course, you will need to wrap the entire code with a try...catch block and make task completed in the finally block.

C#

try
{
    ...
}
catch(Exception e)
{
    var message = e.ToString();
    Debug.WriteLine(message);
}
finally
{
    serviceDeferral?.Complete();
}

Full code of the service will look like this.

C#

public sealed class DialogflowVoiceCommandService : IBackgroundTask
{
    private BackgroundTaskDeferral serviceDeferral;
    private VoiceCommandServiceConnection voiceServiceConnection;
    private Dialogflow Dialogflow;

public async void Run(IBackgroundTaskInstance taskInstance)
{
    serviceDeferral = taskInstance.GetDeferral();
    taskInstance.Canceled += OnTaskCanceled;

    var triggerDetails = taskInstance.TriggerDetails as AppServiceTriggerDetails;

    if (triggerDetails != null)
    {
        var config = new AIConfiguration("YOUR_CLIENT_ACCESS_TOKEN", SupportedLanguage.English);

        Dialogflow = new Dialogflow(config);
        Dialogflow.DataService.PersistSessionId();

        try
        {
            voiceServiceConnection = VoiceCommandServiceConnection.FromAppServiceTriggerDetails(triggerDetails);
            voiceServiceConnection.VoiceCommandCompleted += VoiceCommandCompleted;
            var voiceCommand = await voiceServiceConnection.GetVoiceCommandAsync();
            var recognizedText = voiceCommand.SpeechRecognitionResult?.Text;
            var voiceCommandName = voiceCommand.CommandName;

            switch (voiceCommandName)
            {
                case "type":
                    {
                        var aiResponse = await Dialogflow.TextRequestAsync(recognizedText);
                        await Dialogflow.LaunchAppInForegroundAsync(voiceServiceConnection, aiResponse);
                    }
                    break;
                case "unknown":
                    {
                        if (!string.IsNullOrEmpty(recognizedText))
                        {
                            var aiResponse = await Dialogflow.TextRequestAsync(recognizedText);
                            if (aiResponse != null)
                            {
                                await Dialogflow.SendResponseToCortanaAsync(voiceServiceConnection, aiResponse);
                            }
                        }
                    }
                    break;

                case "greetings":
                    {
                        var aiResponse = await Dialogflow.TextRequestAsync(recognizedText);

                        var repeatMessage = new VoiceCommandUserMessage
                        {
                            DisplayMessage = "Repeat please",
                            SpokenMessage = "Repeat please"
                        };

                        var processingMessage = new VoiceCommandUserMessage
                        {
                            DisplayMessage = aiResponse?.Result?.Fulfillment?.Speech ?? "Pizza",
                            SpokenMessage = ""
                        };

                        var resp = VoiceCommandResponse.CreateResponseForPrompt(processingMessage, repeatMessage);
                        await voiceServiceConnection.ReportSuccessAsync(resp);
                        break;
                    }

                default:
                    if (!<span class="cm-variable-3">string.IsNullOrEmpty(recognizedText))
                    {
                        var aiResponse = await Dialogflow.TextRequestAsync(recognizedText);
                        if (aiResponse != null)
                        {
                            await Dialogflow.SendResponseToCortanaAsync(voiceServiceConnection, aiResponse);
                        }
                    }
                    else
                    {
                        await SendResponse("Cannot recognize");
                    }

                    break;
                }

        }
        catch(Exception e)
        {
            var message = e.ToString();
            Debug.WriteLine(message);
        }
        finally
        {
            serviceDeferral?.Complete();
        }
    }
}

private void OnTaskCanceled(IBackgroundTaskInstance sender, BackgroundTaskCancellationReason reason)
{
    serviceDeferral?.Complete();
}

private void VoiceCommandCompleted(VoiceCommandServiceConnection sender, VoiceCommandCompletedEventArgs args)
{
    serviceDeferral?.Complete();
}

private async <span class="cm-variable-3">Task SendResponse(<span class="cm-variable-3">string textResponse)
{
    var userMessage = new VoiceCommandUserMessage
    {
        DisplayMessage = textResponse,
        SpokenMessage = textResponse
    };

    var response = VoiceCommandResponse.CreateResponse(userMessage);
    await voiceServiceConnection.ReportSuccessAsync(response);
}

}