Windows 10 through 10. Issue # 3. Using Cortana to interact with users

In this issue of our Windows 10 to 10 series of articles , we will discuss how to extend the experience of interacting with your application in Windows 10 to those moments when the application is not even running. The topic of the last article, live tiles and notifications , revealed one of the possible ways, today we will look at how to use Cortana, a personal assistant in Windows 10, for this task. We will use the example AdventureWorks, available on GitHub as the basis for the cited code fragments. .

In this article, we will try to figure out what Cortana is like, how to make it support meaningful interaction with your users, what needs to be done to integrate Cortana into the application, and then consider two ways (of many) how your application can interact with users depending on the scenario.
')

What is Cortana?

One of the most interesting features of Windows 10 is Cortana’s personal assistant, previously available on Windows Phone. Cortana is a key element for communicating with Windows 10 in natural language. With the help of Cortana, users can interact with Windows (both with the OS itself and with your applications), using the usual voice way of communicating with other people.

For example, think about the questions that only your application can give answers to and which people could give in Cortana’s voice: “when is my next trip?”, “Find a radio station”, “is Jack available?”. In turn, your application can answer such questions by providing an answer to Cortana for the utterance and display.

Or imagine the tasks that a person could ask Cortana to do in the context of your application: “cancel my trip to London”, “remember this radio station”, “write Jack, I'll be back later”.

Voice commands can provide quick access to information within the application. This is a kind of deep link in your application. Just as you create tiles today to provide quick access to the contents of your application, you can use voice commands as a quick way to “look inside” the application. And just as you make it possible to pin the next trip created in the app on the Start screen, you can also allow the user to use the voice command for Cortana to access the same trip. This feature will make your users more productive, and interaction with the application more accessible.

By expanding Cortana, you can improve interaction with your users and please them with the ability to perform actions with fast voice commands. However, since this article is not about Cortana ’s user experience , let's figure out how to integrate with it.

Hey, Cortana, let's talk

Since the interaction with Cortana rests on the voice, the created experience of interaction should be as close as possible to the natural dialogue. On MSDN, you'll find a general design guide for Cortana with recommendations for building engagement experiences. To achieve the best result, we advise you to follow the following principles: efficiency, relevance, clarity and trust.

What do they mean in practice?

Efficiency : less is better. Be short and try to reduce the number of words without losing the meaning.
Relevance : do not deviate from the topic. If I asked to add my favorite ABBA song to the playlist, I don’t need to report a low battery at the same time. Better confirm that in a moment I can enjoy ABBA. :)
Clarity : dialogue must fit the audience. Use normal everyday language instead of jargon understood by a narrow group of people.
Trust : Answers must accurately reflect what is happening and respect user preferences. If your application did not complete the task, do not say that you did it. Also, do not return phrases that, from the user's point of view, would not be worth saying out loud.

At the same time, you should consider localizing interaction with Cortana, especially if you have already localized the rest of the application and made it globally accessible. Cortana is currently available in the USA, Great Britain, China, France, Italy, Germany and Spain ( announcement about expansion into additional markets ). Localization will help you encourage users to use Cortana more often to interact with your application.

Teaching Cortana what to answer

Cortana uses the Voice Command Definition (VCD) definition file to understand how the user can interact with your application using voice. This file can be a ready-made XML file or generated from code (you will need to fill in a set of commands in a VCD file when you first start it). The following is an example VCD file:

<?xml version="1.0" encoding="utf-8"?> <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.1"> <CommandSet xml:lang="en-us" Name="AdventureWorksCommandSet_en-us"> <CommandPrefix> Adventure Works, </CommandPrefix> <Example> Show trip to London </Example> <Command Name="showTripToDestination"> <Example> show trip to London </Example> <ListenFor RequireAppName="BeforeOrAfterPhrase"> show trip to {destination} </ListenFor> <Feedback> Showing trip to {destination} </Feedback> <Navigate/> </Command> <PhraseList Label="destination"> <Item> London </Item> <Item> Dallas </Item> </PhraseList> </CommandSet> <!-- Other CommandSets for other languages --> </VoiceCommands>

When activating an application from the OnLaunched application launch event, you need to call the InstallCommandSetsFromStorageFileAsync method to register the commands that Cortana should recognize. Keep in mind that if the device is restored from backup and your application is reinstalled, then voice commands are not restored automatically. To ensure that the voice command description data remain intact, you can initialize the VCD file each time you start or activate the application. But it is better to keep a setting that indicates whether the VCD file is already installed, and check this particular parameter. Below is a simple code example showing how to install a VCD file from an application:

 var storageFile = await Windows.Storage.StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///CortanaVcd.xml")); await Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager.InstallCommandSetsFromStorageFileAsync(storageFile);

Add dynamism

Now that you are familiar with the basics, we can add a bit of variety at runtime . Below is an example of the dynamic switching of a VCD file that needs to be downloaded:

 private Windows.ApplicationModel.VoiceCommands.VoiceCommnadDefinition.VoiceCommandSet commandSetEnUs; if (Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager.InstalledCommandSets.TryGetValue("AdventureWorksCommandSet_en-us", out commandSetEnUs)) { // this code will fully replace the destination list await commandSetEnUs.SetPhraseListAsync("destination", new string[] {"Chicago", "Seattle", "New York", "Phoenix"}); }

How will my application interact with Cortana?

There are several ways how your application can communicate with Cortana. Here are the three most typical ways:

Launch application from Cortana. In addition to simply launching the application, you can also specify a “deep” link or command to execute inside the application.
Simple interaction within the Cortana interface (usually this is saving / querying data through a background process).
The interaction between the user and the application inside Cortana.

Application launch

If you have a difficult task and you want the user to go directly to your application, using Cortana for this is a great idea! Since some complex tasks in reality can be performed faster and more accurately through voice commands, this can really help users.

 protected override void OnActivated(IActivatedEventArgs e) { // Was the app activated by a voice command? if (e.Kind != Windows.ApplicationModel.Activation.ActivationKind.VoiceCommand) { return; } var commandArgs = e as Windows.ApplicationModel.Activation.VoiceCommandActivatedEventArgs; var navigationParameterString = ""; Windows.ApplicationModel.VoiceCommands.VoiceCommand.SpeechRecognitionResult speechRecognitionResult = commandArgs.Result; // Get the name of the voice command and the text spoken string voiceCommandName = speechRecognitionResult.RulePath[0]; string textSpoken = speechRecognitionResult.Text; // The commandMode is either "voice" or "text", and it indicates how the voice command was entered by the user. // Apps should respect "text" mode by providing feedback in a silent form. string commandMode = this.SemanticInterpretation("commandMode", speechRecognitionResult); switch (voiceCommandName) { case "showTripToDestination": // Access the value of the {destination} phrase in the voice command string destination = speechRecognitionResult.SemanticInterpretation.Properties["destination"][0]; // Create a navigation parameter string to pass to the page navigationParameterString = string.Format("{0}|{1}|{2}|{3}", voiceCommandName, commandMode, textSpoken, destination); // Set the page where to navigate for this voice command navigateToPageType = typeof(TripPage); break; default: // There is no match for the voice command name. Navigate to MainPage navigateToPageType = typeof(MainPage); break; } if (this.rootFrame == null) { // App needs to create a new Frame, not shown } if (!this.rootFrame.Navigate(navigateToPageType, navigationParameterString)) { throw new Exception("Failed to create voice command page"); } }

Simple interaction inside Cortana to save or query data

Now that your VCD file is connected to Cortana, and you know how to perform basic interaction, let's see how to do more complex operations. For example, you want Cortana to show the user some data or, on the contrary, save it. The MSDN website has detailed instructions for setting up background tasks for Cortana . Here is a brief summary:

Create a Windows Runtime Component project in your solution.
Create a new class that implements the IBackgroundTask interface, which will work as an application service
In the manifest of your UWP application (Package.appxmanifest) add a new extension for the created service. This step is described in detail in MSDN.

The following is an example of a fragment of the Package.appxmanifest file:

 <Package> <Applications> <Application> <Extensions> <Extension Category="windows.appService" EntryPoint= "AdventureWorks.VoiceCommands.AdventureWorksVoiceCommandService"> <AppService Name="AdventureWorksVoiceCommandService"/> </Extension> </Extensions> <Application> <Applications> </Package>

After starting, the background service of the application has 0.5 seconds to call the ReportSuccessAsync method. Cortana uses the data provided by the application to demonstrate and voice the response indicated in the VCD file. If the application spends more than half a second to return the result, Cortana inserts an intermediate screen, as shown below. Cortana displays a temporary screen until the application returns the result via ReportSuccessAsync, but for no more than 5 seconds. If the application did not call the ReportSuccessAsync method or any of the VoiceCommandServiceConnection methods to provide information to Cortane, the user will receive an error message and the service call will be canceled.

Here is a simple code example for implementing IBackgroundTask in the application service:

 using Windows.ApplicationModel.Background; namespace AdventureWorks.VoiceCommands { public sealed class AdventureWorksVoiceCommandService : IBackgroundTask { public void Run(IBackgroundTaskInstance taskInstance) { BackgroundTaskDeferral _deferral = taskInstance.GetDeferral(); // // TODO: Insert code // _deferral.Complete(); } } }

Interaction within Cortana

At this point, you have learned the basics and are ready to realize even tighter interaction with Cortana . An application can request different screen types to support this or that functionality:

Successful completion
Intermediate
Progress
the confirmation
Uncertainty
Mistake

Let's take a closer look at one of these scenarios: uncertainty. Sometimes it happens that your application has several options that it can return. In such situations, you need to resolve the uncertainty of what to do next. For example, if a user chose music and needed to decide what to play next: ABBA, Nickelback or White Snake, Cortana can help make a choice. The code snippet below from the Adventure Works example shows how to resolve the ambiguity that has appeared within the application service:

 // Create a VoiceCommandUserMessage for the initial question. var userPrompt = new VoiceCommandUserMessage(); userPrompt.DisplayMessage = "Which one do you want to cancel?"; userPrompt.SpokenMessage = "Which Chicago trip do you wanna cancel?"; // Create a VoiceCommandUserMessage for the second question, // in case Cortana needs to reprompt. var userReprompt = new VoiceCommandUserMessage(); userReprompt.DisplayMessage = “Which one did you want to cancel?”; userReprompt.SpokenMessage = "Which one did you wanna to cancel?"; // Create the list of content tiles to show the selection items. var destinationsContentTiles = new List<VoiceCommandContentTile>(); // create your VoiceCommandContentTiles for(int i=0; i < 5; i++) { var destinationTile = new VoiceCommandContentTile(); destinationTile.ContentTileType = VoiceCommandContentTileType.TitleWith68x68IconAndText; // The AppContext is optional. // Replace this value with something specific to your app. destinationTile.AppContext = "id_Vegas_00" + i; destinationTile.Title = "Tech Conference"; destinationTile.TextLine1 = "May " + i + "th"; destinationsContentTiles.Add(destinationTile); } // Create the disambiguation response. var response = VoiceCommandResponse.CreateResponseForPrompt(userPrompt, userReprompt, destinationsContentTiles); // Request that Cortana shows the disambiguation screen. var voiceCommandDisambiguationResult = await voiceServiceConnection.RequestDisambiguationAsync(response); if (voiceCommandDisambiguationResult != null) { // Use the voiceCommandDisambiguationResult.SelectedItem to take action. // Call Cortana to present the next screen in .5 seconds // and avoid a transition screen. }

Summing up

We hope that now you better understand how easy it is to add integration with Cortana into your application and thereby improve the interaction experience for your users. From simple application launching to complex interaction models even without launching the application - in all cases, Cortana can actually be useful. To take advantage of the integration with Cortana, you can start from the simplest - add a new way to work with deep links in your application.

If you feel that working with Cortana in practice can add value to your application, feel free to implement it. As soon as you update your application, do not forget to declare yourself in the DVLUP program in the nomination “ Adding Cortana to your app ” to get points and XP for updating the application. Tell us what you have done via Twitter @WindowsDev with the hashtag # Win10x10 - we will be happy to know what the developers create under Windows.