Actions on Google: write a simple application for Google Assistant on Dialogflow and Cloud Functions for Firebase

At the end of last month, the official release of Google Assistant took place in Russian, so it's time to figure out how to make your applications ( actions ) for Assistant on the standard Google technology stack. In this article, we will look at creating an action in Actions on Google , analyze the process of extracting entities and intents from phrases in Dialogflow , learn how to write handlers of extracted information and work with the network in Cloud Functions for Firebase .





Fig. 1. Application architecture for Assistant.



The development of an Assistant began to actively develop relatively recently, so there are few materials on the network yet, and the number of tools and technologies used significantly increases the threshold of entry. Though this article does not solve, but at least contributes to the solution of the problems mentioned. Let's start with the application architecture for the Assistant (Fig. 1) implemented on the standard Google technology stack:

• Actions on Google - a platform for creating applications for Google Assistant.
• Dialogflow is an NLU engine (Natural Language Understanding), which is responsible for processing natural languages ​​and design dialogs.
• Cloud Functions for Firebase (for convenience, we will use the abbreviation Firebase Functions) - cloud functions for processing complex user interaction logic and for working with third-party services. Firebase Functions and Dialogflow interact through the webhook, so technically you can use any other server solution. However, Firebase Functions is a good alternative, and sometimes a replacement for your own backend. It allows you to create and run services on the infrastructure of Google, without worrying about selecting, scaling, or managing servers. On the one hand, it allows you to focus on the product component of the development and functionality of the service, without wasting time on infrastructure tasks and administration. But on the other hand, as a rule, delegation entails a weakening of control over the situation.

The article focuses on the technical aspect of development, the cost of using the listed services will not be disassembled.

')



Fig. 2. Interaction of components of Google Assistant (Based on material: Google Home and Google Assistant Workshop ).



Within the framework of the described stack, the logic of the action of action looks like this (Fig. 2):

• A user accesses the Google Assistant application and initiates a conversation with a specific action.
• Google Assistant, through Actions on Google, proxies each phrase of the user in a text format in Dialogflow, additionally providing information about the user himself (with prior request and with the user's consent) and the current conversation .
• Dialogflow processes the received phrase, extracts the necessary information from it and, based on ML, makes decisions about which answer will be generated.
• In some cases, Dialogflow can delegate the formation of a response to a server on Firebase Functions, which, in turn, can use third-party services to obtain the necessary information for a response.
• After the answer is generated, Dialogflow returns it to Actions on Google, from where it enters the Google Assistant application.

Idea

Our action will be determined by the phrase, what gifs the user wants to see, and then will look for them through the GIPHY API and return to the user in the form of cards. When implementing the action, we will analyze the solution of the following tasks:

1. Setting and linking Actions on Google, Dialogflow and Firebase Functions.
2. Extract keywords from user phrases (Dialogflow).
3. Creating dialogue scripts (Dialogflow).
4. Work with dialogue context (Dialogflow).
5. Creating and connecting a webhook to generate a response to the user's phrase (Dialogflow, Firebase Function).
6. Display carousel of cards in the interface (Firebase Functions).
7. Downloading information from a third-party service (Firebase Functions).

Primary setup

Fig. 3. Creating Dialogflow agent.



First of all, we need a Google account. Let's start by creating a project in Dialogflow, to do this, click the “Create Agent” button in the console and fill in the required fields (Fig. 3):

• The default language is "Russian - ru."
• Time zone: "(GMT + 3: 00) Europe / Moscow".
• Google Cloud Project: A new GCP for your Dialogflow agent will be created automatically, or you can choose one of the existing GCP projects, if you have any.

Then we press the “Create” button in the upper right corner and wait while the console configures the new project.





Fig. 4. Standard intents.



By default, when creating the Dialogflow agent, two intents are created (Fig. 4):

• "Default Welcome Intent" - is responsible for greeting the user;
• «Default Fallback Intent» - handles unknown phrases that Dialogflow can not be attributed to any other intents.

Creating dialogs in Dialogflow has already been described in detail in articles here , here and here , so I will not focus on its principle of operation.





Fig. 5. Answers for Default Welcome Intent.



Add to the "Default Welcome Intent" a few welcome answers that will help the user understand what the action is for and what functions he can perform. In the "Responses" section, select the "Google Assistant" tab and in the "Suggestion Ships" we will write sample phrases to prompt the user how to communicate with the action (Fig. 5).



Action can be debugged in Google Assistant both on the phone and in the official emulator. To open the emulator, go to the “Integrations” section, click on the “Integration Settings” button in the “Google Assistant” card and click on the “Manage Assistant App”. And in the phone and in the action emulator, you can run the code phrase "Okay Google, I want to talk with my test application."

Basic scenario: search for gifs

We will create a new Search Intent intent that will extract keywords from the user's phrase and transfer them via the webhook server to Firebase Functions. The server, in turn, using the GIPHY API, will find the corresponding gifs and return the result to the user in the form of cards.





Fig. 6. Add training phrases.



To begin with, in the section “Training Phrases” we will add typical training phrases (Fig. 6):

• "I want to look at the dancing giraffes."
• "Find the animated cartoon."
• "Show the seals."
• "Show gifs."
• "Find me animated elephants."
• "Show gifs with pandas."
• "Gifs with poloskunami raccoons."
• "You have seals."
• "Find the funny fall."

Fig. 7. Extract parameters from text.



For the added phrases, we note the search parameter that Dialogflow should select from the text. In this case, the most appropriate type of parameter would be @sys.any , since practically any language construct can act as a search query parameter. We call this parameter query and mark as mandatory (Fig. 7).





Fig. 8. List of leading questions.



In the “Prompts” subsection, we’ll write clarifying questions that Dialogflow will ask if it cannot extract keywords from the phrase (Fig. 8).



Then you should go down to the “Fulfillment” section at the very bottom of the page (not to be confused with the section of the same name in the left menu). Click the "Enable Fullfilment" button, and then enable the "Enable webhook call for this intent" setting. This will allow Dialogflow to delegate the formation of a response to Firebase Functions when it hits the content.



Now we will go to the tab “Fulfillment” in the left menu and turn on the “Inline Editor”, where we will write the logic for the newly created “Search Intent”. To search for gifs by keywords, we will use the query https://api.giphy.com/v1/gifs/search , which returns a list of objects found in JSON format according to the specification . We will display the answer received from GIPHY in the form of Browsing Carousel - a carousel of image cards that, when clicked, opens a web page. In our case, when clicking on a card, the user will go to the GIPHY service page with this animation and a list of similar ones.



The code implementing the above functionality is presented below.

 'use strict'; const GIPHY_API_KEY = 'API_KEY'; const SEARCH_RESULTS = [ '-,    .', ',   .', ',   !' ]; // Import the Dialogflow module from the Actions on Google client library. const { dialogflow, BrowseCarouselItem, BrowseCarousel, Suggestions, Image } = require('actions-on-google'); // Import the firebase-functions package for deployment. const functions = require('firebase-functions'); // Import the request-promise package for network requests. const request = require('request-promise'); // Instantiate the Dialogflow client. const app = dialogflow({ debug: true }); function getCarouselItems(data) { var carouselItems = []; data.slice(0, 10).forEach(function (gif) { carouselItems.push(new BrowseCarouselItem({ title: gif.title || gif.id, url: gif.url, image: new Image({ url: gif.images.downsized_medium.url, alt: gif.title || gif.id }), })); }); return carouselItems; } function search(conv, query) { // Send the GET request to GIPHY API. return request({ method: 'GET', uri: 'https://api.giphy.com/v1/gifs/search', qs: { 'api_key': GIPHY_API_KEY, 'q': query, 'limit': 10, 'offset': 0, 'lang': 'ru' }, json: true, resolveWithFullResponse: true, }).then(function (responce) { // Handle the API call success. console.log(responce.statusCode + ': ' + responce.statusMessage); console.log(JSON.stringify(responce.body)); // Obtain carousel items from the API call response. var carouselItems = getCarouselItems(responce.body.data); // Validate items count. if (carouselItems.length <= 10 && carouselItems.length >= 2) { conv.data.query = query; conv.data.searchCount = conv.data.searchCount || 0; conv.ask(SEARCH_RESULTS[conv.data.searchCount % SEARCH_RESULTS.length]); conv.data.searchCount++; conv.ask(new BrowseCarousel({ items: carouselItems })); } else { // Show alternative response. conv.ask('      ,   - ?)'); } }).catch(function (error) { // Handle the API call failure. console.log(error); conv.ask(',     .'); }); } // Handle the Dialogflow intent named 'Search Intent'. // The intent collects a parameter named 'query'. app.intent('Search Intent', (conv, { query }) => { return search(conv, query); }); // Set the DialogflowApp object to handle the HTTPS POST request. exports.dialogflowFirebaseFulfillment = functions.https.onRequest(app); 

Since the user can access the same intent several times, it is recommended to return various answers to him. For this purpose, the JSON-object Conversation.data was used, which retains its value both when it re-accesses the content and when it accesses other conversation scenarios.





Fig. 9. Initialization of the conversation (left), refinement of the search parameters and further display of the results (center), display of the search results for the new query (right)



Note: to work with third-party services API via Firebase Functions, you need to connect billing, otherwise when trying to work with the network, an error will occur:

“Billing account not configured. External network is not accessible and quotas are severely limited. Configure billing account to remove these restrictions. ”

To do this, in the left menu, click on the “Paid Account” and choose Flame ($ 25 per month) or Blaze (pay-as-you-use) among the proposed tariff plans. I chose the latter option, because as part of the development of the test application, it seemed to me more profitable.

Advanced Script: Pagination

In most cases, the search query GIPHY will find significantly more than ten gifs, so it will be correct to allow the user to see the entire search results, i.e. add pagination.



In the Dialogflow console, we hover over the cell “Search Intent”. A few buttons will appear on the right, click on “Add follow-up intent”. This will allow us to create a conversation branch following the “Search Intent”. Among the elements of the drop-down list, select “more” - the standard attribute to initiate the display of additional information.





Fig. 10. The context of the intent “Search Intent - more”.



Let's move to the newly created intent and make changes to the “Context” section. Since the user can ask to show another gif several times in a row, this intent should be able to be called recursively. For this, in the outgoing context it is necessary to register the same line as that indicated in the incoming one (Fig. 10). In the "Fullfilment" section, you should also enable the "Enable webhook call for this intent" setting.



Now back to “Fillfulment” from the side menu, where we initialize the handler for “Search Intent - more”. We also add the offset parameter to the search function, which will be used for pagination in the GIPHY API.

 const SEARCH_RESULTS_MORE = [ '   !', ',    .', ',   .  ,    .' ]; function search(conv, query, offset) { // Send the GET request to GIPHY API. return request({ method: 'GET', uri: 'https://api.giphy.com/v1/gifs/search', qs: { 'api_key': GIPHY_API_KEY, 'q': query, 'limit': 10, 'offset': offset, 'lang': 'ru' }, json: true, resolveWithFullResponse: true, }).then(function (responce) { // Handle the API call success. console.log(responce.statusCode + ': ' + responce.statusMessage); console.log(JSON.stringify(responce.body)); // Obtain carousel items from the API call response. var carouselItems = getCarouselItems(responce.body.data); // Validate items count. if (carouselItems.length <= 10 && carouselItems.length >= 2) { conv.data.query = query; conv.data.offset = responce.body.pagination.count + responce.body.pagination.offset; conv.data.paginationCount = conv.data.paginationCount || 0; conv.data.searchCount = conv.data.searchCount || 0; // Show successful response. if (offset == 0) { conv.ask(SEARCH_RESULTS[conv.data.searchCount % SEARCH_RESULTS.length]); conv.data.searchCount++; } else { conv.ask(SEARCH_RESULTS_MORE[conv.data.paginationCount % SEARCH_RESULTS_MORE.length]); conv.data.paginationCount++; } conv.ask(new BrowseCarousel({ items: carouselItems })); conv.ask(new Suggestions('')); } else { // Show alternative response. conv.ask('      ,   - ?)'); } }).catch(function (error) { // Handle the API call failure. console.log(error); conv.ask(',     .'); }); } // Handle the Dialogflow intent named 'Search Intent - more'. app.intent('Search Intent - more', (conv) => { // Load more gifs from the privious search query return search(conv, conv.data.query, conv.data.offset); }); 

Fig. 11. Pagination when searching for gifs.

Result

The video of the action is shown below.







Project code and assistant dump is available on Github .