Playing with emotions: Microsoft Cognitive Services + Unity
Friends! Surely many of you have already heard about cognitive services that allow you to solve complex tasks with a single REST API call — to determine emotions and a person’s age from a photo, to make machine translation of text, etc. Often, cognitive services are implemented in applications or web backends. Today, our big friend, VRTech employee and Game developer Grigory Dyadichenko will tell us how to implement cognitive services in Unity games, and also invite you to the Unity developers rally, where you can discuss this in more detail.
In this article I would like to tell about the integration of Microsoft Cognitive Services in Unity; how to make HTTP requests to services through the WWW class (if suddenly someone else didn’t come across it and doesn’t know) and tell me what unexpected problems I faced while developing an application using these services for Google Play.
Microsoft Cognitive Services is a set of cloud services that allow you to perform tasks such as speech, face recognition, emotion recognition, and more. More details can be found here.
')
Once I already wrote an article about cognitive services, and even specifically about the Emotions API. It used a library for UWP, which cannot be used in the Unity project. Therefore recently the idea occurred to me that it would be quite good to write a wrapper for these services for Unity. And I got down to business.
These services are an interesting and inexpensive tool for creating “wow effect” at trade shows, collecting contacts and similar tasks. Working with them, in principle, is much easier than with the same OpenCV. In the context of game development, you can make a cool bun for the player, which allows you to generate an avatar to the player from his photo.
Let us turn to the description of the wrapper itself. At the moment, it is partially covered with the Emotions API and Face API.
Interaction with the solution is very simple. You create the service you need by specifying the SubscriptionKey in the constructor (for convenience, the ScriptableObject was created in the demo scenes to store them), and then you create a quortenine in which you take the data you need.
A free trial version of the subscription key can be obtained on the Microsoft cognitive services site.
So, why is the Cortina? The fact is that the most convenient way to access services is Rest API. The easiest way to do this in Unity is through the WWW class, in which the query runs asynchronously. There are many ways to wait for its implementation.
For example, you can block the main thread, but I suspect that in most cases this is undesirable.
And you can make korutinami, which is implemented in this version of the wrapper.
This method works well, and it suited me when I wrote my application for android. Since the analysis of photos takes some time, so that the user does not sit idle, I decided to integrate advertising. But during the integration of advertising, unexpected problems appeared. The user watches the ad, the profile is analyzed - profit, but it was not there. Here I was waiting for a feature about which I did not know about Unity Ads on the android. The fact is that during the display of advertising, the main stream is blocked, so for the analysis of the profile it was decided to put everything into a separate stream.
There I was waiting for a new, but quite logical discovery. It turns out that the WWW class can only work in the main thread. Therefore it was necessary to write everything on System.Net (version 2.0, since it is in Unity). And I would put this solution in the repository, but they needed to sign an SSL certificate, which is not obvious, and can lead to unintended consequences for the user of the wrapper. If suddenly it will be interesting to someone, then I can put it in a separate project on the githaba, but from the point of view of implementation, there is nothing complicated.
(Not the most beautiful example made in haste)
Another fun discovery was discovered when testing the Face API. I wanted to make this effect a perfect smile.
Face API can return the same set of emotions as the Emotions API. But in the course of the tests, I discovered that the results differ, while the Emotions API works a little more stable and accurate. Therefore, for this effect, the landmarks of the face (in order to correctly put an asterisk) were taken from the Face API, and the emotions from the Emotions API.
In the near future, I plan to return to the implementation of this wrapper and to the search for new jokes related to the use of Microsoft Cognitive Services. In the meantime, there are Demo scenes in the project that show the simplest interaction of EmotionService with a webcam. In addition, some useful utilities for screenshots (a script that takes a screenshot of a particular RectTransform for example)
Returning to the wrapper, you can download and monitor its development in the Github repository . (It is possible after the mitap reach the hands to write documentation)
On June 7, the third Unity of developers in Moscow will be held at VSBI. If you are engaged in development on Unity or it is interesting to you - come! The event is free, registration is required, you can register and learn more information here.
And also, to make follow-ups and see materials from past meetings, you can join the groups:
→ VK: vk.com/unimosmeet
→ FB: www.facebook.com/groups/unimosmeet
Dyadichenko Grigory is the leading Unity developer at VRTech. Organizer of Unity Moscow Meetup. He is fond of algorithms on graphs, game development and everything related to computer graphics.
In this article I would like to tell about the integration of Microsoft Cognitive Services in Unity; how to make HTTP requests to services through the WWW class (if suddenly someone else didn’t come across it and doesn’t know) and tell me what unexpected problems I faced while developing an application using these services for Google Play.
Microsoft Cognitive Services is a set of cloud services that allow you to perform tasks such as speech, face recognition, emotion recognition, and more. More details can be found here.
')
Once I already wrote an article about cognitive services, and even specifically about the Emotions API. It used a library for UWP, which cannot be used in the Unity project. Therefore recently the idea occurred to me that it would be quite good to write a wrapper for these services for Unity. And I got down to business.
These services are an interesting and inexpensive tool for creating “wow effect” at trade shows, collecting contacts and similar tasks. Working with them, in principle, is much easier than with the same OpenCV. In the context of game development, you can make a cool bun for the player, which allows you to generate an avatar to the player from his photo.
Let us turn to the description of the wrapper itself. At the moment, it is partially covered with the Emotions API and Face API.
Interaction with the solution is very simple. You create the service you need by specifying the SubscriptionKey in the constructor (for convenience, the ScriptableObject was created in the demo scenes to store them), and then you create a quortenine in which you take the data you need.
Example
private IEnumerator CheckEmotions() { EmotionService emoServ = new EmotionService(SubscriptionKeys.Instance.EmotionsApiKey); while (true) { yield return new WaitForEndOfFrame(); yield return emoServ.GetEmoInfoCoroutine(_WebCam.Screenshot); var emotions = emoServ.LastEmotions; if (emotions != null) { _MaxEmotionValue.text = GetMaxEmotionOnScreenshot(emotions); } yield return new WaitForSeconds(DELAY); } } A free trial version of the subscription key can be obtained on the Microsoft cognitive services site.
So, why is the Cortina? The fact is that the most convenient way to access services is Rest API. The easiest way to do this in Unity is through the WWW class, in which the query runs asynchronously. There are many ways to wait for its implementation.
For example, you can block the main thread, but I suspect that in most cases this is undesirable.
And you can make korutinami, which is implemented in this version of the wrapper.
Example
private IEnumerator CreateRecognizeRequestAndSaveResponseCoroutine( string contentHeader, byte[] data) { Dictionary<string, string> headers = new Dictionary<string, string>(); headers.Add(Constants.SUB_KEY_HEADER, _SubscriptionKey); headers.Add(Constants.CONTENT_TYPE_HEADER, contentHeader); WWW request = new WWW(_RecognizeRequestUrl, data, headers); yield return new WaitUntil(() => request.isDone); ParseEmotionsFromJson(request.text); } This method works well, and it suited me when I wrote my application for android. Since the analysis of photos takes some time, so that the user does not sit idle, I decided to integrate advertising. But during the integration of advertising, unexpected problems appeared. The user watches the ad, the profile is analyzed - profit, but it was not there. Here I was waiting for a feature about which I did not know about Unity Ads on the android. The fact is that during the display of advertising, the main stream is blocked, so for the analysis of the profile it was decided to put everything into a separate stream.
There I was waiting for a new, but quite logical discovery. It turns out that the WWW class can only work in the main thread. Therefore it was necessary to write everything on System.Net (version 2.0, since it is in Unity). And I would put this solution in the repository, but they needed to sign an SSL certificate, which is not obvious, and can lead to unintended consequences for the user of the wrapper. If suddenly it will be interesting to someone, then I can put it in a separate project on the githaba, but from the point of view of implementation, there is nothing complicated.
(Not the most beautiful example made in haste)
Example in a separate thread
private void CreateRecognizeRequest() { Clear(); _Thread = new Thread(Run); _Thread.Start(); } private void Run() { WebHeaderCollection headers = new WebHeaderCollection(); headers.Add(Constants.SUB_KEY_HEADER, _SubscriptionKey); var request = HttpWebRequest.Create(_RecognizeRequestUrl); request.ContentType = _ContentHeader; request.Headers = headers; request.ContentLength = _Data.Length; request.Method = WebRequestMethods.Http.Post; var dataStream = request.GetRequestStream(); dataStream.Write(_Data, 0, _Data.Length); dataStream.Close(); var response = request.GetResponse(); dataStream = response.GetResponseStream(); StreamReader reader = new StreamReader(dataStream); string responseString = reader.ReadToEnd(); reader.Close(); dataStream.Close(); response.Close(); Debug.Log(responseString); if (!TryParseEmotionsFromJson(responseString)) { Run(); } else { _IsDataReady = true; } } Another fun discovery was discovered when testing the Face API. I wanted to make this effect a perfect smile.
Face API can return the same set of emotions as the Emotions API. But in the course of the tests, I discovered that the results differ, while the Emotions API works a little more stable and accurate. Therefore, for this effect, the landmarks of the face (in order to correctly put an asterisk) were taken from the Face API, and the emotions from the Emotions API.
In the near future, I plan to return to the implementation of this wrapper and to the search for new jokes related to the use of Microsoft Cognitive Services. In the meantime, there are Demo scenes in the project that show the simplest interaction of EmotionService with a webcam. In addition, some useful utilities for screenshots (a script that takes a screenshot of a particular RectTransform for example)
Returning to the wrapper, you can download and monitor its development in the Github repository . (It is possible after the mitap reach the hands to write documentation)
Unity Moscow Meetup # 3
On June 7, the third Unity of developers in Moscow will be held at VSBI. If you are engaged in development on Unity or it is interesting to you - come! The event is free, registration is required, you can register and learn more information here.
And also, to make follow-ups and see materials from past meetings, you can join the groups:
→ VK: vk.com/unimosmeet
→ FB: www.facebook.com/groups/unimosmeet
about the author
Dyadichenko Grigory is the leading Unity developer at VRTech. Organizer of Unity Moscow Meetup. He is fond of algorithms on graphs, game development and everything related to computer graphics.Source: https://habr.com/ru/post/329866/
All Articles