The study of eye movements: eytreking without a video camera and other solutions
Is it possible to collect data on the movements of the eyes of all visitors to the planetarium, without having at the disposal of special equipment? How to study visual attention during the active actions of several people without constructing a complex system with eytrekler-glasses? How to achieve exceptional accuracy and record eye movements with a frequency of 8000 Hz? In our article we will try to answer these questions.
Nowadays, well-known eytreker, based on the method of video recording eye movements: these are trackers of famous brands like EyeLink , Tobii , as well as various hand-made solutions and webcam-based solutions (we wrote about this in a previous article on eytracking) . However, there are still many interesting and unusual approaches to the study of human visual attention.
In 2016, Wang et al. Published a new eye tracking system based on infrared thermography. Efron et al., (1989) showed that the edge of the cornea is 0.45 ° warmer than its center, a similar discovery formed the basis for the development of thermal eytriking.
')
Wang et al., 2016, in order to validate their system, they simultaneously registered eye movements using the methods of videooculography (Eye Link, 500 Hz) and thermography (60 Hz) in ten subjects. The median verification error was 1.4 ° compared with EyeLink, which shows the good potential of this method for its further use.
It is curious that thermography is also successfully used to recognize the emotional state of a person (for example, see Park et al., 2013) and respiratory rate (for example, see Al-Khalidi et al., 2011). Thus, thermal eytracking is becoming a promising method for obtaining multimodal data.
The study of the visual attention of a large number of people at one point in time is a very tempting task. However, its implementation, at first glance, seems to be a difficult and expensive process (at least, already at the stage of procurement, debugging or the creation of equipment with its subsequent synchronization).
Shillcock, Wase (2015) proposed an original method for registering the visual attention of a crowd of people, for the realization of which all they need is a box of pens, a pack of paper and their own stimulus material. The experiment was successfully replicated by Bielecki et al. (2017) in the study of the visual attention of visitors to the planetarium using a 16-meter hemispherical screen.
The essence of the technique is that the presentation is periodically, for a short period of time, interrupted by a slide with a grid, in each of the cells of which there is a letter or some other symbol. The task of the subjects - to write the character that they just saw. Thus, it is possible to obtain heat maps of visual attention of the entire audience as a whole, which makes it possible to analyze its distribution among the participants of the lecture not only depending on its content, but also on the location of the individual participant in the hall.
Of course, this technique is not applicable for "field" experiments, when there is no stimulus material in the form of video or images. However, for the natural conditions of the experiment, there were also some tricky solutions.
The study of oculomotor behavior in the most natural, everyday, everyday conditions is the dream of many researchers of visual attention. If the task of registering eye movements in the case of active actions can be done relatively well by eytreker glasses (for example, Ronaldo is kicking a ball with glasses-tracker), then this experimental design is not suitable for a qualitative study of oculomotor behavior during social interactions, as eye contact The facial expressions of the entire face are important elements of non-verbal communication. To date, the available software eytrekerov, which would receive the trajectory of the gaze approximately the same as when using eytrekerov-devices, does not exist (but our team is in the process of developing and testing the first software eytrekera EyeCatcher, able to replace a full-fledged laboratory tracker) .
The G3E project focuses on extracting human visual attention data from a video, but its initiators abandoned the idea of ​​obtaining trajectories of gaze in the form of changing the coordinates of the pupil position and focused on getting the direction of the gaze in the context of determining whether the subject is looking at the interlocutor or not. This system was developed based on the algorithms for predicting the position of the gaze depending on the position of the head (Ba, Odobez, 2006) and the gaze appearance model (this model includes coupled images with the position of the eye in orbit and the corresponding direction of gaze) (Mora, Odobez , 2013). In addition, to improve the functional parameters of the system, data from annotated video were used (Siegfried R., Odobez, 2017).
Video annotation has found its application not only as an effective tool for improving the quality of the algorithms, but also as an independent method for studying eye movements.
Thus, in a study of visual attention and behavioral strategies during the card game Campos et al., 2015, each of the two players was recorded on cameras. After that, the video was laid out by two annotators in the ELAN program in three categories: look-at-game, look-at-other and look-elsewhere, the consistency of the annotators was calculated using Cohen's kappa and compared the results with behavior strategies (cooperation, competition and avoidance). games). It turned out that with the motivation to cooperate, the time spent on viewing the interlocutor's face is significantly more, as well as mutual eye contacts. When motivated to compete, on the contrary, there are short but frequent views of the interlocutor's face, and the tendency to avoid eye contact is distinguished.
Jarik, Kingstone, 2015 in the study of the dynamics of mutual eye contact after the game in terms of cooperation and competition also turned to the video annotation method (in this study, as in the previous one, two annotators participated). Here, after the cooperative game, the subjects had a shorter duration of eye contact, more often interrupted compared with another group of participants in the experiment.
In terms of alternatives to the method of videooculography, contact methods should also be mentioned (with them, all of the eytracking began!). You can write a separate article on the history of eytracking (eerie suckers attached to the eyeball, or, say, a description of mechanical recorders the size of an entire table, require detailed text with explanations), so we’ll only stop at the current stage of development of this group of methods. Among the contact methods of eye movement registration that are actively used today, it is necessary to single out electromagnetic eytrekery and the EOG method.
A prime example of electromagnetic trackers are Primelec's Swiss trackers, which can record eye movements with extremely high accuracy - 0.0002 ° and a sampling frequency of 8000 Hz. The principle of their work is based on the induction of electric current caused by the movement of the metal circuit in a magnetic field. The movable element is a ring with two internally arranged inductance circuits, which is put on the subject's eyes like a contact lens (as eyewitnesses report, the installation procedure of this magic “lens” is not pleasant). With the help of such devices, it turns out to study various features of micro-movements of the eyes, including their changes in astronauts in weightlessness.
Electrooculogram (EOG) is a method for recording electrical activity that occurs when an eye moves. Since the cornea of ​​the eye has a positive charge on the retina, this creates a constant (root-retinal) potential. When the position of the eye changes, this potential is corrected. EOG is recorded by electrodes, which are installed near the nasal and temporal angle of the palpebral fissure (for registration of horizontal saccades ), as well as above the eyebrow and under the eye - for registration of vertical saccades. This method is still widely used in medical practice, sometimes in combination with the EEG method.
The Neurodata Lab's work group on light tracking and the visual sensory system continues its work.
Material author:
Maria Konstantinova, Researcher at Neurodata Lab , biologist, physiologist, specialist in visual sensory system, oculography and ocular motorics.
Nowadays, well-known eytreker, based on the method of video recording eye movements: these are trackers of famous brands like EyeLink , Tobii , as well as various hand-made solutions and webcam-based solutions (we wrote about this in a previous article on eytracking) . However, there are still many interesting and unusual approaches to the study of human visual attention.
Thermal IT
In 2016, Wang et al. Published a new eye tracking system based on infrared thermography. Efron et al., (1989) showed that the edge of the cornea is 0.45 ° warmer than its center, a similar discovery formed the basis for the development of thermal eytriking.
')
Wang et al., 2016, in order to validate their system, they simultaneously registered eye movements using the methods of videooculography (Eye Link, 500 Hz) and thermography (60 Hz) in ten subjects. The median verification error was 1.4 ° compared with EyeLink, which shows the good potential of this method for its further use.
It is curious that thermography is also successfully used to recognize the emotional state of a person (for example, see Park et al., 2013) and respiratory rate (for example, see Al-Khalidi et al., 2011). Thus, thermal eytracking is becoming a promising method for obtaining multimodal data.
Collective Aytreking
The study of the visual attention of a large number of people at one point in time is a very tempting task. However, its implementation, at first glance, seems to be a difficult and expensive process (at least, already at the stage of procurement, debugging or the creation of equipment with its subsequent synchronization).
Shillcock, Wase (2015) proposed an original method for registering the visual attention of a crowd of people, for the realization of which all they need is a box of pens, a pack of paper and their own stimulus material. The experiment was successfully replicated by Bielecki et al. (2017) in the study of the visual attention of visitors to the planetarium using a 16-meter hemispherical screen.
The essence of the technique is that the presentation is periodically, for a short period of time, interrupted by a slide with a grid, in each of the cells of which there is a letter or some other symbol. The task of the subjects - to write the character that they just saw. Thus, it is possible to obtain heat maps of visual attention of the entire audience as a whole, which makes it possible to analyze its distribution among the participants of the lecture not only depending on its content, but also on the location of the individual participant in the hall.
Of course, this technique is not applicable for "field" experiments, when there is no stimulus material in the form of video or images. However, for the natural conditions of the experiment, there were also some tricky solutions.
Registration of eye movements in natural conditions
The study of oculomotor behavior in the most natural, everyday, everyday conditions is the dream of many researchers of visual attention. If the task of registering eye movements in the case of active actions can be done relatively well by eytreker glasses (for example, Ronaldo is kicking a ball with glasses-tracker), then this experimental design is not suitable for a qualitative study of oculomotor behavior during social interactions, as eye contact The facial expressions of the entire face are important elements of non-verbal communication. To date, the available software eytrekerov, which would receive the trajectory of the gaze approximately the same as when using eytrekerov-devices, does not exist (but our team is in the process of developing and testing the first software eytrekera EyeCatcher, able to replace a full-fledged laboratory tracker) .
The G3E project focuses on extracting human visual attention data from a video, but its initiators abandoned the idea of ​​obtaining trajectories of gaze in the form of changing the coordinates of the pupil position and focused on getting the direction of the gaze in the context of determining whether the subject is looking at the interlocutor or not. This system was developed based on the algorithms for predicting the position of the gaze depending on the position of the head (Ba, Odobez, 2006) and the gaze appearance model (this model includes coupled images with the position of the eye in orbit and the corresponding direction of gaze) (Mora, Odobez , 2013). In addition, to improve the functional parameters of the system, data from annotated video were used (Siegfried R., Odobez, 2017).
Video annotation has found its application not only as an effective tool for improving the quality of the algorithms, but also as an independent method for studying eye movements.
Thus, in a study of visual attention and behavioral strategies during the card game Campos et al., 2015, each of the two players was recorded on cameras. After that, the video was laid out by two annotators in the ELAN program in three categories: look-at-game, look-at-other and look-elsewhere, the consistency of the annotators was calculated using Cohen's kappa and compared the results with behavior strategies (cooperation, competition and avoidance). games). It turned out that with the motivation to cooperate, the time spent on viewing the interlocutor's face is significantly more, as well as mutual eye contacts. When motivated to compete, on the contrary, there are short but frequent views of the interlocutor's face, and the tendency to avoid eye contact is distinguished.
Jarik, Kingstone, 2015 in the study of the dynamics of mutual eye contact after the game in terms of cooperation and competition also turned to the video annotation method (in this study, as in the previous one, two annotators participated). Here, after the cooperative game, the subjects had a shorter duration of eye contact, more often interrupted compared with another group of participants in the experiment.
Good old contact methods
In terms of alternatives to the method of videooculography, contact methods should also be mentioned (with them, all of the eytracking began!). You can write a separate article on the history of eytracking (eerie suckers attached to the eyeball, or, say, a description of mechanical recorders the size of an entire table, require detailed text with explanations), so we’ll only stop at the current stage of development of this group of methods. Among the contact methods of eye movement registration that are actively used today, it is necessary to single out electromagnetic eytrekery and the EOG method.
A prime example of electromagnetic trackers are Primelec's Swiss trackers, which can record eye movements with extremely high accuracy - 0.0002 ° and a sampling frequency of 8000 Hz. The principle of their work is based on the induction of electric current caused by the movement of the metal circuit in a magnetic field. The movable element is a ring with two internally arranged inductance circuits, which is put on the subject's eyes like a contact lens (as eyewitnesses report, the installation procedure of this magic “lens” is not pleasant). With the help of such devices, it turns out to study various features of micro-movements of the eyes, including their changes in astronauts in weightlessness.
Electrooculogram (EOG) is a method for recording electrical activity that occurs when an eye moves. Since the cornea of ​​the eye has a positive charge on the retina, this creates a constant (root-retinal) potential. When the position of the eye changes, this potential is corrected. EOG is recorded by electrodes, which are installed near the nasal and temporal angle of the palpebral fissure (for registration of horizontal saccades ), as well as above the eyebrow and under the eye - for registration of vertical saccades. This method is still widely used in medical practice, sometimes in combination with the EEG method.
The Neurodata Lab's work group on light tracking and the visual sensory system continues its work.
Literature
Al-Khalidi F., Saatchi R., Elphick H., Burke D. 2011. American Journal of Engineering and Applied Sciences 4 (4). P. 586 - 597.
Ba.SO, Odobez J.-M. Sitting room meeting. 2006. 3rd Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI).
Bielecki M., Potega vel. Zabik K., Gochna M., Mikulski J. Mass analysis. 2017. ECEM 2017 Program and Abstracts Eds. Radach R., Deubel H., Vorstius C., Hofmann MJP121.
Campos, J., Alves-Oliveira, P., Paiva A. Looking for a conflict in a dyadic Mixed-Motive Game. 2015. Auton. Agent. Multi-Agent. Syst. 30 (1). P.112-135.
Efron N., Graeme Y., Brennan NA Ocular surface temperature. 1989. Current eye research 8 (9). P.901-906.
Funes Mora, KA, Odobez, J.-M. Person Independent 3D gaze estimation from remote RGBD cameras. 2013. In International Conference on Image Processing. Ieee.
Park KK, Suk HW, Hwang H., Lee J. mock crime using infrared thermal imaging. 2013. Frontiers in Human Neuroscience 7 (70). P.1 - 17.
R. Shillcock, Wase C. 2015. ECEM 2015 Program and Abstracts Eds. Ansorge U., Ditye T., Florack A., Leder HP66.
Siegfried R., Odobez J.-M. Supervised gaze bias correction for gaze coding in interactions. COGAIN Symposium Wuppertal, August 21st 2017.
Wang Q., Boccanfuso L., Li B., Ahn AY, Foster CE, Orr MP, Scassellati B., Shic F. Thermographic Eye Tracking. 2016. Conference paper: Proceedings of the Symposium on Eye Tracking Research and Applications.
Okutin OL, Okutina G.Yu. The main characteristics and possibilities of using Primelec hardware for the study of eye micro-movements (Report on the visit to the department and laboratory of neurology at the University of Zurich Clinic). 2011. Experimental Psychology 4 (3).
Ba.SO, Odobez J.-M. Sitting room meeting. 2006. 3rd Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI).
Bielecki M., Potega vel. Zabik K., Gochna M., Mikulski J. Mass analysis. 2017. ECEM 2017 Program and Abstracts Eds. Radach R., Deubel H., Vorstius C., Hofmann MJP121.
Campos, J., Alves-Oliveira, P., Paiva A. Looking for a conflict in a dyadic Mixed-Motive Game. 2015. Auton. Agent. Multi-Agent. Syst. 30 (1). P.112-135.
Efron N., Graeme Y., Brennan NA Ocular surface temperature. 1989. Current eye research 8 (9). P.901-906.
Funes Mora, KA, Odobez, J.-M. Person Independent 3D gaze estimation from remote RGBD cameras. 2013. In International Conference on Image Processing. Ieee.
Park KK, Suk HW, Hwang H., Lee J. mock crime using infrared thermal imaging. 2013. Frontiers in Human Neuroscience 7 (70). P.1 - 17.
R. Shillcock, Wase C. 2015. ECEM 2015 Program and Abstracts Eds. Ansorge U., Ditye T., Florack A., Leder HP66.
Siegfried R., Odobez J.-M. Supervised gaze bias correction for gaze coding in interactions. COGAIN Symposium Wuppertal, August 21st 2017.
Wang Q., Boccanfuso L., Li B., Ahn AY, Foster CE, Orr MP, Scassellati B., Shic F. Thermographic Eye Tracking. 2016. Conference paper: Proceedings of the Symposium on Eye Tracking Research and Applications.
Okutin OL, Okutina G.Yu. The main characteristics and possibilities of using Primelec hardware for the study of eye micro-movements (Report on the visit to the department and laboratory of neurology at the University of Zurich Clinic). 2011. Experimental Psychology 4 (3).
Material author:
Maria Konstantinova, Researcher at Neurodata Lab , biologist, physiologist, specialist in visual sensory system, oculography and ocular motorics.
Source: https://habr.com/ru/post/344782/
All Articles