Methods of interaction with the system: from punched tapes to neural interfaces
Touch screens and touchpads are things that we interact with throughout our day. However, the first computers did without them: the input tools have gone through an impressive evolutionary path.
Mikhail Fedosov , Art Director of Nanosemantics and speaker of the Product Design Weekend course, gathered for us a brief history of the development of human-machine interfaces, talked about alternative ways to enter data and about systems that may become popular in the near future. Under the cut a lot of text and photos that cause nostalgia.
The first general-purpose electronic digital computer that could be reprogrammed to solve a wide range of problems was created in 1945.
')
The users of the first computers were exclusively engineers and programmers: interaction with the system was carried out by means of plugs, punched cards and punched tapes.
One of the first personal computers was Altair-8800. The car had no keyboard or screen. Users entered programs and data for them in binary format by clicking a set of small keys that could occupy only two positions: up and down. The results were also read in binary code - by luminous lights.
Teletype is an electromechanical printing press used to transmit text messages between two subscribers via the simplest electrical channel.
Computers used teletypes to input and output information. They had no displays - the user could start typing a command in the input window only after receiving a special signal on the tape.
It is interesting that at that time there were no standards for the location of keys and control buttons, and each computer manufacturer built an information input system based on its own considerations.
One of the first PC keyboards, made in 1974 by Tom Knight from the Massachusetts Institute of Technology. The keyboard contained many control keys and modifier keys. Even now the design looks good. Especially noteworthy is the key classification system: the data entry keys are gray, the control keys are blue with white text, the modifier keys are blue with black text.
The computer mouse, created in the USSR in the 80s, was popularly called the “rat”.
The metal ball looks especially harsh.
Naturally, this is not the first mouse in the world, but one of the first created in the USSR. By the way, as one of the key elements of the mechanism, an ordinary bearing made at the Chelyabinsk Tractor Plant was used. Here is an interesting extract from the description of the principles of work:
And the very first computer mouse in the world was invented back in 1964 by the American scientist Douglas Engelbar.
The computer system PLATO IV, which appeared in 1972, had a touch screen, whose work was based on a grid of infrared rays, segmented in 16x16 format. But even such low accuracy allowed the user to enter selected data by clicking on the screen in the right place.
The first tele-autograph, or graphics tablet, was invented in 1888. The most familiar to us on the mechanism of work and the most famous of the previously invented is the tablet RAND Tablet, invented in 1964. The RAND Tablet used a grid of conductors under the surface of the tablet, to which electrical impulses coded with the threefold Gray code were applied. A capacitively coupled pen received this signal, which could then be decoded back into coordinates.
An interesting principle of operation was the acoustic tablet, the pen of which generated sparks by means of a spark gap - an air gap separating the electrodes. The clicks were triangulated by a series of microphones to locate the pen. But the principle of operation was quite complex, and the tablets failed with extraneous noises.
The tablets became especially popular after the release of AutoCAD.
The first speech recognition device appeared in 1952, it could recognize human-readable numbers. And in 1963, a device was presented in the USA that could recognize whole phrases and execute commands.
Now the voice interfaces are actively developing. First, the emergence and development of neural networks contributes to more accurate voice recognition, improving the quality of this technology and, as a result, its popularity. The technology has reached such a high level that even the speech of a person with imperfect pronunciation, who is in a relatively noisy place, is successfully recognized.
Secondly, there are many devices where voice input is sometimes more convenient than any other: for example, navigators or smart speakers.
The third reason for the popularity of voice input is the large market in China, where, due to the difficulties of typing hieroglyphs, voice input and voice messages are widely used.
In addition to such giants as Amazon Alexa, Google Assistant, Siri or Alice Yandex, small but strong players appear on the market, for example, the voice assistant SOVA (Smart Open Virtual Assistant), in the development of which Mikhail Fedosov participated.
The main feature of this smart speaker is decentralized speech recognition, the use of which, according to the developers, allows you to refuse to collect confidential user information completely. Also, SOVA developers use fully open source code to enable third-party developers to improve the intelligence module of the column.
It can be noted that we still use all these data entry devices to a greater or lesser extent. Since the sixties, little has changed ideologically. The most popular ways to enter information are still a computer mouse, touchpad and voice commands, although the frequency of using each of them has changed.
But programmers and engineers tried to create new ways to interact with the user. Some were more successful and went into production, while others remained prototypes. Some devices are something completely innovative, while others are only designed to expand or improve existing functionality. Let's look at some of them.
A trackball is an input device that, in essence, is an inverted mechanical computer mouse. The principle of operation is the same, the only difference is that instead of rolling the mouse over the surface, the user rolls the ball over the mouse itself.
More modern variations of the trackball are integrated into gamepads or mice.
One of the trackball manufacturers, ITAC Systems, claims that after four hours of active work with the mouse, due to fatigue of the wrist, the arm becomes up to 60% weaker, while the use of the trackball does not affect the studied indicators of hand fatigue.
It is because of the tension of the muscles of the arm, many designers use graphic tablets, even if they are not important functions such as pressing force or a large working surface.
Also, many people like the clear response of the device (such people intentionally buy louder keyboards). In 2016, the Korean company Pantech released a smartphone Pantech Sky IM-100, in which the key feature was the control wheel. The video shows how it functions.
But trackballs did not gain much popularity either then or now. The smartphone from Pantech did not become a hit, although it managed to find its own small group of fans.
There are a lot of game controllers that imitate user interaction with real things from our everyday life. Probably the most famous of them is Guitar Hero.
The essence of this game is to press the buttons in time. If the user copes with this task, the track that he "plays" sounds continuous and beautiful. If a person does not hit the buttons, there is a grinding sound, and the melody is interrupted. Some craftsmen even managed to play Dark Soul on the guitar from the Guitar Hero set.
Of the unusual input devices in games, you can mention the device Tony Hawk Ride, imitating a skateboard. Also stands out the device Thrustmaster Flight MFD Cougar for flight simulators - it is mainly used for Microsoft Flight Simulator X.
In fact, these are not even alternatives, but modified customary input devices operating on the same principles, but using other technologies.
A keyboard that is projected on a flat surface. It works with a sensor that recognizes which button the user presses. But this keyboard did not have time to appear, as it had already lost its relevance. First, the user does not receive the usual feedback from the keyboard, and this slows down typing and makes it almost impossible to type blindly. Secondly, everyone has familiar smartphones on which you can quickly type text, and there is no point in carrying with you another box in the form of this device.
Tap Keyboard is worn on the hands and reads the position of the fingers in space. It has analogues, but they do not read the movements of the user badly, and all the advantages of such devices overlap very quickly with this fact. In more detail, the keyboard-brass knuckles are described in a recent article on Habré.
EasySMX Ring Mouse - mouse ring.
It is unlikely that it is suitable for everyday use, but, for example, as a clicker for presentations will be quite useful.
The Leap is a small USB device with cameras and sensors that captures the movement of hands and transfers information to a computer. Developed device by OcuSpec.
So far, the device looks damp, the angle of capture is rather narrow, and the accuracy leaves much to be desired. They have their own market where you can test the device. But the most important thing is that the company continues to develop its sensors and sees the perspective of its device in combination with virtual / augmented reality glasses.
Especially fresh is the fact that the hand "passes" for the object. LeapMotion builds a 3D model of the hand and “erases” part of the object. There are many videos on their YouTube channel showing this technology. Now, at least in the video, the recognition looks much clearer than just with The Leap device. But let's see what will happen in the release versions of the product.
Tobii Eye Tracker 4C helps the computer recognize the direction of the user's gaze. The device consists of cameras and sensors. In personal use, acts as an addition to the keyboard, mouse or gamepad, most often for games. For example, using this device in the game you can aim at opponents. In some games (for example, Elite Dangerous) the camera rotates depending on the direction of the user's gaze, which creates a sense of reality, immersion in the game.
In addition to games, such devices are often used in business, for example, to research a design or product. You can follow where a person looks in the first place when they first meet a product, or make a heat map and, based on this data, increase or decrease the contrast of elements.
Those who used this device note that it works well even in the dark (there is enough light from the monitor), and the glasses that the user can wear do not interfere with the correct operation of the device.
Eye Tracker is an excellent example of how to improve the already existing ways of human interaction with the machine, while not forcing the user to relearn, but only harmoniously complementing the already existing process.
Neurointerface - is the interaction of man and machine through the impulses of the brain. There are many companies that deal with neural interfaces, for example, Ctrl Labs (get information by reading the user's muscular activity) or Emotiv (interact through analyzing the user's brain activity). However, no company at the moment has shown significant progress in this area: as a rule, their capabilities are limited to moving objects around the screen.
The article covers, perhaps, the most common ways of interaction between man and machine. But there are still many not described here, but no less interesting devices. So, there are ways to interact with devices for people with special needs - for example, control with the help of tongue or buttocks. Experimental prostheses are associated with the human neural system and self-learning: the longer a person uses them, the easier it is for him to manage them, and the easier they become to manage.
You can learn more about management interfaces and other elements of product design on our course on Product Design Weekend , which will be held on December 22 and 23. Mikhail Fedosov will tell you how to functionally improve the product, what to look for when evaluating the interface and how not to get lost in trends.
Mikhail Fedosov , Art Director of Nanosemantics and speaker of the Product Design Weekend course, gathered for us a brief history of the development of human-machine interfaces, talked about alternative ways to enter data and about systems that may become popular in the near future. Under the cut a lot of text and photos that cause nostalgia.
Plug-in switching method
The first general-purpose electronic digital computer that could be reprogrammed to solve a wide range of problems was created in 1945.
')
The users of the first computers were exclusively engineers and programmers: interaction with the system was carried out by means of plugs, punched cards and punched tapes.
Toggle switches
One of the first personal computers was Altair-8800. The car had no keyboard or screen. Users entered programs and data for them in binary format by clicking a set of small keys that could occupy only two positions: up and down. The results were also read in binary code - by luminous lights.
TTY and first keyboards
Teletype is an electromechanical printing press used to transmit text messages between two subscribers via the simplest electrical channel.
Computers used teletypes to input and output information. They had no displays - the user could start typing a command in the input window only after receiving a special signal on the tape.
It is interesting that at that time there were no standards for the location of keys and control buttons, and each computer manufacturer built an information input system based on its own considerations.
One of the first PC keyboards, made in 1974 by Tom Knight from the Massachusetts Institute of Technology. The keyboard contained many control keys and modifier keys. Even now the design looks good. Especially noteworthy is the key classification system: the data entry keys are gray, the control keys are blue with white text, the modifier keys are blue with black text.
Manipulator of graphic information (mouse)
The computer mouse, created in the USSR in the 80s, was popularly called the “rat”.
The metal ball looks especially harsh.
Naturally, this is not the first mouse in the world, but one of the first created in the USSR. By the way, as one of the key elements of the mechanism, an ordinary bearing made at the Chelyabinsk Tractor Plant was used. Here is an interesting extract from the description of the principles of work:
"The invention relates to computing and can be used to enter service information. In order to improve the accuracy of the device, directional sensors are introduced into it, made on the basis of a planetary system in the form of ball bearings, providing the formation of output signals that determine the direction of rotation of the axes of the sensors along the X and Y axes, which eliminates the errors of the coordinates entered into the terminals. ”
And the very first computer mouse in the world was invented back in 1964 by the American scientist Douglas Engelbar.
Touch screen
The computer system PLATO IV, which appeared in 1972, had a touch screen, whose work was based on a grid of infrared rays, segmented in 16x16 format. But even such low accuracy allowed the user to enter selected data by clicking on the screen in the right place.
Teleautograph (graphic tablet)
The first tele-autograph, or graphics tablet, was invented in 1888. The most familiar to us on the mechanism of work and the most famous of the previously invented is the tablet RAND Tablet, invented in 1964. The RAND Tablet used a grid of conductors under the surface of the tablet, to which electrical impulses coded with the threefold Gray code were applied. A capacitively coupled pen received this signal, which could then be decoded back into coordinates.
An interesting principle of operation was the acoustic tablet, the pen of which generated sparks by means of a spark gap - an air gap separating the electrodes. The clicks were triangulated by a series of microphones to locate the pen. But the principle of operation was quite complex, and the tablets failed with extraneous noises.
The tablets became especially popular after the release of AutoCAD.
Speech recognition
The first speech recognition device appeared in 1952, it could recognize human-readable numbers. And in 1963, a device was presented in the USA that could recognize whole phrases and execute commands.
Now the voice interfaces are actively developing. First, the emergence and development of neural networks contributes to more accurate voice recognition, improving the quality of this technology and, as a result, its popularity. The technology has reached such a high level that even the speech of a person with imperfect pronunciation, who is in a relatively noisy place, is successfully recognized.
Secondly, there are many devices where voice input is sometimes more convenient than any other: for example, navigators or smart speakers.
The third reason for the popularity of voice input is the large market in China, where, due to the difficulties of typing hieroglyphs, voice input and voice messages are widely used.
In addition to such giants as Amazon Alexa, Google Assistant, Siri or Alice Yandex, small but strong players appear on the market, for example, the voice assistant SOVA (Smart Open Virtual Assistant), in the development of which Mikhail Fedosov participated.
The main feature of this smart speaker is decentralized speech recognition, the use of which, according to the developers, allows you to refuse to collect confidential user information completely. Also, SOVA developers use fully open source code to enable third-party developers to improve the intelligence module of the column.
It can be noted that we still use all these data entry devices to a greater or lesser extent. Since the sixties, little has changed ideologically. The most popular ways to enter information are still a computer mouse, touchpad and voice commands, although the frequency of using each of them has changed.
But programmers and engineers tried to create new ways to interact with the user. Some were more successful and went into production, while others remained prototypes. Some devices are something completely innovative, while others are only designed to expand or improve existing functionality. Let's look at some of them.
Trackball
A trackball is an input device that, in essence, is an inverted mechanical computer mouse. The principle of operation is the same, the only difference is that instead of rolling the mouse over the surface, the user rolls the ball over the mouse itself.
More modern variations of the trackball are integrated into gamepads or mice.
One of the trackball manufacturers, ITAC Systems, claims that after four hours of active work with the mouse, due to fatigue of the wrist, the arm becomes up to 60% weaker, while the use of the trackball does not affect the studied indicators of hand fatigue.
It is because of the tension of the muscles of the arm, many designers use graphic tablets, even if they are not important functions such as pressing force or a large working surface.
Also, many people like the clear response of the device (such people intentionally buy louder keyboards). In 2016, the Korean company Pantech released a smartphone Pantech Sky IM-100, in which the key feature was the control wheel. The video shows how it functions.
But trackballs did not gain much popularity either then or now. The smartphone from Pantech did not become a hit, although it managed to find its own small group of fans.
Game Controllers
There are a lot of game controllers that imitate user interaction with real things from our everyday life. Probably the most famous of them is Guitar Hero.
The essence of this game is to press the buttons in time. If the user copes with this task, the track that he "plays" sounds continuous and beautiful. If a person does not hit the buttons, there is a grinding sound, and the melody is interrupted. Some craftsmen even managed to play Dark Soul on the guitar from the Guitar Hero set.
Of the unusual input devices in games, you can mention the device Tony Hawk Ride, imitating a skateboard. Also stands out the device Thrustmaster Flight MFD Cougar for flight simulators - it is mainly used for Microsoft Flight Simulator X.
Alternatives to familiar devices
In fact, these are not even alternatives, but modified customary input devices operating on the same principles, but using other technologies.
Projection keyboard
A keyboard that is projected on a flat surface. It works with a sensor that recognizes which button the user presses. But this keyboard did not have time to appear, as it had already lost its relevance. First, the user does not receive the usual feedback from the keyboard, and this slows down typing and makes it almost impossible to type blindly. Secondly, everyone has familiar smartphones on which you can quickly type text, and there is no point in carrying with you another box in the form of this device.
Brass Keyboard
Tap Keyboard is worn on the hands and reads the position of the fingers in space. It has analogues, but they do not read the movements of the user badly, and all the advantages of such devices overlap very quickly with this fact. In more detail, the keyboard-brass knuckles are described in a recent article on Habré.
Mouse ring
EasySMX Ring Mouse - mouse ring.
It is unlikely that it is suitable for everyday use, but, for example, as a clicker for presentations will be quite useful.
Motion capture Near future?
The Leap is a small USB device with cameras and sensors that captures the movement of hands and transfers information to a computer. Developed device by OcuSpec.
So far, the device looks damp, the angle of capture is rather narrow, and the accuracy leaves much to be desired. They have their own market where you can test the device. But the most important thing is that the company continues to develop its sensors and sees the perspective of its device in combination with virtual / augmented reality glasses.
Especially fresh is the fact that the hand "passes" for the object. LeapMotion builds a 3D model of the hand and “erases” part of the object. There are many videos on their YouTube channel showing this technology. Now, at least in the video, the recognition looks much clearer than just with The Leap device. But let's see what will happen in the release versions of the product.
Eye tracker
Tobii Eye Tracker 4C helps the computer recognize the direction of the user's gaze. The device consists of cameras and sensors. In personal use, acts as an addition to the keyboard, mouse or gamepad, most often for games. For example, using this device in the game you can aim at opponents. In some games (for example, Elite Dangerous) the camera rotates depending on the direction of the user's gaze, which creates a sense of reality, immersion in the game.
In addition to games, such devices are often used in business, for example, to research a design or product. You can follow where a person looks in the first place when they first meet a product, or make a heat map and, based on this data, increase or decrease the contrast of elements.
Those who used this device note that it works well even in the dark (there is enough light from the monitor), and the glasses that the user can wear do not interfere with the correct operation of the device.
Eye Tracker is an excellent example of how to improve the already existing ways of human interaction with the machine, while not forcing the user to relearn, but only harmoniously complementing the already existing process.
Neural Interfaces
Neurointerface - is the interaction of man and machine through the impulses of the brain. There are many companies that deal with neural interfaces, for example, Ctrl Labs (get information by reading the user's muscular activity) or Emotiv (interact through analyzing the user's brain activity). However, no company at the moment has shown significant progress in this area: as a rule, their capabilities are limited to moving objects around the screen.
Conclusion
The article covers, perhaps, the most common ways of interaction between man and machine. But there are still many not described here, but no less interesting devices. So, there are ways to interact with devices for people with special needs - for example, control with the help of tongue or buttocks. Experimental prostheses are associated with the human neural system and self-learning: the longer a person uses them, the easier it is for him to manage them, and the easier they become to manage.
You can learn more about management interfaces and other elements of product design on our course on Product Design Weekend , which will be held on December 22 and 23. Mikhail Fedosov will tell you how to functionally improve the product, what to look for when evaluating the interface and how not to get lost in trends.
Source: https://habr.com/ru/post/432626/
All Articles