Eye Control software: a natural way to enhance interaction with computers and devices
A short definition of what eye control is
Eye Control, also know as gaze control, is an input method that allows users to operate a computer or other devices using only their eyes. It can also be combined with other input methods such as mouse, keyboard or voice.
Eye Control uses information about the point of gaze of the user to enhance interaction with computers and devices. The enhancement can be categorized by two dimensions:
- degree of interaction support: from augmenting existing mouse, keyboard, touch interaction to full hands-free control
- type of interaction support: from direct (e.g. triggering buttons by gaze) to predictive (e.g. pre-loading content based on gaze focus or scrolling based on reading predictions or supporting next steps in a workflow) to caring/abstract (e.g. detect a flow state in gaze patterns and automatically protect the user from non-urgent interruptions or reminding user to take a break)
Especially the predictive and abstract type of interaction support is a fundamental shift from all established types of computer interaction (mouse, keyboard, touch, voice) in that way, that the computer does not need to wait for an active user input but can actively create models about the cognitive state and intention of the user and take ad hoc action without user input.
With the growing presence of eye-tracking capable devices, eye control will substantially impact application development, e.g., i/o interface design or timing content delivery adjusted to the user's cognitive state (e.g., placing notification and training contents when the user is in a receptive state). So if you have ever asked yourself, how you can control your computer with your eyes, you will find the answers here in this article.
What is eye control software used for?
Software for eye control stretches a very broad range of technologies starting from analysis and refinement of sensor input based on heuristics. Filters and machine learning is used to interface analysis and User Eyperience Design to psychological models and neural networks to model cognitive states and intention of users to active operation of applications and games. Interaction paradigms for eye control are only in the process of being defined with a wide range of interpretations ranging from assistive technology (high robustness, broad range of users) over augmentation of office work or game control (performance and comfort) to personal assistance (user-centric, context-aware). Don’t worry, we will show you some very tangible use cases in the next chapter.
Benefits of using gaze control software:
Similar to the mouse, voice and touch, eye control is an enabling technology for almost all application development with user interaction. Key benefits for software solutions implementing eye control are:
- Enhanced user experience based on eye control as a natural and intuitive input modality
- Increased productivity due to reduction of mouse interaction time
- Better user well-being and performance based on user interaction responsive to the user’s cognitive-affective state (e.g., supporting flow states, active task scheduling adjusted to the current mental capacity)
- Enabling hands-free interaction for augmented/virtual/mixed reality and the Metaverse
- Better application design based on continuous large scale interaction analysis also when no active input is generated
- User intention/workflow prediction
Main Benefits of Eye Control Software
Why using eye control software at the workplace?
Human-machine interaction at office workplaces has not changed for more than 40 years
Probably the most obvious use case for eye control software is the office environment / the workplace. While we naturally use fingers and voice to operate our smartphones or smart speakers, computers are still controlled the same way as in the early 80s of the 20th century – using mouse and keyboard.
Therefore, thousands of mouse clicks and keyboard-mouse changes are part of everyday computer life. E.g., users in service centers perform 3,000 to 5,000 mouse clicks daily, which costs a lot of time (21 percent of a working day on average), causes health problems (such as RSI) and annoys users.
Innovation in the workplace towards better workplace ergonomics by using gaze control systems
Innovation in the workplace towards better workplace ergonomics by using gaze control systems
Through eye control and gaze-based automation of frequent activities such as copy&paste, scrolling or context changes, eye gaze control systems significantly reduce the daily mouse marathon. So basically, gaze control could impact the future of work by redesigning the way we interact with our systems in our day-to-day work. Something we from SAP have already started to investigate in. This might also help to design an inclusive workplace in the “new normal”.
What are the benefits of fewer mouse usage?
- Higher productivity: Reduced mouse usage leads to more consistent workflows, reducing average handling time.
- Higher employee satisfaction: Eye control eliminates the cause of RSI ("mouse arm") and other musculoskeletal diseases, resulting in improved well-being and fewer sick days.
- Higher employer attractiveness: An innovative workplace that supports well-being strengthens the image as an attractive place to work, resulting in lower attrition rates and higher competitiveness in the search for talent.
Gaze control use cases beyond the office workplace
Eye control systems can not only be used beyond improving workplace ergonomics. Innovation in the workplace can also achieved in other areas, like Quality Assurance or at the Assembly Line. A touchless information terminal is another use case for hands-free software usage / gaze control projects. And there are more fields where gaze control could add value. Also the combination with other natural ways of interaction, like voice, offers potential for the technology.
Hands-free operation on the shop floor for quality assurance or assembly workplaces
- Situation: Worker must interrupt manual flow to interact with a machine/computer (e.g., confirm work step, input data).
- Problem: This interruption costs time, is not ergonomic, and can lead to work accidents
- Solution: Touchless and intuitive human-machine-interaction via eye control
- Technology: App runtime based on SAP UI5/ SAP Fiori, powered by SAP Business Technology Platform
- Benefits: Higher productivity, better ergonomics, and increased worker safety
Here you can find a video about the use case: Eye Control for SAP shop floor environments by NUIA
Hannover Fair (Germany): How a system can be used with your bare eyes
Touchless operation for info terminals using gaze control
Who doesn’t know info terminals? For example, at an entrance to a building, the airport, or at various other places. Well after Covid19 hygiene has become an even more important topic to us. Touching an info terminal that has been touched by quite a few people before might not sound appealing to you. This is another use case for eye control software. You can find a demonstration of how this works in the video here: Eye Tracking @Munich Urban Colab. The showcase was built with the the eye-tracking solution NUIA from our SAP partner 4tiitoo.
Touchless operation of an information panel
User intention support to replace mouse and touch usage for improved user interaction
Next to the “pointing benefits” the human gaze can also be utilized to understand the user's intention, to proactively support the user in the daily tasks. This ranges from basic interaction support to more complex prediction of the user’s cognitive situation and what he/she wants to do next.
User intention supporting use case include:
- automated scrolling while the user is reading an email or webpage
- predictive pre-loading of content, e.g., the next page of search results, when the user approaches the bottom of the list and is reading or searching
- optimize search results based on how the user exactly looks on specific search results, or excludes them automatically using the peripheral view
- offering user support or teasing learning nuggets when the user is lost in application operation.
Heatmap for user experience optimization based on eye tracking
The combination of gaze control with other human senses enables additional use cases
While gaze is optimal for pointing and intention recognition, it is not very well suited for entering information or complex commands. This is where voice control comes in as it offers the possibility to communicate complex commands or dictate content in natural language (Natural Language Processing). Therefore, the combination of both senses enables additional opportunities. This could be entering information into an input field by just looking at it and dictating text to looking at a large list view and asking to find specific information. A combination that might completely change the way one interacts with a system in the future.
Eye Tracking Technology is the foundation for Gaze Control
The question might subsequently arise, because somehow the computer must know where you look at. When it comes to necessary hardware and technology components, it actually isn’t that complicated. In order to know where one looks at, and take actions from this, eye control software needs to track your gaze.
What hardware is needed to use eye control software?
For this eye tracking software is used within different hardware devices. So if you ask yourself, what is eye tracking? Then this is your answer. Tracking where your sight goes / where you look at. Established eye tracking technology / hardware devices fall into three categories:
Eye tracking hardware for gaze control software
- Screen-based – A camera that tracks eye movements can be either stationary (computer webcam) or mobile (smartphone), with the commonality that the pupil position is mapped to a screen coordinate. More precise eye tracking is enabled by more sophisticated, infrared-based camera setups (remote eye tracker). A remarkable number of personal devices are already equipped with webcams that could be used for basic eye tracking. In the iOS SDK (since v12) app developers could easily get the face’s gaze position. For MS Windows, Tobii eye tracking devices are deeply integrated to allow not only eye tracking but even gaze control for uses such as scrolling.
- Glasses – Eye tracking glasses aren’t fundamentally different from an eye tracking webcam – the key distinction is that the camera is mounted in front of the user’s eyes. This can lead to more accurate tracking and, more importantly, a mapping of the eye gaze to real world objects while utilizing an additional front facing camera. This is often used in field scenarios, to perform remote and mobile eye tracking or control, for example on the factory/shop floor.
- Rumors in the market: Within the recent years the number of gaze tracking / eye control related patents has been increasing continuously. Who else could be behind such a solution? Well, many rumors revolve around Apple releasing new Apple Glasses. As of 2022 its still rumors though. This is also a main foundation for all discussions around metaverse.
- Glasses + Augmented Reality / XR headsets – By adding a virtual overlay to the real-world image a user sees, eye tracking opens a whole new possibility of interaction with a virtual and real environment. This is not only used for calibration to optimize the 3D experience, but also for acting with the 3D user interface as a replacement for or enhancement of a pointer. We have seen this with Microsoft Hololens 2 and Magic Leap. With Augmented Reality, eye-tracking isn’t just an added feature – it’s becoming a necessity.
Glasses used for eye control
Who offers eye tracking technology?
Established vendors of eye tracking solutions include Tobii AB, SR Research Ltd., iMotions A/S,EyeTech Digital Systems, Inc., EyeTracking, Inc., Mirametrix Inc., Seeing Machines Ltd., Smart Eye AB, LC Technologies, Inc.
SAP partnered up with 4iitoo to create the workplace of the future
What are current limitations of eye tracker software / gaze control systems?
As every technology, eye control software also has its limitations and weaknesses. The following list gives you an impression, where the technology still has room for improvement and might not be a good fit.
- Less precision compared to a mouse.
- Unconscious eye movements might trigger unintended actions if application design does not prevent it.
- Access to subconscious information requires dedicated privacy concepts.
- Initial user calibration needed.
- Users with multifocal glasses experience lower gaze precision.
- Remote eye tracking limited to about 1m interaction distance with standard hardware (while special systems can support up to 3m).
- Cannot be used with IR coated glasses (e.g., most sunglasses) and extreme eye conditions.
- Eye tracker software cannot tell you why someone is looking at something (i.e., whether their visual attention is accompanied by a positive or negative emotion). However, first studies indicate that this limitation could be addressed using additional digital technologies, like machine learning on gaze patterns or by combining it with facial expression recognition.
The future of eye tracking and gaze control systems
Eye tracking and gaze control might also be used in other scenarios in the future. Maybe for some futuristic scenarios the technology even plays a key role. Let’s have a closer look at those potential fields of application.
Smart contact lenses as eye tracker
An exciting innovation to look out for are Smart Contact Lenses, which would be wearable just like any existing contact lenses and able to track eye movements and make the technology more mobile and accessible than ever before. If and how they can be used in a business context is still an open question though.
Use of eye control as part of the Metaverse
High quality Augmented Reality and Virtual Reality wearables will be crucial to the success of the metaverse. Those wearables must be able to ensure a high level of immersion, which the metaverse promises to bring to both the consumer and enterprise worlds. Enterprise metaverse worlds also promise to bring people together in common spaces for the purposes of collaboration, which naturally relies on effective and real-time communication. And as it is well known that most of communication is nonverbal, including facial expressions and eye movements, this is where eye tracking technology and VR gaze control will undoubtedly play a large role in delivering immersive, authentic connections and experiences by helping to understand human attention and expression.
Eye control can also help with more intuitive interaction with metaverse spaces, avatars, and objects as in some cases it's much more natural, intuitive, and easy to just control these with one's gaze rather than awkwardly waving hands or controllers around. It will also help to bring about a much more inclusive and accessible metaverse to those who are not able to use their hands as a means of interaction.
How SAP uses eye control software
Eye control is an exciting and natural human-machine interaction technology that offers many interesting use cases and a lot of value potential. Especially in a business context eye control has found its way into productive use already.
At SAP eye control is also a topic on our technology radar and we see it as relevant for our business applications. So, we partnered with 4tiitoo, who went through our startup acceleration program, to enable users to leverage eye control software.
SAP / business software can be controlled with the eyes
Eye control with NUIA from 4tiitoo - redefining human-machine interaction
SAP partners with 4tiitoo to make gaze control available
In this context 4tiitoo developed the possibility to control the SAP Fiori user interface with eyes. SAP’s CEO Christian Klein showcased this functionality in his presentation to chancellor Merkel at Hannover Messe 2019. Read more about the Co-Innovation: Eye-Tracking Startup Co-Innovates with SAP.
SAP started to use NUIA Eye Control in departments from support center to shared services. Furthermore, the eye control solution was added to the Future of Work program due to its benefits in well-being, inclusion, and work facilitation and equipped flex desk locations with NUIA eye trackers.
Didn’t know that SAP is using this technology? Doesn’t surprise me. If you are interested to learn more about the technology and potential use cases, don’t hesitate to reach out via firstname.lastname@example.org.
You can also download 4tiitoo’s Eye Control software here and try it yourself: Download NUIA Intelligent Workplace