Eye Control

Software for eye control stretches a very broad range of technologies starting from analysis and refinement of sensor input based on heuristics. Filters and machine learning is used to interface analysis and User Eyperience Design to psychological models and neural networks to model cognitive states and intention of users to active operation of applications and games. Interaction paradigms for eye control are only in the process of being defined with a wide range of interpretations ranging from assistive technology (high robustness, broad range of users) over augmentation of office work or game control (performance and comfort) to personal assistance (user-centric, context-aware). Don’t worry, we will show you some very tangible use cases in the next chapter.

Similar to the mouse, voice and touch, eye control is an enabling technology for almost all application development with user interaction. Key benefits for software solutions implementing eye control are:

• Enhanced user experience based on eye control as a natural and intuitive input modality
• Increased productivity due to reduction of mouse interaction time
• Better user well-being and performance based on user interaction responsive to the user’s cognitive-affective state (e.g., supporting flow states, active task scheduling adjusted to the current mental capacity)
• Enabling hands-free interaction for augmented/virtual/mixed reality and the Metaverse
• Better application design based on continuous large scale interaction analysis also when no active input is generated
• User intention/workflow prediction

What are the benefits of fewer mouse usage?

• Higher productivity: Reduced mouse usage leads to more consistent workflows, reducing average handling time.
• Higher employee satisfaction: Eye control eliminates the cause of RSI ("mouse arm") and other musculoskeletal diseases, resulting in improved well-being and fewer sick days.
• Higher employer attractiveness: An innovative workplace that supports well-being strengthens the image as an attractive place to work, resulting in lower attrition rates and higher competitiveness in the search for talent.

Eye control systems can not only be used beyond improving workplace ergonomics. Innovation in the workplace can also achieved in other areas, like Quality Assurance or at the Assembly Line. A touchless information terminal is another use case for hands-free software usage / gaze control projects. And there are more fields where gaze control could add value. Also the combination with other natural ways of interaction, like voice, offers potential for the technology.

Hands-free operation on the shop floor for quality assurance or assembly workplaces

• Situation: Worker must interrupt manual flow to interact with a machine/computer (e.g., confirm work step, input data).
• Problem: This interruption costs time, is not ergonomic, and can lead to work accidents
• Solution: Touchless and intuitive human-machine-interaction via eye control
• Technology: App runtime based on SAP UI5/ SAP Fiori, powered by SAP Business Technology Platform
• Benefits: Higher productivity, better ergonomics, and increased worker safety

Here you can find a video about the use case: Eye Control for SAP shop floor environments by NUIA

Touchless operation for info terminals using gaze control

Who doesn’t know info terminals? For example, at an entrance to a building, the airport, or at various other places. Well after Covid19 hygiene has become an even more important topic to us. Touching an info terminal that has been touched by quite a few people before might not sound appealing to you. This is another use case for eye control software. You can find a demonstration of how this works in the video here: Eye Tracking @Munich Urban Colab. The showcase was built with the the eye-tracking solution NUIA from our SAP partner 4tiitoo.

Link: https://news.sap.com/2021/08/eye-tracking-control-computer-with-gaze/

User intention support to replace mouse and touch usage for improved user interaction

Next to the “pointing benefits” the human gaze can also be utilized to understand the user's intention, to proactively support the user in the daily tasks. This ranges from basic interaction support to more complex prediction of the user’s cognitive situation and what he/she wants to do next.

User intention supporting use case include:

• automated scrolling while the user is reading an email or webpage
• predictive pre-loading of content, e.g., the next page of search results, when the user approaches the bottom of the list and is reading or searching
• optimize search results based on how the user exactly looks on specific search results, or excludes them automatically using the peripheral view
• offering user support or teasing learning nuggets when the user is lost in application operation.

The combination of gaze control with other human senses enables additional use cases

While gaze is optimal for pointing and intention recognition, it is not very well suited for entering information or complex commands. This is where voice control comes in as it offers the possibility to communicate complex commands or dictate content in natural language (Natural Language Processing). Therefore, the combination of both senses enables additional opportunities. This could be entering information into an input field by just looking at it and dictating text to looking at a large list view and asking to find specific information. A combination that might completely change the way one interacts with a system in the future.

What hardware is needed to use eye control software?

For this eye tracking software is used within different hardware devices. So if you ask yourself, what is eye tracking? Then this is your answer. Tracking where your sight goes / where you look at. Established eye tracking technology / hardware devices fall into three categories:

• Screen-based – A camera that tracks eye movements can be either stationary (computer webcam) or mobile (smartphone), with the commonality that the pupil position is mapped to a screen coordinate. More precise eye tracking is enabled by more sophisticated, infrared-based camera setups (remote eye tracker). A remarkable number of personal devices are already equipped with webcams that could be used for basic eye tracking. In the iOS SDK (since v12) app developers could easily get the face’s gaze position. For MS Windows, Tobii eye tracking devices are deeply integrated to allow not only eye tracking but even gaze control for uses such as scrolling.
  

• Glasses – Eye tracking glasses aren’t fundamentally different from an eye tracking webcam – the key distinction is that the camera is mounted in front of the user’s eyes. This can lead to more accurate tracking and, more importantly, a mapping of the eye gaze to real world objects while utilizing an additional front facing camera. This is often used in field scenarios, to perform remote and mobile eye tracking or control, for example on the factory/shop floor.
• Rumors in the market: Within the recent years the number of gaze tracking / eye control related patents has been increasing continuously. Who else could be behind such a solution? Well, many rumors revolve around Apple releasing new Apple Glasses. As of 2022 its still rumors though. This is also a main foundation for all discussions around metaverse.
  

• Glasses + Augmented Reality / XR headsets – By adding a virtual overlay to the real-world image a user sees, eye tracking opens a whole new possibility of interaction with a virtual and real environment. This is not only used for calibration to optimize the 3D experience, but also for acting with the 3D user interface as a replacement for or enhancement of a pointer. We have seen this with Microsoft Hololens 2 and Magic Leap. With Augmented Reality, eye-tracking isn’t just an added feature – it’s becoming a necessity.

Established vendors of eye tracking solutions include Tobii AB, SR Research Ltd., iMotions A/S,EyeTech Digital Systems, Inc., EyeTracking, Inc., Mirametrix Inc., Seeing Machines Ltd., Smart Eye AB, LC Technologies, Inc.

As every technology, eye control software also has its limitations and weaknesses. The following list gives you an impression, where the technology still has room for improvement and might not be a good fit.

Weaknesses

• Less precision compared to a mouse.
• Unconscious eye movements might trigger unintended actions if application design does not prevent it.
• Access to subconscious information requires dedicated privacy concepts.
• Initial user calibration needed.
• Users with multifocal glasses experience lower gaze precision.

Limitations

• Remote eye tracking limited to about 1m interaction distance with standard hardware (while special systems can support up to 3m).
• Cannot be used with IR coated glasses (e.g., most sunglasses) and extreme eye conditions.
• Eye tracker software cannot tell you why someone is looking at something (i.e., whether their visual attention is accompanied by a positive or negative emotion). However, first studies indicate that this limitation could be addressed using additional digital technologies, like machine learning on gaze patterns or by combining it with facial expression recognition.
  
  

Eye tracking and gaze control might also be used in other scenarios in the future. Maybe for some futuristic scenarios the technology even plays a key role. Let’s have a closer look at those potential fields of application.

Smart contact lenses as eye tracker

An exciting innovation to look out for are Smart Contact Lenses, which would be wearable just like any existing contact lenses and able to track eye movements and make the technology more mobile and accessible than ever before. If and how they can be used in a business context is still an open question though.

Use of eye control as part of the Metaverse

High quality Augmented Reality and Virtual Reality wearables will be crucial to the success of the metaverse. Those wearables must be able to ensure a high level of immersion, which the metaverse promises to bring to both the consumer and enterprise worlds. Enterprise metaverse worlds also promise to bring people together in common spaces for the purposes of collaboration, which naturally relies on effective and real-time communication. And as it is well known that most of communication is nonverbal, including facial expressions and eye movements, this is where eye tracking technology and VR gaze control will undoubtedly play a large role in delivering immersive, authentic connections and experiences by helping to understand human attention and expression.

Eye control can also help with more intuitive interaction with metaverse spaces, avatars, and objects as in some cases it's much more natural, intuitive, and easy to just control these with one's gaze rather than awkwardly waving hands or controllers around. It will also help to bring about a much more inclusive and accessible metaverse to those who are not able to use their hands as a means of interaction.

Eye control is an exciting and natural human-machine interaction technology that offers many interesting use cases and a lot of value potential. Especially in a business context eye control has found its way into productive use already.

At SAP eye control is also a topic on our technology radar and we see it as relevant for our business applications. So, we partnered with 4tiitoo, who went through our startup acceleration program, to enable users to leverage eye control software.