Eye-Based IT



Background

Recently the ALS patient Birger Jeppesen wrote the book " Is heaven a better place to be? (Danish title: Er der mon bedre i himlen?)". This book was written using an eye-controlled computer user interface. The cost price of this equipment is approximately US$ 3000 in addition to the price of the PC. Furthermore, this system requires a substantial training periode and some human assistance. It is the aim of the project "Eye-Based IT" to make eye-controlled computer equipment accessible to a broader audience of disabled people, at a lower cost price, and in a more user friendly form. Traditional communication equipment like pointers mounted in the forehead have a number of obvious drawbacks: they are slow, tedious, and fatiguing to use for longer periods, and they do often require assistance for mounting the equipment. However, they are cheap and work in conjunction with standard interfaces such as keyboards. Speech recognition systems are currently not available in Danish, and they cannot be used unless the patient has a clear pronunciation. It is the aim of the project "Eye-Based IT" to develop cheap, fast, and flexible tools that can be used by disabled people of minimum mobility and with a minimum of assistance. Furthermore, eye-based user interfaces may also show useful in professional settings and it may be an alternative interaction device for people with arm or shoulder strain problems.


Technology

A system that measures the users eye movements when he or she observes a computer screen or a blackboard may be used for eye gaze typing. Existing gaze typing systems are relatively expensive (US$ 3000 to 25000 plus the cost of a PC) and some of them does not include the special Danish characters æ, ø, and å. See list of products and "Eye Controlled Media" by Theo Engell-Nielsen & Arne John Glenstrup for a general description of the potential of this technology. It is the aim of the project "Eye-Based IT" to develop systems based on ordinary video cameras or web-cams and to develop eye tracking software that will run on a state-of-the-art PC. The price of the total system will be low and the software can be distributed via the Internet. The drawbacks are that the system cannot track the eye movement very precisely. In general, it will not be possible to use standard mouse operated software. Only the more expensive systems can obtain such a precision. They have not become cheaper during the last 5 years as they are based on specialized hardware components. The system developed in the project "Eye-Based IT" will take use of a second computer monitor placed on top of the ordinary monitor. Next to this a plate with characters may be placed. The user selects a character on the plate by looking at it. It now appears on the extra computer monitor. The user acknowlegde it by fixating the monitor. This systems is very flexible and new symbols and characters can easily be included. It is design goal that well trained users may reach a typing speed of one character a second under optimal circumstances. The final system can be downloaded from a website. It is the plan to develop the tracking system so it can be used with standard word processors, email systems, and Internet browsers. Should this not be feasible, a specialized integrated word processor, email system, and Internet browser will be developed under the project. Furthermore, it will be investigated how the technology may be incorporated in interactive television systems.


PhD-project: visual tracking of gaze direction.

By use of a camera mounted on top of a standard PC, the eye and iris must be tracked. It shall be investigated if these informations can be used for determining the gaze direction. Eventually other facial features can be taken into account. The basic techniques in this PhD-project are real-time tracking (snakes and condensation techniques) and computer vision (self-calibration and tensor techniques).


Collaborating partners

The system will be developed in collaboration with Professor Kenji Itoh, Tokyo Institute of Technology. Professor Itoh will be responsible for the implementation of the Japanese character set (approximately 400 units). During the development representatives from user organizations will take part in the project. Their advise will be dictating the evolution of the project.


Time schedule

>
February-March 2000 Appointment of PhD-student in computer vision
March-April 2000 Analysis of demands in collaboration with user organizations and establishment of experimental lab.
April 2000 - December 2000 Development of tracking-algorithms.
January 2001 User test of tracking-system.
February 2001 - July 2001 Design of user interface for plate-monitor-system. Patents.
August 2001 Presentation of prototype system
September 2001 Project meeting in Tokyo with Japanese partner
September 2001 - December 2001 Optimization of tracking system. Programming of drivers for tracking system.
January 2002 - June 2002 End-user tests.
April 2002 Project meeting in Copenhagen with Japanese partners.
August 2002 - December 2002 Programming of web site. Eventually programming of email/browser system. User test of web site.
January 2003 Upload of web site. Ph.d.-thesis ready.
February - April 2003 Lectures and courses for organizations of disabled.



Staff and partners:

Associate professor Mads Nielsen, main Ph.D.-supervisor
Associate professor John Paulin Hansen co-supervisor, project responsible, design of user interfaces.
Ph.d.student in computer vision systems.
Programmer of drivers and user interfaces
Professor Kenji Itoh, Tokyo Institute of Technology
Research Associate Hirotaka Aoki, Tokyo Institute of Technology