Machine Gesture and Sign Language Recognition

Machine gesture and sign language recognition is, as the name suggests, about recognition of gestures and/or sign language using computers. A number of hardware techniques are used for gathering information about body positioning; typically either image-based (using cameras, moving lights etc) or device-based (using instrumented gloves, styli, position trackers etc.), although hybrids are beginning to come about.

However, getting the data is only the first step. The second step, that of recognising the sign or gesture once it has been captured is much more challenging, especially in a continuous stream. In fact currently, this is the focus of the research.

This page tries to include links to online information on this subject on the 'Net, and to stay up-to-date as far as conferences and so on are concerned.

If you have any suggestions, corrections or additions, please don't hesitate to mail me.

Index

• Recent Changes

• Online resources and researchers in gesture and sign language

• Online resources in allied fields

• Conferences in this and associated fields

• Adding or changing material on this page.

Recent Changes

2 October 2002

Added Information about Ipke's great list of Gesture Conferences.

21 November 2001

Added Richard Bowden to the list of researchers in gesture recognition.

Online resources and researchers in gesture and sign language recognition.

• Tony Heap has developed an effective system for tracking hand position using a video camera. While not directly involving gesture recognition, it is an important preprocessing step for video-based gesture recognition. At his Web page, you can find a brief description of the software, can download a Tech Report about it, and even download demo software for an SGI from there.

• Christopher Lee and Yangsheng Xu at Carnegie-Mellon University have been working on a system that recognises gestures from a Cyberglove in real-time. Currently, it can learn up to fourteen gestures fairly easily. You can visit their home page, or download their paper on the system.

• Graeme Sweeney has helped set up the Gesture Archive, as a result of Gesture Workshop '96. Not only has he set up a good set of links, but he has also set up the gesture data repository, for collected gesture data. I've put my data there, and I recommend that others upload their data as well.

• Thad Starner at the Media Lab at MIT has done some work on recognition of ASL using a camera and hidden Markov models. He has the Tech Report and his Master's thesis available in PostScript format. This is currently the state of the art in my opinion. There is also the abstract of his Master's thesis here.

• A very useful introduction to the concept of Whole-Hand Input is by D. J. Sturman also at MIT, whose comprehensive PhD thesis is a great starting point. You can get it (in PostScript) from here.

• Another person who did his Master's Thesis at MIT was Alan Wexelblat, who constructed a generalised feature-based recogniser using two CyberGloves and three Bird trackers. You can get his Master's thesis in PostScript. Also of great use is his bibliography. Together with Dr Marc Cavazza, Alan also ran a Workshop at CHI '95 -- entitled Gesture at the User Interface.

• Brigitte Dorner did her Master's thesis at Simon Fraser University, Canada, on recognising movements of a glove, termed the Color Glove. You can get her thesis here. At the same time, Eli Hagen has been working on a flexible deductive database to ASL, which you can get here. Both are in PostScript. Currently they are working together to get a complete translation system for ASL up and running.

  Also working at SFU is Axel Mulder who also has a great set of Web Pages. His interest is in musical instruments and gesture.

• Lynn Messing is also doing work trying to recognise ASL (although as a preliminary step she's trying to recognise Signed English). Already she has a finger-spelling recognition system with 96 per cent accuracy. She's also, together with Garland Stern, been working on Sister Mary of the Internet which displays previously captured signs. They plan to get her to do the opposite of recognition, i.e. given a modified Stokoe (prn. stoa-KEE) notation of the signs, it performs the sign. Lynn was also chair of the WIGLS workshop.

• Peter Vamplew is developing a sign language recognition system for Auslan, termed SLARTI. He's at the University of Tasmania.

• Sidney Fels did some research on converting gestures to sound in his Ph D thesis, completing a system called GloveTalk-II -- available here. A complete list of Sidney's publications is available at ATR.

• Randy Pausch (author of "Virtual Reality on $5 a day") developed a system called CANDY designed to allow children with cerebral palsy a means to communicate. You can get that thesis from here.

• James Kramer, founder and now president of Virtual Technologies, designed the CyberGlove that is probably the best glove currently available. Virtual Technologies also produce the GesturePlus system which can recognise finger-spelling at up to 4 characters a second. VirTex can be contacted at info@virtex.com. They also have a Web page.

• Chris Hand's homepage on Gesture Recognition This has a lot of useful information if you are interested in this area, including Chris' own paper on gesture recognition using grammars.

• Josh Bers developed a system for recognition of gestures for animating computer characters.

• Yanghee Nam at the KAIST in Korea is doing research into hand gesture recognition. He, like Thad Starner is looking at their recognition using Hidden Markov Models; although he is not using them specifically for recognition of sign languages.

• Richard Bowden is researching a number of issues related to temporal tracking of hands for BSL finger spelling recognition, markerless motion capture and automatic conversion from British Sign Language to American Sign Language.

• My research: My research involves recognising Auslan (Australian Sign Language), using single discrete signs and a PowerGlove.

Associated Web services

• A useful source of information is the mailing list GESTURE-L. Click here to subscribe to the newsgroup. Include the words subscribe gesture-l. You can also search the archive and the Welcome message. Update: Gesture-l is being shut down; but hopefully, an alternative will be set up. Several people have volunteered. Stay tuned for more info as it comes to hand.

• Much of the technology used and some of the topics overlap into the area of virtual reality. The moderated newsgroup sci.virtual-worlds is a great source of information on some of the technical aspects, such as gloves and so on. The newsgroup is moderated by Toni Emerson who works for the HIT Lab at the University of Washington. Of particular usefulness is the ftp archive they have over there. HIT Lab is dedicated to providing info about VR.

• Sherman Wilcox is working on a multimedia dictionary for ASL. Currently he has a working application that runs on Macs.

Conferences in this and associated areas

• Gesture In Human-Computer Interaction: A list put together by Ipke Wachsmuth of previous and future conferences on Human-Computer Interaction. Very useful.

• Gesture Workshop '96: This conference was held in York, England on Tuesday 19th March 1996.

• European Conference on Disablity, Virtual Reality and Associated Technologies: This was held in Reading, England on 8th-10th July 1996. Not directly relevant, but since the instrumented gloves are an "associated technology", maybe a few papers on gesture recognition could get a look-in.

• WIGLS is the Workshop on the Integration of Gesture and Language in Speech, organised by Lynn Messing (see above).

• Second International Conference on Face and Gesture Recognition: This was held in Killington, Vermont, USA from October 13 to October 16.

• The Bielefeld Gesture Workshop 1997 was held from 17-19th September 1997.
• Gesture Workshop '99, the Third Gesture Workshop occurred on 17 - 19 March 1999, at Gif-sur-Yvette, near Paris.

Upcoming conferences in this and associated areas

Adding Material