SixthSense - a wearable gestural interface (MIT Media Lab)

http://code.google.com/p/sixthsense/

 

 

 

 

http://vediktronics.blogspot.kr/2011/12/sixth-sense-technology.html

 

Sixth Sense Technology It’s the beginning of a new era of technology where engineering will reach new milestones. Just like in the science fiction movies where display of computer screen appears on walls, commands are given by gestures, the smart digital environment which talks to us to do our work and so on, these all will be possible very soon. You imagine it and sixth sense technology will make it possible. Isn’t it futuristic? Now it’s time for sci-fi movie directors to think ahead because the technology shown in there fiction movies soon will become household stuff. Before few years back it was considered to be supernatural or tantalizing imagination. But now it has been made possible. Thanks to Pranav Mistry, a genius who introduced mankind to this futuristic technology. What is sixth sense? Sixth Sense is a wearable gestural interface that enhances the physical world around us with digital information and lets us use natural hand gestures to interact with that information. It is based on the concepts of augmented reality and has well implemented the perceptions of it. Sixth sense technology has integrated the real world objects with digital world. The fabulous 6th sense technology is a blend of many exquisite technologies. The thing which makes it magnificent is the marvelous integration of all those technologies and presents it into a single portable and economical product. It associates technologies like hand gesture recognition, image capturing, processing, and manipulation, etc. It superimposes the digital world on the real world. Sixth sense technology is a perception of augmented reality concept. Like senses enable us to perceive information about the environment in different ways it also aims at perceiving information. Sixth sense is in fact, about comprehending information more than our available senses. And today there is not just this physical world from where we get information but also the digital world which has become a part of our life. This digital world is now as important to us as this physical world. And with the internet the digital world can be expanded many times the physical world. God hasn’t given us sense to interact with the digital world so we have created them like smart phones, tablets, computers, laptops, net books, PDAs, music players, and others gadgets. These gadgets enable us to communicate with the digital world around us. But we’re humans and our physical body isn’t meant for digital world so we can’t interact directly to the digital world. For instance we press keys to dial a number; we type text to search it and so on. This means for an individual to communicate with the digital world he/she must learn it. We don’t communicate directly and efficiently to the digital world as we do with the real world. The sixth sense technology is all about interacting to the digital world in most efficient and direct way. Hence, it wouldn’t be wrong to conclude sixth sense technology as gateway between digital and real world.  Before Wear Ur World (WuW) came there were other methods like speech recognition software, touch recognition etc., which empowered us with direct interfacing. This WuW or sixth sense device invented by Pranav Mistry is a prototype of next level of digital to real world interfacing. It comprises of a camera, a projector, a mobile cum computing device and colored sensors which are put on the fingers of a human being. The device efficiently senses the motion of the colored markers. Using them it provides us the freedom of directly interacting with the digital world. This technology enables people to interact in the digital world as if they are interacting in the real world. Why choose sixth sense technology? Humans take decisions after acquiring inputs from the senses. But the information we collect aren’t enough to result in the right decisions. But the information which could help making a good decision is largely available on internet. Although the information can be gathered by connecting devices like computers and mobiles but they are restricted to the screen and there is no direct interaction between the tangible physical world and intangible digital world. This sixth sense technology provides us with the freedom of interacting with the digital world with hand gestures. This technology has a wide application in the field of artificial intelligence. This methodology can aid in synthesis of bots that will be able to interact with humans. How does sixth sense works? The sixth sense technology uses different technologies like gesture recognition, image processing, etc. At present the commercial product isn’t launched but the prototype is prepared. The sixth sense prototype is made using very common and easily available equipments like pocket projector, a mirror, mobile components, color markers and a camera. The projector projects visual images on a surface. This surface can be wall, table, book or even your hand. Thus, the entire world is available on your screen now. When user moves their hands to form different movements with colored markers on the finger tips, the camera captures these movements. Both the projector and the camera are connected to the mobile computing device in the user’s pocket. Recognition is made using computer vision technique. These markers act as visual tracking fiducials. The software program processes this video stream data and interprets the movements into gestures. The gestures are different from one another and are assigned some commands. These gestures can act as input to application which is projected by the projector. Since, the projector is aligned downwards for compactness; therefore images would be formed at the user’s feet if mirror wasn’t used. The mirror reflects the image formed by the projector to front. The entire hardware is fabricated in the form of a pendent. The entire product cost around $ 350 and that also because of projector. It works very similar like a touch screen phone with entire world as the screen. Evolution of Sixth Sense Technology Steve Mann is considered as the father of Sixth Sense technology who made a wearable computer in 1990. He implemented the Sixth Sense technology as the neck worn projector with a camera system. He was a media lab student at that time. Then his work was carried forward by Pranav Mistry, an Indian research assistant in MIT Media Lab. He came up with exciting new applications from this technology. Sixth sense technology was developed at media labs in MIT and coined as Wear Ur World (WUW). The inventors have filed patent under the name Wear Ur World (WUW) in February 2010. “Rather than waiting for that time to come, I want people to make their own system. Why not?,” Mistry says in an article on Rediff Business. “People will be able to make their own hardware. I will give them instructions how to make it. And also provide them key software…give them basic key software layers…they will be able to build their own applications. They will be able to modify base level and do anything”. So it can be expected that the software will be open source and there will be a wide market of apps too. Applications Fingers as brush: The user can draw anything on paint with the help of his fingers. This drawing can be 3D also. Hence, no need to use mouse. Capture photos with fingers: using the fingers the user can capture photos hence, no need to carry an additional gizmo. The box created by the fingers act as frame for capturing photo. Palm is the new dialer: this technology enables the user to call without using the dialer. The dialer will be projected on palm and the user can dial the number using other hand. Read Books easily: Check out the ratings of the Book you are going to buy, it checks the ratings from the internet. And another amazing thing is that it reads the book for you. Video Newspapers: like the video newspapers of Harry Potter this technology identifies the news headline and then projects the relevant video. Check your Flight Status: Just place the ticket in front of the projector and it checks its status from the internet. Clock: the user just needs to make gesture of clock and the watch will be projected on the user’s hand. Access anywhere internet: the users can browse internet on any surface even on their palm. Conclusion This technology has seamless applications. This can be used as a replacement of the 5th senses for handicapped peoples. This can provide easy control over machineries in industry. This will have different application for different developers just depending upon how he imagines and what he wants. So, considering its widespread applications the inventor Pranav Mistry has decided to make its software open source. This will enable individuals to make their own application depending upon needs and imagination. As this technology will emerge may be new devices and hence forth new markets will evolve. Some existing devices and technologies will be discontinued but one thing is guaranteed it will write a new chapter in history of science and technology.

 

 

http://www.pranavmistry.com/projects/sixthsense/#PUBLICATIONS - SixthSense - a wearable gestural interface (MIT Media Lab)

 

integrating information with the real world Pranav Mistry Fluid Interfaces Group | MIT Media Lab   ABOUT 'SixthSense' is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information. We've evolved over millions of years to sense the world around us. When we encounter something, someone or some place, we use our five natural senses to perceive information about it; that information helps us make decisions and chose the right actions to take. But arguably the most useful information that can help us make the right decision is not naturally perceivable with our five senses, namely the data, information and knowledge that mankind has accumulated about everything and which is increasingly all available online. Although the miniaturization of computing devices allows us to carry computers in our pockets, keeping us continually connected to the digital world, there is no link between our digital devices and our interactions with the physical world. Information is confined traditionally on paper or digitally on a screen. SixthSense bridges this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this information via natural hand gestures. ‘SixthSense’ frees information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer. The SixthSense prototype is comprised of a pocket projector, a mirror and a camera. The hardware components are coupled in a pendant like mobile wearable device. Both the projector and the camera are connected to the mobile computing device in the user’s pocket. The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks user's hand gestures and physical objects using computer-vision based techniques. The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducials) at the tip of the user’s fingers using simple computer-vision techniques. The movements and arrangements of these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces. The maximum number of tracked fingers is only constrained by the number of unique fiducials, thus SixthSense also supports multi-touch and multi-user interaction. The SixthSense prototype implements several applications that demonstrate the usefulness, viability and flexibility of the system. The map application lets the user navigate a map displayed on a nearby surface using hand gestures, similar to gestures supported by Multi-Touch based systems, letting the user zoom in, zoom out or pan using intuitive hand movements. The drawing application lets the user draw on any surface by tracking the fingertip movements of the user’s index finger. SixthSense also recognizes user’s freehand gestures (postures). For example, the SixthSense system implements a gestural camera that takes photos of the scene the user is looking at by detecting the ‘framing’ gesture. The user can stop by any surface or wall and flick through the photos he/she has taken. SixthSense also lets the user draw icons or symbols in the air using the movement of the index finger and recognizes those symbols as interaction instructions. For example, drawing a magnifying glass symbol takes the user to the map application or drawing an ‘@’ symbol lets the user check his mail. The SixthSense system also augments physical objects the user is interacting with by projecting more information about these objects projected on them. For example, a newspaper can show live video news or dynamic information can be provided on a regular piece of paper. The gesture of drawing a circle on the user’s wrist projects an analog watch. The current prototype system costs approximate $350 to build. Instructions on how to make your own prototype device can be found here (coming soon). PICTURES                           . . . some more pictures are coming soon.   . . . more videos are coming soon, too. PUBLICATIONS P. Mistry, P. Maes. SixthSense – A Wearable Gestural Interface. In the Proceedings of SIGGRAPH Asia 2009, Sketch. Yokohama, Japan. 2009 P. Mistry, P. Maes. SixthSense – A Wearable Gestural Interface. In the Proceedings of SIGGRAPH Asia 2009, Emerging Technologies. Yokohama, Japan. 2009 P. Mistry. The thrilling potential of SixthSense technology. TEDIndia 2009. Mysore, India 2009 P. Mistry, P. Maes, L. Chang. WUW - Wear Ur World - A Wearable Gestural Interface. In the CHI '09 extended abstracts on Human factors in computing systems. Boston, USA. 2009 P. Maes, P. Mistry. Unveiling the "Sixth Sense," game-changing wearable tech. TED 2009. Long Beach, CA, USA 2009 AWARDS   SixthSense wins  2009 INVENTION AWARD  by Popular Science      TR35 2009 Young Inventor Under 35 Award  by Technology Review PRESS COVERAGE  WIRED News   MIT Students Turn Internet Into a Sixth Human Sense  FOX News   MIT Team Shows 'Sixth Sense' Wearable Computer  BBC   SixthSense blurs digital and the real  The Sydney Morning Herald   MIT researchers make 'sixth sense' gadget  The Inquirer   MIT makes sixth sense gadget  Mumbai Mirror   Wear ur world  THE TIMES OF INDIA   'Sixth sense' gizmo makes sci-fi a reality  Gizmodo   Sixth Sense Technology May Change How We Look at the World Forever  Fast Company   MIT's Sixth Sense Machine Makes Reality Better  Slashdot   MIT Researchers Create a Cheap "6th Sense" Device  engadget   MIT's "sixth sense" augmented reality device demonstrated on video  engadget   MIT concocts wearable "sixth sense" device, Bruce Willis is like "what?"  SLASH GEAR   Wearable projection computer project: Internet ’sixth sense’  World Changing   Pattie Maes Invents a Sixth Sense  SOFTPEDIA   Next-Gen 'Sixth Sense' Device Created at MIT  Gearlog   The Internet as an Extension of Self  DVICE.com   Minority Report-like wearable interface gives you a sixth sense  MAXIMUMPC   TED 2009: The Sixth Sense is Actually a Minority Report-like Internet Interface  redOrbit   MIT Develops Device That Offers 'Sixth Sense'  AFP   MIT researchers make 'sixth sense' gadget  TMC   MIT Researchers Unveil Computing Device that Turns Any Surface into a Touchscreen  betanews   MIT develops an even more augmented reality device  GEEK.com   MIT develop Wear Ur World, 6th sense hand gesture gadget  GIGAOM   MIT develop Wear Ur World, 6th sense hand gesture gadget  Crunch Gear   Video: That MIT “sixth sense” device in action  GottaBeMobile.com   MIT Student Doing Amazing “Sixth Sense” Work  sina.com   美推出可穿戴电脑将网络变成人类第六感  io9.com   MIT Student Turns His Body Into a Computer  Wired Vision   身体と世界と情報を統合：情報をどこにでも投影するウェアラブル・マシン：動画  CNET NEWS   MIT's 6th Sense device could trump Apple's multitouch  New York Times   At TED, Virtual Worlds Collide With Reality  hexun.com   “少数派报告”成为现实 用手指控制虚拟屏(组图)  CNET JAPAN   MIT研究グループ、「第6感」ガジェットを開発  CNET TAIWAN   MIT的Sixth Sense裝置挑戰蘋果的多重觸控技術  news.com.au   MIT's 'Wear Ur World' (WUW) device puts internet literally at your fingertips  The Sun   New Minority Report internet  The Gazette, CANADA   MIT researchers make 'Minority Report' gadget  JOINS NEWS   손가락 컴퓨터가 펼치는 4차원 세계  EXPRESS COMPUTER   Windows 7, Netbooks, Kindle 2 and WUW  The Tech, MIT   New Invention Gives People 'Sixth Sense'  Le Monde, France   La réalité augmentée du bout des doigts  Mail Today, India   Indian geek invents 'SIXTH SENSE' gizmo  COMPUTERWORLD   Future shock: The PC of 2019  computerra, Russia   Все свое носи с собой  American Observer   SixthSense Makes the Entire World your Computer  Clarin.com   Diseñan la primera computadora que se maneja con gestos  Business Standard   Indian IT geek develops digital 'sixth sense' at MIT  IndianExpress   The world at your fingertips  CORE77   MIT's wearable "Sixth Sense" device turns any surface  CIO   MIT Wearable Gadget Gives You Sixth Sense, a la Minority Report  PCWORLD   MIT Wearable Gadget Gives You Sixth Sense  The Christian Science Monitor   Sixth Sense: A Web you can wear  ACM News   MIT Student Develops Wearable Interface That Relies on Hand Gestures  CNN IBN   Superstars of Science  The Times of India   Gujarati boy rules world's senses  Financial Cronicle   The world @ your fingertips  CNN IBN   Superstars of Science  Sina.com   麻省理工開發出可佩戴式計算機  ELTIEMPO.COM   Sexto Sentido Digital CONTACT US Pranav Mistry pranav@mit.edu | www.pranavmistry.com     미니 프로젝터를 활용한 웨어러블 프로젝션 컴퓨터 프로젝트 WUW (Wear Ur World)   http://www.pranavmistry.com/projects/sixthsense/ 손동작과 음성인식으로 콘트롤되는 Wearable Gestural Interface 프로젝트가 MIT 학생들 (MIT 미디어랩에서 Fluid Interface를 연구하는 Pranav Mistry, Liyan Chang, Pattie Maes (지도 교수))에 의해 개발중이고, TED에서 최근 데모가 공개되었다. 프로토타입의 구성품은 3M의 피코 프로젝터, 휴대폰, 로지텍 퀵캠, 손가락끝에 장착된 4가지 칼라의 매직마커 등이고, 등에는 노트북을 메고 있다. 맨 아래 PDF 링크를 클릭하면, 상세 논문 내용을 볼 수 있다.     wuw_wearable_gestural_interface_mit.pdf ※ 아래는 TED에 공개된 또하나의 Sixth sense 프로젝트 강연 비디오... 이번에는 교수가 나와서 직접 설명한다.