Like a mystic Dr Strange-like superhero, whose simple hand gestures make the world around them spring to life, the Zero Ring, makes things happen.
It may sound like it’s part of an infinity gauntlet, but it actually contains motion sensors that track how your hand moves around. Linked by Bluetooth to your mobile phone, this gesture signal is compared with template gestures saved in the phone. If they match then the phone sends a message to make something useful happen. One gesture might cause your smart-enabled light bulb to turn on. Another might turn the central heating up a notch. Critically, the ring has an on-off switch to prevent it working when you don’t want it to! After all you don’t want all your hand movements to be tracked and so control things all the time.
This wearable tech is one of the first consumer goods to be operated by recognising simple gestures as orders. As the technology progresses, more complex hand patterns will be possible opening up whole new ways to interact.
Just don’t have it switched on when you are having a scratch … or your personal universe might just collapse by mistake.
QMUL’s Sophie Skach is interested in both textiles and computers. In a recent summer project she explored how much your chair could tell about you.
When sitting talking, we change postures a lot, and maybe that gives vital clues to what is actually happening when groups of people are talking. By embedding sensors in the fabric of a chair, perhaps it can tell something about the social roles involved – who is dominant in the conversation, who is passive, who is speaking, who is listening, who isn’t. Intelligence starts with being able to sense the world, so sytems like this may one day help make computers more intelligent, making it easier for them to work out, unobtrusively, the nuances of what we humans are up to.
One day soon your chair may be keeping an eye on you!
It’s fun to add emoticons to messages, and they help ensure people understand our feelings. They are helping some people understand feelings face-to-face too, with a bit of help from an Artificial Intelligence.
Reading faces
We take it for granted that we can look at someone’s face and tell whether they are happy or sad, angry or surprised. Autistic children, however, often struggle to understand people’s expressions. When anxious we also all tend to avoid eye contact. Some autistic children do that all the time. They are then even less likely to see the clues in people’s faces, and so start to understand emotions. This can make it harder to make friends.
From robots to glasses
Many hi-tech ways have been tried to help autistic children learn about emotions. One, for example, involves letting them play with robot ‘friends’ as some find the cartoon-like expressions on a robot face more comfortable and easier to follow. A different approach is based on wearable technology. Researchers at Stanford University have created a program for autistic children that works out a person’s expression and displays an emoticon of it in a pair of smart glasses.
An AI reading faces for you
A camera in the glasses records what the wearer sees and the Artificial Intelligence (AI) program detects any faces. This kind of technology is also used in smartphones to detect faces in your photo collection. It uses ‘machine learning’: the program learns what a face is by being shown lots of images, some with and some without faces. The program uses all that data to work out the patterns in an image that mean there is a face. It then uses that pattern to spot new faces.
In a similar way it can be trained on faces with different expressions. A training set of faces are used that are labelled with the emotion in that image. This allows the program to spot what pattern in a face makes a happy face, what makes a sad face, and so on. Having recognised an expression, the glasses finally act as a screen and show an emoticon, such as a smiley, corresponding to that expression. Superimposing digital images on the real world like this is called augmented reality. It makes looking at faces like a game and means that the child can use the emoticon to understand what the person in front of them is feeling. It also means they can start to learn for themselves – almost like the AI! The AI is labelling the faces for them, just as people had done for it. With the glasses, autistic children can be sure what each face is actually saying rather than having to guess. Eventually they might then form their own rules and so do it on their own.
Making a difference
The Stanford system was trialled with autistic children in their own homes. They used the system for several months and their parents found it made a clear difference. By the end many of the children were engaging much more with their family including making a lot more eye contact.
Emoticons are making a real difference to their lives.
Ariana Grande, has added something new to her sell out stadium tours. She is controlling her vocals using gloves. Yep, gloves! To add reverb to her voice, Ariana pinches her thumb and forefinger. She changes background sounds by a sweep of the hand.
Imogen Heap, a Grammy award winning UK recording artist with a passion for technology, is behind the gesture control gloves that Florida born pop diva Ariana is wowing audiences across the world with.
Using technology to augment and change vocals is not new, sound engineers with banks of buttons and sliders have manipulated and improved performances for years, but now the artist can do it for themselves, using wearable tech with with bluetooth to control their sounds live.
So puff out your chest, robin and hear the humans notch up the sound gymnastics, we are not just limited to our vocal cords. Have a go at making wearables that control sound yourself. Maybe try Sonic Pi with a BBC micro:bit and search for the BBC’s ‘Strictly micro:bit live lesson’ for more on making your own wearable tech.
Google’s Fitbit is a smart wristwatch which doesn’t just tell you the time but can also monitor your movements and your heart beat. A particular time of day when your heart beat slows down and you move much less is at night when you’re fast asleep in bed.
Not everyone sleeps well though. Some people struggle to get to sleep and then wake up often during the night and so they feel tired during the day. The FitBit’s “Sleep Profiles” is an AI-supported sleep tracking tool (available to Premium subscribers) that may be able to help them. If the sleeper regularly wears their watch in bed it can monitor their sleep and build up a picture of how long it takes them to fall asleep, how often they wake up and offer some suggestions on how to get a better night’s rest.
So far Google has analysed 22 billion hours of sleep data from Fitbit users (who all agree to share their information so that they and everyone else can benefit from that shared knowledge). They used unsupervised machine learning to find out more about the data. This method gives an artificial intelligence lots of information but doesn’t tell it what to do with it. Instead they asked the AI to cluster groups of data together for the scientists to analyse and interpret. The result was six clusters of data showing the most common different ways that people sleep.
To make it easy for users to understand what the data meant, and how closely their own sleep pattern matched one of the clusters, Fitbit named each cluster after an animal. They took a bit of care over selecting animals to use as they wanted people to have more positive associations (no one wants to be called a sloth for example!) and came up with bear 🐻 tortoise 🐢dolphin 🐬giraffe 🦒parrot 🦜and hedgehog 🦔. People’s ‘sleep animals’ don’t stay the same though (just like our sleep) and you might be a dolphin one month and a tortoise the next. Tortoise-sleepers spend longer in bed but also take longer to fall asleep, and dolphin-sleepers sleep very lightly and tend to spend more time awake in bed.
Elena Perez, one of the product managers for Fitbit, said that parents of little children had told her that they’d seen the icon of the sleeping animal appear on their parents’ watch and knew that it was time to go to bed. Sweet dreams…
Did you know?
Dolphins and many birds use ‘unihemispheric sleep’ which means that one half of their brain (like humans their brains are also divided into two hemispheres) falls asleep first and the other stays awake. Then the hemispheres swap over!
Beetles are one of the most prolific species on the planet. As the famous geneticist J.B.S. Haldane is supposed to have said: God has an inordinate fondness for beetles. One of the reasons they are so successful is that, unlike us, their skeleton is outside their body, not inside! This kind of skeleton is called an exoskeleton. Humans are now trying to get in on the act. In the computer science version exoskeletons are robots that you wear.
Animal shells
All sorts of animals have evolved all sorts of different exoskeletons. We call the big ones shells. Many insects, like beetles, have exoskeletons. So do crabs, scorpions, snails and clams. Tortoises are particularly interesting as they have both an internal skeleton, like us, and a shell too.
Animals use exoskeletons for lots of reasons. Most obviously it protects them from predators. It can also help stop them drying out in the sun, and stop them getting wet in the rain. They are used by some animals for sensing the world, and help animals like locusts to jump. Some tortoises and armadillos use them for digging and other animals use them to feed. It’s not surprising, with so many uses that there are a lot of them about.
Human shells
Generally, exoskeletons seem like a pretty good idea! So it’s not surprising that we humans want them too. A suit of armour is actually just a simple version of an exoskeleton designed to protect a knight from ‘predators’. It’s not much different to a tortoise protected inside its shell. The difference to the ones humans make now is our modern exoskeletons are powered and controlled by computers. They really are a robot you wear. They react to your movements.
As with animals’ shells, powered exoskeletons help humans do all sorts of things, not just act as armour. By being powered they give us extra strength, allowing us to lift weights far heavier than we could otherwise, and can turn our small movements in to larger ones. That means they can, for example, help people who have problems moving about to walk (see ‘The Wrong Trousers’) or help nurses lift patients in and out of bed. They are used by surgeons to do operations when they are in a different place to the patient, removing the shakiness of their hands, and by rescue workers working in dangerous situations. There are even ones designed to help astronauts exercise in space. They make movement harder rather than easier to force them to exercise despite the lower gravity.
All in all, copying beetles, but with our own computing twist, seems like a pretty good idea.
Computer Scientists and digital artists are behind the fabulous special effects and computer generated imagery we see in today’s movies, but for a bit of fun, in this series, we look at how movie plots could change if they involved Computer Scientists. Here we look at an alternative version of the Christmas film, Elf, starring Will Ferrell.
***Spoiler Alert***
Christmas Eve, and a baby crawls into Santa’s pack as he delivers presents at an orphenage. The baby is wearing only a nappy, but this being the 21st century the babys’s reusable Buddy nappy is an Intelligent nappy. It is part of the Internet of Things and is chipped, including sensors and a messaging system that allow it to report to the laundry system when the nappy needs changing (and when it doesn’t) as well as performing remote health monitoring of the baby. It is the height of optimised baby care. When the baby is reported missing the New York Police work with the nappy company, accessing their logs, and eventually work out which nappy the baby was wearing and track its movements…to the roof of the orphenage!
The baby by this point has been found by Santa in his sack at the North Pole, and named Buddy by the Elves after the label on his nappy. The Elves change Buddy’s nappy, and as their laundry uses the same high tech system for their own clothes, their laundry logs the presence of the nappy, allowing the Police to determine its location.
Santa intends to officially adopt Buddy, but things are moving rapidly now. The New York Police believe they have discovered the secret base of an international child smuggling ring. They have determined the location of the criminal hideout as somewhere near the North Pole and put together an armed task force. It is Boxing Day. As Santa gets in touch with the orphanage to explain the situation, and arrange an adoption, armed police already surround the North Pole and are moving in.
The New York Police Commissioner, wanting the good publicity she sees arising from capturing a child smuggling ring, orders the operation to be live streamed to the world. The precise location of the criminal hideout, so operation, is not revealed to the public, which is fortunate given what follows. As the police move in the cameras are switched on and people the world over, are glued to their screens watching the operation unfold. As the police break in to the workshops, toys go flying and Elves scatter, running for their lives, but as Santa appears and calmly allows himself to be handcuffed, it starts to dawn on the police where they are and who they have arrested. The live stream is cut abruptly, and as the full story emerges, and apologies made on all sides. Santa is proved to be real to a world that was becoming sceptical. A side effect is there is a massive boost in Christmas Spirit across the world that keeps Santa’s sleigh powered without the need for engines for many decades to come. Buddy is officially adopted and grows up believing he is an Elf until one fateful year when …
In reality
The idea of the Internet of Things is that objects, not just people, have a presence on the Internet and can communicate with other objects and systems. The idea provides the backbone of the idea of smart homes, where fridges can detect they are out of milk and order more, carpets detect dirt and summon a robot hoover, and the boiler detects when the occupants are nearing home and heats the house just in time.
Wearable computing, where clothes have embedded sensors and computers is also already a reality, though mainly in the form of watches, jewellery and the like. Clothes in shops do include electronic tags that help with stock control, and increasingly electronic-textiles based on metallic fibres and semi-conducting inks, are being used to create clothes with computers and electronics embedded in them.
Making e-textiles durable to be washed is still a challenge. Smart reusable nappies may be a while in coming.
For Star Wars Day (May 4th), here is a Star Wars inspired research from the archive…
Virtual reality can give users an experience that was previously only available a long time ago in a galaxy far, far away. Josh Holtrop, a graduate of Calvin College in the USA, constructed a Jedi training environment inspired by the scene from Star Wars in which Luke Skywalker goes up against a hovering droid that shoots laser beams at him. Fortunately, you don’t have to be blindfolded in the virtual reality version, like Luke was in the movie. All you need to wear over your eyes is a pair of virtual reality goggles with screens inside.
When you’re wearing the goggles, it’s as though you’re encased in a cylinder with rough metal walls. A bumpy metallic sphere floats in front of the glowing blade of your lightsaber – which in the real world is a toy version with a blue light and whooshy sound effects, though you see the realistic virtual version. The sphere in your goggles spins around, shooting yellow pellets of light toward you as it does. It’s up to you to bring your weapon around and deflect each menacing pulse away before it hits you. If you do, you get a point. If you don’t, your vision fills with yellow and you lose one of your ten lives.
Tracking movement with magnetism
It takes more than just some fancy goggles to make the Jedi trainer work, though. A computer tracks your movement in order to translate your position into the game. How does it know where you are? In their system, because the whole time you’re playing the game, you’re also wandering through a magnetic field. The field comes from a small box on the ceiling above you and stretches for about a metre and a half in all directions. Sixty times every second, sensors attached to the headset and lightsaber check their position in the magnetic field and send that information to the computer. As you move your head and your sabre the sensors relay their position, and the view in your goggles changes. What’s more, each of your eyes receives a slightly different view, just like in real life, creating the feeling of a 3D environment.
Once the sensors have gathered all the information, it’s up to the software to create and animate the virtual 3D world – from the big cylinder you’re standing in to the tiny spheres the droid shoots at you. It controls the behaviour of the droid, too, making it move semi-randomly and become a tougher opponent as you go through the levels. Most users seem to get the hang of it pretty quickly. “Most of them take about two minutes to get used to the environment. Once they start using it, they get better at the game. Everybody’s bad at it the first sixty seconds,” Josh says. “My mother actually has the highest score for a beginner.”
The atom smasher
Much as every Jedi apprentice needs to find a way to train, there are uses for Josh’s system beyond gaming too. Another student, Jess Vriesma, wrote a program for the system that he calls the “atom smasher”. Instead of a helmet and lightsaber, each sensor represents a virtual atom. If the user guides the two atoms together, a bond forms between them. Two new atoms then appear, which the user can then add to the existing structure. By doing this over and over, you can build virtual molecules. The ultimate aim of the researchers at Calvin College was to build a system that lets you ‘zoom in’ to the molecule to the point where you could actually walk round inside it.
The team also bought themselves a shiny new magnetic field generator, that lets them generate a field that’s almost nine metres across. That’s big enough for two scientists to walk round the same molecule together. Or, of course, two budding Jedi to spar against one another.
Wearable computing is now increasingly common whether wearing smart watches or clothes that light up. The pioneer of the latter was Japanese artist, Tanaka Atsuko, with her 1950s art work, Electric Dress. It was anything but light though, weighing 50-60kg, clothing her head to foot in a mixture of fluorescent and normal light bulbs.
She was a member of the influential Gutai (meaning concrete as opposed to abstract) Art Association and Zero Society of Japanese artists who pioneered highly experimental performance and conceptual art, that often included the artist’s actual body. The Electric Dress was an example of this, and she experimented with combining art and electronics in other work too.
Atsuko had studied dress-making as well as art, and did dress making as a hobby, so fashion was perhaps a likely way for her to express her artistic ideas, but Electric Dress was much more than just fashion as a medium for art. She had the idea of the dress when surrounded by the fluorescent lights in Osaka city centre. She set about designing and making the dress and ultimately walked around the gallery wearing it when it was exhibited at the 2nd Gutai Art Exhibition in Tokyo. Once on it flashed the lights randomly, bathing her in multicoloured light. Wearing it was potentially dangerous. It was incredibly hot and the light was dazzling. There was also a risk of electrocution if anything went wrong! She is quoted as saying after wearing it: “I had the fleeting thought: Is this how a death-row inmate would feel?”
It wasn’t the first time, electric lights had been worn, since as early as 1884 you could hire women, wearing lights on their heads powered by batteries hidden in their clothes, to light up a cocktail party, for example. However, Tanaka Atsuko’s was certainly the most extreme and influential version of a light dress, and shows how art and artists can inspire new ideas in technology. Up to then, what constituted wearable computing was more about watch like gadgets than adding electronics or computing to clothes.
Now, of course, with LEDs, and conductive thread that can be sewn into clothes and special micro-controllers, an electric dress is both much easier to make, and with programming skill you can program the lights in all sorts of creative ways. One example is a dress created for a BBC educational special of Strictly Come Dancing promoting the BBC micro:bit and showing what it was capable of with creativity. Worn by professional dancer, Karen Hauer, in a special dance to show it off, the micro:bit’s accelerometer was used to control the way the LEDs covering the dress in place of sequins, lit up in patterns. The faster she spun while dancing the more furious the patterns of the lights flashing.
Now you can easily buy kits to create your own computer-controlled clothes with online guides to get you started, so if interested in fashion and computer science why not start experimenting. Unlike Tanaka Atsuko you won’t have to put your life at risk for your art and wearable computing, overlapping with soft robotics is now a major research area, so it could be the start of a great research career.
What could a blind or partially-sighted person get from a visit to an art gallery? Quite a lot if the art gallery puts their mind to it. Even more if they make use of technology. So much so, we may all want the enhanced experience.
The best art galleries provide special tours for blind and partially-sighted people. One kind involves a guide or curator explaining paintings and other works of art in depth. It is not exactly like a normal guided tour that might focus on the history or importance of a painting. The best will give both an overview of the history and importance whilst also giving a detailed description of the whole picture as well as the detail, emphasising how each part was painted. They might, for example, describe the brush strokes and technique as well as what is depicted. They help the viewer create a really detailed mental model of the painting.
One visually-impaired guide who now gives such tours at galleries such as Tate Britain, Lisa Squirrel, has argued that these tours give a much deeper and richer understanding of the art than a normal tour and certainly more than someone just looking at the pictures and reading the text as they wander around. Lisa studied Art History at university and before visiting a gallery herself reads lots and lots about the works and artists she will visit. She found that guided tours by sighted experts using guided hand movements in front of a painting helped her build really good internal models of the works in her mind. Combined with her extensive knowledge from reading, she wasn’t building just a picture of the image depicted but of the way it was painted too. She gained a deep understanding of the works she explored including what was special about them.
The other kind of tour art galleries provide is a touching tour. It involves blind and partially-sighted visitors being allowed to touch selected works of art as part of a guided tour where a curator also explains the art. Blind art lover, Georgina Kleege, has suggested that touch tours give a much richer experience than a normal tour, and should also be put on for all for this reason. It is again about more than just feeling the shape and so “working out its form that”seeing” what a sighted person would take in at a glance. It is about gaining a whole different sensory experience of the work: its texture, for example, not a lesser version just of what it looks like.
How might technology help? Well, the company, NeuroDigital Technologies, has developed a haptic glove system for the purpose. Haptic gloves are gloves that contain vibration pads that stimulate the skin of the person in different, very fine ways so as to fool the wearer’s brain into thinking it is touching things of different shapes and textures. Their system has over a thousand different vibration patterns to simulate different feelings of touching surfaces. They also contain sensors that determine the precise position of the gloves in space as the person moves their hands around.
The team behind the idea scanned several works of art using very accurate laser scanners that build up a 3D picture of the thing being scanned. From this they created a 3D model of the work. This then allowed a person wearing to feel as though they were touching the actual sculpture feeling all the detail. More than that the team could augment the experience to give enhanced feelings in places in shadow, for example, or to emphasise different parts of the work.
A similar system could be applied to historical artifacts too: allowing people to “feel” not just see the Rosetta Stone, for example. Perhaps it could also be applied to paintings to allow a person to feel the brush strokes in a way that could just not otherwise be done. This would give an enhanced version of the experience Lisa felt was so useful of having her hand guided in front of a painting and the brush strokes and areas being described. Different colours might also be coded with different vibration patterns in this way allowing a series of different enhanced touch tours of a painting, first exploring its colours, then its brush strokes, and so on.
What about talking tours? Can technology help there? AIs can already describe pictures, but early versions at least were trained on the descriptions people have given to images on the Internet: “a black cat sitting on top of the TV looking cute”, The Mona Lisa: a young woman staring at you”. That in itself wouldn’t cut it. Neither would training the AI on the normal brief descriptions on the gallery walls next to works of art. However, art books and websites are full of detail and more recent AIs can give very detailed descriptions of art works if asked. These descriptions include what the picture looks like overall, the components, colours, brushstrokes and composition, symbolism, historical context and more (at least for famous paintings). With specific training from curators and art historians the AIs will only get better. What is still missing for a blind person though from the kind of experience Lisa has when experiencing painting with a guide, is the link to the actual picture in space – having the guide move her hand in front of the painting as the parts are described. However, all that is needed to fill that gap is to combine a chat-based AI with a haptic glove system (and provide a way to link descriptions to spatial locations on the image). Then, the descriptions can be linked to positions of a hand moving in space in front of a virtual version of the picture. Combine that with the kind of system already invented to help blind people navigate, where vibrations on a walking stick indicate directions and times to turn, and the gloves can then not only give haptic sensations of the picture in front of the picture or sculpture, but also guide the person’s movement over it.
Whether you have such an experience in a gallery, in front of the work of art, or in your own front room, blind and partially sighted people could soon be getting much better experiences of art than sighted people. At which point, as Georgina Kleege, suggested for normal touch tours, everyone else will likely want the full “blind” experience too.
cs4fn 