Delicious computing: gestural computing with bananas and pizzas…

A photograph of ripening yellow bananas

by Paul Curzon, Queen Mary University of London

Imagine being able to pick up an ordinary banana and use it as a phone. That’s part of the vision of ‘invoked computing’, which is being developed by Japanese researchers. A lot of the computers in our lives are camouflaged – smartphones are more like computers than phones, after all – but invoked computing would mean that computers would be everywhere and nowhere at the same time.

The idea is that in the future, computer systems could monitor an entire environment, watching your movements. Whenever you wanted to interact with a computer, you would just need to make a gesture. For example, if you picked up a banana and held one end to your ear and the other to your mouth, the computer would guess that you wanted to use the phone. It would then use a fancy speaker system to direct the sound, so you would even hear the phone call as though it were coming from the banana.

Sometimes you might find yourself needing a bit more computing power, though, right? Not to worry. You can make yourself a laptop if you just find an old pizza box. Lift the lid and the system will project the video and sound straight on to the box.

At the moment the banana phone and pizza box laptop are the only ways that you can use invoked computing in the researchers’ system, but they hope to expand it so that you can use other objects. Then, rather than having to learn how to use your computers, your computers will have to learn how you would like to use them. And when you are finished using your phone, you could eat it.


This article was originally published on CS4FN and can also be found on page 2 of CS4FN Issue 15, Does your computer understand you?, which you can download as a PDF. All of our free material can be downloaded here: https://cs4fndownloads.wordpress.com/


Related Magazine …

This blog is funded through EPSRC grant EP/W033615/1.

Your own electrical sea: sensing your movements

Your own electrical sea
by Paul Curzon, Queen Mary University of London

A silhouetted man holding up an umbrella as a lightning storm rages around him against a slate grey sky. He is holding a briefcase.
A man sheltering from an electrical storm Image by Gerd Altmann from Pixabay

You can’t see them, but there are waves of electricity flowing around you right now. Electricity leaks out of power lines, lights, computers and every other gadget nearby. Soon a computer may be able to track your movements by following the ripples you make in your own electromagnetic sea. Scientists at Microsoft Research in the US have figured out a way to sense the position of someone’s body by using it as an antenna.

Why would you want a computer to do this? So that you could control it just by moving your body. This is already possible with systems like the Xbox Kinect, but that works by tracking you with a camera, so you have to stay in front of it or it loses you. A system that uses your body as an electric antenna could follow you throughout a room, or even a whole building.

First you need an instrument that can sense the changes you make in your own electrical field as you move around. In the future, the researchers would like this to be a little gadget you could carry in your pocket, but the technology isn’t quite small enough yet. For this experiment, they used a wireless data sensor that’s about twice the size of a mobile phone. The volunteers wore it in a little backpack. All the electrical data it picked up were transmitted to a computer that would run the calculations to figure out how the user was moving.

Get moving

In their first experiment, the researchers wanted to find out whether their gadget could sense what movements their volunteers made. To do this, they had the volunteers take their sensing devices home and use them in two different rooms: the kitchen and the living room. Those two rooms are usually different from one another in interesting ways. Living rooms are usually big open spaces with only a few small appliances in them. Kitchens, though, are often small, and cram lots of big electricals in the same room. The electrical sensors would really have to work hard to make sense through the interference.

Once the experiment was ready to go, each volunteer ran through a series of twelve movements. Their exercises included waving, bending over, stepping to the right or left, and even a bit of kicking and punching. The sensor would collect the electrical readings and then send them to a laptop. What happened after that was a bit of artificial intelligence. The researchers used the first few rounds of movements to train the computer to recognise the electrical signatures of each movement. Later on, it was the computer’s job to match up the readings it got through the sensor to the gestures it already knew. That’s a technique called machine learning.

One of the surprising things that made the sensor’s job tougher was that electrical appliances change what they are doing more often than you think. Maybe a refrigerator switches its cooling on and off, or a computer starts up its hard disk. Each of these changes means a change in the electrical waves flowing through the room, and the computer had to recognise each gesture through the changing noise.

Where’d you go?

The next step for the system was to see if it could recognise which room someone was standing in when they performed the movements. There were now eight locations to keep straight – two locations in one large room and six more scattered throughout the house. It was up to the system to learn the electrical signature for each room, as well as the signature for each movement. That’s pretty tough work. But it worked well – really well. The system was able to guess the room almost 100% of the time. What’s more, they found that the location tracking even worked on the data from the first experiment, when they were only supposed to be looking at movements. But the electrical signatures of each room were built into that data too, and the system was expert enough to pick them out.

Putting it all together

In the future the researchers are hoping that their gadgets will become small enough to carry around with you wherever you are in a building. This could allow you to control computers within your house, or switch things on and off just by making certain movements. The fact that the system can sense your location might mean that you could use the same gestures to do different things. Maybe in the living room a punch would turn on the television, but in the kitchen it would start the microwave. Whatever the case, it’s a great way to use the invisible flow of energy all around us.

 


This article was originally published on CS4FN and can also be found on pages 14-15 of CS4FN Issue 15, Does your computer understand you?, which you can download as a PDF. All of our free material can be downloaded here: https://cs4fndownloads.wordpress.com/