Category: issue 19 – Touch it feel it hear it

CS4FN Advent 2023 – Day 16: candy cane or walking aid: designing for everyone, human computer interaction

Welcome to Day 16 of the CS4FN Christmas Computing Advent Calendar in which we’re posting a blog post every day in December until (and including) Christmas Day. A little later than planned (sorry!) because we’ve been screening the last of the making of the Royal Institution Christmas Lectures 2023, all about artificial intelligence. You’ll be able to watch those on 26, 27 and 28 December 2023 on BBC Four at 8pm. Thanks to everyone who came along.

In this series of posts we’re both celebrating the breadth of computing research but also the history of our own CS4FN project which has been inspiring young people about computing and supporting teachers in teaching the topic, in part by distributing free magazines to subscribing UK schools since 2005 (ask your teacher to subscribe for next year’s magazine).

Today’s advent calendar picture is of a candy cane which made me think both of walking aids and of support sticks that alert others that the person using it is blind or visually impaired.

A white candy cane with green and red stripes.

We’ve worked with several people over the years to write about their research into making life easier for people with a variety of disabilities. Issue 19 of our magazine (“Touch it, feel it, hear it!”) focused on the DePiC project (‘Design Patterns for Inclusive Collaboration’) which included work on helping visually impaired sound engineers to use recording studio equipment, and you can read one of the articles (see ‘2. The Haptic Wave’) from that magazine below.

Another of our CS4FN magazines (issue 27, called “Smart Health: decisions, decisions, decisions“) was about Bayesian mathematics and its use in computing, but one of those uses might be an app with the potential to help people with arthritis get medical support when they most need it (rather than having to wait until their next appointment) – download the magazine by clicking on its title and scroll to page 16 & 17 (p9 of the 11 page PDF). Our writing also supports the (obvious) case, that disabled people must be involved at the design and decision-making stages.

1. Design for All (and by All!)

by Paul Curzon, QMUL. This article was originally published on the CS4FN website.

Making things work for everyone

Designing for the disabled – that must be a niche market mustn’t it? Actually no. One in five people have a disability of some kind! More surprising still, the disabled have been the inspiration behind some of the biggest companies in the world. Some of the ideas out there might eventually give us all super powers.

Just because people have disabilities doesn’t mean they can’t be the designers, the innovators themselves of course. Some of the most innovative people out there were once labelled ‘disabled’. Just because you are different doesn’t mean you aren’t able!

Where do innovators get their ideas from? Often they come from people driven to support people currently disadvantaged in society. The resulting technologies then not only help those with disabilities but become the everyday objects we all rely on. A classic example is the idea of reducing the kerbs on pavements to make it possible for people in wheelchairs to get around. Turns out of course that they also help people with pushchairs, bikes, roller-blades and more. That’s not just a one-off example, some of the most famous inventors and biggest companies in the world have their roots in ‘design for all’.

Designing for more extreme situations pushes designers into thinking creatively, thinking out of the box. That’s when totally new solutions turn up. Designing for everyone is just a good idea!

2. Blind driver filches funky feely sound machine! The Haptic Wave

by Jane Waite, QMUL. This article was originally published on the CS4FN website.

The blind musician Joey Stuckey in his recent music video commandeers then drives off in a car, and yes he is blind. How can a blind person drive a car, and what has that got to do with him trying to filch a sound machine? So maybe taking the car was just a stunt, but he really did try and run off with a novel sound machine!

As well as fronting his band Joey is an audio engineer. Unlike driving a car, which is all about seeing things around you – signs, cars pedestrians – being an audio engineer seems a natural job for someone who is blind. Its about recording, mixing and editing music, speech and sound effects. What matters most is that the person has a good ear. Having the right skills could easily lead to a job in the music industry, in TV and films, or even in the games industry. It’s also an important job. Getting the sound right is critical to the experience of a film or game. You don’t want to be struggling to hear mumbling actors, or the sound effects to drown out a key piece of information in a game.

Mixing desks

Once upon a time Audio engineers used massive physical mixing desks. That was largely ok for a blind person as they could remember the positions of the controls as well as feel the buttons. As the digital age has marched on, mixing desks have been replaced by Digital Audio Workstations. They are computer programs and the trouble is that despite being about sound, they are based on vision.

When we learn about sound we are shown pictures of wavy lines: sound waves. Later, we might use an oscilloscope or music editing software, and see how, if we make a louder sound, the curves get taller on the screen: the amplitude. We get to hear the sound and see the sound wave at the same time. That’s this multimodal idea again, two ways of sensing the same thing.

Peter Francken in his studio. Image from Wikimedia Commons

Mixing desks

Once upon a time Audio engineers used massive physical mixing desks. That was largely ok for a blind person as they could remember the positions of the controls as well as feel the buttons. As the digital age has marched on, mixing desks have been replaced by Digital Audio Workstations. They are computer programs and the trouble is that despite being about sound, they are based on vision.

When we learn about sound we are shown pictures of wavy lines: sound waves. Later, we might use an oscilloscope or music editing software, and see how, if we make a louder sound, the curves get taller on the screen: the amplitude. We get to hear the sound and see the sound wave at the same time. That’s this multimodal idea again, two ways of sensing the same thing.

But hang on, sound isn’t really a load of wavy lines curling out of our mouths, and shooting away from guitar strings. Sound is energy and atoms pushing up against each other. But we think of sound as a sound wave to help us understand it. That’s what a computer scientist calls abstraction: representing things in a simpler way. Sound waves are an abstraction, a simplified representation, of sound itself.

Sound waveform image by Gordon Johnson from Pixabay

The representation of sound as sound waves, as a waveform, helps us work with sound, and with Digital Audio Workstations it is now essential for audio engineers. The engineer works with lines, colors, blinks and particularly sound waves on a screen as they listen to the sound. They can see the peaks and troughs of the waves, helping them find the quiet, loud and distinctive moments of a piece of music, at a glance, for example. That’s great as it makes the job much easier…but only if you are fully sighted. It makes things impossible for someone with a visual impairment. You can’t see the sound waves on the editing screen. Touching a screen tells you nothing. Even though it’s ultimately about sounds, doing your job has been made as hard as driving a car. This is rather sad given computers have the potential to make many kinds of work much more accessible to all.

Feel the sound

The DePIC research team, a group of people from Goldsmiths, Queen Mary University of London and Bath Universities with a mission to solve problems that involve the senses, decided to fix it. They’ve created the first ever plug-in software for professional Digital Audio Workstations that makes peak level meters completely accessible. It uses ‘sonification’: it turns those visual signals in to sound! decided to fix the problems. They brought together Computer Scientists, Design experts, and Cognitive Scientists and most importantly of all audio engineers who have visual impairments. Working together over two years in workshops sharing their experiences and ideas, developing, testing and improving prototypes to figure out how a visually impaired engineer might ‘see’ soundwaves. They created the HapticWave, a device that enables a user to feel rather than see a sound wave.

The HapticWave

The HapticWave combines novel hardware and software to provide a new interface to the traditional Digital Audio Workstation. The hardware includes a long wooden box with a plastic slider. As you move the slider right and left you move forward and backwards through the music. On the slider there is a small brass button, called a fader. Tiny embossed stripes on the side of the slider let you know where the fader is relative to the middle and ends of the slider. It moves up and down in sync with the height of the sound wave. So in a quiet moment the fader returns to the centre of the slider. When the music is loud, the fader zooms to the top of the handle. As you slide forwards and backwards through the music the little button shoots up and down, up and down tracing the waveform. You feel its volume changing. Music with heavy banging beats has your brass button zooming up and down, so mind your fingers!

So back to the title of the article! Joey trialled the HapticWave at a research workshop and rather wanted to take one home, he loved it so much he jokingly tried distracting the researchers to get one. But he didn’t get away with it – maybe his getaway car just wasn’t fast enough!

3. An audio illusion, and an audiovisual one

This one-minute video illustrates an interesting audio illusion, demonstrating that our brains are ‘always using prior information to make sense of new information coming in’.

The McGurk Effect

You can read more about the McGurk effect on page 7 of issue 5 of the CS4FN magazine, called ‘The Perception Deception‘.

Advert for our Advent calendar
Click the tree to visit our CS4FN Christmas Computing Advent Calendar

EPSRC supports this blog through research grant EP/W033615/1.

Digital lollipop: no calories, just electronics!

by Jane Waite, Queen Mary University of London

Can a computer create a taste in your mouth? Imagine scrolling down a list of flavours and then savouring your sweet choice from a digital lollipop. Not keen on that flavour, just click and choose a different one, and another and another. No calories, just the taste.

Mug of lollipops image by Vinzenz Lorenz M from Pixabay

Nimesha Ranasinghe, a researcher at the National University of Singapore is developing a Tongue Mounted Digital Taste Interface, or digital lollipop. It sends tiny electrical signals to the very tip of your tongue to stimulate your taste buds and create a virtual taste!

One of UNESCO’s 2014 ’10 best innovations in the world’, the prototype doesn’t quite look like a lollipop (yet). There are two parts to this sweet sensation, the wearable tongue interface and the control system. The bit you put in your mouth, the tongue interface, has two small silver electrodes. You touch them to the tip of your tongue to get the taste hit. The control system creates a tiny electrical current and a minuscule temperature change, creating a taste as it activates your taste buds.

The prototype lollipop can create sour, salty, bitter, sweet, minty, and spicy sensations but it’s not just a bit of food fun. What if you had to avoid sweet foods or had a limited sense of taste? Perhaps the lollipop can help people with food addictions, just like the e-cigarette has helped those trying to give up smoking?
Perhaps the lollipop can help people with food addictions

But eating is more than just a flavour on your tongue, it is a multi-modal experience, you see the red of a ripe strawberry, hear the crunch of a carrot, feel sticky salt on chippy fingers, smell the Sunday roast, anticipate that satisfied snooze afterwards. How might computers simulate all that? Does it start with a digital lollipop? We will have to wait and see, hear, taste, smell, touch and feel!

Taste over the Internet

The Singapore team are exploring how to send tastes over the Internet. They have suggested rules to send ‘taste’ messages between computers, called the Taste Over Internet Protocol, including a messaging format called TasteXML They’ve also outlined the design for a mobile phone with electrodes to deliver the flavour! Sweet or salt anyone?

More on

Related Magazine …

This article was originally published on the CS4FN website and also appears on page 14 of Issue 19 of the CS4FN magazine “Touch it, feel it, hear it” which you can download as a PDF below, along with all of our other free material here.

EPSRC supports this blog through research grant EP/W033615/1.

Patterns for Sharing – making algorithms generalisable

A white screen with 8 black arrows emanating from a smaller rectangle drawn in marker pen, representing how one idea can be used in multiple ways

Patterns for Sharing

by Paul Curzon and Jane Waite, Queen Mary University of London

A white screen with 8 black arrows emanating from a smaller rectangle drawn in marker pen, representing how one idea can be used in multiple ways
Image adapted from original by Gerd Altmann from Pixabay

Computer Scientists like to share: share in a way that means less work for all. Why make people work if you can help them avoid it with some computational thinking. Don’t make them do the same thing over and over – write a program and a computer can do it in future. Invent an algorithm and everyone can use it whenever that problem crops up for them. The same idea applies to inclusive design: making sure designs can be used by anyone, impairments or not. Why make people reinvent the same things over and over. Let others build on your experience of designing accessible things in the past. That is where the idea of Design Patterns and a team called DePIC come in.

The DePIC research team are a group of people from Queen Mary University of London, Goldsmiths and Bath Universities with a mission to solve problems that involve the senses, and they are drawing on their inner desire to share! The team unlock situations where individuals with sensory impairments are disadvantaged in their use of computers. For example, if you are blind how can you ‘see’ a graph on a screen, and so work with others on it or the data it represents. DePIC want to make things easier for those with sensory impairments, whether it be at home, leisure or at work, they want to level the playing field so that everyone can take part in our amazing technological world. Why shouldn’t a blind musician feel a sound wave and not be restricted because they can’t see it (see ‘Blind driver filches funky feely sound machine!’). DePIC, it turns out, is all about generalisation.

Generalise it!

Generalisation is the computational thinking idea that once you’ve solved a problem, with a bit of tweaking you can use the solution for lots of other similar problems too. Written some software to put names and scores in order for a high score table? Generalise the algorithm so it can sort anything in to order: names and addresses, tracks in a music collection, or whatever. Generalisation is a powerful computational thinking idea and it doesn’t just apply to algorithms, it applies to design too. That is the way the DePIC team are working.

DePIC actually stands for Design Patterns for Inclusive Collaboration. Design Patterns are a kind of generalisation: so design ideas that work can be used again and again. A Design Pattern describes the problem it solves, including the context it works in, and the way it can be solved. For example, when using computers people often need to find something of interest amongst information on a screen. It might, for example, be to find a point where a graph reaches it’s highest point, find numbers in a spreadsheet of figures that are unusually low, or locate the hour hand on a watch to tell the time. But what if you aren’t in a position to see the screen?

Anyone can work with information using whatever sense is convenient.

Make good sense

One solution to all these problems is to use sound. You can play a sound and then distort it when the cursor is at the point of interest. The design pattern for this would make clear what features of the sound would work well, its pitch say, and how it should be changed. Experiments are run to find out what works best. Inclusive design patterns make clear how different senses can be used to solve the same problem. For example, another solution is to use touch and mark the point with a distinctive feel like an increase in resistance (see the 18th century ‘Tactful Watch’!).

The idea is that designers can then use these patterns in their own designs knowing they work. The patterns help them design inclusively rather then ignoring other senses. Suddenly anyone can work on that screen of information, using whatever senses are most convenient for them at the time. And it all boils down to computer scientists wanting to share.

 

This article was originally published on the CS4FN website and a copy can also be found on page 9 in Issue 19 of the CS4FN magazine “Touch it, feel it, hear it“, which you can download free as a PDF along with all of our other free material here.