Smart speakers like Alexa might know a joke or two, but machines aren’t very good at sounding funny yet. Comedians, on the other hand, are experts at sounding both funny and exciting, even when they’ve told the same joke hundreds of times. Maybe speech technology could learn a thing or two from comedians… that is what my research is about.
To test a joke, stand-up comedians tell it to lots of different audiences and see how they react. If no-one laughs, they might change the words of the joke or the way they tell it. If we can learn how they make their adjustments, maybe technology can borrow their tricks. How much do comedians change as they write a new show? Does a comedian say the same joke the same way at every performance? The first step is to find out.
The first step is to record lots of the same live show of a comedian and find the parts that match from one show to the next. It was much faster to write a program to find the same jokes in different shows than finding them all myself. My code goes through all the words and sounds a comedian said in one live show and looks for matching chunks in their other shows. Words need to be in the same exact order to be a match: “Why did the chicken cross the road” is very different to “Why did the road cross the chicken”! The process of looking through a sequence to find a match is called “subsequence matching,” because you’re looking through one sequence (the whole set of words and sounds in a show) for a smaller sequence (the “sub” in “subsequence”). If a subsequence (little sequence) is found in lots of shows, it means the comedian says that joke the same way at every show. Subsequence matching is a brand new way to study comedy and other types of speech that are repeated, like school lessons or a favourite campfire story.
By comparing how comedians told the same jokes in lots of different shows, I found patterns in the way they told them. Although comedy can sound very improvised, a big chunk of comedians’ speech (around 40%) was exactly the same in different shows. Sounds like “ummm” and “errr” might seem like mistakes but these hesitation sounds were part of some matches, so we know that they weren’t actually mistakes. Maybe “umm”s help comedians sound like they’re making up their jokes on the spot.
Varying how long pauses are could be an important part of making speech sound lively, too. A comedian told a joke more slowly and evenly when they were recorded on their own than when they had an audience. Comedians work very hard to prepare their jokes so they are funny to lots of different people. Computers might, therefore, be able to borrow the way comedians test their jokes and change them. For example, one comedian kept only five of their original jokes in their final show! New jokes were added little by little around the old jokes, rather than being added in big chunks.
If you want to run an experiment at home, try recording yourself telling the same joke to a few different people. How much practice did you need before you could say the joke all at once? What did you change, including little sounds like “umm”? What didn’t you change? How did the person you were telling the joke to, change how you told it?
There’s lots more to learn from comedians and actors, like whether they change their voice and movement to keep different people’s attention. This research is the first to use computers to study how performers repeat and adjust what they say, but hopefully just the beginning.
Florence Nightingale, the most famous female Victorian after Queen Victoria, is known for her commitment to nursing, especially in the Crimean War. She rebelled against convention to become a nurse at a time when nursing was seen as a lowly job, not suitable for ‘ladies’. She broke convention in another less well-known, but much more significant way too. She was a mathematician – the first woman to be elected a member of the Royal Statistical Society. She also pioneered the use of pictures to present the statistical data that she collected about causes of war deaths and issues of sanitation and health. What she did was an early version of the current Big Data revolution in computer science.
Soldiers were dying in vast numbers in the field hospital she worked in, not directly from their original wounds but from the poor conditions. But how do you persuade people of something that (at least then) is so unintuitive? Even she originally got the cause of the deaths wrong, thinking they were due to poor nutrition, rather than the hospital conditions as her statistics later showed. Politicians, the people with power to take action, were incapable of understanding statistical reports full of numbers then (and probably now). She needed a way to present the information so that the facts would jump out to anyone. Only then could she turn her numbers into life-saving action. Her solution was to use pictures, often presenting her statistics as books of pie charts and circular histograms.
Whilst she didn’t invent them, Florence Nightingale certainly was responsible for demonstrating how effective they could be in promoting change, and so subsequently popularising their use. She undoubtedly saved more lives with her statistics than from her solitary rounds at night by lamplight.
She had collected data on the reason each person died but to present the data in ways that were convincing she also had to act as a human computer doing computation on the basic data. For each month based on the raw data, she computed annual rate of mortality per 1,000. Then to present it in a circular histogram, where the area represents deaths she calculated the appropriate radius for each segment, allowing the charts to then be drawn.
Big Data is now a big thing. It is the idea that if you collect lots of data about something (which computers now make easy) then you (and computers themselves) can look for patterns and so gain knowledge and, for people, ultimately wisdom from it. Florence Nightingale certainly did that. Data visualisation is now an important area of computer science. As computers allow us to collect and store ever more data, it becomes harder and harder for people to make any sense of it all – to pick out the important nuggets of information that matter. Raw numbers are little use if you can’t actually turn them into knowledge, or better still wisdom. Machine Learning programs can number crunch the data and make decisions from it, but its hard to know where the decisions came from. That often matters if we are to be persuaded. For humans the right kind of picture for the right kind of data can do just that as Florence Nightingale showed.
‘The Lady of the Lamp’: more than a nurse, but also a remarkable statistician and pioneer of a field of computer science…a Lady who made a difference by rebelling with a cause.
Machine Learning is the technology driving driverless cars, recognising faces in your photo collection and more, but how could it help machines invent new instruments? Rebecca Fiebrink of Goldsmiths, University of London is finding out.
Rebecca is helping composers and instrument builders to design new musical instruments and giving them new ways to perform. Her work has also shown that machine learning provides an alternative to programming as a way to quickly turn design ideas into prototypes that can be tested.
Suppose you want to create a new drum machine-based musical instrument that is controlled by the wave of a hand: perhaps a fist means one beat, whereas waggling your fingers brings in a different beat. To program a prototype of your idea, you would need to write code that could recognize all the different hand gestures, perhaps based on a video feed. You would then have some kind of decision code that chose the appropriate beat. The second part is not too hard, perhaps, but writing code to recognize specific gestures in video is a lot harder, needing sophisticated programming skills. Rebecca wants even young children to be able to do it!
How can machine learning help? Rebecca has developed a machine learning program with a difference. It takes sensor input – sound, video, in fact just about any kind of sensor you can imagine. It then watches, listens…senses what is happening and learns to associate what it senses with different actions it should take. With the drum machine example, you would first select one of the kinds of beats. You then make the gesture that should trigger it: a fist perhaps. You do that a few times so it can learn what a fist looks like. It learns that the patterns it is sensing are to be linked with the beat you selected. Then you select the next beat and show it the next gesture – waggling your fingers – until it has seen enough examples. You keep doing this with each different gesture you want to control the instrument. In just a few minutes you have a working machine to try. It is learning by example how the instrument you are wanting works. You can try it, and then adjust it by showing it new examples if it doesn’t quite do what you want.
It is learning by example how the instrument you are wanting works.
Rebecca realised that this approach of learning by example gives a really powerful new way to support creativity: to help designers design. In the traditional ways machine learning is used, you start with lots of examples of the things that you want it to recognize – lots of pictures of cats and dogs, perhaps. You know the difference, so label all these training pictures as cats or dogs, so it knows which to form the two patterns from. Your aim is for the machine to learn the difference between cat and dog patterns so it can decide for itself when it sees new pictures.
When designing something like a new musical instrument though, you don’t actually know exactly what you want at the start. You have a general idea but will work out the specifics as you go. You tinker with the design, trying new things and keeping the ideas that work, gradually refining your thoughts about what you want as you refine the design of the instrument. The machine learning program can even help by making mistakes – it might not have learnt exactly what you were thinking but as a result makes some really exciting sound you never thought of. You can then explore that new idea.
One of Rebecca’s motivations in wanting to design new instruments is to create accessible instruments that people with a wide range of illness and disability can play. The idea is to adapt the instrument to the kinds of movement the person can actually do. The result is a tailored instrument perfect for each person. An advantage of this approach is you can turn a whole room, say, into an instrument so that every movement does something: an instrument that it’s impossible not to play. It is a play space to explore.
Playing an instrument suddenly really is just playing.
Paul Curzon, Queen Mary University of London based on a 2016 talk by Rebecca Fiebrink
Lots of people think that Computer Science and IT are strictly for men only. That’s really bizarre given that right from the start women like Grace Hopper and Ada Lovelace played pivotal roles in the development of computers, and women are still at the leading edge today. To be a successful modern IT Pro you have to be a good team player, not to mention good at dealing with clients, which are skills women are generally good at.
‘Geeky male computer scientist’ is of course just a stereotype, like ‘helpless female in need of rescue by male hunk’, ‘scientist as mad eccentric in white coat’, or ‘evil robot wanting to take over the world’.
Where do false stereotypes come from? Films play a part in the way their (usually male, non-scientist) directors decide to represent characters.
Students on a ‘Gender in Computer Science’ course at Siena College in the US watched lots of films with Computer Science plots from as far back as 1928 to see how the way women, computers and computer scientists are portrayed has changed over time. Here are their views on some of those films.
Do you agree – when you are done read what the real IT Pros think of their jobs…and remember stereotypes are fiction, careers are what you make of them and real robots are (usually) nice!
1928: Metropolis
In a city of the future the ruling class live lavishly while the workers live poorly in the underworld. An evil scientist substitutes a robot for a female worker activist. It purposely starts a riot as an excuse so reprisals can be taken. All hell breaks loose until the male hero comes to the rescue…
X Computers: Evil
X Women as IT Pros: Helpless
X Computer Scientists: Evil
“Women are more or less portrayed as helpless … The computer scientist … as evil”
1956: Forbidden Planet
An all-male crew travel to Altair-4 to discover the fate of the colony there. They discover all that is left is scientist Dr Morbius, his beautiful daughter Altaira and a servant robot called Robby, programmed to be unable to harm humans. But what have Morbius’ machines and experiments to do with the colony’s fate?
✓ Computers: Helpful & Harmless
X Women as IT Pros: love interest
X Computer Scientists: Evil
“Altaira plays a typical woman’s role…helpless…unintelligent …Barbie-like”
1971: THX 1138
In an Orwellian future, an android controlled police state where everyone is made to take drugs that suppress emotion. LUH 3417 and THX 1138 stop taking their drugs, fall in love and try to escape…
X Computers: Evil Police
X Women as IT Pros: Few
X Computer Scientists: Heartless
“The computer scientists are depicted as boring, heartless and easily confused”
1982: Blade Runner
In the industrial wastelands of a future Los Angeles, large companies have all the power. Robotic ‘Replicants’ are almost indistinguishable from humans but have incredible strength and no emotions. Deckard (Harrison Ford) must find and destroy a group of Replicants that have developed emotions and so threaten humanity as they rebel against being ‘slaves’.
X Computers: Evil
X Women as IT Pros: None
X Computer Scientists: Caused the problem
“A woman plays the minor role of a replicant…but is portrayed as a topless dancer”
1986: Short Circuit
A comedy adventure about a robot that comes ‘alive’ after a power surge in a lightening storm. The robot, called ‘Number 5’ built for use by the US military and tries to escape its creators as it doesn’t want to ‘die’. It is helped by Stephanie Speck (Ally Sheedy) who realises, that like the animals she loves, it is sentient and helps it escape from the scientists of company Nova that built it.
✓ Computers: Nice
X Women as IT Pros: None
X Computer Scientists: Thick-headed
“The male computer scientists are often thick-headed… introverted…no life skills…There were no female computer scientists”
1995: Hackers
A group of genius teenage hackers become the target of the FBI after they unknowingly tap into a high-tech embezzling scheme that could cause a horrific environmental disaster. Dade Murphy and Kate Libby (Angelina Jolie) square off in a battle of the sexes and computer skills.
X Computers: Used illegally
✓ Women as IT Pros: Elite…but illegal
X Computer Scientists: Criminals
“Angelina plays a hard hitting, elite hacker who is better than everyone in her group except Dane who is her equal”
So it wasn’t great. Robots were killers, scientists evil. Computer scientist’s were introverted and thickheaded. Women were either sexbots or helpless love interest to be rescued by the hunky male star. 1995’s film Hackers was about as good as it got. At last a woman had expert computing skills. It’s hardly surprising some girls were led to believe computing isn’t for them with a century-long “conspiracy” aiming to convince them their role in life is to be helpless.
As our area on women in computing shows the truth is far more interesting. Women have always played a big part in the development of modern technology. So have things improved in films in the 21st century? There are more films with strong action-heroine stars now, though until very recently few films passed the Bechdel test: do two women ever talk together about anything other than a man? So can we at least find any 21st century films with realistic main character roles for women as computer experts? Here goes…
1999-2003: Matrix Trilogy
Hero Neo discovers reality isn’t what it seems. It is all a virtual reality. Trinity is there to be his romantic interest – she’s been told by the Oracle that she will fall in love with the “One” (that’s him). It’s not looking good. In film 2 Neo has to save her. Oh dear. At least she is supposed to be a super-hacker famous for cracking an uncrackable database. Oh well.
X Computers: Enslaving humanity
✓ Women as IT Pros: Elite…but illegal (there to be saved)
✓ Computer Scientists: The resistance
2009: The Girl With the Dragon Tattoo
This is the story of super-hacker Lisbeth Salander. Both emotionally and sexually abused as a child she looks after herself, and that includes teaching herself to be an expert with computers. She uses her immense skills to get what she wants. She is cool and clever and absolutely not willing to let the men treat her as a victim. Wonderful.
X Computers: used for hacking
✓ Women as IT Pros: Elite…but illegal look after themselves)
X Computer Scientists: hackers
2014: Captain America: The Winter Soldier
This film is all about a male hunk, so it’s not looking good, but then early on we see Agent Natasha Romanoff, (also known as superheroine the Black Widow). She is the brains to Captain America’s brawn and from the start she is clearly the expert with computers. While Captain America beats people up, her mission is to collect data. And she even gets her own film series…eventually!
X Computers: used for hacking
✓ Women as IT Pros: Elite…superheroes
X Computer Scientists: hackers
2015: Star Wars: Episode VII – the Force Awakens
Rey is a scavenger with engineering skills. She is very smart, and can look after herself without expecting men to save her. She’s not a hacker! Instead, she creates and mends things. She repurposes parts she finds on wrecked spaceships to sell to survive. She learnt her engineering skills tinkering in old ships and fixes the Millennium Falcon’s electro-mechanical problems. She is even the main character of the whole film!
✓ Computers: make the universe work
✓ Women as IT Pros: Elite, scavenges and fixes things
✓ Computer Scientists: at least some build and fix things
There are plenty of moronic films, made by men who can’t portray women in remotely realistic ways, but at least things are a bit better than they were last century. The women are already here in the real world. They are slowly getting there in the movies. Let’s just hope the trend speeds up, and we have more female leads who create things, like the real female computer scientists.
cs4fn 