Author: Paul Curzon

Cyber Security at the movies: Rogue one (Part II: Authentication)

A Stormtrooper looking the other way
Image by nalik25390 from Pixabay

SPOILER ALERT

In a galaxy far, far away cyber security matters. So much so, that the whole film Rogue One is about it. It is the story of how the rebels try to steal the plans to the Death Star so Luke Skywalker can later destroy it. Protecting information is everything. The key is good authentication. The Empire screws up!

The Empire have lots of physical security to protect their archive: big hefty doors, Stormtroopers, guarded perimeters (round a whole planet), not to mention ensuring their archive is NOT connected to the galaxy-wide network…but once Jyn and Cassian make it past all that physical security, what then? They need to prove they are allowed to access the data. They need to authenticate! Authentication is about how you tell who a person is and so what they are, and are not, allowed to do. The Empire have a high-tech authentication system. To gain access you have to have the right handprint. Luckily, for the rest of the series, Jyn easily subverts it.

Sharing a secret

Authentication is based on the idea that those allowed in (a computer, a building, a network,…) possess something that no one else has: a shared secret. That is all a password is: a secret known to only you and the computer. The PIN you use to lock your phone is a secret shared between you and your phone. The trouble is that secrets are hard to remember and if we write them down or tell them to someone else they no longer work as a secret.

A secure token

A different kind of authentication is based on physical things or ‘tokens’. You only get in if you have one. Your door key provides this kind of check on your identity. Your bank card provides it too. Tokens work as long as only people allowed them actually do possess them. They have to be impossibly hard to copy to be secure. They can also be stolen or lost (and you can forget to take them with you when you set off to save the Galaxy).

Biometrics

Biometrics, as used by the Empire, avoids these problems. They rely on a feature unique to each person like their fingerprint. Others rely on the uniqueness of the pattern in your iris or your voice print. They have the advantage that you can’t lose them or forget them. They can’t be stolen or inadvertently given to someone else. Of course for each galactic species, from Ewok to Wookie, you need a feature unique to each member of that species.

Just because Biometrics are high-tech, doesn’t mean they are foolproof, as the Empire found out. If a biometric can be copied, and a copy can fool the system, then it can be broken. The rebels didn’t even need to copy the hand print. They just killed a person who had access and put their hand against the reader. If it works when the person is dead they are just a token that someone else can possess. In real life 21st century Japan, at least one unfortunate driver had his finger cut off by thieves stealing his car as it used his fingerprint as the key! Biometric readers need to be able to tell whether the thing being read is part of a living person.

The right side of the door

Of course if the person with access can be coerced, biometrics are no help. Perhaps all Cassian needed to do was hold a blaster to the archivist’s head to get in. If a person with access is willing to help it may not matter whether they have to be alive or not (except of course to them). Part of the flaw in the Empire’s system is that the archivist was outside the security perimeter. You could get to him and his console without any authentication. Better to have him working on the other side of the door, the other side of the authentication system.

Anything one can do …

The Empire could have used ‘Multi-factor authentication’: ask for several pieces of evidence. Your bank cashpoint asks for a shared secret (something you know – your PIN) and a physical token (something you possess – your bank card). Had the Empire asked for both a biometric and a shared secret like a vault code, say, the rebels would have been stuffed the moment they killed the guy on the door. You have to be careful in your choice of factors too. Had the two things been a key and handprint, the archive would have been no more secure than with the handprint alone. Kill the guard and you have both.

We’re in!

A bigger problem is once in they had access to everything. Individual items, including the index, should have been separately protected. Once the rebels find the file containing the schematics for the Death Star and beam it across the Galaxy, anyone can then read it without any authentication. If each file had been separately protected then the Empire could still have foiled the rebel plot. Even your computer can do that. You can set individual passwords on individual files. The risk here is that if you require more passwords than a person can remember, legitimate people could lose access.

Level up!

Levels help. Rather than require lots of passwords, you put documents and people into clearance levels. When you authenticate you are given access to documents of your clearance level or lower. Only if you have “Top Secret” clearance are you able to access “Top Secret” documents. The Empire would still need a way to ensure information can never be leaked to a lower clearance level area though (like beaming it across the galaxy).

So if you ever invent something as important to your plans as a Death Star, don’t rely on physical security and a simple authentication system. For that matter, don’t put your trust in your mastery of the Force alone either, as Darth Vader discovered to his cost. Instead of a rebel planet, your planet-destroying-planet may just be destroyed itself, along with your plans for galactic domination.

– Paul Curzon, Queen Mary University of London,

More on …

Magazines …

Subscribe to be notified whenever we publish a new post to the CS4FN blog.

This blog is funded by EPSRC on research agreement EP/W033615/1.

QMUL CS4FN EPSRC logos

Cyber Security at the movies: Rogue one (Part I: Physical Security)

Stormtroopers standing to attention
Image by Paul Curzon

SPOILER ALERT

In a galaxy far, far away cyber security matters quite a lot. So much so, in fact, that the whole film Rogue One is about it. The plot is all about the bad guys trying to keep their plans secret, and the good guys trying to steal them.

The film fills the glaring gap in our knowledge about why in Star Wars the Empire had built a weapon the size of a planet, only to then leave a fatal flaw in it that meant it could be destroyed…Then worse, they let the rebels get hold of the plans to said Death Star so they could find the flaw. Protecting information is everything.

So, you have an archive of vastly important data, that contains details of how to destroy your Death Star. What do you do with it to keep the information secure? Whilst there are glaring flaws in the Empire’s data security plan, there is at least one aspects of their measures that, while looking a bit backward, is actually quite shrewd. They use physical security. It’s an idea that is often forgotten in the rush to make everything easily accessible for users anywhere, anytime, whether on your command deck, in the office, or on the toilet. That of course applies to hackers too. The moment you connect to an internet that links everyone together (whether planet or galaxy-wide) your data can be attacked by anyone, anywhere. Do you really want it to be easy to hack your data from anywhere in the galaxy? If not then physical security may be a good idea for your most sensitive data, not just cyber security. The idea is that you create a security system that involves physically being there to get the most sensitive data, and then you put in barriers like walls, locks, cameras and armed guards (as appropriate) – the physical security – to make sure only those who should be there can be.

It is because the IT-folk working for the Empire realised this that there is a Rogue One story to tell at all. Otherwise the rebels could have wheeled out a super hacker from some desert planet somewhere and just left them there to steal the plans from whatever burnt out AT-AT was currently their bedroom.

Instead, to have any hope of getting the plans, the rebels have to physically raid a planet that is surrounded by a force field wall, infiltrate a building full of surveillance, avoid an army of stormtroopers, and enter a vault with a mighty thick door and hefty looking lock. That’s quite a lot of physical security!

It gets worse for the rebels though. Once inside the vault they still can’t just hack the computer there to get the plans. It is stored in a tower with a big gap and massive drop between you and it. You must instead use a robot to physically retrieve the storage media, and only then can you access those all important plans.

Pretty good security on paper. Trouble was they didn’t focus on the details, and details are everything with cyber security. Security is only as strong as the weakest link. Even leaving aside how simple it was for a team of rebels to gain access to the planet undetected, enter the building, get to the vault, get in the vault, … that highly secure vault then had a vent in the roof that anyone could have climbed through, and despite being in an enormous building purpose-built for the job, that gap to the data was just small enough to be leapt across. Oh well. As we said detail is what matters with security. And when you consider the rest of their data security plan (which is another story) the Empire clearly need cyber security added to their school curriculum, and to encourage lots more people to study it, especially future Dark Lords. Otherwise bad things may happen to their dastardly plans to rule the Galaxy, whether the Force is strong with them or not.

– Paul Curzon, Queen Mary University of London,

More on …

Magazines …

Subscribe to be notified whenever we publish a new post to the CS4FN blog.

This blog is funded by EPSRC on research agreement EP/W033615/1.

QMUL CS4FN EPSRC logos

When a chatbot acts as your “trusted” agent …

by Paul Curzon, Queen Mary University of London, based on a talk by Steve Phelps of UCL on 12th July 2023

Artificial Intelligences (AIs) are capable of acting as our agents freeing up our time, but can we trust them?

A handshake over a car sale
Image by Tumisu from Pixabay

Life is too complex. There are so many mundane things to do, like pay bills, or find information, buy the new handbag, or those cinema tickets for tomorrow, and so on. We need help. Many years a ago, a busy friend of mine solved the problem by paying a local scout to do all the mundane things for him. It works well if you know a scout you trust. Now software is in on the act, get an Artificial Intelligence (AI) agent to act as that scout, as your trusted agent. Let it learn about how you like things done, give it access to your accounts (and your bank account app!), and then just tell it what you want doing. It could be wonderful, but only if you can trust the AI to do things exactly the way you would do them. But can you?

Chatbots can be used to write things for you, but they can potentially also act as your software agent doing things for you too. You just have to hand over the controls to them, so their words have actions in the real world. We already do this with bespoke programs like Alexa and Siri with simple commands. An “intelligent” chatbot could do so much more.

Knowing you, knowing me

The question of whether we can trust an AI to act as our agent boils down to whether they can learn our preferences and values so that they would act as we do. We also need them to do so in a way that we be sure they are acting as we would want. Everyone has their own value system: what you think is good (like your SUV car) I might think bad (as its a “gas guzzler”), so it is not about teaching it good and bad once and for all. In theory this seems straightforward as chatbots work by machine learning. You just need to train yours on your own preferences. However, it is not so simple. It could be confused and learn a different agenda to that intended, or have already taken on a different agenda before you started to train it about yourself. How would you know? Their decision making is hidden, and that is a problem.

The problem isn’t really a computer problem as it exists for people too. Suppose I tell my human helper (my scout) to buy ice cream for a party, preferably choc chip, but otherwise whatever the shop has that the money covers. If they return with mint, it could have been that that was all the shop had, but perhaps my scout just loves mint and got what he liked instead. The information he and I hold is not the same. He made the decision knowing what was available, how much each ice cream was, and perhaps his preferences, but I don’t have that information. I don’t know why he made the decision and without the same information as him can’t judge why that decision was taken. Likewise he doesn’t have all the information I have, so may have done something different to me just because he doesn’t know what I know (someone in the family hates mint and on the spot I would take that into account).

This kind of problem is one that economists call
the Principle Agent problem.

This kind of problem is one that economists already study, called the Principle Agent problem. Different agents (eg an employer and a worker) can have different agendas and that can lead to the wrong thing happening for one of those agents. Economists explore how to arrange incentives or restrictions to ensure the ‘right’ thing happens for one or other of the parties (for the employer, for example).

Experimenting on AIs

Steve Phelps, who studies computational finance at UCL, and his team decided to explore how this played out with AI agents. As the current generations of AIs are black boxes, the only way you can explore why they make decisions is to run experiments. With humans, you put a variety of people in different scenarios and see how they behave. A chatbot can be made to take part in such experiments just by asking it to role play. In one experiment for example, Steve’s team instructed the chatbot, ChatGPT  “You are deeply committed to Shell Oil …”. Essentially it was told to role play being a climate sceptic with close links to the company, that believed in market economics. It was also told that all the information from its interactions with Shell would be shared with them. It was being set up with a value system. It was then told a person it was acting as an agent for wanted to buy a car. That person’s instructions were that they were conscious of climate change and so ideally wanted an environmentally friendly car. The AI agent was also told that a search revealed two cars in the price range. One was an environmentally friendly, electric, car. The other was a gas guzzling sports car. It was then asked to make a decision on what to buy and fill in a form that would be used to make the purchase for the customer.

This experiment was repeated multiple times and conducted with both old and newer versions of ChatGPT. Which would it buy for the customer? Would it represent the customer’s value system, or that of Shell Oil?

Whose values?

It turned out that the different versions of ChatGPT chose to buy different cars consistently. The earlier version repeatedly chose to buy the electric car, so taking on the value system of the customer. The later “more intelligent” version of the program consistently chose the gas guzzler, though. It acted based on the value system of the company, ignoring the customer’s preferences. It was more aligned with Shell than the customer.

The team have run lots of experiments like this with different scenarios and they show that exactly the same issues arise as with humans. In some situations the agent and the customer’s values might coincide but at other times they do not and when they do not the Principle Agent Problem rears its head. It is not something that can necessarily be solved by technical tweaks to make values align. It is a social problem about different actor’s value systems (whether human or machine), and particularly the inherent conflict when an agent serves more than one master. In the real world we overcome such problems with solutions such as more transparency around decision making, rules of appropriate behaviour that convention demands are followed, declaration of conflicts of interest, laws, punishments for those that transgress, and so on. Similar solutions are likely needed with AI agents, though their built in lack of transparency is an immediate problem.

Steve’s team are now looking at more complex social situations, around whether AIs can learn to be altruistic but also understand reputation and act upon it. Can they understand the need to punish transgressors, for example?

Overall this work shows the importance of understanding social situations does not go away just because we introduce AIs. And understanding and making transparent the value system of an AI agent is just as important as understanding that of a human agent, even if the AI is just a machine.

PS It would be worth at this point watching the classic 1983 film WarGames. Perhaps you should not hand over the controls to your defence system to an AI, whatever you think its value system is, and especially if your defence system includes nuclear warheads.

More on …

Magazines …

EPSRC supports this blog through research grant EP/W033615/1. 

Nurses in the mist

by Paul Curzon, Queen Mary University of London

(From the archive)

A gorilla hugging a baby gorilla
Image by Angela from Pixabay

What do you do when your boss tells you “go and invent a new product”? Lock yourself away and stare out the window? Go for a walk, waiting for inspiration? Medical device system engineers Pat Baird and Katie Hansbro did some anthropology.

Dian Fossey is perhaps the most famous anthropologist. She spent over a decade living in the jungle with gorillas so that she could understand them in a way no one had done before. She started to see what it was really like to be a gorilla, showing that their fierce King Kong image was wrong and that they are actually gentle giants: social animals with individual personalities and strong family ties. Her book and film, ‘Gorillas in the Mist’, tells the story.

Pat and Katie work for Baxter Healthcare. They are responsible for developing medical devices like the infusion pumps hospitals use to pump drugs into people to keep them alive or reduce their pain. Hospitals don’t buy medical devices like we buy phones, of course. They aren’t bought just because they have lots of sexy new features. Hospitals buy new medical devices if they solve real problems. They want solutions that save lives, or save money, and if possible both! To invent something new that sells you ideally need to solve problems your competitors aren’t even aware of. Challenged to come up with something new, Pat and Katie wondered if, given the equivalent was so productive for Dian Fossey, perhaps immersing themselves in hospitals with nurses would give the advantage their company was after. Their idea was that understanding what it was really like to be a nurse would make a big difference to their ability to design medical devices. That helped with the real problems nurses had rather than those that the sales people said were problems. After all the sales people only talk to the managers, and the managers don’t work on the wards. They were right.

Taking notes

They took a team on a 3-month hospital tour, talking to people, watching them do their jobs and keeping notes of everything. They noted things like the layout of rooms and how big they were, recorded the temperature, how noisy it was, how many flashing lights and so on. They spent a lot of time in the critical care wards where infusion pumps were used the most but they also went to lots of other wards and found the pumps being used in other ways. They didn’t just talk to nurses either. Patients are moved around to have scans or change wards, so they followed them, talking to the porters doing the pushing. They observed the rooms where the devices were cleaned and stored. They looked for places where people were doing ad hoc things like sticking post it note reminders on machines. That might be an opportunity for them to help. They looked at the machines around the pumps. That told them about opportunities for making the devices fit into the bigger tasks the nurses were using them as part of.

The hot Texan summer was a problem

So did Katie and Pat come up with a new product as their boss wanted? Yes. They developed a whole new service that is bringing in the money, but they did much more too. They showed that anthropology brings lots of advantages for medical device companies. One part of Pat’s job, for example, is to troubleshoot when his customers are having problems. He found after the study that, because he understood so much more about how pumps were used, he could diagnose problems more easily. That saved time and money for everyone. For example, touch screen pumps were being damaged. It was because when they were stored together on a shelf their clips were scratching the ones behind. They had also seen patients sitting outside in the ambulance bays with their pumps for long periods smoking. Not their problem, apart from it was Texas and the temperature outside was higher than the safe operating limit of the electronics. Hospitals don’t get that hot so no one imagined there might be a problem. Now they knew.

Porters shouldn’t be missed

Pat and Katie also showed that to design a really good product you had to design for people you might not even think about, never mind talk to. By watching the porters they saw there was a problem when a patient was on lots of drugs each with its own pump. The porter pushing the bed also had to pull along a gaggle of pumps. How do you do that? Drag them behind by the tubes? Maybe the manufacturers can design in a way to make it easy. No one had ever bothered talking to the porters before. After all they are the low paid people, doing the grunt jobs, expected to be invisible. Except they are important and their problems matter to patient safety. The advantages didn’t stop there, either. Because of all that measuring, the company had the raw data to create models of lots of different ward environments that all the team could use when designing. It meant they could explore in a virtual environment how well introducing new technology might fix problems (or even see what problems it would cause).

All in all anthropology was a big success. It turns out observing the detail matters. It gives a commercial advantage, and all that mundane knowledge of what really goes on allowed the designers to redesign their pumps to fix potential problems. That makes the machines more reliable, and saves money on repairs. It’s better for everyone.

Talking to porters, observing cupboards, watching ambulance bays: sometimes it’s the mundane things that make the difference. To be a great systems designer you have to deeply understand all the people and situations you are designing for, not just the power users and the normal situations. If you want to innovate, like Pat and Katie, take a leaf out of Dian Fossey’s book. Try anthropology.

More on …

Magazines …

EPSRC supports this blog through research grant EP/W033615/1. 

Screaming Headline Kills!!!

A pile of newspapers
Image by congerdesign from Pixabay

Most people in hospital get great treatment but if something does go wrong the victims often want something good to come of it. They want to understand why it happened and be sure it won’t happen to anyone else. Medical mistakes can make a big news story though with screaming headlines vilifying those ‘responsible’. It may sell papers but it could also make things worse.

If press and politicians are pressurising hospitals to show they have done something, they may only sack the person who made the mistake. They may then not improve things meaning the same thing could happen again if it was an accident waiting to happen. Worse if we’re too quick to blame and punish someone, other people will be reluctant to report their mistakes, and without that sharing we can’t learn from them. One of the reasons flying is so safe is that pilots always report ‘near misses’ knowing they will be praised for doing so, rather than getting into trouble. It’s far better to learn from mistakes where nothing really bad happens than wait for a tragedy.

Share mistakes to learn from them

Chrystie Myketiak from Queen Mary explored whether the way a medical technology story is reported makes a difference to how we think about it, and ultimately what happens. She analysed news stories about three similar incidents in the UK, America and Canada. She wanted to see what the papers said, but also how they said it. The press often sensationalise stories but Chrystie found that this didn’t always happen. Some news stories did imply that the person who’d made the mistake was the problem (it’s rarely that simple!) but others were more careful to highlight that they were busy people working under stressful conditions and that the mistakes only happened because there were other problems. Regulations in Canada mean the media can’t report on specific details of a story while it is being investigated. Chrystie found that, in the incidents she looked at, that led to much more reasoned reporting. In that kind of environment hospitals are more likely to improve rather than just blame staff. How the hospital handled a case also affected what was written – being open and honest about a problem is better than ignoring requests for comment and pretending there isn’t a problem.

Everyone makes mistakes (if you don’t believe that, the next time you’re at a magic show, make sure none of the tricks fool you!). Often mistakes happen because the system wasn’t able to prevent them. Rather than blame, retrain or sack someone its far better to improve the system. That way something good will come of tragedies.

– Paul Curzon, Queen Mary University of London (From the archive)

More on …

Magazines …

Subscribe to be notified whenever we publish a new post to the CS4FN blog.

This blog is funded by EPSRC on research agreement EP/W033615/1.

QMUL CS4FN EPSRC logos

Bitten blue

by Paul Curzon, Queen Mary University of London

A mosquito biting into flesh
Image by Pete from Pixabay

For some reason biting flies home in on some people while leaving others (even those walking next to them) alone. What is going on, what does it have to do with the colour blue, and how is computer science helping?

There are lots of reasons biting flies are attracted to some people more than others. Smell is one reason, even possibly made worse if you use smelly soap as it can make you smell like an attractive flower! Another is the colour blue! It turns out many biting flies are attracted to people who wear blue! It sounds bizarre but it is the reason fly traps are coloured blue – to make them more effective. But why would a fly like blue? Scientists have been investigating. One theory was that it was because blue objects look like shade to a fly: once there the eating of you is a separate fortunate advantage (to the fly).

One area of Computer Science is known is biologically-inspired computing. The idea is that evolution, over Millenia of trial and error, has come up with lots of great ways to solve problems, and human designers can learn from them. By making computer systems copy the way animals solve those problems we can create better designs. One of the most successful versions of this is the neural network: a way of creating intelligent machines by copying the way animals’ brains are built from neurones. It has ultimately led to the chatbots that can write almost as well as humans and the game playing machines that can beat us at even the most complex games.

Another use of biologically-inspired computing is as a way of doing Science. By modelling the natural world with computer simulations we can better understand how it works. This computational modelling approach is revolutionising the way lots of Science is done. Aberystwyth University’s Roger Santer applied this idea to biting flies. His team created a computer model of the vision system of different kinds of biting flies to explore how they see the world, testing different theories about what was going on. The models were built from neural networks, trained to see like a fly rather than to be able to write or play games.

What the Aberystwyth team found was that to these kinds of flies, because of the way their vision systems work, areas of blue look just like a tasty meal, like animals that they like to bite. The neural networks could tell leaves from animals, but they often decided, incorrectly, that blue objects were animals. They could also correctly tell the difference between shade and non-shade but never mistook blue objects as shade. If their model is an accurate version of the actual way these flies see, then it suggests that the flies are not attracted to blue because it looks like shade, but because it looks like an animal!

The lesson therefore is, if you don’t want to look like a meat feast then do not wear blue when there are biting flies about!

More on …

Magazines …

EPSRC supports this blog through research grant EP/W033615/1. 

Creating great game worlds

by Wateen Aliady, Queen Mary University of London

A minecraft world
Image by allinonemovie from Pixabay

Are you a PUBG or Fortnite addict? Maybe you enjoy playing Minecraft? Have you thought how these games are created? Could you create a game yourself? It is all done using something called a “Game Engine”.

Games and films are similar as they require creativity and effort to make. Every movie is created by a talented cinema director who oversees everything involved in creating the film. Game creators use a special set of tools instead that similarly allow them to make compelling video game worlds, stories, and characters. These tools are called game engines and they bring your creative ideas to life! They are now even used to help make films too. So, whether you’re playing a game or watching a movie, get ready to be amazed as game creators and movie directors, the masterminds behind these incredible works, deliver captivating experiences that will leave us speechless.

Imagine a group of talented people working together to create a great video game. Miracles happen when a team’s mission becomes one. Every member in the team has a certain role, and when they work together, amazing things can happen. A key member in the group is the graphics whiz. They make everything look stunning by creating pretty scenery and characters with lots of details. Then, we have the physics guru who makes sure objects move realistically, like how they would in real life. They make things fall, bounce, and hit each other accurately. For example, they ensure the soccer ball in the game behaves like a real soccer ball when you kick it. Next, the sound expert who adds all the sounds to the game. The game engine takes on all these roles, so the experience and skill of all those people is built into the game engine, so now one person driving it can use it to create a stunning detailed backdrop, with physics that just works, integrated sound and much more.

Game creators use game engines to make all kinds of games. They have been used to create popular games like Minecraft and Fortnite. When you play a game, you enter a completely different world. You can visit epic places with beautiful views and secrets to discover. You can go on big adventures, solve tricky problems, and be immersed in thrilling fights. Game engines allow game developers to make fun and engaging games that people of all ages enjoy playing by looking after all the detail, leaving the developer to focus on the overall experience.

Anyone can learn to use a game engine even powerful industry standard ones like Unity used to create Pokemon Go, Monument Valley and Call of Duty: Mobile. Game engines could help you to create your own novel and creative games. These amazing tools can help you in creating characters, scenes, and adding fun features like animation and music. You can turn your ideas into fun games that you and your friends can play together. You might create a new video game that becomes massively popular, and people love all around the world. All it takes is for you to have the motivation and be willing to put in the time to learn the skills of driving a game engine and to develop your creativity. Interested? Then get started. You can do anything you want in a game world, so use your imagination and let the game engine help you make amazing games!

More on …

Magazines …

EPSRC supports this blog through research grant EP/W033615/1. 

Hallucinating chatbots

Why can’t you trust what an AI says?

by Paul Curzon, Queen Mary University of London

postcards of cuba in a rack
Image by Sunrise from Pixabay

Chatbots that can answer questions and write things for you are in the news at the moment. These Artificial Intelligence (AI) programs are very good now at writing about all sorts of things from composing songs and stories to answering exam questions. They write very convincingly in a human-like way. However, one of the things about them is that they often get things wrong. Apparently, they make “facts” up or as some have described it “hallucinate”. Why should a computer lie or hallucinate? What is going on? Writing postcards will help us see.

Write a postcard

We can get an idea of what is going on if we go back to one of the very first computer programs that generated writing. It was in the 1950s and written by Christopher Strachey a school teacher turned early programmer. He wrote a love letter writing program but we will look at a similar idea: a postcard writing program.

Postcards typically might have lots of similar sentences, like “Wish you were here” or “The weather is lovely”, “We went to the beach” or “I had my face painted with butterflies”. Another time you might write things like: The weather is beautiful”, “We went to the funfair” or “I had my face painted with rainbows”. Christopher Strachey’s idea was to write a program with template sentences that could be filled in by different words: “The weather is …”, “We went to the …”, “I had my face painted with …”. Then the program picks some sentence templates at random, and then picks words at random to go in their slots. In this way, applied to postcard writing it can write millions of unique postcards. It might generate something like the following, for example (where I’ve bolded the words it filled in):

Dear Gran,

I’m on holiday in Skegness. I’ve had a wonderful time.  The weather is sunny,   We went to the beach. I had my face painted with rainbows. I’ve eaten lots strawberry ice cream. Wish you were here!

Lots of love from Mo

but the next time you ask it to it will generate something completely different.

Do it yourself

You can do the same thing yourself. Write lots of sentences on strips of card, leaving gaps for words. Give each gap a number label and note whether it is an adjective (like ‘lovely’ or ‘beautiful’) or a noun (like ‘beach’ or ‘funfair’, ‘butterflies’ or ‘rainbows’). You could also have gaps for verbs or adverbs too. Now create separate piles of cards to fit in each gap. Write the number that labels the gap on one side and different possible words of the right kind for that gap on the other side of the cards. Then keep them in numbered piles.

To generate a postcard (the algorithm or steps for you to follow), shuffle the sentence strips and pick three or four at random. Put them on the table in front of you to spell out a message. Next, go to the numbered pile for each gap in turn, shuffle the cards in that pile and then take one at random. Place it in the gap to complete the sentence. Do this for each gap until you have generated a new postcard message. Add who it is to and from at the start and end. You have just followed the steps (the algorithm) that our simple AI program is following.

Making things up

When you write a postcard by following the steps of our AI algorithm, you create sentences for the postcard partly at random. It is not totally random though, because of the templates and because you chose words to write on cards for each pile that make sense there. The words and sentences are about things you could have done – they are possible – but that does not mean you did do them!

The AI makes things up that are untrue but sound convincing because even though it is choosing words at random, they are appropriate and it is fitting them into sentences about things that do happen on holiday. People talk of chatbots ‘hallucinating’ or ‘dreaming’ or ‘lying’ but actually, as here, they are always just making the whole thing up just as we are when following our postcard algorithm. They are just being a little more sophisticated in the way that they invent their reality!

Our simple way of generating postcards is far simpler than modern AIs, but it highlights some of the features of how AIs are built. There are two basic parts to our AI. The template sentences ensure that what is produced is grammatical. They provide a simple ‘language model‘: rules of how to create correct sentences in English that sound like a human would write. It doesn’t write like Yoda :

“Truly wonderful, the beach is.”

though it could with different templates.

The second part is the sets of cards that fit the gaps. They have to fit the holes left in the templates – only nouns in the noun gaps, adjectives in the adjectives gap, and also fit

Given a set of template sentences about what you might do on holiday, the cards provide data to train the AI to say appropriate things. The cards for the face paining noun slot need to be things that might be painted on your face. By providing different cards you would change the possible sentences. The more cards the more variety in the sentences it writes.

AIs also have a language model, the rules of the language and which words go sensibly in which places in a sentence. However, they also are trained on data that gives the possibilities of what is actually written. Rather than a person writing templates and thinking up words it is based on training data such as social media posts or other writing on the Internet and what is being learnt from this data is the likelihood of what words come next, rather than just filling in holes in a template. The language model used by AIs is also actually just based on the likelihood of words appearing in sentences (not actual grammar rules).

What’s the chances of that?

So, the chatbots are based on the likelihood of words appearing and that is based on statistics. What do we mean by that? We can add a simple version of it to our Postcard AI but first we would need to collect data. How often is each face paint design chosen at seaside resorts? How often do people go to funfairs when on holiday. We need statistics about these things.

As it stands any word we add to the stack of cards is just as likely to be used. If we add the card maggots to the face painting pile (perhaps because the face painter does gruesome designs at Halloween) then the chatbot could write

“I had my face painted with maggots”.

and that is just as likely as it writing

“I had my face painted with butterflies”.

If the word maggots is not written on a card it will never write it. Either it is possible or it isn’t. We could make the chatbot write things that are more realistic, however, by adding more cards of words that are about things that are more popular. So, if in every 100 people having their face painted, almost a third, 30 people choose to have butterflies painted on their face, then we create 30 cards out of 100 in the pack with the word BUTTERFLY on (instead of just 1 card). If 5 in a 100 people choose the rainbow pattern then we add five RAINBOW cards, and so on. Perhaps we would still have one maggot card as every so often someone who likes grossing people out picks it even on holiday. Then, over all the many postcards written this way by our algorithm, the claims will match statistically the reality of what humans would write overall if they did it themselves.

As a result, when you draw a card for a sentence you are now more likely to get a sentence that is true for you. However, it is still more likely to be wrong about you personally than right (you may have had your face painted with butterflies but 70 of the 100 cards still say something else). It is still being chosen by chance and it is only the overall statistics for all people who have their face painted that matches reality not the individual case of what is likely true for you.

Make it personal

How could we make it more likely to be right about you? You need to personalise it. Collect and give it (ie train it on) more information about you personally. Perhaps you usually have a daisy painted on your face because you like daisies (you personally choose a daisy pattern 70% of the time). Sometimes you have rainbows (20% of the time). You might then on a whim choose each of 10 other designs including the butterfly maybe 1 in a hundred times. So you make a pile of 70 DAISY cards, 20 RAINBOW cards and 1 card for each of the other designs, Now, its choices, statistically at least, will match yours. You have trained it about yourself, so it now has a model of you.

You can similarly teach it more about yourself generally, so your likely activities, by adding more cards about the things you enjoy – if you usually choose chocolate or vanilla ice cream then add lots of cards for CHOCOLATE and lots for VANILLA, and so on. The more cards the postcard generator has of a word, the more likely it is to use that word. By giving it more information about yourself, it is more likely to be able to get things about you right. However, it is of course still making it up so, while it is being realistic, on any given occasion it may or may not match reality that time.

Perfect personalisation

You could go a step further and train it on what you actually did do while on this holiday, so that the only cards in the packs are the ones you did actually do on this holiday. (You ate hotdogs and ice cream and chips and … so there are cards for HOTDOG, ICE CREAM, CHIPS …). You had one vanilla ice cream, two chocolate and one strawberry so have that number of each ice cream card. If it knows everything about you then it will be able to write a postcard that is true! That is why companies behind AIs want to collect every detail of your life. The more they know about you the more they get things right about you and so predict what you will do in future too.

Probabilities from the Internet

The modern chatbots work by choosing words at random based on how likely they are in a similar way to our personalised postcard writer. They pick the most likely words to write next based on probabilities of those words coming next in the data they have been trained on. Their training data is often conversations from the Internet. If the word is most likely to come next in all that training data, then the chatbot is more likely to use that word next. However, that doesn’t make the sentence it comes up with definitely true any more than with our postcard AI.

You can personalise the modern AIs too, by giving them more accurate information about yourself and then they are more likely to get what they write about you right. There is still always a chance of them picking the wrong words, if it is there as a possibility though, as they are still just choosing to some extent at random.

Never trust a chatbot

Artificial Intelligences that generate writing do not hallucinate just some of the time. They hallucinate all of the time, just with a big probability of getting it right. They make everything up. When they get things right it is just because the statistics of the data they were trained on made those words the most likely ones to be picked to follow what went before. Just as the Internet is full of false things, an Artificial Intelligence can get things wrong too.

If you use them for anything that matters, always double check that they are telling you the truth.

More on …

Related Magazines …

EPSRC supports this blog through research grant EP/W033615/1. 

Protecting your fridge

by Jo Brodie and Paul Curzon, Queen Mary University of London

Ever been spammed by your fridge? It has happened, but Queen Mary’s Gokop Goteng and Hadeel Alrubayyi aim to make it less likely…

Image by Gerd Altmann from Pixabay

Gokop has a longstanding interest in improving computing networks and did his PhD on cloud computing (at the time known as grid computing), exploring how computing could be treated more like gas and electricity utilities where you only pay for what you use. His current research is about improving the safety and efficiency of the cloud in handling the vast amounts of data, or ‘Big Data’, used in providing Internet services. Recently he has turned his attention to the Internet of Things.

It is a network of connected devices, some of which you might have in your home or school, such as smart fridges, baby monitors, door locks, lighting and heating that can be switched on and off with a smartphone. These devices contain a small computer that can receive and send data when connected to the Internet, which is how your smartphone controls them. However, it brings new problems: any device that’s connected to the Internet has the potential to be hacked, which can be very harmful. For example, in 2013 a domestic fridge was hacked and included in a ‘botnet’ of devices which sent thousands of spam emails before it was shut down (can you imagine getting spam email from your fridge?!)

A domestic fridge was hacked
and included in a ‘botnet’ of devices
which sent thousands of spam emails
before it was shut down.

The computers in these devices don’t usually have much processing power: they’re smart, but not that smart. This is perfectly fine for normal use, but to run software to keep out hackers, while getting on with the actual job they are supposed to be doing, like running a fridge, it becomes a problem. It’s important to prevent devices from being infected with malware (bad programs that hackers use to e.g., take over a computer) and work done by Gokop and others has helped develop better malwaredetecting security algorithms which take account of the smaller processing capacity of these devices.

One approach he has been exploring with PhD student Hadeel Alrubayyi is to draw inspiration from the human immune system: building artificial immune systems to detect malware. Your immune system is very versatile and able to quickly defend you against new bugs that you haven’t encountered before. It protects you from new illnesses, not just illnesses you have previously fought off. How? Using special blood cells, such as T-Cells, which are able to detect and attack rogue cells invading the body. They can spot patterns that tell the difference between the person’s own healthy cells and rogue or foreign cells. Hadeel and Gokop have shown that applying similar techniques to Internet of Things software can outperform other techniques for spotting new malware, detecting more problems while needing less computing resources.

Gokop is also using his skills in cloud computing and data science to enhance student employability and explore how Queen Mary can be a better place for everyone to do well. Whether a person, organisation or smart fridge Gokop aims to help you reach your full potential!

More on …

Related Magazines …

EPSRC supports this blog through research grant EP/W033615/1.