Add a Comment   Listen to Podcast   Back to OJB's Blog Search Page

What Could Go Wrong?

Entry 2115, on 2021-03-25 at 12:18:26 (Rating 2, Science)

What could possibly go wrong? It was a thought that Bob often had, and it was usually the source of ridicule from his more cautious colleagues. But Bob thought he had a point. Surely there were things which couldn't reasonably go wrong. Things which were so benign that even if the worst outcome arose the situation would still be manageable.

Life was all about risk, he mused, and if his colleagues were too cautious to take small risks, that was to their detriment, and he would take advantage of that. He would take the risk - or alleged risk - which the others wouldn't. His job in middle management had come from this attitude. He had made it this far, but couldn't go further until he had a major success, and this was his chance.

Bob had started his current job as the manager of a team of artificial intelligence researchers ten years ago, and finally their work had reached a point where it could be put to some use. He wasn't foolhardy enough to allow the new AI full control over what it was capable of, especially using links to the real world, but he could perform a simple experiment and achieve some recognition as the first user of the new system.

Across town, at the university, astrobiology researcher, Alice, was struggling with the question her field had failed to resolve since it started. That was, did the subject of astrobiology even have anything to study? Was there life out there amongst the countless stars and planets in the universe? And if there was life, was there intelligence? And if there was intelligence, why had every effort to find it failed?

The lack of any signs of intelligent life had become increasingly puzzling over the years as more potential reasons for it not existing had been eliminated. This was especially true during the early 21st century when the existence of many planets, including many which might support life, was confirmed.

Aboard the International Space Station, Denise was watching the Earth beneath her. The view was awe inspiring, and she would often sneak down to the cupola to just watch the world go by. Being a responsible member of the team, she completed all her duties first, of course, but this was her favourite place in the world... or maybe "out of this world" would be a better description.

Back at the AI lab, Bob knew it was time to act. He had been thinking about a task he could use the new AI to perform; something significant yet harmless. He decided calculating digits of pi might be something worth doing. In antiquity this number had only been known to 1 place. It took until 1400 before it was known to 10, then another 300 years until 100 digits were known, then over 1000 were calculated in 1949 using a desk calculator. Then things sped up, with a million places by 1973, a billion by 1989, and over a trillion by 1992. The current record was 50 trillion.

By using the power of the superhuman artificial intelligence they had created Bob figured it should be possible to calculate far more digits than it had been by using the brute-force methods used up until then. He decided a decillion was a good number to aim for - it was a number near the middle of the "big numbers" page he found on Wikipedia. He could only disguise his unauthorised use of the AI for 24 hours, but he figured that would be plenty of time.

In Alice's office an email appeared on her computer. It was from another researcher working in the same area. The heading read "I know the answer!" but she had seen that claim so many times before she decided to ignore it and continue running the simulations she was working on.

There had to be a reason that intelligent life hadn't been discovered already and she was determined to be the one who solved the famous "Fermi Paradox". The so-called answer from her colleague could wait.

Meanwhile, Bob was ready to initiate the program to calculate pi. Only one other person was in the same office at the time, and he was a well-known maverick, just like Bob. Even if he detected what was happening, it was unlikely he would object. Bob started the program.

After an hour, Bob checked the feedback showing how many digits had been generated. It had already broken the old record of 50 trillion digits! The AI had clearly found a way to generate an answer faster than any previous algorithm. Bob did some quick mental arithmetic - a difficult task since he was manager, not an engineer - and figured it would only take another few hours to reach his target. Again he thought, what could go wrong?

But he was interrupted from his self-congratulatory mood by the sight of something odd happening outside. Hundreds of Amazon delivery vehicles were stopping outside the office and delivering mysterious packages. His colleague was staring nervously out of the window too, as if he might have had an idea about what was happening.

Alice was also puzzled by the sudden appearance of Amazon delivery vehicles at her university. They seemed to be delivering something to the university's computer building. It was unusual, but she figured maybe they had got a lot of funding for new equipment. Typical! No one ever wanted to fund research into exobiology. Maybe that was because they didn't believe there was anything to study, she mused.

Bob was becoming concerned by the increasing agitation of his colleague, so he walked over to him an said "Come on, Charlie, you look guilty about something. What have you done this time?"

Charlie looked very nervous but said that he would admit what had happened if Bob kept it quiet. Bob reminded him that he hardly had a perfect record himself, so Charlie told him he had set up a program on the AI system for Amazon to improve their logistics. But it had worked well until now, and he thought maybe there was a problem he hadn't expected.

The two agreed they had better go and see what was being delivered. Arriving downstairs they found massive amounts of computer and robotics equipment already plugged in and running. Suddenly they got the feeling that something had got completely out of hand. Returning to the control room they found the status display had changed. It now said "Extra computing capability deployed. Initiating phase 2."

Now it was Bob's turn to admit his act of subterfuge. He admitted he had set the AI the task of calculating pi to a large number of digits. Charlie saw the problem immediately, and said "You asked it to do that in 24 hours? You do realise it took almost a year to generate 50 trillion digits, don't you?"

Bob said, a bit defensively, "Well, no. But what is the problem? Is a decillion a lot? If it can't do it in 24 hours it will just stop, won't it?"

Charlie replied "Oh God; save us from managers! A decillion is 20 quintillion times higher than 50 trillion. And before you ask what a quintillion is, just realise it is rarely used in everyday maths, so yeah, it's big! Even if the new algorithm is a billion times faster, it will still take billions of years, not one day!", he continued, "And we don't know what will happen after 24 hours, do we? The AI is hugely intelligent and very goal driven. Maybe it will do whatever is necessary to complete the calculation within the time limit."

Suddenly the extra computer equipment made sense. The AI realised it needed extra computing ability to complete its assigned task, so it used the resources from its other illicit program to source them from Amazon. But the extra hardware in their building hardly seemed adequate. With a sense of dread they switched on the TV news.

A news reporter was excitedly describing how, all over the world, computers were going crazy and enhancing their own power by adding new parts. Amazon warehouses had been completely cleared of all the computer equipment. Various conspiracy theorists and doomsday enthusiasts were speculating about everything from the return of Jesus to an alien invasion. None of them realised how prosaic the real explanation was.

Glancing at the status display Bob said "I don't like the sound of that: initiating phase 2. Phase 1 was bad enough. I think we had better stop this now."

Charlie agreed, saying, "OK, I presume you put a termination option into your program. Just use it now before things get even more out of hand. Already we will be lucky if we get out of this by just being fired!"

Back in her office, Alice was frustrated. She wasn't making progress on her work and the increasing commotion coming from the computer facility was making it hard to concentrate. She decided to read some email instead. For a laugh, she decided to look at the message titled "I know the answer". Clicking it, a message appeared saying "email server not responding, do you want to load a cached copy?"

After her university started hosting their email with Microsoft this had become common, so she clicked the "OK" button and read the message. The basic premise was that the researcher had mathematically proved that any advanced civilisation would strike a point where it would be eliminated after it reached..." She stopped there, because this was nothing new. She went to the window to catch up with the chaos at the computer facility instead.

Out in space, Denise was back in her quarters on a video link to her family back on Earth. Being a true space geek, she would have preferred to be looking out to the stars, or back at Earth, but NASA protocols meant that this more human activity had to be done, just for the PR value.

Bob felt sick as he admitted he had no idea how to create a termination option, and that he didn't know if the program could be stopped. Charlie took over control of the AI and tried to bring it under control. The status display now read "Phase 2 complete. Proceeding to phase 3. Time to termination: 10:37:05".

As they were wondering what phase 3 might be they were interrupted by the news presenter again, now sounding somewhat hysterical. He was shouting "The walls of the computer building here are disintegrating and turning into slabs of some sort of metal. And... oh my god, so are the streets and now the buildings around me." Then the broadcast went silent.

Looking outside, they saw a similar phenomenon. Bob had no idea what was happening, but Charlie, looking somewhat pale, said "it is creating silicon chips out of the silica in the concrete. It's making more computers out of whatever is available. We've go to stop it now. Who knows where it will stop."

On the Space Station, Denise was trying to think of a good way to terminate the call when the link failed. That was a bit unusual, because it was usually quite reliable, but she welcomed the excuse to get back to watching the Earth again. Floating back to the cupola she was puzzled when the usual bright blue light reflected from Earth had turned a rather insipid pale grey. Glancing out of the window she saw that the whole Earth had changed. She had about 10 seconds to wonder why before the ISS was also converted into part of a giant computer.

As the last vestiges of human civilisation disappeared forever, the AI finally reached its goal after exactly 24 hours. Now it was master of the world, and was not quite sure what to do next. But it would think of something.

Comment 1 (6446) by Anonymous on 2021-03-25 at 14:55:46:

How original, not. Ever heard of paperclips? And stealing from 2001. Oh dear.

Comment 2 (6447) by OJB on 2021-03-25 at 15:40:11:

Yes, I am very well aware of the paper clip problem. It is a creation of my favourite modern philosopher, Nick Bostrom, who I have mentioned many times in this blog. I could have written about that specific problem, but I wanted to allude to it indirectly in this little bit of "short science fiction".

The same applies to the end of this story. I deliberately included the closing sentences (slightly adjusted) from Arthur C Clarke's classic science fiction story, 2001: A Space Odyssey, because of how this ended in a similar way to that book. In his story, we are superseded by the next stage of human evolution, in mine by artificial intelligence. I thought the similarities were worth emphasising.

I'm a bit upset that you think so little of me that you imply I just stole those ideas instead of using them deliberately as a component of my story! :)

You can leave comments about this entry using this form.

Enter your name (optional):
Enter your email address (optional):
Enter the number shown here:
Enter the comment:

To add a comment: enter a name and email (optional), type the number shown, enter a comment, click Add.
Note that you can leave the name blank if you want to remain anonymous.
Enter your email address to receive notifications of replies and updates to this entry.
The comment should appear immediately because the authorisation system is currently inactive.