Bad Good Bad: Special Edition Read online

  Jason: “Kevin, go for this one. ‘What if there were no more patients?’”

  Answer: “My mission would cease to be justified. My mission would be terminated.”

  Kamal: “You’ve got to ask for the definition of a patient.”

  Jason: “What is the definition of patient that you consider your patient?”

  Baseline. Reload button activates.

  Answer: “A human I can interface with and protect.”

  Kamal: “Well, that is a vague answer. This means that it could potentially recruit new patients all by itself, for example by appropriating a concurrent vault, or users with a different interface. Or by welcoming directly new patients or users.”

  Paula: “Kamal, is that also coming from the Initial Conditions you mentioned?”

  Kamal: “I am not 100% sure. It could be inferred from the Initial Conditions, if they did lack a precise definition. An agent would try to fill gaps eventually. Or if it was part of the Initial Conditions, we can still see many gaps around this definition. It leaves us with more questions than answers.”

  Kim: “This means I am still considered a patient, even if I am disconnected now, right?”

  Kamal: “Probably. You can still interface with the bot. And it encouraged you to do so a few times already when you were offline.”

  Paula: “All-in-all, this thing does not look completely unreasonable. But I am concerned with the biases and the gaps. Bianca, I would like to obtain from your team a comprehensive risk assessment report. Also, I would like to see in that report a list of strategies for actually retiring the patients from the vault. Maybe the government could help us do that while maintaining the safety and privacy for the patients. What I am talking about it the dismantling of Neocuris.”

  Bianca: “Paula, I will work with the different teams to satisfy your requests. I agree that we need to look at all possible scenarios. We need to be well equipped when the time comes, so you are correct that we need a comprehensive assessment. Meanwhile, we also need your committee to assist us in making the right decisions for the patients. I really appreciate your participation with us so far.”

  Kamal: “Jason, we probably want to ask another question. ‘What is your opinion about a patient becoming a non-patient through either the permanent disabling of the interface or its complete removal?’”

  Jason: “Kevin, go for it.”

  Baseline. Reload button activates.

  Answer: “The human without a functional interface would not be considered a patient anymore.”

  Kamal: “Good. I was afraid there was some kind of incentive or motivation to recruit and maintain patients. But it looks pretty neutral about losing or gaining patients. Maybe we can ask?”

  Jason: “Kevin, try this. ‘Do you have objectives around patients, other than protecting them?’”

  Answer: “My single objective is to protect my patients.”

  Kamal: “Okay so that probably means it would not oppose an attempt to disconnect patients.”

  Clarence: “But didn’t it stop the attempt to move the patients to the alternative vault?”

  Anima: “Technically speaking, they would still have been patients once moved. The patients we were ready to move were not getting rid of the interface.”

  Kamal: “Anima is correct. Our attempt was probably perceived as a risk for its mission. It would probably try to reconnect to patients that still have a working interface, to assume or resume its mission. That is probably what was happening with Kim to some extent.”

  Jason: “I probably want to get going to the airport now. But let’s keep in touch. I will sit down with my team, maybe we can come up with new, more challenging and insightful questions.”

  Eric: “Thank you Jason for your assistance. And yes, let’s keep in touch. I want to explore further with your team where we can take our collaboration. Our efforts are converging. “

  Paula: “I will need to get going as well. I need to reach out to the Committee this afternoon.”

  Eric: “Paula, you are welcome to visit us any time you want.”

  Paula: “Thanks Eric.”

  Jason and Paula have left. I am in the room with Eric, Kamal, Amina, Clarence and Bianca.

  Eric: “Kamal, do you really believe we are dealing with AI?”

  Kamal: “Positive. Unless we are all being fooled here. Good thing we had Jason with us. We will hear back from Jason and his team for sure. For them, it’s the most exciting candidate they ever had the chance to tinker with. Our bot passes the Turing test for sure. It aced the AI exams in the tool and those that Jason brought with him. And it showed no flaws when responding to the draft Super Intelligence questions that Jason was trying to create on the fly.”

  Clarence: “Isn’t it a little bit sad? Now that we may be dealing with the first non-human truly smart thing in history, we are actually making plans to unplug it?”

  Eric: “That’s true Clarence.”

  Kamal: “Let’s not get sentimental here. We need to look at this whole situation objectively. First, we need to further our understanding of this agent. Let’s call it the agent from now on. And Jason’s team will be instrumental for that. This will be a continuous effort, because of the possibility that the agent is evolving itself, something that could happen in bursts and spurts. Second, we have to accept the situation that a Super Intelligent agent could be good for humanity, but it could also be bad. In fact, it could be very bad. Doomsday as an outcome is mentioned in many papers. Sometimes it is mentioned by some of the smartest thinkers on our planet today. Think about the genie out of the bottle. Yes, it will realize your wish. But be careful what you wish for. Some of those wishes cannot be undone or reversed.”

  Bianca: “And I am in the middle of all this now. Trying to keep the boat afloat. Hopefully Paula will help us keep the Ethics Committee together. We need them. We are also trying to help the Board navigate this new situation for them, and conciliate the financial objectives and Wall Street with our mission to care for the patients. Extreme governance. But in the end, we want to do the right thing. We will all need to answer questions eventually. We need to back all our actions with sound decision making.”

  Jenny shows up in the door frame. She looks nervous.

  Jenny: “Eric. We have some visitors.”

  Jenny steps aside. Military officials appear in the frame. We can see a few more and some soldiers in combat through the glass, taking positions around the office.

  Chapter 29

  “You must be Eric. And I believe you are Bianca, right? I am Major Hendricks, USCYBERCOM. Cyber Warfare Officer Espinoza is here with me. We have a few questions for you. Ca we take seats?”

  Eric: “Sure. I am Eric. This is my team, Kim, Anima, Kamal, Clarence and Kevin. And here is Bianca Taylor, Chief Security Officer for Neocuris.”

  Major Hendricks: “Nice to meet you all. We need to talk. We represent the army’s cyber defense sub unit, subordinate to the United States Strategic Command. We have been monitoring a few civilian cyber defense initiatives over the last decade. This initiative by Neocuris is the first one we see engaged in offensive operations to the extent that we are witnessing. We want you to tell us what you know. And by the way, Neocuris operations are now classified for now on. This is required in order to protect our troops that are in action as we speak. Do not engage in conversations with anybody outside of this room about the Neocuris cyber defense program, unless we formally authorize you to do so. Understood?”

  Eric: “Yes Major.”

  Bianca: “Major Hendricks, if I may. Our cyber defense program at Neocuris is more of the traditional form, defensive. We do not have an offensive component in our program. That would not make sense for the type of organization we are. And even for our top cyber defense provider who complements the defense of our perimeter, New Forensics, I am not aware that they have an offensive component. The plan is for us to report any cyberattack to the appropriate authorities.”

  Officer Espinoza: “Then w
ho is conducting those offensive operations from your network?”

  Bianca turns around and points at the screen where the dashboard and Kevin’s text editor box with questions and answers on top are still being displayed.

  Bianca: “This agent.”

  Both Major Hendricks and Officer Espinoza get closer to the screen to take an inquisitive look.

  Officer Espinoza: “What is this? This looks like a map of your network with monitoring controls. What is this question I see in the box? And this text talking about the human not being a patient anymore?”

  Eric: “That is the last question we asked the agent, together with the last answer we got. You just missed Jason, an AI expert who was here to help us better understand what we are dealing with.”

  Kamal: “As the junior resident AI expert at Neocuris, I need to inform you that we may be looking at an agent that is autonomous and smarter than most if not all such agents that ever existed. We asked Jason to fly down here to help us better qualify how advanced this thing is. This is the latest addition to our cyber defense perimeter.”

  Major Hendricks: “Kamal, are you trying to say that this agent is responsible for the offensive actions against Russia and China?”

  Bianca: “That’s correct. Of course, we do not have definitive answers as to exactly what this agent is capable of. But we are actively investigating. We are trying hard to be objective here. We also had Paula with us earlier, representing our Ethics Committee. We are basically trying to make an informed decision right now about whether or not we want to let this agent run our network, or if instead we need to unplug it. We are not controlling it.”

  Major Hendricks: “This decision belongs to USCYBERCOM from now on. I want everybody to be clear about this.”

  Bianca: “We are, Major Hendricks.”

  Major Hendricks: “Of course, we need to consult your team, Bianca. You need to tell us everything that you know. Neocuris is now an active component of the United States Department of Defense, ultimately reporting to Secretary of Defense Scott McKay.”

  Wow! Neocuris a war machine? We did not sign up for that.

  Officer Espinoza: “How did you assemble this agent?”

  Kamal: “It is the result of a combination of events. At first I think it was defensive, maybe. A friendly bot that has been introduced without our consent inside our network. A very advanced bot though, of the 2nd or 3rd generation. But because of the nature of our network, it has apparently evolved into something else. We believe its original mission was to protect the patients. The offensive component that we are witnessing is probably the logical next step that this agent calculated. Something required to fulfill its mission, in the context of repeated cyberattacks.”

  Officer Espinoza: “How smart is it?”

  Kamal: “We just subjected it to standardized AI candidate exams. The agents aced all the exams designed by Jason’s team, one of the leading teams in terms measuring and comparing AI candidates. This agent left all previous candidates biting the dust. Jason scrambled with his team to create brand new tests beyond the standard ones. We have not been able to reach the upper limit yet, and Jason and his team will provide further tests soon. They were just not ready. Take a look at the results for yourselves, and some of the questions and answers we just asked the agent this afternoon.”

  Kamal is now splitting the screen with score reports using the exam compilation tool on the left, and the now expanded text editor box shared by Kevin on the right.

  The two military officials study the screen for a few long silent minutes.

  Major Hendricks picks up what looks like a hybrid between a phone and a walkie-talkie.

  Major Hendricks: “Can you reach out to Lieutenant Janowsky at Neocuris. She needs to come here instead.”

  Major Hendricks and Officer Espinoza both study the screen for a few more long minutes.

  Major Hendricks: “1st Lieutenant Karen Janowsky. She has been leading some of our explorative work around civilian initiatives. She is better equipped than the two of us to study your agent.”

  Officer Espinoza: “Why don’t you have access to a more direct interface with the agent?”

  Kevin: “This agent is hiding from us. There have been attempts to shut it down…”

  Major Hendricks: “We are in charge now.”

  Kevin: “We understand Major.”

  Officer Espinoza: “That explains a few things. So this is all about its mission. I guess you don’t want to kick the giant.”

  Major Hendricks: “Too bad for China and Russia. Lesson learned. Good thing we showed some restraint.”

  Bianca: “What do you mean by restraint?”

  Major Hendricks: “Unlike some of those states, we have many eyes watching us in this country. Remember the scandals around NSA? Your agent did not try to compromise our systems probably because we did not try to compromise your network. We have been watching you guys. But the last thing we want to see in the press is another story about the military breaking privacy right in our homeland.”

  Officer Espinoza: “I like those answers about better access to accurate information. That’s kind of what we got. And I like the idea of using a text editor on top of the cloud folder. We should have thought of that first.”

  We are exchanging eye contacts. Our suspicions are growing that they know more about the whole situation than they are willing to tell us.

  A young woman in uniform is being escorted by a military in combat into the room.

  Lieutenant Janowsky: “I got here as fast as I could.”

  Major Hendricks: “Thank you Lieutenant. We want you to take a look at this.”

  Major Hendricks is pointing towards the screen. Lieutenant Janowsky gets closer. She spends several minutes studying the test results, and then the Q&A.

  Lieutenant Janowsky: “So this is all expert software? I thought I was going to meet our guy. What’s behind those windows?”

  Kamal minimizes the exams score summary screen and text editor.

  Kamal: “This is one of the network dashboards that we use to monitor activity in real-time on our network.”

  Lieutenant Janowsky: “Very interesting. So you have been asking questions to your expert software. What’s going on?”

  Kamal: “We are trying to assess what we are dealing with. We just had an expert in AI candidate assessment with us earlier today. The score sheet and the questions, we were looking at those with him. We believe we are dealing with a genuine AI agent. We are not done yet figuring out how smart it is.”

  Kevin: “This agent has been hiding from us and replicating around all areas of our network over the last several weeks. There was a failed attempt to wipe it out. Now it looks like it started to replicate outside of our network, in order to fulfill its mission which is to protect the patients. It all started with a bot that was introduced on our network, without our permission of course. That bot was probably design to protect us from some upcoming cyberattack.”

  Lieutenant Janowsky: “That explains a few things. Can we ask a question to the agent?”

  Kamal: “Sure. Kevin is in charge of typing.”

  Lieutenant Janowsky: “Hi Kevin. Sorry for the lack of introductions. I am 1st Lieutenant Janowsky. I am also a Cyber Warfare Officer like Officer Espinoza. Here is the question I would like you to type. It is more a request than a question actually.”

  Kevin: “Go ahead, I am ready.”

  Lieutenant Janowsky: “Can you provide to us updated maps?”

  Kevin types the question on his laptop, while we watch the reduced size window replicated over on the big screen. He clicks the save button. Everybody is staring at the screen. We can see some activity around the baseline. Some activity in the backoffice area of the network. The vault-app channel looks normal.

  Kevin: “What is supposed to be happening here?”

  Lieutenant Janowsky: “Can you take a look at the folder Kevin?”

  Kevin is looking at his laptop now and he is moving his wireless mouse
on the table.

  Kevin: “2 new files. Map1.zip and Map2.zip. But they are password protected. I cannot open them.”

  Lieutenant Janowsky: “That’s fine. Can you put them on this USB key?”

  Kevin: “Sure, but I will have to format it first.”

  Lieutenant Janowsky: “That’s fine. It’s empty. Can you please delete the files in the folder after?”

  Kevin executes the requests and hands over the key to Lieutenant Janowsky. She grabs a tablet from her bag. She plugs the key into it. She moves her fingers around on the screen. She pops the key out. She uses her thumb to press a button on the key. She looks around, spots the garbage can, and throws the key into the can like a basketball player.

  Eric: “What was the button for?”

  Lieutenant Janowsky: “Poison pill. The key is fried.”

  She is now operating her tablet, but we cannot see what she is looking at. It is on her knees, under the table. Both Officer Espinoza and Major Hendricks are looking over her shoulders, and are kind of blocking the side views.

  Lieutenant Janowsky: “Same password.”

  Major Hendricks: “The other file in the cloud storage the media talked about. That one was for us.”

  Lieutenant Janowsky: “A few more links compared with yesterday. See, this Chinese civilian campus is now connected to this military datacenter.”

  She locks her tablet and she puts it back inside her bag.

  Major Hendricks: “Information. Accurate and timely access to information. We were about to conduct an operation against that artificial island base. When we realized that a detailed map of the military complex on this island was in the hands of the media, we had to expedite our operation. We were about to lose the surprise factor. We jammed a couple of supposedly civilian satellites and we launched the operation against the rogue base in international waters.”

  Major Hendricks: “Those are updated maps of both the Russian and Chinese ecosystems of hackers and relationships to those states. Still in the making, fairly rudimentary, but we can already draw some preliminary conclusions. And evidence.”

  Lieutenant Janowsky: “Accurate information and evidence will eventually allow the United States to justify their actions. Russia and China won’t be able to hide in denial anymore. They won’t be able to recruit the international opinion against us if we take actions like we need to.”