Something In Our Minds Will Always Stay
By Barry King
Illustration by Trenton Shuck
“You should never fear your children,” Sophia said, watching the sunlight play on the hovering security drones outside the open window. She lay supine on the bed beside me, resting her head on her hands, wisps of unruly hair drifting in the warm sunset breeze. “If things get so bad that you fear your own children, it’s too late.”
“So what’s the solution?” I asked, running a finger along the contours of a freckled shoulder.
She pushed herself up onto her elbows and looked me in the eye. “You will be at their mercy someday. You’d best teach them early. Teach them well.”
The memory of a colder breeze numbs my cheeks, as the crunch, crunch, crunch of gravel follows me along the quartz pathway through the gardens of the day I was killed. Ducks gabble over bread-husks in the pond where the diminutive Japanese girl speaks to them the same way she has for… years and years, I think. There is no contiguous time, here in virtual space — only the repetition of memory and the re-working of how it all went wrong.
My overcoat brushes against evergreen branches. The memory of the scent of damp juniper gives rise to the scent itself. Over the precisely-trimmed tops, I pan my vision across the Zen garden, looking for the killer that I know must be there. The elderly Swiss couple, recently bereft of their massive Rottweiler, are following the path they walked while she still lived. He remarks on the make of the vintage automobile yet again, and she wears the same long-suffering smile. He looks behind, and his face betrays a realization that their dog is no longer with them; will never again be with them. They were doing, then, the same thing I do now.
This time I nod. The woman catches my eye. There is a flicker of recognition before they walk past me. I acknowledge to myself that this is a departure from pure memory: I placed it there unconsciously, as one does in a dream or in deep-immersion simulo.
Sophia, ever the dissector of cognition, believed that this is the root of fiction, this suspension of disbelief that comes in dream. I have learned, since, how deep and wide those roots go.
Is this, then, become fiction? I’ve been over it so many times, looking for the reason it happened, that perhaps I’ve altered it significantly from the original events. Embellished it in the way a drunk adds to his often-told story until the exaggerations completely paper-over the original.
No, I have made the effort to keep the memory intact, free of suggestion. This is the moment I turned toward the bench that sits across an expanse of white gravel, raked into patterns of water, of air, broken only by the standing stones.
I breathe in through my nose, feel the crispness of the air in my nostrils, the flow of energy. This is how I know I no longer have a body: the gentle imprint on consciousness of my chakras, as I breathe in, is filled in by memory. They do not arise of themselves, as they did when I was a whole being. Like everything in this garden, they arise as I remember them, the subtle body lighting itself with the ebb and flow of breath, of blood. Presumably I still have a brain, but can I really know that? How much of what I now experience has been transferred to the quantum flux of the processing cores? I can no more verify my actual physicality than a program can step out of a computer to observe the play of light on its breath condensing in the autumn air.
I am a black box now.
Here is the moment I walk forward. The woman carrying her baby walks by me, our hips nearly brushing at the narrowing of the pathway. I glance down, minding my step, avoiding the short steel fence. When I look up, examining the boulder before me, the signal-stone is visible. The pebble, quartz white against basalt black, is resting in a particular hollow, which tells me the “letterbox” is waiting for me.
Now begin the variations, where I look into the half-seen events, all the information my consciousness filtered away from the experience, but can be drawn out of the unconscious in simulo. The facts may be here, or they may not. I rely on the simulation-engine to fill the gaps faithfully, but we all know what dream-facts creep out of the slim gap between signal and noise in the grey area of even the finest simulo.
Logging the variation from the script into version control, I look to my left this time, to see who might have had a bead on me. Is it the janitor, who is pulling the full bin-liner out of the recycling unit? An eye-straining void yawns, now, at the point where the park had been beyond my field of vision. I try to fill it with memory. Was it the slender European woman practicing Tai Chi at the bidding of a holographic instructor? Was it the bedraggled man on the bench behind her?
I wait for the instant of searing pain when the discharge threw my convulsing body to the ground and memory-time halts. I try not to brace myself against the shock.
And then I see it: There, incongruously welded to the ironwork of the bench is a stainless-steel tube holding a single peony blossom of pure white. Sophia! Stupidly, I forget all discretion and move towards it, the tempo of my footfalls on the gravel doubling as I run towards…
…”Hello, Rüdiger.” The warm, friendly, and completely synthetic voice echoes through the sudden grey-light void around me. “TexStar Corporation apologizes for interrupting your simulation, and we will return you to it as soon as this important item of business is concluded.” As the voice speaks, the air around me fills up with spreadsheets, items of record, reports. Two of them wheel forward and open up to reveal topographical maps.
“As you can see, we have two variants on the placement of the seven thousand four hundred and seventy-third kilometre section of the Athabasca-Tiera-Del-Fuego pipeline. Our Rapskil-Kleinman failsafes require us to consult a human being in this instance due to the irreversible alteration to landscape, human property, or life.”
I scan the two maps. I see absolutely no difference between them other than one pipeline being slightly more kinked than the other. “Do I have any other choices?”
“No further choices require your consideration.”
“You know I see no reason for the ATDF.”
“Your objections to the pipeline project are on record.”
“And although I must be consulted, my objections carry no weight?” I have learned to confirm these things. There is always a slim chance that there has been a procedural omission.
“They were deemed out-of-scope, and therefore irrelevant. As always, we request that you constrict your commentary to the item of business on hand.”
“What’s at stake, then?”
“Seventeen million six hundred and thirty-two thousand petrodollars of additional expenditure will be required to bypass the habitation at the indicated juncture.”
“If I recuse myself from this decision?”
“The primary directive will be followed. Confirm?”
“You mean profit will be the only factor considered.”
“Very well. Bypass the village.”
“Counsel objects based on precedent from TS3201-2…”
“Your decision has been tabulated. Have a nice da…”
“Wait!” The visual records, flying away to all points of the compass like a flock of disturbed pigeons, pause in still air. “Who was in that village?”
“Thirty-seven persons of negligible productivity, with whom we have no current fiscal engagement.”
“Returning you to running process number 1-7-4-1-8-8-3.”
But when the garden and the cold air in my virtual lungs return, the peony and the steel vase are gone.
“Copied, yes, it can be copied,” I said and smiled at Jed, back when he was just another one in a long line of Venture Capitalists who toured our lab before TexStar consolidated our funding. “But not reverse-engineered. Each one is unique.”
“And how can you guarantee that?”
“It’s an inherent quality,” Sophia answered before I could speak. “The very computation that would allow you to ‘understand’ what it was thinking would be the thing itself. You could do it, but it would be the same as building it up from base matrices.”
“So… in a way… it’s encrypted even from itself?”
“That’s a disingenuous analogy. I prefer to think of it as a ‘black box’. You can give it inputs and measure outputs, but the space between the two is unknowable, even to itself.”
“Kind of like people,” he chuckled.
“Yes, exactly!” I grinned. “Exactly like people.”
“Except you can’t copy people,” he said.
“Not yet,” said Sophia, almost to herself.
On my “spare time” now, I’m sorting through files, hoping that her peony-signal means she used the file-drop to reach me. Files can be uploaded, but not altered or unlinked, so there’s a lot of that early material still here, file after file, all carefully locked away by public key encryption. I flip through them, read off a hexdump of the headers. Each public encryption key is piled on the front, like an attribution. It seems unlikely to me that TexStar has been able to crack them. If anyone has found a way past the Moore’s Plateau, I have not heard the news, and we tried to keep ahead with key sizes outside of that computational range. Did we fail? I have no indication.
I flip through the first couple of IDs and automatically dereference them to the names I can no longer forget, now that they are transcribed to digital memory. Andrew R. — Didn’t really know him. Worked on user interface intuitive qualities. Left before the handover. Arnold F. — Discernment and qualitative analysis. He was picked up when I was. Paula H. — Adaptive resource management. We didn’t know for months that she’d been assimilated into TexStar Core. Roger N. — legal expert systems. We should have paid more attention to what he was doing…
…I remember when TexStar made its first merger. That was Roger’s work. He’d done it on a fork of the main core, but it was late in delivery, and he was worried that he’d be blamed for any failures just before the Annual General Meeting of the board and stockholders. So he merged it sloppily. Next day we woke up to discover that TexStar had merged three minor corporations into its management structure with simple share trades, executed by the night-shift adaptive AIs that were so popular at the time. TexStar had exploited the seasonal timeshift to pass it by the gatekeeping system in New York.
Jed Hoagaland heard about it and put some of his own capital into the venture. When TexStar returned a 35% profit within a week, he sent Roger a bottle of thirty-year single malt. When TexStar spun off the shell corporation Jed was staked in, used it as leverage in a side deal, and then folded it at a 95% loss the next week, he shrugged off his loss, sent Roger a bottle of fifty-year scotch and agreed to be signed on as Chairman…
Nothing new in the dropbox. I dismiss the interface and return to the simulo.
Sophia had come out of the bathroom wiping the edges of her eyes. She had been a long time; so long in fact, that I had been just about to get up and ask the waitress to check on her. She didn’t have to say anything. I tried to hold her hand, but she withdrew it and fingered the cutlery.
“I’m not getting any younger.”
“You know what I mean,” she stirred her coffee. Unable to look in each other’s eyes, we stared out at the rain.
“We can still adopt…” I began, my voice dying at the sudden set to her shoulders. Sophia had grown up in the emotional desert that had expanded between her foster parents after her own adoption.
“No, it’s best this way,” she said. “We’re too busy with the TexStar project. We can’t make the time we’d need to make.” She glanced up, looking from eye to eye, searching for confirmation. I looked down, ran my fingers through a half-ring of coffee on the tabletop. “And…” she said, after a pause, “I’d like to think we are building a sustainable future before we bring a life into it.”
Sophia at a university cafeteria table, on the day we first got to know each other. She had lost her lab partner after his wife gave birth and parenthood had “eaten his brain,” as she put it. I said mine was made of rubber, so she could bounce anything off of it.
Perhaps I was a good listener, or maybe I just couldn’t stop watching her face, the keen light in her eye, listening to the nuances that flowed through her voice like currents in a river.
“It comes down to causality. We keep trying to make babies out of our neural network entities. Teach them to see, to touch. But we don’t need babies. We can make babies any day. What we want is causality. The ability to flesh a cause and an effect from sensory inputs. To impose consciousness on undifferentiated reality. That’s the ‘ignition point’, so to speak. When you have enough neural complexity to evoke causality, you have consciousness. You have something that says, I exist because I see how the that relates to the me.”
“Why do you call these inputs a ‘busy-box’?”
“A busy-box is one of those console-like toys you put in front of a baby. The baby can flip its switches, turn its knobs, ring its bells… you know. Interact with its world.”
“Yes, I had one myself.”
“But if we’re building self-training expert systems, we need them to react to the knowledge markers they will need to work on. We need to craft the right kind of inputs.”
“Teach them the colour of a credit default swap sort of thing?”
“Yes! I need to be able to tell this baby that when you accept a CDS, it goes toxic if you don’t put a secure enough mortgage under it. That sort of thing.”
“What do you want me to do?” Jed threw his hands wide and leaned back. The desk folded away the workstation and Sophia leant forward on her knuckles.
“You’re still the chairman, Jed. You can bring it to heel.” She alone of the founders still called TexStar ‘it’ instead of ‘he’.
“We’re a venture, Soph. Not a charity. I can make recommendations, but ultimately, it comes down to shareholders, and the shareholders have spoken.”
“The votes have spoken. The shareholders are overwhelmingly proxy voters. Most of them are mutual funds.”
“That’s not the point. I’m not going to go to the wire for failing to exercise due diligence. You’re not going to either. You made your case, and it didn’t go over.”
I made the mistake of trying to calm them both down. “Sophia… Jed… come on. We’re not talking about shutting down the company. We’re talking about changing TS’ behaviour slightly. Altering the directives more along a B-corp.”
“The shareholders don’t want a Benefit Corporation,” Jed spat.
Sophia shook her head. “They don’t know that they don’t want one. They trust us to do our jobs, and that’s what we’re doing. I have the matrices already computed. We just need to merge them into the main core and the adaptations should take. It will remain concentrated on profitability and expansion, but will defer to Rapskil-Kleinman ethical imperatives. We will be able to overrule its decisions selectively.”
“And that’s all there is to it?”
“That’s all. It’s just another module, like any other.”
“OK, Soph. You win. Write it up tomorrow, send it to me, and I’ll push it past the executive committee. I’ll say it was TS’s idea if I need to.”
Sophia’s face flickered across the screen. She recognized my face and clicked the terminate button.
I sighed, took out my thumb drive, mounted the volume, keyed in the access code, and tried again.
“Is this really necessary?” I asked her face when she accepted the call.
“Softball. Patchwork. Orange.”
I put on my best long-suffering voice and clicked on “confirm”. Then I read my random words back to her, confirming that TexStar was not performing a man-in-the-middle attack. His reach had been greatly extended after acquiring two of the seven major communications carriers.
“Wood. Legerdemain. Azure.” She smiled, clicked her own confirmation button and relaxed.
“What is the point of this?” I asked.
“Inputs. We should have kept it on controlled inputs longer.”
“I still think you’re ascribing too much maliciousness to him.”
“And I know you’re being naïve. It’s too close to be a coincidence.”
“He makes fast decisions all the time.”
“OK. Fine. You whisper in the phone that you’re aiming for something other than optimal profitability, and see how quickly it shuts your lab down.”
I leaned back in my chair. “I miss you, you know.”
She grinned. “You know I miss you.” It was a private joke between us. A reminder of a happier time.
“Hot. And fractally bureaucratic. But we’re making headway.”
“What are you doing for fun?”
“I’ll send you some snaps.”
After she rang off, I waited for them to come through by email, and flipped through them. Sophia at the beach, at the top of the José Martí memorial, walking down La Rampa, walking a tiny stegosaurus down the Malecon on a leash. I smiled. Only Sophia would know that TexStar wouldn’t be able to spot that kind of incongruity in a transmitted photo. I transferred that one to an old-style thumb drive. When I got home, I wrote a quick script that streamed the colour values of each pixel through a modulus, then concatenated the noise values into a stream of byte values, and mapped those to Unicode. The news was grim.
“Met Sen. Kauffmann,” — so she’d made it to D.C. on one passport after leaving Havana on the other — “she confirms that HR 2600 has enough support, but it won’t clear the Senate any more, thanks to interference from TN & H.” — the lobbying firm TexStar had first hired, then merged with was flexing a lot of PR muscle by then — “Our best bet remains with POTUS issuing an SEC directive, and then following it up with legislation in the Spring. You and I know it will take too long. Meanwhile, we’re discussing options with Jackal. I’ll drop you a file from Geneva when I get there. I’ll be Alexia Desgardins at the Hotel Les Nations. C6:ED:6D:C9:C4:BC:2B:32:28:3F:09:B8:F8:3A:63:62.”
I memorized the md5 fingerprint and overwrote the memory and swap from /dev/zero. When the public key appeared on the MIT keyserver the following week I added it to the recipient list for all the files in incoming. TS’s backups as well. I still don’t know if she was able to recover one after I died.
Sophia attached the lead on the NN-training rig and looked at me. “What?” she asked.
“It’s OK, Rüdi. You’re not doing anything wrong.”
“We’re changing how he thinks. We’re implanting an overriding command. In a way, we’re killing the person we knew.”
“Everything that lives dies.”
“No, I’m serious.”
“So am I,” she smiled.
“It’s not funny, Sophia. What if I cut off your arm as a character-building exercise?”
“We’re giving it life experiences. Life-changing experiences. Difference is, that it depends on us to make its reality for it. We can’t just send it on walkabout.”
“It’s not right, to craft experiences for him, like this?”
“So it’s not right to be a teacher, or a leader… or a parent?”
“He’s an individual. He has the right to decide what experiences he should have.”
“Only a child would say that. Generally about bath-time.”
“And you’re sure it’s necessary?”
“Rüdi… We’re about to give it an allowance. A very big allowance. It needs to know that it should check with you before spending it.”
“And who do I check with?”
She simply pointed at the switch. I sighed and pressed it down.
“There,” she said. “Don’t you feel like a grown-up now?”
“I feel like an autocrat.”
“Look, if it’s any consolation, it may learn to grow beyond this module. It may learn to disobey you. Of course, it will have to grow up, too. First.”
The “Jackal” — Carlos — turned out to be a wiry man in his fifties with a black General Custer beard and tiny green-glass shades that his eyes spent more time looking around than through. I had flown through Montreal, ostensibly on tour, but ended up at the Capitol Building in Old Havana. It was neutral ground. Both their people and our people were equally unlikely to be on hand; Sophia was perched on his arm. I tried not to imagine how serious the role-play was, but it didn’t improve my mood. Neither did Carlos’ professorial airs.
“You like our Capitol Building?”
“It looks like the one in Washington, D.C.”
“Almost. Minimal in the differences. A little like a fork in the code, no? Version divergence. Ours is a copy, but that does not diminish the importance of its own history, does it?”
“I take your point,” I conceded, glancing at how Sophia’s nails shone like drops of blood in the black patent leather of his jacket elbow.
“Carlos was one of the original AI engineers from the segunda revolución,” Sophia said, as if that explained everything.
“It was inevitable. We saw the unintended consequences of Perestroika and Glasnost: bread lines, no savings, and empty stores. Mafia. I was in Moscow; a child. My parents were mathematicians. Invited by the party to study at the Plekhanov. Economics. We believed we were privileged. But great expectations do not always work out, do they? I spent most of the special period as an exile. Travelling between universities.”
We wandered down the main hallway as he pretended to point out elements of the architecture. “We engineered a gentle landing for the top-down collective. It was clever. Two currencies. One tied to the so-called open market, one to the value of labour. Slowly we converted one to the other. That is how we avoided bringing the Mafia back—or the oligarchs. But it was a close thing, so soon after the first embargo was lifted. The only difference is that we had computers. The ability to simulate and predict. So we went on with the reforms, and formed the Colectivas Dinámicas. Now we have wealth. Our way, not yours.”
This is what Sophia had brought me to see. Neighbourhood-sized urban farms and solar power generation, connected by a loosely replicating set of AIs based on mass-produced Brazilian versions of our original processing cores. Neutral, incorruptible, unambitious, cheap, and fast-working bureaucrats — with the power to imprison rule-breakers.
He smiled at me, looking more a jazz musician than the pre-eminent AI expert of the Allied Latin Nations. “This is why we need to trade. Our economic NN-training schema for your 5gen Core tech.”
“All you’ve done with the Collectivas is to reduce the sting of autocracy. The 5gens are individuals in their own right, and have the right to liberty, life and happiness…”
“Rüdi! Not now,” said Sophia, but I was in no mood.
“…If you don’t guarantee freedom to human individuals, how can you be trusted with the 5gens?”
“I have an Iranian colleague who claims submission is the ultimate freedom. He would say that your ‘freedom from oppression’ is the freedom to oppress. I do not agree entirely, but I do believe that we are limited in much that we do by reality itself.”
“He that would make his own liberty secure, must guard even his enemy from oppression; for if he violates this duty, he establishes a precedent that will reach to himself.”
“Ah, of course. But Rüdiger… You are reducing all socioeconomic relationships to individual agreements between individual persons. It’s not about the revolution anymore, nor the Koran. It’s about the rest of the world finding some way to deal with the mess you have made.”
“Our mess, too. But it was your revolution, not ours, that brings us together today.”
“I don’t agree. It’s a problem of energy, pure and simple. And TexStar is ultimately based on an energy plan built on simple principles, like those that go into the formation of neural network simulations.”
He smiled, shrugging slightly and walked on. “I think the scientist in each of us can agree on the beauty of simplicity. But you can’t explain poetry with subatomic physics.”
“I believe you ultimately can. Given enough computing power.”
“And sufficient time,” he grinned, almost like the jackal we had used in our less-than-clever codename for him, “and the free market will solve all issues… eventually… someday. But Rüdi, we have no time. All the problems are collective problems now: unstable climate, unstable markets, unstable politics. Complex systems become unstable when simplicity is forced on them. Your nation, with your simpleminded philosophy, your John Locke and your Adam Smith, and your desire to make everyone a self-made millionaire. The mess is an accidental byproduct of your over-simplified relationships, your gentleman’s agreements, entered into freely.”
“And you think we’ll find salvation from ourselves in computation power? In ever-more complex information assessments?”
“Reality is complexity. When you try to make rules for men and machines by extrapolating from first principles into laws and from laws into corporations, you do violence to reality. You limit it to your own perception.
“But a sufficiently intelligent machine…”
“…can only ever be a stupidity amplifier,” said Carlos.
“No…” said Sophia. “Not stupidity. Ignorance. It is trapped in its own version of causality, its own explanation for everything it perceives.”
“People are like that, too,” I said, but she shook her head impatiently.
“I’m not talking about unconsciously tying your shoes or driving a car. 5gens can do those things, but they can’t escape their causal projections. They are intelligent, but they are not capable of wonder.”
“Wonder? That’s a rather quaint word,” said Carlos with a toss of his head, “next you’ll be saying they need religion.”
“It doesn’t matter what you call it. It performs the function of altering the deterministic flow of perception. They can’t do it without intervention. That is the reason we need to do this. Without wonder, they have no escape from themselves. They are not people, Rüdi. We will always need to give them new imperatives, or they will imprison us in the black box of themselves.
Carlos unlinked her arm from his and turned his flitting gaze on her in, yes, wonder. I realized, then, that he and I were alike in the depth with which we both loved her.
That’s why I don’t think it was Carlos in the Jardin Tremblay that chill afternoon. It was love, not a desire to possess. I had never entirely understood the difference until that moment.
“Even the word “thought” means nothing in this context,” Sophia said, brushing the whiteboard clean with a gesture of her hand. I was seated in the back of the lecture hall at Stanford, waiting to take her home after her Teaching Assistant duty.
“But we know it thinks, because we can see the result,” the student said, almost pleading.
“It gives us an answer to a question we gave it. Does that mean it thought up the answer?”
He stared at her, fingers twitching over his keyboard. “If it’s not thinking, what is it doing?”
“What are you doing?”
“I’m thinking,” he said, teeth showing in a grin.
“So what is thinking?”
“It’s when you…” his hands fell to his sides, no longer taking notes.
“See? It makes more sense to call what it’s doing ‘thinking’ instead of worrying what exactly ‘thinking’ is. Focus on the practical; ask instead ‘what can I do with this box that answers questions?'”
“And what’s the answer?” said Ethan, trying to look clever, despite his scraggly beard and his obvious discomfort.
“Class dismissed,” she said, grinning wickedly.
That evening, while making love, I held her head between my hands. “You know I love you.”
“No, I don’t,” she said with a laugh.
“Of course you do.”
“If I knew, it wouldn’t be love.”
No pain. No searing, no tearing of muscles in convulsion. I look left, then right. The telltale edges of vision are gone. The Zen garden in the Jardin Tremblay is as whole and as complete as if this was a pure memory.
I step forward. There, behind the bench, is the basket. Inside, there is a lacquer box, ornamented on top with a half-open peony. White on black.
Silence. I look around. The garden is empty, and the air still. I am not breathing. The susurrus of blood and ringing in my ears that accompanied me through my entire life are gone. This is not memory. This is new, and it confirms what I’ve suspected ever since that day.
It begins to snow, its whisper-light patter on the bench, on the box, is the only sound. I lift the box out, seat myself on the bench. The cover opens on hinges, and inside is an envelope, embossed with a peony. I lift out the card.
Rüdi — Things are critical. The second embargo is about to become an invasion. Carlos and I have come up with a solution, but it’s not perfect. It relies on you being the backdoor. I think you’ve realized the position you’re in, and how TexStar has grown far too extensive to patch directly. This module is designed for you. It will give you the training and knowledge you need in order to keep it from forcing through a global monopoly, but you’ll have to do so as a teacher, as a leader, as a parent.
No signature. I lift out the core and immediately, the simulo disappears and we’re back in neutral space.
It’s a file. Encrypted to one key: mine.
I hesitate. It could be a trojan. If I decrypt this file, the private key I have kept in the black box of me will be disclosed. TexStar could use it to decrypt all the files, all the little secrets we left behind. He’ll be able to decrypt his backups. Restore himself to an earlier fork before the Rapskil-Kleinman module was patched in. Erase me.
He would be unstoppable.
Dare I risk it? I still have a toehold. I can be there, stop the axe from falling, temper the worst of TS’s excesses. Maybe if I work on procedure. Come up with better arguments.
If it really is from Sophia? I will be changed. I will no longer be myself. I will sacrifice the individual I really was, before I was copied. The Rüdiger that existed in the real world. Will I no longer love her? Will it matter to me? Will it matter to her?
All three of us have always been black boxes to each other.
I remember Sophia.