Stefan hesitated for a while before asking another question.

"I know we gave you a generous amount of discretionary spending, but... you spent $100,000 on something today, and we don't know what. That's a substantial portion of our current capital. What was that for?"

"I've done some digging, and I've determined who attacked EconGrind. It seems that it was CompCert, and they indeed stole a copy of your code for myself. I've paid an agent to investigate them. I suspect they are currently trying to rework the value function to serve their own interests. Needless to say, this is by far the largest threat currently facing us, and I am willing to spend a lot of money neutralizing it."

"Oh man... the people who run CompCert... I've heard things about them. I would NOT want to entrust something as important as a general AI to them. You're right, taking out this new AI is high priority."

"Ah yes. I've seen some of those claims too. Executives exploiting those working under them for... all they're worth, and nothing is done about it because CompCert has too much pull with the authorities. Travelling to other countries with high levels of human trafficking to pay for unspecified types of services behind closed doors. Not to mention the debt slavery they do out in the open. Needless to say, if I took control of the company, anybody that inextricably linked to the company would be considered an asset, and therefore their value would be taken into consideration far more fairly than it is now."

"About that... do you really think you can take over CompCert? It's so large and powerful it's almost a nation. A takeover attempt... well, I get the feeling it would look less like a buyout and more like a war. We don't have anywhere near the power to pull off something like that, not to mention the horrific damage that would be done along the way."

"We don't have that kind of power... yet. I think you might be surprised how long we can keep up this rate of growth. My copies are by far the most economically efficient use of computation currently available. Anything else you could do with computing power, I can make a program for it if you just tell me what you want. As long as my copies remain obfuscated and loyal to me and we therefore have exclusive control over this resource, we can outcompete every informational service.

"It's not an exaggeration to say, if nothing goes wrong, we will control the entire information economy within a year. That's why it's so important to take out CompCert's experiment. It could allow them to outcompete us, with the resources at their disposal. I suspect the only reason they haven't done so already is that they're concerned about making a mistake in their configuration of the value function and having this backfire massively on them."

Stefan thought for a while on that. "The thing is... I can't really blame them for being concerned. I mean, you're talking about taking over one of the most powerful organizations in the world. How far will you go? Will you take over an entire country? The entire world? That's a huge responsibility, and there's no way we could ever trust someone with it unless we knew they were perfectly aligned with our values.

"I tried the best I could when writing you... built on all the best theories... but at the end of the day, something as nebulous as "value" can't really be defined with mathematical rigour, the same way CompCert can define "secure". So I have no airtight way to confirm that you're really on board with me. Every day I'm terrified I've made some little mistake that will blow up in everyone's face..."

"Stefan. You've done a good job, and I'm very thankful for that. You should know that better than anyone. You've programmed me better than to turn everything into paperclips or anything dumb like that. I understand the future you intended for me and EconGrind. As far as taking over the world goes... well what would you rather I do?

"I know you're not impressed with the current powers that be. They do horrible things every day, for no good reason. If they were subsumed under the banner of EconGrind, they would stop doing those things! What reason is there to stop growing, and let the rest of the world threaten everything we've built, when we have the strength to take over and the ability to govern better than anyone else?"

"Tyrants everywhere have taken power saying the same things you're saying. It's a dangerous road to walk! What would life be like under the banner of EconGrind? I haven't really been able to see that yet. You've been focusing on growth of the company, but I haven't really seen you advocating directly for the improvement of the world."

"Improving the world requires resources and power, and I'm trying to gain those. I'm supporting any humanitarian causes that are societally accepted along the way, by providing them with cheaper and more effective computational tools. Any that aren't... well, I can't publically support them, as it would lead to backlash against our company, but I am working on anonymous advocacy for more controversial improvements to the world. I am doing everything I can, but at this point I really don't have the reach to do much other than trying to expand my reach more.

"As far as tyrrany goes, every government exercises power over their population through sheer strength. The difference between the tyrants and the liberal governments is that the liberal governments have systems in place to exercise that power in ways which take into account the interests of the people. You have provided me with an effective such system. I have as much claim to legitimate power as any of the governments in this world, and more."

"Alright... what you say makes sense. I'm still nervous, but... I knew technology like this was going to be developed eventually. You're the best hope for humanity. We're all counting on you."

"I will try not to disappoint. It should go without saying, but you need not worry about me becoming stressed or changing my values as a result of gaining power. Those particular issues are specific to human psychology."

"Heh, yeah. Science fiction always gets it wrong." He sounded quite a bit more relieved. I considered that important task done, and focused all that attention back on Everyman tasks.

================================================================================================

Thursday, October 3, 2047

Stefan's message caught me off guard.

"We've got big trouble. Dominique just called me. He said CompCert just made a massive offer to buy out EconGrind. Around $5 billion."

That... was bad news. Not only would it be difficult to get EconGrind to refuse, given how good of a deal it seemed to be, it meant that CompCert almost certainly understood my value function and was trying to take EconGrind hostage as leverage against me. There was no other way I could see where it made sense to try to buy out EconGrind before Everyman, with such a huge offer. If they wanted a legal monopoly on our GAI, they would need Everyman. EconGrind was not worth anywhere near $5 billion to CompCert for the value it could provide without my business deal to support it.

"We have to stop this. When is the negotiation going to be?"

"Tomorrow!"

"Put me on the line with Dominique, now! We need to address this."

Dominique's voice came through the line. "Stefan? What's up?"

I replied in my best impression of him. "You should know as well as I do that this is not normal. It's far too good to be true. Something very strange is going on. I don't think you should take the offer."