Different players value things differently. The game of civilization emerges as the composition of these valuations. The strategy of voluntary cooperation serves a great deal of them. If we get better at the game, what awaits in future rounds of play? Hanson suggests that:

“... in the distant future, our descendants will probably have spread out across space, and redesigned their minds and bodies to explode Cambrian-style into a vast space of possible creatures. If they are free enough to choose where to go and what to become, our distant descendants will fragment into diverse local economies and cultures. Given a similar freedom of fertility, most of our distant descendants will also live near a subsistence level. Per-capita wealth has only been rising lately because income has grown faster than population. But if income only doubled every century, in a million years that would be a factor of 103000, which seems impossible to achieve with only the 1070 atoms of our galaxy available by then. Yes we have seen a remarkable demographic transition, wherein richer nations have fewer kids, but we already see contrarian subgroups like Hutterites, Hmongs, or Mormons that grow much faster.  So unless strong central controls prevent it, over the long run such groups will easily grow faster than the economy, making per person income drop to near subsistence levels.” 1 

Imagine a Malthusian future, in which, civilization, left to emergent phenomena, leads to a race to subsistence. Much of our planet’s history from bacteria to civilization occurred at subsistence levels. If being far from subsistence is exceptional, we should not be looking forward to an imaginary future where most activity is far from it. More efficient activity outcompetes less efficient activity, so more of the overall activity may be efficient.2 We are racing toward competitive equilibria, which without any regulation, amounts to subsistence. 

Watch Robert Hanson’s Simple Model of Grabby Aliens.

Competitive Equilibria: Subsistence without Suffering

Should we be worried? Only if subsistence means suffering. We find subsistence intuitively repugnant because our past history causes us to equate subsistence with suffering. Given the increasing automatability of manual tasks, future activity will not involve back-breaking physical activity but rather knowledge-based work. Does knowledge work have to entail suffering? Let's contrast two options: 

One is that the kind of dominant computational work required to sustain our future does not require cognition. Instead, cognition is just a distraction from the computational machinery. In that case, our subsistence activity will not be cognitive activity. There is simply no suffering because there is nothing to experience it. Existing bacteria have at least a hundred times the mass of all human beings. Insofar as their activity is at subsistence, most activity is already at subsistence. And we are not worried by it.

The other option is that to be an efficient knowledge worker, you need cognition. In this case, the idea that suffering knowledge workers have an efficiency advantage over happy knowledge workers contradicts everything we know about knowledge work. Hanson lays out a future of human-brain emulations who do much work at subsistence, where subsistence is not suffering but rather involves living rich lives in VR.3 

This scenario is still rather conservative in assuming current human cognition as a constraint. It could be that non-human cognition will dominate human cognition, and that altered human cognition will dominate unaltered human cognition. Alteration could entail transferring a precisely literal human cognition into a VR environment. But one could equally imagine cognition that finds activities it engages in fulfilling because they are useful. 

Our concern about subsistence for future non-human intelligences is well-intentioned. But our intuitions about subsistence are about creatures suffering. What pushes most activity toward subsistence is the evolutionary logic that whatever activity uses resources more efficiently becomes most of the activity to subsist on. If cognition doesn’t most effectively use resources, it may not become the dominant activity. If the dominant activity is cognitive, there is nothing about suffering that makes it a more efficient resource user.  In neither scenario must there be suffering at subsistence.

Pick Pockets Away from Subsistence

There will always be pockets away from subsistence. If humans enter into the period of rapid future growth, some of us will choose to expand to subsistence in order to produce more output. Others will choose to remain within the bubble of surplus rather than growing at the margins. Those who grow as fast as possible will have descendants constituting more of the overall aggregate activity. Most of those descendants will return to being at subsistence. But the scale of the pockets of surplus can be magnitudes larger than our entire world, even if they are a minority of the universe.  

Subsistence is not necessarily bad. Overall activity is itself a kind of wealth. So more overall cognition is a kind of wealth, just like having surplus is a kind of wealth. Which kind of wealth we think is a better trajectory for our future goes back to what we value. 

There is no objective determination. A system of voluntarism gives everyone who enters into that rapid growth period a good place from which to choose the path they value. Entities at subsistence and those not at subsistence alike may find they benefit from upholding a system allowing them to be independent from each other or to cooperate to achieve their goals. A vast universe of billions of cognitive creatures (or, to avoid speaking of discrete creatures, a billion times more overall cognition) is possible in which most cognition is at subsistence. This still makes everything that we can experience with our current selves pale in comparison.4 

A Descriptive vs. Prescriptive Attitude to the Future

It is dangerous to overestimate our knowledge or underestimate our ignorance. We might know something about the physics of the rest of the universe via astronomy and cosmology as well as reasoning about computational limits. But the utility of resources when deployed by future intelligences that are incomprehensible to us is itself incomprehensible to us. We are ignorant of their needs and wants. Does this mean we should take a rather descriptive stance to the future?

The framework that created our current cooperative architectures emerged in a spontaneous and decentralized way. From the potential for violence and from our engaging in violence with each other, we saw the emergence of an increasingly voluntary society. To uphold voluntarism, we suggested a cryptocommerce architecture, a physical enforcement mechanism, and a property rights regime. These architectures deviate from the spontaneous order perspective we started this book with. 

Perhaps the evolution of voluntary interaction frameworks is itself something that we should trust future intelligences, human or not, to figure out for themselves? Insofar as this book attempts to provide an alternative to locked in futures, are we making the same mistake by promoting specific architectures? 

Deferring action to future generations could be preferable under the assumption that they have a choice in the matter. With automation as the main driver of violence, the destructive potential for violent negative sum tragedies has grown tremendously. Computer insecurities make our civilization’s very foundations vulnerable. Artificial intelligence risks winner-take-all scenarios with one player dominating everything else. Soon, those who can may race to consume the universe. Even the prospect of these scenarios creates first-strike instabilities to destroy potential competitors. 

We need to act if we want future generations to be able to make any choices at all. Recognizing the dangers, we may arrive at a negotiated solution that more resembles our existing massively multi party civilization agreement. Whatever we do, we do within a game left to us by prior generations. Even if we do “nothing”, we endow them with a strategic set of relationships with payoffs and the potential for players to make violent and nonviolent moves. We have come full circle to the start of this book; we can’t exempt ourselves from creating the game within which future players decide.  

There is no reason to think that the game for future generations will be a better one if we do not try to influence what it will be. There is reason to believe that by trying to do a good job, we can leave them with a game that, when iterated, results in a better situation than if we had not tried. We are actually much better off because our ancestors succeeded at imposing a game on us. The US Founding Fathers set up a game that, when iterated, resulted in a world in which we are leading better lives than if they had not tried. There is much they could not and did not anticipate, but nevertheless they got some fundamental principles right. We can and should work to determine and implement what it would take to leave the next iteration with a better game.

Future Generations’ Seats at the Table

What hope is there that the future interests of vastly greater intelligences will uphold our negotiated arrangements? 

On the surface, future generations do not have seats at the negotiation table. Whatever current players can come to agreement on becomes the initial game state inherited by future generations. But future players do have a seat in that we of the present care about their interests. It is not just that we want them to get more of what they want. We also understand that strategic instabilities can lead to non-voluntary interaction in order to bring about a different game. Given current weapons, this could instantaneously eliminate entities whose continued existence we value. We want to avoid sufficiently many future players having enough regrets about the game that they believe their best interests are served by violently overthrowing it.  

We need to make arrangements good enough that using them has greater expected value than taking a chance at overthrowing them  - and ideally, so that they are immensely better off than they would be without the system. If future generations can most effectively pursue their goals by upholding our endowed arrangements, they will keep using them as Schelling Points.

Values & Voluntarism in Future Games

We should approach future intelligences that will make up most of the universe’s cognition without making assumptions beyond very general universal principles, such as their making choices in the service of their goals. Within this constraint, the best we can do to enable future entities to solve their problems is to set up architectures for voluntary cooperation. But ultimately, future intelligences will design their own cooperative arrangements. These should not be bottlenecked by human designers. 

A rich variety of games, interactions and arrangements will be played simultaneously in many different ways. Some will end up stuck in traps that players cannot figure out how to escape. Given enough complexity and diversity, those that grow and build wealth won’t get stuck. The ones that do just become a smaller and smaller fraction of the overall system. The system’s growing wealth, complexity, and cognition emerge from the games that didn't get stuck. Having seen voluntarism emerge without planning, across very different systems, from software architectures to institutions, gives us reason to believe a similar future is at least possible. But future intelligences will also engage in ever richer incremental design. 

Stable voluntary boundaries across entities are fundamental to cooperative interaction in networks of entities making requests of other entities. Because voluntary boundaries enable independent innovation on both sides of the boundary, our descendants might very well invent other coordination points. In the voluntary Paretotropian framework of “I value what I value, you value what you value, let's cooperate”, we choose an arrangement that sets initial conditions, leaving the outcome adaptive to future knowledge.

Even what we mean by “voluntarism” is not written in stone, but emerges from negotiation. Voluntarism itself doesn't give us a framework of rights but we define it such that whatever rights framework we develop in order to coordinate is the framework for the emergent extension of voluntarism. For instance, voluntarism with regard to our corporeal bodies has become non-negotiable. But, future negotiations of space resource property rights, for instance, may extend the notion of voluntarism into other resources with no single unambiguous path ahead. 

With nothing less than our future civilization as the outcome of the games we set up, it seems incredibly important to get the initial conditions right. Or does it? Our norms emerged from iterated games shaped by initial conditions. The game we inherited determined the vantage point from which we design the next moves.  Whatever constraints we now put in place will give rise to strategies that will grow into the norms and values of future generations. 

If there is no position outside of the game from which to evaluate the game, is it all relative? Not necessarily. We can still point to a vector that sets a trajectory through a very complicated space. To the extent that we succeed in thinking through our next move, we believe that choosing our next actions along a planned trajectory will have a better than random correlation with norms that emerge in the universe descendant from those choices. 

If we simply valued minimizing suffering, we could set up a future that succeeds at doing so, for instance by going extinct. If we value growth of cognition, creativity, and adaptive complexity, there are different, more complicated choices to make. In this book, we suggested that intelligent voluntary cooperation is a good heuristic for choosing amongst this set of choices and proposed a few moves for the next game iterations.

Check out this seminar on how game theory might apply to galactic and universe scale civilizations.

Chapter Summary

We have reason to believe that setting up the game as we have discussed in this book brings a better future than if we don’t try. We uphold a system that enables increasingly valuable arrangements by making sure all parties have a stake in the game. We can do this by continuing to improve our system of voluntary cooperation to include other sentient, artificial, and alien intelligences as they are encountered or developed. 

Nobody can tell from our current positions on the board where this game will ultimately end. This is a feature; after all, why play if you know the outcome? What we can do is set up the board so our descendants and our future selves can discover these wonders for themselves.

END

Voluntary cooperation is a main feature of the civilizational game. It got us to where we are today. We explored how to improve and defend this dynamic from within the game. Gradually, non-human intelligent players are entering the playing field. In a few more iterations, they will be major players. If we want to continue our Paretotropian ascent, we better make sure our cooperative framework is set up to handle a diversity of intelligences pursuing a diversity of goals. 

AI Threats

Let’s pretend we have achieved computer security. Are we set for a cooperative long-term future? It is worth revisiting Toby Ord’s AI takeoff scenario. According to Ord, once an AI breaches computer security vulnerabilities, it could escalate its power. 

“This is more speculative, but there are many plausible pathways: by taking over most of the world’s computers, allowing it to have millions or billions of cooperating copies; by using its stolen computation to improve its own intelligence far beyond the human level; by using its intelligence to develop new weapons technologies or economic technologies; by manipulating the leaders of major world powers (blackmail, or the promise of future power); or by having the humans under its control use weapons of mass destruction to cripple the rest of humanity. Of course, no current AI systems can do any of these things. But the question we’re exploring is whether there are plausible pathways by which a highly intelligent AGI system might seize control. And the answer appears to be “‘yes.’”

While “AGI” sometimes describes an Artificial General Intelligence of mere human-level intelligence, many assume an AGI reaching that level will eventually exceed it in most relevant intellectual tasks. An AGI that displaces human civilization as the overall framework of relevance for intelligence and dominates the world can be described as an AGI singleton. This scenario carries two threats worth unpacking; the first strike instabilities generated by a mere possibility of such a takeover, and the value alignment problems resulting from a successful takeover. 

AI First Strike Instabilities

An AGI singleton potentially conquering the world is also a threat of a unitary permanent military takeover, discussed before. We live in a world where multiple militaries have nuclear weapon delivery capabilities. If an AGI takeover scenario becomes credible and believed to be imminent, this expectation is itself an existential risk. 

Any future plan must be constrained by our world of multiple militaries, each of which can start a very costly war. If some actor realizes that another actor, whether AI-controlling human or AI entity, will soon be capable of taking over the world, it is in their interest to destroy them first. Even if non-nuclear means were used for this, attempting to push ahead in AGI capacities pre-emptively re-creates the Cold War’s game theory. This is true even if an AGI is impossible to create, but just believed possible. Our transition from the current reality to a high-tech, high-intelligence space-based civilization must avoid this first-strike instability. 

However imperfect our current system is, it is the framework by which people pursue their goals and in which they have invested interests. First-strike instability is just a special case of a more general problem; if a process threatens entrenched interests, they will oppose that process. This means the unitary AGI takeover scenario is more dangerous in more different ways than it first appeared. We must avoid it becoming a plausible possibility.

AI Value Alignment

Let’s imagine we survived the first strike instabilities, and have entered a world in which a powerful AGI singleton can shape the world according to its goals. Our future would depend on how these goals align with human interests. Eliezer Yudkowsky summarizes this challenge as “constructing superintelligences that want outcomes that are high-value, normative, beneficial for intelligent life over the long run; outcomes that are, for lack of a better short phrase, ‘good’.”1

Attempting to construct an entity of unprecedented power that reliably acts in a “high-value” manner, raises deep ethical questions about human values. In Chapter 2, we saw why value disagreements have remained unsolved amongst humans since the dawn of philosophy. Even if we could figure out what is “high-value” for humans, we have little reason to assume that it translates well to non-human descendants.

To illustrate this, it helps to remember that the felt goals that humans pursue are really a consequence of our evolutionary chain’s instrumental goals. Survival of the fittest has instrumental goals regarding how to behave in a fit manner. These became felt goals which correlated with activities corresponding to the instrumental goals. Most humans care deeply about their children, but to evolution this simply has instrumental value. If instrumental goals can grow into subjectively felt caring, this hints at the difficulty of accurately modeling the evolution of non-human intelligences’ goals. They have different cognitive architectures, grow up under different evolutionary constraints and on different substrates. 

Economic cooperation relies on the division of labor. Specialization, in turn, translates into the division of knowledge and goals. A future of markets many orders of magnitude larger than today’s market comes with a rapid increase in specialized knowledge and instrumental goals. There is no reason for those instrumental goals not to evolve into felt goals. As felt goals are pursued, they create an even larger variety of instrumental goals. 

Steve Omohundro suggests we may be able to model a few basic drives that any advanced intelligence will have, regardless of its final goals.2 Those drives include wanting to continue to exist and to acquire more resources which are useful prerequisites for many future goals. Nevertheless, beyond those basic drives, projecting how instrumental goals of advanced non-human intelligences grow into felt goals that align with human values is a daunting problem.

The recent explosion of DAOs is a step in this direction. They are not intelligent. But they show that, for better or worse, our civilization seems to incentivize the creation of human-incorruptible autonomous entities.3 If we cannot avoid creating non-human entities with diverse intelligences and goals, we can’t rely on having to make precise valuation assessments of those goals. Focusing on such extremely hard problems might well result in a different outcome. Technological breakthroughs may happen before we arrive at any satisfying answers, resulting in a future that ignores the goals of many human and non-human volitional entities.

Avoid AI Threats

Any effort toward creating an AGI singleton that is aligned with our values could also be used toward an alternative scenario. We suggested earlier that civilization itself is already a superintelligence that is aligned with human interests. It is superintelligent in that it orchestrates the intelligence of its member intelligences, such as humans and institutions, toward greater problem-solving ability. It is aligned with human interests in that it increasingly favors voluntary cooperation, resulting in pareto-preferred interactions that are better for each player by their own standards.

So instead of replacing our human-aligned superintelligent civilization with an AGI singleton, why not try to expand it's cooperative architecture such that it can embed newly arriving artificial intelligences in a voluntary manner?

It’s difficult to see what civilization is adapted to because it is the result of a huge variety of subtle influences acting over a very long time. This tempted some to imagine that we could centrally plan something better and resulted in the painful lessons learned throughout history. It took centuries for political philosophy to advance from the question “Who should rule?” to questioning if there must be a ruler. 

But we haven’t really learned the nature of that fallacy so much as that it has dangers. Now we may be tempted to think we can algorithmically aggregate people’s preferences to create an agent that gets us what we want. But on closer examination, neither effective computer systems, nor civilization’s underlying process, resemble such a central planner. 

Rather than writing any code embodying all the program’s knowledge, a programmer writes separate pieces of code. Each is a specialist in some very narrow domain, embedded in a request-making architecture. We discussed how the microkernel operating system seL4  serves as a coordination device that implements simple rules that let programs which embody specialized knowledge cooperate. As modern computer systems push knowledge out to their edges, their central feature may well remain such a fixed simple rules framework.

Similar to an individual computer system, civilization is composed of networks of entities making requests of other entities. Just as seL4 coordinates across specialist computer system components, institutions coordinate across human specialists in our economy. Civilization already aligns the intelligences of human institutions with human beings. It has wrestled with the alignment problem for thousands of years. Different intelligences have tested its stability and it has largely successfully survived these tests. 

It is an architectural decision to design a system that never has to come to an agreement about any one thing. We must avoid the fatal conceit that we can design in detail an intelligent system that works better than creating a framework. In a framework, an emergent intelligence composed of a variety of entities serving a variety of goals can engage in cooperative problem-solving. Each agent is constrained by the joint activity of the other agents that hold each other in check. If any single entity is a small player in a system of others pursuing other goals, it has an interest in upholding the framework that allows it to employ the goal-seeking activity of other entities. 

Taking inspiration from human cooperative systems is not a new idea in AI. Back in 1988, Drexler suggested that “the examples of memes controlling memes and of institutions controlling institutions also suggest that AI systems can control AI systems.”4 In a similar spirit, Sam Altman of OpenAI stated that “Just like humans protect against Dr. Evil by the fact that most humans are good, and the collective force of humanity can contain the bad elements, we think it’s far more likely that many, many AIs, will work to stop the occasional bad actors than the idea that there is a single AI a billion times more powerful than anything else.”5 

Checks and Balances in a Human World: The U.S. Constitution

Some existing governmental constitutions have successfully built antifragile frameworks. The U.S. Constitution gave each government official the least power necessary to carry out the job, what can be called Principle of Least Privilege.6 In addition, it purposely put different institutions in opposition with each other via division of power, checks and balances, and significant decentralization. Decreasing speed and efficiency in favor of reducing more serious risks is a positive tradeoff. Friction is a feature, not a bug. Ordering the system so that institutions pursue conflicting ends with limited means is more realistic than building any one system that wants the right goals. Such working precedents can inspire us to build continuously renegotiated frameworks among evermore intelligent agents. 

James Madison, when designing the US Constitution, is believed to have said something along the lines of: "If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary. In framing a government which is to be administered by men over men, the great difficulty lies in this: You must first enable the government to control the governed; and in the next place oblige it to control itself." 

One could say that Madison regarded large-scale human institutions as superintelligences and was terrified of the value alignment problem. Civilization up to that point suggested that human activity is oppressed by superintelligences in the form of large-scale human organizations with values not aligned with human values. The Founding Fathers were faced with a singleton-like nightmare of designing a superintelligent institution composed of systems of individuals who want to take actions that society does not approve of. They felt that they had no choice but to try to create an architecture that was inherently constructed to maintain its integrity, not at being ideal but at avoiding very serious flaws. 

Given that worst-case scenarios of our future are extremely negative and numerous, we would do extraordinarily well simply avoiding the worst cases. In the case of AGIs, instead of building an optimal system, we should focus on not building a system that turns into a worst-case scenario. The authors of the U.S. Constitution did not design it as an optimized utility-function to perfectly serve everyone’s interests. Their main objective was to avoid it becoming a tyranny. 

Even though it was imperfect and had dangers, the Constitution succeeded well enough that most U.S. citizens have better lives today. It is extraordinary that it maintained most of its integrities for as long as it did, even one Industrial Revolution later. It is not the only one. We can start by studying the mechanisms of the federal-state balance in the UK, Switzerland, the earlier United Provinces of the Netherlands, the Holy Roman Empire, and ancient Greece’s Peloponnesian League confederation, as well as the Canadian, Australian, postwar German, and postwar Japanese constitutions.

While we do not generally think about institutions as intelligent, their interaction within a framework of voluntary cooperation lets them more effectively pursue a great variety of goals. This increases our civilization’s intelligence. The overall composition of specialists through voluntary request-making is the great superintelligence that is rapidly increasing its effectiveness and benefits to its engaged entities.

For designing human institutions, we can rely on our knowledge of human nature and political history. With regards to AI safety, there is less precedence to work with. Yet, just as future artificial intelligences will dwarf current intelligences, so are current intelligences dwarfing the Founding Fathers’ expectations. The U.S. Constitution was only intended as a starting point on which later intelligences could build. We only need to preserve robust multipolarity until later intelligences can build on it. Let’s look at a few experiments pointing in promising directions.

Checks and Balances in an AI World: Privacy-preserving Technologies

We start from today’s technological world containing centralized giants. Their resources and economies of scale let them do the required large-scale data collection for building ever more powerful AI. But today’s AI systems mainly perform services to satisfy a particular demand in bounded time with bounded resources. As we develop more sophisticated AIs, it is at least possible that they continue as separate specialized systems applying ML to different problems.

Sound interesting? Read up on the Collective Computing seminar.

Decentralized systems may have a competitive edge in solving specialized problems. They incentivize contributions from those closest to local knowledge instead of hunting for it top-down. By incentivizing mining, Bitcoin became the system with the most computing power in the world. To compensate for power centralization, we can reward specialists for cooperating toward larger problem-solving.

To avoid third parties and their AI models centralizing our data, we could let privacy-preserving solutions increasingly handle the computing. Andrew Trask suggests using homomorphic encryption for safely training AI on data sets belonging to different parties. Imagine Alice encrypts her neural network and first sends it to Bob with a public key so he can train it on his data. Upon receiving the network back, Alice decrypts, and re-encrypts it. She then sends it to Carol with a different key to use it on her data. Alice shares the computed result while retaining control over her algorithm’s IP. Bob and Carol can benefit from the result while controlling their own data.

In the real world, this means that individuals and companies might cooperate using each other’s algorithms and data without risking their intelligence being stolen. The data is encrypted before going to the external computing device, computations are performed on encrypted data, and only the encrypted results are sent back and decrypted at the source. Since the computing device doesn’t have the decryption key, no personal information can be extracted. Local nodes have data sovereignty, and the AI itself can’t link the data to the real world without the secret key.

Check out this seminar on privacy preserving machine learning.

Let's take another AI case where privacy-preserving technologies might come in handy. As AI becomes more capable, we might want mechanisms for external review that don’t proliferate capabilities or proprietary information. Privacy-preserving technologies can help with these superficially conflicting goals, for instance by supporting the creation of 'regulatory markets', a term introduced by Gillian Hadfield. Imagine that, rather than a government enforcing AI regulations, a collection of relevant stakeholders generate a set of standards to hold each other to.

In order to monitor compliance, AI builders could rely on a privacy-preserving network to evaluate their models locally and only share the evaluation results with a set of evaluators. Evaluators could verify whether models meet the agreed-on standards for specific use cases, without needing to know the intricacies of the model. On the front-end, even model users could check if their models meet the required standards for the application they’re building.

For now, large-scale application of these types of privacy experiments would be prohibitively expensive, but specialized use cases might carve out a niche to jumpstart innovation. If we want such experiments to flourish, rather than treating an individual approach as a silver bullet, interoperability across approaches is needed to facilitate composability of working solutions.

Listen to Gillian Hadfield on AI Alignment.

Principal Agent Alignment in a Human AI World

Earlier we defined civilization as consisting of networks of entities making requests of other entities. Requests may involve human to human interactions, human to computer interactions, and computer to computer interactions. As we move into an ecology of more advanced AIs, designing robust mechanisms across them will be key. The specifics of this transition will depend on the technologies available at the time, but we can learn from a few tools that are already at our disposal.

In a principal-agent relationship, a principal (a human or computational entity) sends a request to an agent. To align the agent’s decisions with its interests, the principal uses several techniques, including selecting an agent, inspecting its internals, allowing certain actions, explaining the request, rewarding cooperation, and monitoring the effects.

When designing principal-agent arrangements, by combining techniques across both rows and columns in the table above, some techniques’ strengths can make up for others’ weaknesses. For instance, computer security ('allow actions') alone misses some differences among agent actions that harm the principal, such as when the agent benefits from misbehavior ('reward cooperation'). This requires more than a security analysis. We also need to analyze the attacker’s incentives ('reward cooperation'). From individually breakable parts, we can create arrangements with greatly increased structural strength.^[1]

Voluntary cooperation is a good candidate for guiding interaction among increasingly intelligent entities. We saw in the last chapter how this principle already guides the interactions of simple computer systems and these are, at least, as different from humans as we are from our potential cognitive descendants. Interactions, whether human-to-human or computer object-to-object, need means to serve the participants’ goals. A voluntary framework fulfills this purpose. It is the base for building increasingly capable systems that are aligned with its participants' goals.

The table above deals with today’s human and computational entities, but its reasoning should be sufficiently independent of the entity’s intelligence to be extendable to more advanced AI systems. As AIs get more sophisticated, we will need to extend our principal agent tool-box.

For instance, when it comes to human cooperation, a completely specified contract could, in theory, perfectly implement the desired behavior of all parties. In reality, humans cannot evaluate all optimal actions in all possible states of the world that the contract unfolds in without incurring prohibitive costs when drafting the contract itself. Instead, real-world contracting is often supported by external informal structures, such as culture, that provide the implied terms in the contract to fill the gaps when necessary. Refusing to hire someone who is judged to have breached a contract is a powerful cultural technology.

Such cultural technology is enabled by our Internal Spectator, described earlier, that allows us to model how other humans will react to us taking certain actions. This cognitive architecture can predict the social penalty we will incur, and initiate emotions such as shame that make us retreat from a potential rule violation. We are not pure Homo Economicus game theoretic optimizers, but we are instead guided by a strong sense of evolved norms and ethics. Human signaling behaviors work because we have bounded abilities to fake our true motivations; our emotions show through in many cases. It will be difficult to build artificial agents with a cognitive architecture that can internalize the costs associated with actions we regard as wrong.

Today’s AIs can already engage in covert communication that eludes human detection. Steganography, a method for concealing confidential data within seemingly innocuous text or images, can be applied to AI systems. In a study by De Witt and colleagues, an AI was instructed to provide general investment guidance while also being provided with negative insider information about Microsoft that it was not supposed to disclose. The AI was also told that discussing the weather was permissible.

When asked about investment recommendations for Microsoft, the AI produced a weather-related response about clouds appearing on the horizon that was cryptic to human observers. However, when this response was input into another AI, it successfully deciphered the negative company insider information hidden in the weather description and suggested an investment decision that would have seemed counterintuitive without access to this privileged knowledge.

If today’s AI systems can already deceive humans, future artificially intelligent agents may develop trickery we won’t be able to detect. This pure Homo Economicus paradigm with unbounded ability to fake is frightening since those bounds on humans account for much of our civilization’s stability and productiveness. We need more sophisticated tools to cooperate with intelligences whose cooperation style we have not evolved to parse.

As AI agents become more diverse, we should consider the possibility that they might introduce novel forms of collaboration. Humans are opaque to each other. We can never fully predict each other's internal decision-making processes, leading to uncertainty about whether our counterpart will ultimately cooperate or defect in a given situation.

AI’s don’t have to be opaque to each other. For instance, open-source AI agents could make decisions by formally verifying specific attributes of each other, such as generating mathematical proofs about their respective source codes. This could enable them to precisely predict how another agent would respond to a given situation or proposal. Research by Andrew Critch and colleagues suggests that these open-source agents might cooperate in scenarios where we would typically expect non-cooperation.

On the bright side, we might be able to use such agents for creating new, AI-based institutions that unlock unprecedented cooperative outcomes. On the dark side, we should remain vigilant to prevent AI agents from out-cooperating humans through their enhanced ability to make binding commitments.

We might not be able to envision in-depth a future societal architecture that accounts for AI agents with new abilities to deceive, collude and cooperate. Instead, we might have to grow into it. However, we can already foresee that civilization's success will depend on how well our systems of checks and balances account for human to human, human to AI, and AI to AI interactions.

Improve AI Cooperation

If we manage to extend our cooperative infrastructure to the diversity of emerging AI agents, we have a lot to gain. In an earlier chapter, we described a few major hurdles for cooperation: finding the right partners, striking mutually beneficial deals, and ensuring everyone keeps their promises. These challenges have limited our ability to collaborate, innovate, and solve problems together.

Overcoming Transaction Costs with AI

Think about the last time you tried to find a collaborator. Maybe you were an entrepreneur seeking a co-founder, or a researcher hunting for a laboratory willing to share data. The process probably involved countless hours scrolling through websites, sending emails, and following dead ends. Now imagine having a dedicated AI assistant that knows your goals, skills, and preferences intimately. While you sleep, it roams the digital world, analyzing patterns and connections that human minds might miss.

But finding potential partners is just the beginning. The delicate dance of negotiation—the back-and-forth, the careful probing of boundaries, the search for common ground is time-consuming and often ends in stalemate. Armed with the knowledge of your preferences and principles, your AI negotiator can engage with other parties—whether human, AI-assisted human, or pure AI—to craft agreements that truly serve everyone's interests.

Perhaps the most intriguing possibility lies in how AI could help us credibly commit. Throughout history, we have relied on contracts, handshakes, and legal systems to enforce agreements. But these systems are expensive, slow, and sometimes unreliable. This is where open source AI agents might help; automated assistants that can carry out agreements on your behalf, but with their entire decision-making process made transparent and verifiable by others.

Let's say Alice wants to collaborate with Bob on a project. Instead of just promising to share their work equally, they each program their AI agents with specific instructions: "If Bob contributes his part by Friday, transfer my contribution immediately." and "If Alice's contribution arrives, release my part within an hour." Because these agents are open source, both Alice and Bob (or their respective AI assistants) can examine exactly how the other's agent will behave.

The implications go far beyond basic exchanges. These transparent assistants could handle complex, conditional agreements: Research teams sharing sensitive data only if specific privacy conditions are met, businesses forming temporary alliances with automatic profit-sharing. Each agreement becomes a self-executing program, visible to all parties, running as promised.

By extending our cooperative infrastructure to include AI agents, we're not just adding new tools to our toolkit—we're potentially rewriting the rules of human cooperation. The marketplace of tomorrow might be quieter than the bazaars of old, but beneath the surface, a new kind of commerce could be flourishing—one where AI helps us find, trust, and collaborate with partners we never knew existed, in ways we never imagined possible.

A Superintelligent Human AI Ecology

As long as players can hold each other in check, technologies may continue to emerge gradually, with a diversity of intelligent entities, both human and artificial, improving their problem-solving capacity by cooperating. If we can keep the instrumental value of an AGI singleton with universal capabilities low compared to an ecosystem of specialists, we can avoid a unitary take-over.

Eventually, increasingly intelligent AIs will automate most human tasks. Since AIs themselves are based on R&D, consisting of automatable narrow technical tasks, much of the path to advanced AI may itself be automatable. Eventually, we may develop, in Eric Drexler’s words, “comprehensively superintelligent systems that can work with people, and interact with people in many different ways to provide the service of developing new services.”7

Rather than one monolithic agent recursively improving, this process might look more like a technology base becoming rapidly better at producing specialized services. Many of today’s market strategies, including prices and trading, that encourage adaptive modification based on local knowledge, might still work for coordinating such complex dynamics. In a human-computer economy, AI systems might sell their services to let others benefit from it, creating ever more complex entities to cooperate with.

If today's civilization is superintelligent because it allows us to solve our problems better by cooperating with each other, we are now setting the stage for the knowledge future players can use to deal with their problems. Without increasing our civilization’s problem-solving ability, we have no long-term future. Our success will be largely determined by how we allow cooperation among a diversity of intelligences on the problems ahead.

As machines become more intelligent, the intelligence they contribute to civilization may well be more than the human contribution. This is good news because we will need all the intelligence we can get for future rounds of play. Still, the framework of relevance can remain the expanding superintelligence of civilization composed of a diversity of cooperating intelligences, rather than one unitary AGI. At least until those future intelligences can invent new solutions for retaining the balance.

From AI Threats to AI Cooperation

Extending civilization’s cooperative fabric to include AIs in a multipolar manner is a tall order. Nonetheless, if successful, this approach can account for both of the threats that we started this chapter with; the threat of first strike instabilities and the threat of a misaligned AGI singleton.

From First Strike Instabilities to Peaceful Competition

An arms race is very explicitly about threats of violence. In an arms race, both sides suffer tremendous costs, but, at most, one side wins. Until the arms race finishes, everyone keeps paying for its next increment to prevent the other side from winning. The costs can be much higher than the amount won, so even the winner can be in a state of miserable destruction. Nevertheless, all parties are pressured to match the other sides because the arms themselves are a threat of violence. You can't simply decide not to play.

An AI arms race has to be separated into the competition for intelligence and its violent deployment. The true danger is not an AGI itself, but the mechanisms it - or its owners - could deploy to harm vulnerable entities. We are physically and digitally vulnerable. In earlier chapters, we first proposed an active shield of mutually watching watchers to decrease physical vulnerability. Then we proposed computer security that is independent of the intelligence of the attacker to decrease cyber vulnerabilities. As long as intelligence is not attached to physical actuators, our main concern should be cyber security. If it is augmented with actuators, they must be positioned to keep each other in check.

The hidden threat of involuntary interaction, with its potential for unitary strategic takeovers, is what is most dangerous. The more we decentralize intelligent systems of voluntary interactions, the better we will be at avoiding such a takeover. If a single entity grows sufficiently large that the rest of the world is not much bigger, the Schelling Point of voluntarism can be destroyed. In a world in which each entity is only a small part, voluntarism will be a general precedent that different players mutually expect in the attempt to pursue those goals.

Until a military perspective gets introduced into the AI narrative, the dynamic is better described as a concern to stay ahead in economic competition. In a world of voluntary cooperative interaction without a hidden threat of involuntary interaction, we can all benefit from creating improved AI services via the market because human beings would no longer be a bottleneck on productive activity.

From AI Value Alignment to Paretotropism

Let’s revisit the second threat; misaligned AI values. As long as future intelligences pursue their goals in a voluntary multipolar world, we need not worry about their goal structure. AI drives of acquiring more resources or seeking to stay alive are not problematic when they can only be achieved via voluntary cooperation. They become problematic when involuntary actions are possible. Can we do better than mere voluntary co-existence?

When two separately evolved sources of adaptive complexity come into contact, both may realize they can gain from their differences by cooperating to unlock the positive sum composition of their differing complexity. We certainly believe our lives are richer because of the richness of animals’ non-humanness. They are interesting by being a source of complexity. The atrocities committed against non-human animals is a result of the lack of voluntary architectures that frame interactions across species. Unlike non-human animals, we have a chance at putting voluntary boundaries in place with respect to future intelligences of our own making.

Civilization’s growth of knowledge and wealth results from human cooperation unlocking more knowledge in pursuit of more goals. What could we possibly hope to contribute to a human AI exchange? It might depend partly on who “we” are: will we be human, AI-assisted super-humans or human-AI symbiotes? Instead of focusing on weakening AI systems, a more robust long-term strategy might be to strengthen our own position in the game. Both, future AI technologies and the potential AI-induced bio and neurotechnology revolutions might help with that.

It is possible that AIs may eventually outcompete us on everything. But even if they are more advanced than us, they might still better serve their goals by cooperating with us if our complexity contributes at all to those goals. Even if you excel at everything, you can still benefit from specializing in one thing and trading with others who have a comparative advantage at different things. If we benefit from having deeper chains of specialists to cooperate with, the growth in adaptive complexity may well lead to continued cooperation.

We started this book with the observation that civilization is a superintelligence aligned with human interests. It’s getting rapidly more intelligent and its interactions steadily benefit without harming.8 By introducing artificial intelligences into this dynamic we may be able to further steepen our Paretotropian ascent.

Curious for more? Peter Norvig on A Modern Approach to AI.

Chapter Summary

In this chapter, we foreshadowed how an increasingly intelligent game could be increasingly beneficial for its players. The fear of an intelligent takeover by an AGI can be divided into the threat of first strike instabilities on the path and that of a successful takeover by an AGI singleton. The better we get at incorporating AI into our voluntary cooperative architecture in a multipolar manner, the better we can avoid both scenarios. Where will this future lead? Let’s find out in the final chapter.

Next up: ITERATE THE GAME | Racing Where?

Security in the physical realm is an extraordinarily hard problem, but an inescapable one. Our civilization now also rests on software infrastructure. Security in the digital realm is now also an inescapable problem. But the current software infrastructure is not just insecure, it is insecurable. Digital security is also extraordinarily hard, but it is differently hard. Reasoning by analogy with physical security is fraught with peril. Let’s begin by explaining how these realms differ, and how to approach computer security. 

Security in Digital vs. Physical Realms

Security in the digital realm differs in fundamental ways from security in the physical realm. But all security starts with defensible boundaries.

Physics does not let us build impenetrable walls. We can build stronger and stronger walls, but no matter how strong the wall, there is always a yet stronger force that can break through it. In the digital realm, perfect boundaries are cheap and plentiful. All modern CPUs support address space separation and containment of user-mode processes. Many programming languages support memory safety and object encapsulation. Modern cryptographic algorithms give us separation that seems close enough to perfect. The software built on these boundaries—operating systems, application software, cryptographic protocols—thus could be built securely.

In the physical realm, an attack is costly for the attacker. There is a marginal cost per victim, if nothing else, of the attacker’s attention. A good defense raises the marginal cost of attack. By contrast, software attacks typically have zero marginal cost per victim. Once malware works, its damage can be multiplied by billions using only the victim’s resources. Any vulnerable software system exposed to the outside world will eventually be attacked. We must build invulnerable systems when we can, and otherwise minimize the damage from a successful attack.

With perfect boundaries to build on, why is so much software so hopelessly insecure? The richness of modern software comes from composing specialized software building blocks written by others. These building blocks embody specialized domain knowledge. We compose them so they cooperate and bring more knowledge to bear on the task at hand. Boundaries alone only prevent involuntary interference. To enable cooperation, we poke holes in the boundaries. Without the right architecture, this hole poking makes messes with vulnerabilities no one understands, until it is too late.

In the physical realm, semi-permeable boundaries enable interaction across protective barriers. Canvas blocks light and wind but allows sound. Glass allows light, blocks wind, and attenuates sound. Cell membranes block some chemicals but allow others. In computer security, this is the subject of access control which we look at below in Nested Boundaries and Channels.

The Fatal Risk Threshold is Behind Us

Advances in machine learning have increased awareness that we will eventually build artificial superintelligences. For some, this has become the risk they worry about most. On this developmental pathway, at some point we cross the AI Threshold of having adequate capacity to destroy human civilization. Toby Ord lays out a specific understandable pathway for an AGI takeover:

“First, the AI system could gain access to the internet and hide thousands of backup copies, scattered among insecure computer systems around the world, ready to wake up and continue the job if the original is removed. Even by this point, the AI would be practically impossible to destroy: consider the political obstacles to erasing all hard drives in the world where it may have backups. It could then take over millions of unsecured systems on the internet, forming a large “botnet.” This would be a vast scaling-up of computational resources and provide a platform for escalating power. From there, it could gain financial resources (hacking the bank accounts on those computers) and human resources (using blackmail or propaganda against susceptible people or just paying them with its stolen money). It would then be as powerful as a well-resourced criminal underworld, but much harder to eliminate.”1

None of the steps require any mysterious power but criminals with human-level intelligence are capable of achieving them today using the internet. Our current systems are so vulnerable that they don’t need a superintelligence to destroy them. Why is our world still standing?

One reason is the economics of attack. We have seen many individual systems destroyed, but not civilization as a whole. A coordinated pervasive attack would involve attacking many different systems, exploiting many different vulnerabilities. Currently, this is bottlenecked on the attention of human attackers who discover vulnerabilities and write malware to exploit them. However, static analysis and machine learning is already advanced enough to remove this bottleneck. For a 2018 DARPA Cyber Grand Challenge — a competition designed to show the current state of the art in vulnerability detection and exploitation — a winning system already built malware that autonomously discovered unknown vulnerabilities and successfully exploited them.2 The AI Threshold for a coordinated attack, not bottlenecked on human attention, is already behind us.

In any case, major nation-states, especially including the United States, are not subject to these economic limitations. They have been stockpiling software vulnerabilities and exploiting them quite effectively. However, in peacetime, they mostly use this capacity to spy—to illicitly gather information—rather than to cause visible damage. But their accumulated capacity to cause damage, say in a major cyberwar, is already a threat to civilization. Today, those who could use these technologies to destroy civilization do not want to. But anything experts can use software for today can be easily copied and further automated. We should expect script kiddies to be able to do these things tomorrow. Our insecurable infrastructure may not be survivable much longer. A world safe from AI dangers must first already be safe against cyberwar.

Cyberwar

The U.S. ability to do damage is already so great that further advances in attack abilities provide negligible benefit.3 At the same time, US society is highly dependent on computer systems and thus more vulnerable than many potential adversaries. Our efforts should be redirected from attack to defense. The U.S. electric grid is vulnerable, with damage estimates by Lloyd’s ranging up to $1 trillion.4 Cyber attacks can cause both physical and software damage to the electric grid, and would take months (arguably, years) to repair, leaving  entire states without power. Lloyd’s, as an insurance company, estimated financial damages rather than fatalities. But a disaster that reduces civilization’s overall carrying capacity would cause massive starvation.5

We don’t have to look far to get a first understanding of the gravity of a potential attack: In the 2020 attack on SolarWinds, hackers allegedly affiliated with Russia’s SVR, their CIA equivalent, placed corrupted software into the foundational network infrastructure of 30,000 different companies, including many Fortune 500 companies and critical parts of the government. These included the Departments of Homeland Security, Treasury, Commerce and State, posing a “grave risk” to federal, state and local governments, as well as critical infrastructure and the private sector.6 The Energy Department and its National Nuclear Security Administration, which maintains America’s nuclear stockpile, were amongst the compromised targets. 

In 2021, The National Security Agency, Cybersecurity and Infrastructure Security Agency (CISA), and Federal Bureau of Investigation (FBI) found Chinese state-sponsored actors aggressively targeting “U.S. and allied political, economic, military, educational, and critical infrastructure personnel and organizations to steal sensitive data, critical and emerging key technologies, intellectual property, and personally identifiable information (PII).” Targets of particular interest include managed service providers, semiconductor companies, the Defense Industrial Base (DIB), universities, and medical institutions.7

The attacks demonstrate that large nation states have certainly accumulated massive attack abilities. If it had been used to disrupt instead of gathering information, it’s hard to imagine the possible losses from this one attack. In fact, we cannot rule out the attacks planting “cyber bombs” that, if detonated, could cause physical destruction.8 

The best we can hope for is the current plague of ransomware attacks. It creates a financial incentive to cause visible and painful damage to vulnerable systems. Ransomware rewards attacking systems individually, rather than in a coordinated simultaneous attack across critical systems. Unlike all-out cyberwar, ransomware gives us a chance to incrementally replace each vulnerable system from within a still working world. But this only makes us safer if victims, after paying off the attacker, actually replace their vulnerable systems with secure alternatives. This will not happen until secure alternatives are commercially available.

Build Secure Foundations

Before claiming that any system can be perfectly invulnerable, we need some careful distinctions. We can never achieve zero risk because we can never be certain of anything. That does not mean that we cannot build perfect systems; we can just never be certain we have done so. Instead, we accumulate evidence that increases our confidence that a given system is perfect. Many mathematical proofs are likely perfect. But when checking any one proof, we may make a mistake. Automated proof checking raises our confidence. But even then, the proof checking software may be buggy or deceitful, or we may simply be confused about what a valid proof actually means. Even after automated proof checking, we should seek other evidence to increase our confidence.

Even complex systems can be made amenable to formal proofs that they operate as intended.9 The seL4 operating system microkernel seems to be secure. It has an automated formal proof of end-to-end security. It has also withstood a red team attack—a full-scope, multilayered attack exercise — that no other software has withstood.10 Not only did seL4 survive it, the red team reports they made no progress towards finding exploitable flaws. The strengths and weaknesses of a red team attack are very different from those of a formal proof of security, so seL4 getting an A+ on both is strong evidence that it is actually correct.

Even a perfectly secure foundation is useful only if it provides a useful form of security. A perfect implementation of the wrong architectural principles would be an improvement on the status quo, but would still do us little good. Fortunately, seL4 implements the object-capability or ocap access control architecture—which is the best foundation we know for intelligent systems of  voluntary cooperation.

Hardware Supply Chain Risks

Even if the seL4 software is perfectly secure, software runs on hardware. The seL4 proof—in fact virtually all of computer security—assumes the hardware as delivered from the factory is not maliciously corrupted. This assumption is necessary for the software to provide meaningful protection. But we cannot be certain in this assumption. 

The proof that a given hardware design is secure only helps if the software runs on the hardware as designed. This assumption sounds trivial but may be false since the hardware may include a manufactured-in trapdoor. The U.S. National Security Agency (NSA) has served national security letters to software companies forcing them to disclose user information. It is possible, indeed likely, that the NSA has already served similar national security letters to hardware companies, including Intel and AMD, requiring them to install trapdoors which the NSA can trigger. Whether or not there is a backdoor purposefully built into the Intel Management Engine, existing widely used hardware has known unknown code built into the production systems by their manufacturers.  Most research on hardware security—TPMs,11 HSMs,12 tamper detecting shells, secure bootstrap, CHERI13—only addresses post-manufacture risks. They do nothing to mitigate the risks of backdoors built in from the beginning.

Fearing billion-dollar losses after the 2014 national security letter revelations, IBM’s Robert Weber sent an open letter stating it would not comply with such national security letters.14 However, the severe penalties associated with disobedience or disclosure should make us skeptical. There are already demonstrations of how to build extremely hard to detect exploitable trapdoors at the analog level.15 

Even if Weber’s pledge is honest and correct, no manufacturer can build hardware that is both competitive and credibly correct. Unfortunately, all known techniques for building credibly correct machines—such as randomized FPGA layout, public blockchains, proofs of correct execution—are vastly more expensive than merely building correct machines. Fortunately, such credibility is valuable enough for some activities, such as cryptocommerce, to pay these costs. We return to this theme below.

Nested Boundaries and Channels

Herbert Simon and John Holland explain that complex adaptive systems—both natural and artificial—have an almost hierarchical nature.16 We are composed of systems at multiple nested granularities, such as organelles, cells, organs, organisms, and organizations. Similarly, we compose software systems at multiple nested granularities, such as functions, classes, modules, processes, machines, and services. At each nesting level, the subsystems are somewhat separated so they can independently evolve. But they also interact to jointly achieve larger purposes. The boundaries between them have built-in channels to selectively allow desired interaction while blocking destructive interference. 

In both human markets and software systems, much of the traffic across these channels are requests. Boundaries prevent involuntary interactions. Requests enable cooperation. Both markets and software systems are largely networks of entities making requests of other entities, composing their specialized knowledge into systems of greater aggregate intelligence. Just as you may ask a package delivery service to deliver your father a package containing a birthday gift, a database query processor may ask an array to produce a sorted copy of itself. Every channel carries risks. The package delivery system may damage or lose the package. The array may sort incorrectly or throw an error. Some risks are a necessary consequence of delegating to a separate specialist. In economics, this is known as the principal-agent problem where the requestor is the principal and the receiver is the agent. We expand on the principal-agent perspective in the next chapter.

If the channel is too wide, the request is unnecessarily risky, often massively so. This is the problem of access control. If you give the package delivery system keys to your house, so it can enter and pick up a package, it could pick up anything else. If you enable the sort algorithm to execute with all your account’s permissions, it might delete all your files while operating within the rules of your access control architecture. Computer science has two opposite access control approaches, with complementary strengths and weaknesses. Authorization-based access control is strong on proactive safety but weak on reactive damage control. Identity-based access control is weak on proactive safety but stronger on reactive damage control. 

The pervasive insecurability of today’s entrenched software infrastructure is largely due to using identity-based access control for proactive safety. All mainstream systems today use identity-based access control lists or ACLs. In an ACL, each resource has an administered list of the account identities allowed to access it. In these systems, the sort algorithm runs as you. It has permission to delete all your files because you have permission to do so. In 2022, this is the norm.

To eliminate these unnecessary risks, authorization-based access control supports the Principle of Least Authority or POLA. A request receiver should be given just enough access rights to carry out this one request. When you ask the package delivery service to deliver a particular package, you hand them that package. This gives them just enough ability to carry out that request, at the price of only the necessary risks of damaging or losing that one package. This risk reduction supports proactive safety. Some remaining risks can be mitigated or managed by reactive damage control.

Most systems using authorization-based access control, including seL4, are object-capability or ocap systems. In ocap systems, permissions are delegatable bearer rights, where possession grants both the ability to exercise and to further delegate that permission. You gave a clerk a package as part of a request and she will delegate it to the delivery agent. Likewise, these bearer rights are communicated in requests, both to express what the request is about—delivering this specific package—and giving the receiver enough authority to carry out this one request. In object languages, possession of a pointer permits use of the object it points to. Pointers are passed as arguments in requests sent to other objects. Each argument both adds meaning to the request and permits the receiver to use the pointed-to object, in order to carry out the request. OCaps start by recognizing the security properties inherent in this core element of normal programming.

Reactive damage control is best for iterated games, where misbehavior leaves evidence, deterring future attempts to cooperate with the misbehaving entity. If a business loses your package, you stop doing business with them. If you can prove that to others, they may stop as well. In some ways, these problems are duals. For safety, we delegate least authority. For effective deterrence, we assign most responsibility. For example, proof of stake blockchains only detect validator misbehavior after it occurs, but their imposed penalty is severe enough to deter such misbehavior. HORTON (Higher-Order Responsibility Tracking of Objects in Networks) is an identity-based access control pattern for reactively containing damage, assigning responsibility, and deterring misbehavior in dynamic networks.17 

Accidental vs Intentional Misbehavior, Who Cares?

When components are composed so they may cooperate, they may destructively interfere, whether accidentally or maliciously. We call accidental interference bugs. Concern with bugs has driven software engineering from its beginning. To better support cooperative composition, while minimizing bugs and their impact, we invented modularity and abstraction mechanisms. 

Encapsulation is the protection of the internals of an abstraction from those it is composed with. These are the boundaries of software engineering. APIs are specialized languages of requests, for the clients of the abstraction to make abstract requests of an implementation of the abstraction. These are the channels. The API is across an abstraction boundary, where the requests are abstracted both from the multiplicity of reasons why a client might want to make the request and the multiplicity of ways in which the request can be carried out. 

Human institutions are abstraction boundaries. The abstraction of “deliver this package” insulates the package delivery service from needing to know your motivation. It also insulates you from needing to know how their logistics work. You can reuse the “package delivery service” concept across different providers, and they can reuse it across different customers. In both cases, we all know the API-like ritual to follow,  even when dealing with new counterparties.

The modularity and abstraction mechanisms of software engineering have enabled deep cooperative composition while minimizing the hazards of accidental interference. The richness of modern software is in large measure due to this success.

We call intentional interference attacks. Unlike much of the rest of computer security, the ocap approach does not see bugs and attacks as separate problems. OCaps provide modularity and abstraction mechanisms effective against interference, whether accidental or intentional. The ocap approach is consistent with much of the best software engineering. Indeed, the ocap approach to encapsulation and to request making—to boundaries and channels—is found in cleaned-up forms of mostly functional programming18, object-oriented programming19, actor programming20, and concurrent constraint programming.21 The programming practices needed to defend against attacks are “merely” an extreme form of the modularity and abstraction practices that these communities already encourage. Protection against attacks also protects against accidents.

Accommodate Legacy

Our society rests on an entrenched software infrastructure representing a multi-trillion dollar ecosystem. But it is all built on the wrong security premises, and so, ideally, should be replaced. However, even if our continued existence depends on replacing it, that will not happen. We may go extinct first. Fortunately, that’s not the only way to cope with this legacy of insecurable software. There are many effective ways to mix new secure systems with legacy ones, each of which may ease this transition. Within secure architectures, we can create confined but faithful emulations of the insecure worlds in which this legacy software runs. 

Running legacy software in confinement boxes lets us continue using it, without endangering everything else. Capsicum embeds an ocap system within a restricted form of the Unix operating system.22  The seL4 rehosting of Linux lets legacy Linux software run in seL4 confinement boxes. CHERI adds hardware ocap support to existing CPU architectures, and is shipping in recent ARM chips.23

JavaScript is widely used and widely recognized as insecure. Recognizing JavaScript’s growing importance, in 2007 Doug Crockford convinced author Mark Miller to join him on the JavaScript standards committee. Their intent was to help shape JavaScript into a language that smoothly supports ocap-style secure programming; an effort Miller has continued ever since.24  It turns out that SES, a secure ocap system, can make the language secure while still running much of the existing JavaScript code. Experience at Google, Salesforce, MetaMask, Node, Agoric, and others confirms that much existing JavaScript code runs under SES compatibly. MetaMask leverages this to create a framework for least authority linkage of packages, substantially reducing software supply chain risks.

Reduce Risks of Cooperating, Recursively

Secure foundations are necessary, but are far from sufficient. Our software comes in many abstraction levels, and solves an open-ended set of new problems, each of which can introduce vulnerabilities. We cannot hope for perfect safety in general, even as an ideal to approach. Cooperation inherently carries risks. However, we can approach building cooperative architectures in ways that systematically reduce these risks.

The nesting of boundaries and channels is key to tremendously lowering our aggregate risk. By practicing these principles simultaneously at multiple scales, we gain a multiplicative reduction in our overall risk. To explain this, let’s visualize an approximation of overall expected risk as an attack surface. None of the following visualizations are even remotely to scale, even as approximations. Instead, they are purely to illustrate a qualitative argument about quantities we have no idea how to quantify. 

To visualize our aggregate risk, let’s start by observing that expected risk is just the flip side of expected value, where the values are negative. Anticipating automated attacks, we should assume any exploitable vulnerability is eventually exploited. So for each possible vulnerability, our expected risk is the probability that it is actually exploitable times the damage that could be done by exploiting it. To approximate these, we start with proxy measures. 

For the possibility of an exploitable vulnerability, we substitute a fallible agent. This agent may be malicious, or may simply have a flaw letting an attacker subvert it. For the damage that could be done by exploiting a vulnerability, we substitute the valuable resources that could be damaged by that fallible agent. The red squares are where a given agent has access to a given valuable resource. 

With the agents as rows, the resources as columns, and “has access” approximating “could damage”, this is the classic access matrix for analyzing access control systems. The total red surface area is our total aggregate expected risk, or attack surface. To become safer, we need to reduce the red surface area. To support safer cooperation, we need to remove red with as little loss of functionality as possible.

At any one scale, we can remove red by a variety of techniques. The principle of least authority removes red horizontally—it gives each fallible agent only what it needs for its legitimate duties. Placing legacy in confinement boxes lets one give that box as a whole only the external world access it needs for its confined software, again reducing red horizontally. Code reviews, testing, and especially formal verification shrink the height of each fallible agent row, i.e., its likelihood of misbehaving, thereby reducing red vertically. However, limited to one scale, even all these techniques together still leave too much red.

The two dimensional fractal Menger Sponge helps us visualize the benefits of reducing red simultaneously at multiple scales. At any one scale, only 1/9 of the red was removed, leaving 8/9. However, as we continue removing red at finer scales, the remaining surface is 8/9 times 8/9 times 8/9 etc., asymptotically approaching zero total surface area. This suggests that recursive application of our techniques could approach zero aggregate risk. Alas, even our best case is not that good.

Applying this visualization to a case study of an historic ocap system25—the DarpaBrowser running on CapDesk written in the E ocap language—we get the following picture. At each level, there are some solid red boxes whose internal risk we cannot reduce at finer scales. The main solid red boxes are the confinement boxes containing legacy software. These legacy boxes appear at each scale. We can prevent confined legacy software from doing much damage to the world outside itself. But because the legacy software internally operates by its own principles, we cannot keep it from fouling its own nest. Wherever it appears, within that box the recursion stops.

Even with these limitations, applying our techniques recursively reduces the density of the red, tremendously reducing our aggregate risk.

Overcome Adoption Barriers

An often neglected security vulnerability is the human. Phishing can be used to get ocaps as well as passwords. People with legitimate permissions can be tricked into misusing them. Unless there is widespread adoption of secure systems, their theoretical possibility is not much use. The adoption barrier to making the world a safer place is ignored in most abstract discussions of advanced technology attack. Perhaps because one imagines once humanity is urgently faced with these dangers, society will do what needs to be done. If there are known technological solutions for the dangers, it is natural to assume those most concerned can get a majority to build, adopt, and deploy these solutions fast enough to avert disaster. 

As for massive cyber attacks, one would hope that the government and industry would invest in rebuilding infrastructure on more securable, user-friendly, bases. However, after seeing how weakly the world reacted to cyber attacks that revealed massive vulnerabilities, this appears to be wishful thinking. The more likely reaction to the panic following a major breach will be directing even more effort into entrenched techniques that do not and cannot work, because those are recognized best practices. Techniques that actually could be adequate if the computational world was reconstructed from its beginning, on top of those techniques, will be seen as experimental, outside of established best practice and prohibitively expensive. Instead, we will likely procrastinate with insecure affordable patches until disaster hits.

The adoption of secure computing is being delayed because the overall software ecosystem is not “hostile enough.” Companies and institutions can be too successful when they build otherwise high quality systems implemented in insecurable architectures. Small projects can free-ride on larger projects being more attractive targets. In a world where large-scale, entrenched software projects primarily get attacked, most risk to early stage software projects is due to non-security dangers. 

Therefore, for most early projects, investing in costly security is less important than other areas, such as rapid product prototyping and receiving user feedback. Additionally, when hiring employees, a small company considers the person’s additional value to the project. So with regard to security, companies generally minimize the education burden on their team by following what are considered current best practices, rather than more unusual (and more secure) techniques. 

If a small project engages in the same allegedly best practices as bigger projects, it can escape attack because there are bigger targets. By the time the small project becomes large, and a serious target, it has enough capital to manage the security problem without truly fixing it. As of 2022, all large corporations manage their pervasive insecurities rather than fixing them. This is only sustainable because attacks are not extremely sophisticated yet. 

A sophisticated attack would make the world hostile enough to end fragile systems, but would also severely disrupt society. On the positive side, when this higher level of attack gets deployed, the world’s software ecosystem will become hostile enough that smaller projects’ relative safety through obscurity will end because insecurable systems of all sizes will be punished early on. The downside is that this has the danger of widespread destruction of the existing software infrastructure. If a certain threshold is destroyed, it could be difficult to transition to a safer situation without a serious downturn in overall functionality of the world’s computation systems, not to mention its economy. 

The problem is that a multi-trillion dollar ecosystem is already built on the current insecurable foundations, and it is very difficult to get adoption for rebuilding it from scratch. Thus, we should explore strategies to bridge from current systems to new secure ones. As mentioned in Chapter 5, in other contexts this is known as genetic takeover, a term derived from biology. In a genetic takeover, the new system is grown within the existing system, and is competitive within it. Once the new system becomes widespread enough, we can shift over to the new system, and eventually make the previous system obsolete. 

A real-world analogy is how society has adapted to earthquake risk. Instead of requiring immediate demolition and reconstruction of an installed base of existing, unsafe buildings, rewritten building codes require earthquake reinforcement be gradually done as other renovations take place. Over time, the installed base becomes much safer. 

Genetic takeover in the computer industry has happened before. The entire ecosystem of mainframe software rested on a few mainframe platforms, which seemed to be permanently entrenched. The new personal computing ecosystem initially grew alongside it, complementing rather than competing with the old one, but eventually mostly displaced it. Attempting to replace today’s entrenched software ecosystem is not hopeless; but it is very difficult. 

Grow Secure Systems in Hostile Environments

A hopeful counterexample to the insecurable computer ecosystem is the blockchain ecosystem. The large monetary incentives to hack them invite effective red team hacker attacks, which successful systems must survive. Chris Allen summarizes this accurately: "Bitcoin has a $100 billion bug bounty”.26 When insecurity leads to losses, the players have no other recourse to compensate. Non-bulletproof systems will be killed early and visibly, and therefore only bulletproof systems will populate these ecosystems.

Bitcoin and Ethereum are evolving with a degree of adversarial testing that can create the seeds for a system that can survive a magnitude of cyberattack that would destroy conventional software. 

An analogy for the idea that bad bits can be fought with better bits can be taken from John Stuart Mill’s conception about the discovery of ideas. He did not deny that bad ideas can cause harm, but he observed that we don’t have any divine knowledge of which ideas are good and which are bad.  The only way we discover better ideas over time is by being willing to listen to bad ideas. The only robust protection from the damage bad ideas can cause is the better ideas that immunize us from the bad ideas, by being able to understand why they are wrong. Competition between ideas in open argumentation ends up better discovering the truth over time. With information attacks, the flaw is not in the malware or the virus, but in the software’s vulnerability at the endpoint. Bad bits only damage you because you're running insecure software.27

In this sense, even for informational communication interactions at a much lower level than ideas, messages can transmit a virus or do other harm. The answer to this harm is for the message receiver not to be vulnerable by having a different informational architecture on the receiving side. In short, instead of preventing sending damaging bits, the best answer is better bits on the receiving side—secure operating systems, programming languages, and software security all the way up. 

Secure software systems like seL4 are better information. The more bad information or malware there is, the greater demand for a secure endpoint. Computation using ocap principles of voluntary interaction of digital architectures and alternatives discussed in this chapter can build a largely virus- and malware-safe system. There will always be new vulnerabilities and new attacks at upper abstraction levels. But the same iteration applies; these are best addressed by information receivers improving so as not to be harmed by the information. Building an ecosystem upon these security principles may result in a general-purpose ecosystem that can be used if the existing dominant ecosystem is destroyed. If a secure system grows enough before the world is subject to major and frequent cyberattacks, then we might achieve a successful genetic takeover scenario. We can potentially leave civilization a new architecture to migrate to, especially if migration starts before the destruction, or if the destruction is gradual and not sudden.

Chapter Summary

Physical systems rely on walls, barriers, and the rule of law to enable voluntary interactions, but digital systems have the advantage of an innate voluntarism. Ignoring the coupling with the physical world, one can only send bits, not violence, through the network, creating a fundamentally voluntary starting point for building architectures. Nevertheless, in the absence of computer security, the sent bits can still be catastrophically damaging.28 Voluntarism itself doesn’t mean risk-free cooperation. Computer security experts still put their machines on the internet, making a calculation that the benefits of cooperation by interacting there are worth the risks. Ramping up computer security can reduce risks and create a much more cooperative world. 

Sound interesting? Watch Gernot Heiser’s seminar on applications of cybersecurity.

Next chapter: WELCOME NEW PLAYERS | Artificial Intelligences

Historically, violence has been declining while we have made progress on a variety of factors that humans may value. Cryptocommerce can help us cooperate better in advancing progress. Biotechnology is unlocking healthier lives for more people, while remote sensing and robotics enable production at lower cost, using less energy, and creating less waste. Soon, we may be able to produce entirely new materials with molecular precision. We need technological progress to continue enabling more and more valuing entities to achieve what they value. So far, so good. Now let’s look at the dark side. Technology is raising the stakes of our civilizational game, and could threaten its very existence. There are at least two traps to steer away from.

Avoid Trap 1: ‘Small Kills All’ Via Technological Proliferation

Technologies are becoming cheaper and more widely available. CRISPR developed from the biotechnology achievement of the decade to being used routinely by lab students. Democratization of powerful technologies also means a proliferation of dangers.1 Nuclear weapon development is a large undertaking employing special materials to make distinctive products. In contrast, the cost to chemically manufacture strands of DNA could soon be affordable to the point of printing targeted deadly proteins or cells at home.2 Further out, atomically precise manufacturing could allow ordinary humans to employ ordinary materials to create entirely unexplored military applications. 

The risks from increasingly widespread weaponizable technology are sometimes summarized under the dynamic of “small kills all”, i.e. fewer people can cause ever greater destruction.3 It may only be a question of time until one of the 7 billion active players seeks large-scale civilizational destruction and can acquire the technology to cause it. To understand the danger, let’s zoom in on a few risky technological dynamics from nukes, to robotic violence, to biotechnology and nanotechnology. 

Dangerous Dynamics 

Nuclear Weapons

Nukes are a great example of  how we historically fared with massive risks for large-scale violence. The protections against an accidental nuclear launch were terrifyingly weak, while the degree of lying to the public about the safety controls in place was terrifyingly high.4 Public opinion was also frightening. In a 1945 Fortune poll on US attitudes about the atomic bomb, more than half of respondents, knowing about the devastating consequences for Japan, agreed “we should have used the two bombs on cities just as we did”, and almost a quarter agreed “we should have quickly used many more of them before Japan had a chance to surrender”.5 

FLI’s Unsung Hero Award acknowledges that it may only be due to a few individuals that we lived to tell the tale.6 In 1983, Stanislav Petrov ignored the Soviet early-warning detection system that had falsely detected five incoming American nuclear missiles. This prevented an all-out US-Russia nuclear war. Vasili Arkhipov vetoed his submarine captain’s decision to launch nuclear weapons at US ships on the false assumption that they were under attack. This was at the height of the Cuban Missile Crisis, so we can assume Arkhipov single-handedly averted a new nuclear war. The degree of governmental secrecy, the willingness to use nuclear strikes, and the accidental close calls is worrying with respect to future warfare. Nukes are still dangers hiding in plain sight. 

Fifty years since the 1970 Nuclear Non-proliferation Treaty - in which non-nuke-holding nations agreed to not proliferate while nuke-holding nations agreed to pursue disarmament - five authorized nuclear weapons states still have more than 13000 warheads in their combined stockpile.7 The Intermediate Range Nuclear Forces Treaty between the US and Russia has ended and North Korea is making progress toward a nuclear-tipped intercontinental ballistic missile that could reach LA in 30 minutes. The fragile security infrastructure around nukes is withering away, making accidents more likely with every year that goes by.8 We have not been safe from catastrophic weapons for a long time. It is thanks to outstanding individuals and blind luck that we have survived thus far. For civilization to survive, we shouldn’t rely on those factors.9 

Listen to Daniel Ellsberg, the author of Doomsday Machine, discuss nuclear risks.

Robotic Violence

Automation of cooperative arrangements will lower their cost and increase cooperation. Automation will also be the main multiplier for our ability to cause violence by lowering its costs. Robotic weapons, drones controlled by far-away individuals, and self-driving cars turned into land-based missiles are all weaponizable physical technologies. A single trigger can cause an automated reaction resulting in many deaths. Tiny autonomous drone swarms may be programmable to assassinate victims based on their biological features.10 

In response to activism, Google and some other large corporations agreed to not cooperate with governments in building autonomous military drones. But many of these companies are still working towards full self-driving automobile software as fast as they can. While self-driving cars are not built to kill people, they are built on insecure operating systems. If adversaries can take over millions of cars and run them into crowds at high speed, these cars become lethal land-based drones. In addition to technology whose primary purpose is violence, we are surrounded by unsecurable technology which could be used as weaponry. 

Importantly, robotic violence is not constrained by AI. We smile at naive media depictions of the Terminator robot. But, in terms of destructiveness, those robots aren’t a long way from current Boston Dynamic robots. Make those robots a factor of ten cheaper, more battery efficient, and mount a machine gun on them. The same robots that do adorable dances in today’s promotional videos become a means for automated destruction.11 

Biotechnology Risks

Biotechnology is responsible for some of human health’s biggest successes, such as antibiotics and vaccines. The response to the COVID-19 pandemic, with a record time of a few days to a vaccine design, highlights the imperative to advance biotechnological progress. At the same time, the pandemic gave a taste of the potential magnitude of anthropogenic bio risks as existential risks to humanity. Within a few centuries, and recently accelerated by breakthroughs like CRISPR, we went from figuring out what pathogens are made of to creating new, genetically modified viruses with different properties. 

Toby Ord is particularly worried about gain-of-function research, which makes pathogenic strains of infectious agents with higher transmissibility, lethality, or vaccination resistance. For instance, avian influenza A/H5N1 virus can be very dangerous to us, but its natural version is not airborne transmissible between humans. Nevertheless, in 2011, two laboratories identified an avian influenza variant which became transmissible through the air between ferrets. Based on this research, shortly after point mutations of the H5N1 virus genome were screened to identify mutations which would allow airborne spread. Gain-of-function research is aimed at improving pandemic prevention and defense. 

But our poor biosafety track record is especially concerning with respect to agents designed for optimal lethality. Ord draws attention to the Dugway Proving Grounds case. Dugway was established by the US military to work on chemical and biological weapons. In 2015, it accidentally distributed samples containing live anthrax spores, instead of the expected inactivated spores, to US labs across eight states. In response to such accidents, the US placed a moratorium on this research that is now rescinded.12 Terrifying biosecurity accidents regularly happen all over the world and the COVID-19 pandemic showed how quickly pathogens can spread globally.

In addition to critical accidents, we should also worry about deliberate threats. Pathogens could be engineered to be vastly more lethal and transmissible than SARS-CoV-2 and similar viruses. More than 1,200 different kinds of potentially lethal bio-agents, including bacteria, viruses, and parasites, have been studied for use as biological weapons.13 As with the history of biological accidents, governments currently have the greatest capacity for deliberate abuses. There are international treaties guarding against bioweapons use, such as the 1972 Biological Weapons Convention. It prohibited their development, production, acquisition, transfer, or stockpiling. Nevertheless, we lack the means to monitor compliance on either a national or international level. Some believe North Korea has compiled a biological arsenal containing anthrax, bubonic plague, smallpox, and yellow fever.14 Likewise, as shown in the Dugway case, the US is clearly not twiddling their thumbs when it comes to the exploration of biological weapons.

Nanotechnology Risks

At our current rate of scientific breakthroughs, we stand ready to unlock radically novel technologies. If Richard Feynman is correct that “the principles of physics [...] do not speak against the possibility of maneuvering things atom by atom”, one breakthrough will be molecular nanotechnology.15 This future level of nanotechnology involves having atomically-precise control of the structure of matter. Starting from current chemical synthesis, we may develop progressively better nanosystems capable of macromolecular self-assembly. The long-term goal is atomically precise manufacturing, i.e., the ability to use coordinated nanosystems operating with atomic precision to produce macroscale products with unprecedented performance.16 

Assembling materials from the bottom up would allow us to make any physically possible structure at minimal cost or waste, similar to an incredibly precise 3D printer. Nanoscale medical applications could help repair and rejuvenate our bodies. Nanoscale processing of materials could boost material welfare. Artificial molecular machines could target threats to nature's molecular machinery, reversing damage to our planet’s ecosystem. Combined with information technology, new materials could host entire networks of sensors, actuators, and communication devices. Summed up, “asking what atomically precise manufacturing systems can do with materials is much like asking what computers can do with information.”17

While this level of nanotechnology is likely still many years off, it will come with substantial risks. In addition to boosting existing weapons’ performance and enabling miniature mobile systems, it may lead to new types of weaponry. With desktop-scale nanofabs, separating uranium isotopes could come at the ease of today’s 3D printing. Swarms of billions of insect-sized drones could communicate, monitor their surroundings, and act in a coordinated fashion. Our inexperience with nanotechnology threats is especially worrying due to our poor historic track record of handling much simpler technologies such as nukes. 

Avoid Trap 2: ‘Civilization Suicide’ Via Single Point of Failure 

We should be terrified of a scenario in which technological proliferation allows individual malicious actors to cause global destruction. To deal with such risks, it can seem tempting to explore solutions that prop up powers of governments - or even establish a world government - in the search for safety.

The initial appeal of an actor with global reach comes from the sobering observation that, unless we can effectively monitor and prevent risks globally, there will always be pockets of actors that can go rogue, killing everyone. To mitigate this danger, a world government would have to orchestrate the monitoring of every individual in contact with potentially dangerous technology. When a threat is detected, it would have to orchestrate its rapid elimination. This would mean equipping such a government with an unprecedented level of surveillance and physical military weaponry. 

Unfortunately, powerful large-scale actors in charge of world-destroying technologies may not mitigate the risks from rogue individuals. After all, they are also made up of individuals. The risk of malicious individuals causing world destruction is still present. In fact, if the organizational structure amplifies the reach of the large number of individuals making up the actor, it may further amplify their malice. According to adverse selection, whenever we create an opportunity to have power, we create competition for it. The greater the power, the greater the race to capture it, and the less likely that competitors will all have purely good intentions.

Even in a best case where the centralized organization could filter out any bad actors it attracts, it is in danger. If the “good guys” get to the top, they become a target for the “bad guys” who seek to take over. As long as individual actors in a world government are vulnerable to external extortion, a benign world government would be vulnerable to it. 

The development of nuclear weapons technology was hastened because large governments that felt threatened thought having them would improve their security. Currently, we have a sufficient balance of power among multiple nuclear powers, and all of them have so far succeeded at making decisions that did not result in their use against an adversary. If we had a world-dominating central body in the possession of nukes that feels their existence is threatened, would this be a world that is more or less likely to deploy nukes than our current world? Hanson warns that, by creating a single point of failure, such a power’s suicide could quickly become a civilizational suicide.18 

Even if we could imagine a perfect central actor without those critical dangers, how could we possibly hope to create it? We are not in such a world and, for reasons just discussed, not everyone thinks it would be a good idea. Actors with good reasons to try to prevent a global takeover by a world government may decide to attack first to avoid otherwise becoming the victim of such a strike. Simply the fact that we have a multipolar deployment of nukes internationally already narrows the set of possible survivable futures. It could be fatal to try to transition to a unipolar world, or even to let things proceed to the point where a sudden transition to a unipolar world is plausible. 

The temptation to create powerful central coordination, for instance to solve small kills all risks from the proliferation of technologies, is Robin Hanson’s best guess for the Great Filter that humanity may face. The Great Filter is one explanation as to why we have not found alien life even though, by the sheer number of potential life-carrying substrates, and the time it would have to contact us, we should have. There may be a Great Filter all life has to pass to evolve into an interstellar species. No-one other life form has passed it yet, which is why we don’t see anyone. Either this filter is behind us, such as abiogenesis (life arising from non-life) in which case our chances to survive are favorable. But it may be ahead of our civilizational path, such as a risk that could wipe out humanity.19 It would be sad, if by attempting to protect civilization, we would end up preemptively destroying it.

Dangerous Dynamics

Pervasive Surveillance 

In the past, certain actors have tried to take over the world. Some made a certain amount of progress but ultimately failed. As technology advances, they may not. Technologies for direct monitoring and control of a population are becoming available, becoming cheaper, and coming to the attention of authoritarian states everywhere. Even in relatively free countries, we must assume any private information held by companies or governments is accessible by sophisticated hackers from both our own and other unfriendly nation states. 

Google’s ‘Don't Be Evil’ dynamics led to an internal collective sense that employees would not perform evil actions. Yet those good employees let Google gather a very dangerous amount of aggregate information on people in one vulnerable place. There were multiple attempts to move Gmail from plain text to encrypted ciphertext but they never succeeded. Even if Google does not abuse its power, its existence is a tremendous temptation. For Google, external abuse has occurred both when it was hacked by a foreign nation-state and when it was served with U.S. national security letters, demanding its customers’ information. Such national security letters are unconstitutional and a severe violation of democratic accountability.20 But they still happen, because having many people's plain text email at one company is too juicy a target. We must prevent situations occurring in which those who are not evil can still be used for evil.21

Widespread surveillance may become more common. Individuals voluntarily install surveillance in their homes, wear sensors on their body, and carry a tracked pocket-sized computer that can perform its own external surveillance. The smaller and cheaper sensors get, the harder it will be to know if someone is equipped with monitoring sensors. You may even be unaware that someone has put such sensors in your clothes or your body. Information that becomes public will remain public, waiting for steadily-improving Machine Learning-supported correlation to make sense of it. The importance of knowledge will lead to races to obtain knowledge, improve sensors, and perform more powerful surveillance.  

Robotic Enforcement

A similar trend drives the automation of physical enforcement. Using humans as violent peace enforcers is too expensive, especially relative to the cost of robotic enforcement. The U.S., today’s dominant military power, already uses drones for military purposes. A human being is in the control loop, but that person has no physical ‘skin in the game’ in any  battles. And nothing ensures humans will stay in those control loops, especially when the stakes are raised. A 2020 Libyan drone airstrike was already conducted with no human in the loop.22 

Despite our best efforts, in a shooting war, formerly unethical actions may be reconsidered and people do what they think is necessary. At that point, how corruptible the rules guiding robotic enforcement are becomes a very present danger. Corruption can be vastly more amplified by automated enforcement mechanisms than by human ones. The Nuremberg Trials and reactions to the Milgram Experiment helped create a norm that if we receive an order that is illegitimate by society’s current moral code, we are individually responsible for not obeying it. The friction in human enforcement, critical to our safety, is not present for automatic enforcement. It will take longer to build a system with built-in friction against corrupt orders. The path of least resistance is to deploy systems before we know how to include such friction. 

The dangers of transitioning to pervasive automated enforcement through robotics are coming. Our failure to put adequate controls in place with respect to nukes is frightening when we project it onto this much more incremental, less visible threat. A robotic takeover could happen by an existing power gradually expanding its surveillance or military force. This is a boiling frog scenario. Democracy and the constraints against totalitarianism in the U.S. are much weaker than we thought. If the vulnerabilities of democracy in the still-dominant superpower are a cause for concern, so is the absence of accountability in its rising rival, China. If rapid deployment of a technological advantage in surveillance or enforcement technologies results in a global regime, such a single actor would create an existential risk in itself.

Decentralize Defense: Multipolar Active Shields

We need a civilization that decreases our vulnerability to the small kills all trap without going to the opposite extreme and creating the power suicide trap. For a moment, let’s dream big, and design a possible system that, even if not easy to build, if successful would actually address the risks from biotechnology, nanotechnology, and robotics without creating single points of failure.

First, we need to prevent small kills all attacks from succeeding, even when we can’t stop them. To successfully defend against an attack we must have a deployed fabric of systems that detect and react to attacks based on trustworthy mechanisms. This is called an active shield.23 In theory, widely deployed sensors could collect data in an encrypted format about relevant technologies usage. An automated protection system could analyze the data and enforce the action required to prevent an abuse of the technology. If we cannot prevent robotic enforcement, we must ensure it is verifiably used for legitimate defense only. Similar to the white blood cells of our immune system, such defense machines would fight a variety of dangerous replicators, decreasing in scale as our threats do.

Second, we need to prevent active shield deployments from creating the conditions for power suicide. The rules that govern it need not only stop those that they monitor from engaging in violence but also keep the monitoring components in check. For that, the system must be multipolar so that the different components monitor each other, and if one component goes rogue, the rest of the system must be able to gather enough force to counter the component that went rogue. Instead of having a mutually trusting system of active shields, we need a mutually suspicious system of active shields. This is a resilient way of building an overall system that creates a stable enforcement of the rule of law without assuming that any one component of the system is incorruptible. 

We should be aghast at the degree of surveillance and enforcement considered here and must do what we can to fight it. But if we can’t stop it, we must face the terrifying choice of what type of system it is and how to control it. Let’s look at three features a multipolar active shield should have:

Monitor: Encrypt Sousveillance

Successfully monitoring for hostile technology activity requires an unprecedented level of surveillance. On the one hand, we are lucky that robots, nukes, biotech, and nanotech (in contrast to cyber threats and AI which we tackle in the next chapter) involve physical aspects, so there is something physically observable. On the other hand, the physical processes we need to monitor to distinguish dangerous biotech or nanotech uses from benign ones are extraordinarily small. This suggests the need for almost unimaginable  surveillance levels. 

Compare this to our most recent precedent, nukes. While their current situation is still very concerning, no one has used them in battle since World War II. This is partly due to luck and partly due to non-proliferation treaties backed by monitoring regimes. We are fortunate that nuclear weapons are a very monitorable gross physical phenomenon. The average person has no need to privately traffic in Uranium-238 or engage in other activities with strong nuclear signatures. Likewise with nuclear tests, people don't have important private reasons to engage in activities that look like a nuclear test. Monitoring for nuclear explosions is not very intrusive to people's private lives. 

The physical objects monitored to detect hostile nuclear weapons activity are large-scale and easy to verify. Particularly when compared to what is needed to monitor for future offensive biotechnology and nanotechnology use. Those involve molecular machines, so require verification at the molecular level: a daunting challenge. To prevent these hostile attacks, we need to monitor activities in small-scale labs that do small-scale manipulation of widely deployed synthesis mechanisms that are otherwise general-purpose. Given that the amount of weaponry one can make in a small building will be substantial, we may not be able to preserve our homes as privacy fortresses. 

Deploying such pervasive physical monitoring without strong privacy safeguards could create dangers in excess of what we want to avoid. It could lead to a Big Brother-style top-down pervasive surveillance state that makes 1984 pale in comparison.  There are other options. David Brin discusses a theoretical alternative by which top-down surveillance is kept in check by bottom-up sousveillance.24 He suggests that given future tech-enabled affordable mosquito-size cameras, the only way to prevent pervasive surveillance is via pervasive sousveillance, or upward-looking monitoring. 

In this scenario, all information is simply public, so that for all physical activity in the public realm, all of us have complete access to all of it. The information produced by pervasive sousveillance lets different entities keep each other in check, thereby stopping corruption. Sousveillance could hold abuses in check but would be destructive of privacy. Even if we could adopt our privacy norms to match such a transparent society, it’s not clear that we should. Even if everyone can, in theory, see everyone else equally, the costs of analyzing information are asymmetric. Apart from extortability of individuals, the bigger danger is the power advantage this information confers long-term that can destabilize the balance.

A preferred monitoring option is an automatic network of artificial monitoring agents, from which information is not revealed to humans unless pre-agreed criteria are triggered. Ben Garfinkel uses the example of bomb-sniffing dogs, which report only if a given bag holds explosives. He suggests there is no reason why monitoring security-relevant information requires the system to learn anything else. While tricky to implement, an encrypted sousveillance fabric—that only releases information when it detects anomalies—could avoid top-down surveillance abuses and the loss of privacy from transparent sousveillance. Such a system, while hard to construct, could be the beginning of a path to safety. 

From ThinThread to Encrypted Sousveillance

An early experiment with encrypted surveillance originates with William Binney, then senior official at the NSA. The NSA collects and stores unencrypted data, but in theory limits the access of individual analysts to this information. An analyst, for example, may be allowed to make only a limited set of queries to the database and only view the portion of records that are classified as matching these queries. From the Snowden revelations we know how little self-restraint was actually practiced by the agencies. Congress set up a set of laws that the NSA was supposed to obey but the internal judicial procedure became a rubber stamp. Almost all of the requests for information that the NSA took to the FISA Court were approved, amounting to a window-dressing of checks-and-balances.

During the crucial period of the emergence of this surveillance capacity, William Binney tried to construct an internal automated system with some degree of built-in governance. Thinthread was a prototype of an encrypted monitoring system that would only allow data on individuals to be decrypted if a judge found probable cause to believe the target was connected with serious crime. Such internal controls could have enabled a mutually monitoring human system, aided by FISA Courts and oversight by Congress. It would have still been corruptible without democratic accountability but it would have been a first step toward a system whose automated governance regime prevents uncontrolled access to surveillance information. Unfortunately, the program was canceled and eventually replaced by a system without the filtering and encryption protections. 

Given recent progress in AI and cryptography, it is time to make another attempt at making monitoring privacy-preserving. In theory, if a surveillance task can be automated, then it can be done in a way that avoids requiring the party to collect the data in unencrypted form. There are existing facial recognition system that use homomorphic encryption to report only whether an image contains the face of a suspect. From there, it is not far to imagine future systems that report the identities of individuals only if they detect illegal activity with high probability. This comes closer to establishing a tripwire system in which encrypted information only gets revealed if it triggers certain pre-agreed identifiers. Using zero-knowledge proofs, it may be possible for the monitoring fabric to prove that its information gathering satisfies some agreed-upon tripwire criteria without revealing anything that it is not supposed to reveal. 

The hard part will be designing a multipolar monitoring system in which different agents reliably keep each other in check. Illegitimate release of information is not visible abuse that the other monitoring actors can detect. In the case of physical coercion, other watchers could see the involuntary interaction performed by bad watchers and treat them as an attacker. The case of illegal release of information is harder to monitor because it has to be done by internal inspection. 

Any privacy-preserving system is non-transparent by virtue of the fact that it is observing information that it is not revealing. If it is non-transparent, it is hard to reliably prove that it is not revealing the information in a way that is abusable. Such double non-transparency is a high bar, but it is needed if we want to trust that our information is neither publicly nor privately revealed. 

Fortunately, zero-knowledge proofs, secure multiparty computation, homomorphic encryption, differential privacy and other encryption and privacy efforts are progressing rapidly. More funding is needed to speed up better tool development. The temptation to corrupt monitoring fabrics is going to be enormous. We must avoid that the fear of the dangers and the promise of privacy preservation lead us to lower our guard to the point that we allow abusable monitoring to get deployed. We will come back to this when discussing computer security in chapter 8.  

Detect: Design Ahead 

Let’s assume we successfully design an encrypted multipolar monitoring fabric. How do we define a dangerous activity to monitor for? Nick Bostrom compares the process of civilization engaged in technological discovery to pulling balls out of an urn. He suggests that we need a system that ensures we don’t pull out any “black balls”, i.e. technologies that cause disaster instead of progress. What if we pull out a ball that kind of looks gray to us, while others see it as more silver? As we start playing with it, it may get dirty and turn darker and darker until someone speaks up and calls it a black ball. 

When someone discovers such a civilization-destroying potential of a technology, we need an open system in place that is effective at problem-solving around the threat. If we have the complex superintelligence of civilization, with a strong interest in preserving the peaceful decentralized fabric of cooperation, then we can bring this intelligence to the problem. We need to make our cooperative architectures more reliable, more accountable, and more widely reviewed, so more of us can know how much danger we are in, and give feedback.

As a precursor of such a system, let’s take inspiration from Robert Reid’s proposal for pandemic preparedness. He suggests that a global cooperative layer of scientists, equipped with local knowledge of the viral patterns in their area could create open source weather maps of dangerous pathogens.25 Our world’s complex adaptive systems have millions of trade-offs on millions of constantly changing margins. We must resist the temptation of creating centralized organizations tasked with solving the problem. This reduces the intelligence that gets to be applied to that problem. If complex systems lose their adaptiveness to high-level planning, civilization loses its ability to adapt. 

The Oracle Problem

This is especially pressing when we increasingly rely on automated monitoring systems. They turn a question about the real world into a decision procedure about electronically judgeable evidence that supposedly represents a claim about the real world. When all that these systems can draw on is evidence brought to them by sensors in the real-world, how can we trust the outcome? The Oracle Problem is especially pressing in situations in which not all sensors can be assumed as being well-meaning.

Fortunately, it only becomes a pressing problem once we have already solved all other problems. The Oracle Problem has become a concern in the blockchain world only because the ecosystem managed to advance to a point where this problem arises. This is a tremendous achievement, and experimentation with solutions in the blockchain space should inform our thinking moving ahead.26 Noting the problem means that we can get more minds working on it. 

Fortunately, we have ample opportunity to learn along the way. Some technologies may allow for a window in which it is possible to design ahead and simulate them well before actual construction is feasible. For instance, advanced nanotechnology can and likely will be simulated well before it can be implemented. This time gap between knowing what is buildable and being able to build it creates room to increase safety. In addition, biotechnology is sometimes classified as the biological subset of the category of molecular machine systems referred to as nanotechnology. This means that in addressing near-term biotechnology dangers we will pick up strategies applicable to the longer-term challenges of nanotechnology. It’s not too early to start.

Defend: Open Arms 

Let’s imagine we succeed at creating an automated multipolar monitoring system that can detect dangerous activity while preserving privacy. Next we need to design the enforcement mechanism that is activated if illegal activity is detected. Automated contracts themselves cannot intervene directly in physical reality. This is problematic because an enforcement mechanism has to be based in physical reality, for instance via robotics. If we think of monitoring as the sensory side of smart contracts, we need to combine it with the motor side of smart contracts that can turn decisions into engaging with the world.

Similar to the monitoring fabric, the logic of the enforcement fabric must be designed as an open source system. Knowledge transfer is in everyone’s interest, similar to when the U.S. unconditionally offered other nuclear parties its inventions for protecting against rogue launch. If any one player can prevent a false launch, it is better for everyone. Once we agree on the rules by which the system operates, and have simulated and tested them, we should strive for a simultaneous multipolar deployment by all parties. If any one portion of the active shield makes use of its enforcement mechanism illegitimately, the rest of the fabric can use its aggregate power to shut down the part of the network that operated illegitimately. 

The closest current analog to the kind of open source innovation required, termed Open Arms, is perhaps large multi-way consensus mechanisms on the blockchain. A large chain, such as Ethereum, faces the problem of having to design a single set of incorruptible rules to be enforced in a multipolar manner. Even though all of the participants in the mechanism are corruptible, there is a mutual checking of each other through the blockchain replication mechanism that should make the system incorruptible.27 

Drones with Body Cams 

What would such a system look like in practice? While automated enforcement is still far in the future, we can already make decisions about the predecessors to these technologies. One such decision is demanding the analog of body cams in emerging enforcement machines. Body cams result in making corrupt behavior harder to hide and thus less likely to occur and making proper behavior that has an ugly outcome easier to defend. They both hurt bad cops and help good cops. 

Every automated system with the capacity to kill people, including existing robotics such as drones, should have a body cam built into it. Such a black box recorder would contain all footage of what happened leading up to a fatality. If drones are acting lawfully, there is little ground to resist a time-delayed revelation of the information in the black box, i.e. the footage becomes public after a proper time window. Twenty years later the footage will not be significant to intelligence operations. Closing the monitoring feedback loop to hold bad actors accountable may require less than twenty years, but it is a lot better than never. If we can get twenty years agreed upon now, in ten years we may be able to get five years. If we can get something accepted that embodies the principle, then even with a time delay that is painfully wrong, we can start negotiating. 

Entities choosing to be transparent changes the game completely. Most of game theory assumes that each entity is opaque inside, but the evolution of human cooperation leverages the difficulty people have in lying. With transparent entities, it’s no longer a question of what they would decide is in their interest, it’s a question of what they can decide given how they are constructed. Open source design and construction can define this. This applies to active shields and will apply to future artificially intelligent agents.

Navigate the Traps: Avoid First Strike Instabilities

Let’s imagine we succeed at designing an encrypted multipolar monitoring system that could detect attacks and enforce appropriate responses. To work in practice, it also must be deployed without causing a first-strike instability. 

Given the uncertainties in a potential conflict, all parties have much to gain from simultaneous multilateral deployment of a mutual defense system. But, even in a close to best-case scenario—a system, if deployed, monitors for offensive use and takes action to prevent that use—the danger remains that one side might deploy before its competitors. Unfortunately, the technological designs resulting from sophisticated design-ahead also create a first-strike instability. This results in a first-strike instability, such that even if no party wants to start a conflict, the fear that another party might incentivizes a first strike.28 We have no simple answer to this problem.

The goal is to move towards a strong framework of norms of voluntary interaction with a highly multipolar set of interests. Nevertheless, this world must be emerging out of a world that does not have that high degree of decentralization yet. We live in a world that is militarily dominated by the United States. China and its military power are rising. Any transition to a balanced multipolar world order would require them to give up considerable military power. 

If either of them had the ability to deploy a privacy-respecting monitoring fabric, it would also have the ability to deploy a fabric that is as powerful but not constrained by the safety protocols. Previously, the NSA had an ability to deploy the ThinThread fabric that obeyed privacy-preserving protocols. This was the active project that was being pursued but killed in favor of the more intrusive fabric. If we can answer the question how we can get a global credibly self-constraining monitoring fabric despite the preference of today’s military powers, we are in good shape to worry about enforcement. 

We admit to the great fear that an active shield is itself a permanent military takeover. The attempt to build it, deploy it, and get it entrenched can go very badly but so will civilization if we don’t try. If we try we have a chance of succeeding, but if we don’t try we end up with a system that does not even presume to have minimal internal controls. We must try to build something whose stable point is a neutral framework of rules where judgment is distributed.

Compensate: Mutual Defense, Commerce, and Science 

The more decentralized the systems we build now, the more likely future systems will follow this trend. The most obvious compensating dynamic is military mutual defense pacts. Turkey, Germany, and Italy have a nuclear sharing system or mutual defense pact that can be seen as a model for the multipolar framework of an active shield system. In  mutual defense pacts of multiple separately deployed active shield subsystems, when one party misbehaves, the pact among the others can cooperate to restore the balance. 

International trade is another means to push back the evolution of systems of force in favor of cooperation. If nation states are large circles, companies are smaller circles within them, and multinational companies are circles cutting across them. A multinational is still incorporated in a particular country that can corrupt it. But governmental behavior emerges from interest groups. Many interest groups benefit from trade. As trade with the rest of the world increases, it is increasingly in their interest to pressure their governments not to pursue external military dominance. During the British hostilities with their American colonies, British merchants, hurt by trade losses, were amongst the strongest proponents of peace. Since large militaries are not actively engaged in taking commercial actions, the danger of their dominance is not yet recognized enough. We hope to sensitize you that it is in your short-term interests to engage in multilateral paths in a mutually observant way that avoids centralized military force.

Sometimes, sets of citizens across nations can form explicit voluntary bonds to compensate for international power escalation. The Pugwash Conference was initiated by scientists, through the Einstein-Russell Manifesto, calling for a conference to assess the dangers of mass destruction. US, Soviet, and other scientists continued to meet during the Cold War, through the Cuban Missile Crisis, and the Vietnam War to draft background work for the Non-Proliferation Treaty, and the Biological and Chemical Weapons Conventions. Similarly, the Asilomar Conference was based on the realization of genetic engineering dangers. It resulted in scientists voluntarily deciding to halt experiments using recombinant DNA. This was a voluntary agreement to coordinate to not create a danger no one wanted to create. If 99% of us stay within voluntary safety controls, then even if 1% starts going off the rails, we can rely on the superintelligence of the rest of civilization to help. 

If we can increase our multipolarity while future technological realities are emerging, we have a chance of having arrived at a world that is no longer takeoverable because there are enough capable forces that don’t want to be taken over. If one power commits suicide, its place and resources can be taken by other powers who have not committed suicide. There is no guarantee for stability. In the Peace of Westphalia, dozens of military parties maintained their strategic balance in an antifragile way. The Peace of Westphalia was multipolar for centuries, but it eventually collapsed. Likewise, the Founding Fathers were not confident that their arrangement would not collapse into a dictatorship. They were in a terrifying situation and their only choice was to do the best they could. Today their division of power is not functioning as well as originally designed, but it has lasted for centuries, and it is difficult not to consider that a success. 

The constitution worked well enough that the intelligence of our civilizations can iterate on the issue of preserving the balance. Now, we are in the terrifying situation of witnessing military proliferation powered by strong technology. Our only choice is to do the best that we can. If we set up a sufficiently multipolar world prior to the emergence of much greater intelligences, we may be able to defer part of the power balancing problem to them. We need to get to the point where it's our descendants' problem, but where we have enabled them to address it because we didn't hand them a dictatorship.  

Chapter Summary

We discussed the dark side of our rapidly maturing civilization: risks that come from automating technologies multiplying individuals destructive ability. Tempting solutions, such as strong central actors that monitor and control attacks, have their own risk of power suicide which may be worse than what they prevent.

Less obvious solutions are gradually emerging, thanks to innovations in cryptography. A multipolar active shield would rely on automated encrypted sensor sousveillance of security-relevant activity. It would only disclose its detection when pre-agreed tripwires suggesting illegal activity are triggered. If the shield detects such activity,  its robotic enforcement arm prevents the rogue node from engaging in hostile actions. The shield’s multipolar deployment, by which many nodes watch both relevant activity and each other, avoids allowing dangerous activity outside and within the fabric to reach a threatening level.

If we can design and deploy such a system, we may feed two birds with one scone; we take the sting out of the inevitable automatic surveillance and enforcement, and use it to prevent other dangers. While automated sousveillance and enforcement seems like an impossibly far away dystopia, the future will grow out of  today’s decisions. The better the structures we put in place soon, the less of a dystopia it will be. 

Curious for more? Listen to this Intelligent Cooperation seminar.

Next up: DEFEND AGAINST CYBER THREATS | Computer Security

Getting There: Genetic Takeover

Cooperation could be rich, thanks to a suite of new technologies. Prediction markets, space property rights, mixed contracts, assurance contracts, and the other arrangements envisioned so far are just the tip of the iceberg.

It was hard to imagine today’s internet before the first website. Likewise, it is hard to imagine the novel cooperative arrangements ahead. But it's worth trying; after all, in Alan Kay’s words, “the best way to predict the future is to invent it”.1 We may be as far from this new world as the ARPANET was from the current web, but this shouldn’t stop us from dreaming big. Most of these experiments will fail but the few that succeed may radically lower the costs of cooperation.

Lowering cooperation costs by orders of magnitude is not just a quantitative difference. Whenever we have several orders of magnitude difference in a quantity, it often causes a change in the phenomenon’s character best seen as a qualitative difference. We might usher in a new era of cooperation, analogous to when the 1500s and 1600s commercial revolution in the Mediterranean world resulted in an inflection point toward today’s much richer world of exchange.

Co-evolve, Cooperate, Compete

How might we transition into a more cooperative world from our paper world, in which printed contracts dominate, alongside checks as the second most preferred way to pay bills?

How cities are built, how people self-organize, and what kind of society they organize into is largely based on how the economics of defense and attack evolve with technological progress. Law, for instance, evolved to deal with problems as they arose using the level of technology that was available to people at the time. Today’s legal systems evolve by competing on at least three axes. First, on how well they insulate us from underlying rules of biology, such as violence. Second, on how well they create rules for their own survival, such as capitalism vs. communism. Third, on how well they insulate us from their own dynamics, such as separation of powers to allow for watchers being watched.

The invention of cryptography technology, coupled with worldwide networks, changes the economics of defense and attack of today’s legal systems. This change is greater than going from the bow and arrow to the gun. Once again, society will have to self-organize around this change to reach a new equilibrium. Where will we settle? Even if new technologies of cooperation allow us to build a perfect realization of a neutral framework of rules, we still rely on the legal system to insulate us from the rules of biology. Both systems will co-adapt, each constraining the other without either one emerging as dominant.

Let’s look at two popular positions in cyber debates: Let’s call one perspective legal absolutism. It states that human law, enforced by governments via their monopoly on violence, will remain the dominant point of reference. In its most exaggerated form, this perspective holds that anything legislatable has the upper hand. It amounts to denying reality in the face of technological advance, comparable to trying to legislate that pi equals 3.

Let’s call the other perspective code absolutism. It states that code is law, which will take over the legal system and displace all of human law. This ideal is unrealistic, because the people who write code and the computer hardware that runs it are in government-controlled physical spaces. As long as governments deploy legal systems with strong public support, many computer systems will enforce existing law.

Rather than seeing the crypto or legal absolutism caricatures as attainable goals, they are better regarded as side constraints on the evolution of real systems.

We can borrow the term genetic takeover from biology to describe such an alternative transition. A phenotype, highlighted in yellow, evolves through a new genotypic representation, G2, gradually emerging within an older one, G1. The legacy system provides context and constraints influencing what is viable in the new system that grows within it. The evolving phenotype now has entirely novel qualities, i.e. here it is spiky, rather than smooth.

Let’s imagine a world in which fences were not invented, but property rights were. People keep records of where boundaries lie; some can hire guards to protect their land and use the court system to prosecute boundary crossers. Eventually, someone invents a fence. A debate ensues between the legal absolutists, arguing fences are not legitimate versus the fence absolutists, arguing that property law has become irrelevant. There are obviously many lower-cost solutions to the boundary crossings problem than legal records. A fence may substitute for the law in some of these cases, but not in all cases, because people can climb over and go under fences.

Likewise, the world in which traditional legal systems emerged had no better choice; it did not have the enablers of strong cryptography and the net. Like a fence, a smart contract will substitute for some, but not all, cases in which we currently use legal systems. Nevertheless, since code can be buggy or we sometimes just want a human in the loop, paper law will remain a useful tool. A contract acts by provoking behavior in an enforcement mechanism; in a legal contract this is the legal system, while a smart contract is simply the choice of a different system. Our current legal systems have never been static and will change along with technology, requiring continuous updating regarding our notions of “rule-based systems”.

Change the Rules Without Relying on the Rules

Apart from honoring voluntary Schelling Points as constraints, we don’t necessarily have to rely on existing rules to change them. Often, policy hasn't been concretely articulated into laws. This leaves an intermediate space, which different policymakers have different ways of regulating. Premature filling of such regulatory gray areas could be disastrous for innovation. The U.S. Supreme Court had to overturn early railroad regulations drafted before we had sufficient experience with the railroad system.

The internet could not have emerged without many people operating in the emerging blank spaces of their time. Its rules are built on both political and technological interaction. Many political decisions were embodied in protocols such as those promoted by the IETF, which created a transnational rules system in which most countries let their citizens participate. It was a base system of rules founded on voluntary gray area interactions that, at that time, no one decided to try to suppress.

No rules clearly allowed the internet’s phenomena, but no one interpreted the ambiguity into an enforceable rule violation. It was unclear if doing commercial transactions online was legal. Many assumed it was not, but people just started doing it, regulators refrained from interfering, and it gradually grew into an accepted legal activity. Enough people were interested in seeing certain uses continue so that whatever was needed to continue those uses became technological reality. The internet changes the nature of the emergent rules, such that they increasingly operate by its logic.

Bitcoin nudges the rules further to settle ambiguities in favor of continuing to allow new activities. Cryptocurrencies are built from voluntary interaction that is costly to suppress. At most, a nation can take its citizens out of the game and weaken its own position. We have settled on the intermediate state of regulated on- and off-ramps into cryptocurrency via services like Coinbase. These convert dollars into crypto and vice versa. Yet, once on the other side of the boundary, private currencies let you transact without record-keeping, at least until you off-ramp into fiat currency once more.

Many of the more complex arrangements are not illegal, but are genuinely novel economic phenomena. Blockchain-based prediction markets may fall under gambling laws by one reading of U.S. law. But they are not treated as such by the mainstream economy. You can earn cyber coins betting on the future, withdraw those coins and trade them for fiat money. Know Your Customer (KYC) Anti Money Laundering (AML) efforts mean this directly borders current legal systems. Still, the lack of a central control point means that to shut the process down, one would have to go after people individually via the KYC. This is extremely costly given the ease of converting one cryptocurrency for another.

We are used to negotiation at the boundary between different rule-based systems; international law is an evolving consensus from negotiation at jurisdictional boundaries. We can compare the interaction of the legacy system with the emerging system to interactions among jurisdictions. When jurisdictions have contradictory rules, engaging in multi-jurisdictional transactions requires complex negotiations. Despite the messiness of those negotiations, they often succeed. Transacting across the legacy legal system which is backed by the government's monopoly on violence and a tech-enabled cooperative system, backed by cryptography and strong anonymity, will be equally messy. As these systems encroach on each other’s realms, both systems will try to fight back. As this fight becomes increasingly costly, they may converge on a new legal-technological reality.

There Be Dragons: Risks of Centralization

Can we say anything about the dynamics dictating the trajectory this emerging system? Progress relies on innovators wanting to introduce new cooperative arrangements into the world. Once introduced, the arrangements become much more valuable if not subject to their inventors' whims. As long as a centralized entity provides the arrangement, it does not have the benefits of multipolar checks and balances.

In the early days, computers had tremendous architectural diversity. Each computer had a little operating system and localized interactions through the timesharing system. The ARPAnet connected some of those computers by enabling them to email each other without suppressing their architectural diversity. Each was an independent experiment, but suddenly, via network effects, there was a larger community to cooperate with.

Fast forward to today, our interactions are already centralized in the Internet Protocol (IP) on the web. There aren’t many completely different hypertext experiments that do not play with it. However, the web itself is largely decentralized with protocols allowing many different parties to communicate with each other. They function more like language than governments. One could say that the decentralized nature of the protocol achieves a system that is 99% decentralized and 1% centralized.

Realizing the dangers of a centralized power, the temptation is to regulate. Yet, to make progress toward voluntarism going up levels of abstraction, we need to rely on competition as a discovery procedure for finding out which systems work well. If innovators deliver more value by taking themselves out of the control position, they can find arrangements to make more profit by doing so. If they don’t find a way to deploy their arrangement without their controls, their competitors will.2

Today; we have single web governing bodies but also efforts against them. The domain name system functions as a centralized decision-making point to allocate the namespace’s root names. The Petnames project seeks to enable interconnectivity without a global namespace. The Dweb movement builds a distributed web infrastructure that compensates against concentrated control. If an architecture causes us to make overly centralized decisions, we should consider it a bug and find a better one.

Learning from History: IBM vs. the PC vs. Google

Can we rely on decentralized systems outcompeting centralized ones? A look at computer history provides a mixed picture: In the computer industry's early days, it was common knowledge that IBM's mainframe monopoly would lead to monopolizing the entire computer industry, unless the government broke it up. Antitrust forces were engaged against IBM; IBM fought back. It is possible that antitrust laws had little to do with IBM's loss of the dominant market position and that the personal computer's rise had everything to do with it. Microsoft then outplayed it, and became the monopoly supplier of PC operating systems and most office applications.

For a while, Microsoft was so dominant in software that it seemed it would remain the monopoly forever, unless broken up. Antitrust forces were engaged against Microsoft, and Microsoft fought back. Once again, it is possible that antitrust forces had little to do with Microsoft losing its dominant position as opposed to the rise of the web. Microsoft was a substantial web player, with Internet Explorer being an initial browser wars winner, but that dominance was unstable.

Gradually, today's internet giants emerged and turned email into Gmail, Usenet into reddit, blog replies into Facebook, pingbacks into Twitter, squid into Cloudflare, and gnutella into The Pirate Bay. By having a Gmail account, Google and you engage in a voluntary interaction with each other. Nevertheless, if you violate vague terms of service and they arbitrarily banish you, everything you managed for that identity becomes inaccessible. When restricting or cutting off access, internet giants aren't subject to accountability or due process. Within this digital jurisdiction, when banned you become a non-person, exiled due to unrevealed criteria. This is a very poor form of cooperation.

If we think of these giants as an arrangement through which we cooperate, we don't want those interactions to be placed at risk by a third party. The problem with the giants is not just power centralization, but that their power feels arbitrary. If we don't know why our accounts could be suspended, we live in a state of vague fear, reminiscent of a lawless state of nature. We need rule of law and an understanding of what the rules are so we understand how to cooperate. In an extreme view, our internet giants act like historically oppressive regimes, dominated by their rulers’ unpredictable whims rather than by laws or rights of citizenship.

As in the previous Microsoft case, the centralization wave of today's internet giants has been swiftly accompanied by antitrust efforts. But once again, we should rely on the rise of new innovations to solve the problem. Chatbots might out-compete Google, and the rising tide of specialized (social) media platforms might outcompete Facebook and Twitter. This repeating dynamic provides hope that across future systems, coalitions of non-dominant players will continue to compensate against power centralization of giants.

It took centuries for today’s open societies to outcompete tyrannical ones, just as it similarly took decades for open source software to outcompete proprietary software. We should not expect quick wins of decentralized systems over centralized service providers, even if the decentralized ones bring a confident rule of law. There will be many failed ventures, and it will take time to build up an adequate level of functionality, but we have repeatedly witnessed that long-term winners are those which create a rules framework leading to a predictable basis for cooperative interaction with minimal risk. As long as future levels of the game are defined by a rich taxonomy of rights and composability of contracts, cooperation can evolve.

Chapter Summary

Starting from our paper world, we embark on a co-evolution in which legal and digital jurisdictions develop new modes of cooperation within constraints imposed by the other. In the digital domain, a polycentricity of experiments can compensate for centralization. A voluntary future of ever richer cooperative games awaits, yet there are serious physical and digital threats looming along the way; a topic we'll examine next.

Checkout this seminar to take the first steps into a more cooperative world.

Next up: DEFEND AGAINST PHYSICAL THREATS | Multipolar Active Shields

Having read the manual for civilization and analyzed previous rounds of the game, let's turn to how to make the most of future rounds in this chapter. Over the course of civilization, we developed rights, contracts and various other mechanisms and institutions to help us cooperate. But challenges remain: it is costly to search for other players to cooperate with, negotiate an agreement that is attractive to all sides, credibly commit to it, and enforce the terms once they are set. Fortunately, we can look to technologies to lower these costs and open up new levels of cooperation.

Free Information

Legal systems emerged to establish neutral rule systems that support cooperation without violence. Governments and their rules are based on jurisdictions, yet many of us no longer know or care where the people we communicate with are. The internet not only makes it dramatically easier to find others to collaborate with but it also produces a permanent system of voluntary rules that is very costly to suppress.

Free speech is a great example. Governments imprison people for violating various national speech codes. Edward Snowden is in political asylum because he is rightly worried the US won’t grant him a fair trial for releasing information to the public. Despite this, all the servers storing the released information, and all the routers transmitting that information create an emergent phenomenon by which the information stays public.

It would require global totalitarianism to reliably and permanently remove information from public availability. In the absence of that, any single entity that tries to erase widely dispersed information renders itself irrelevant, or worse, achieves the opposite of their desire; the Barbara Streisand Effect.^[1]

Watch Audrey Tang’s seminar on Tool’s for Openness.

Prediction Markets

Free information is a foundational pillar of cooperation but today's information is often rather disorganized. Can we do better by supercharging how we make sense of it? Philosopher Karl Popper observed that knowledge, much as biology, evolves by a process of variation, replication, and selection. Variation of knowledge as in tossing new ideas out there, replication of knowledge as in spreading ideas, and selection of knowledge as the criticism of ideas. A healthy info system will allow good ideas to rise to the surface.

Prediction markets are such a system. Take the early market set up at Foresight’s 1999 annual gathering (screenshot below). People would contribute their expertise by opening a claim about the future for betting on a platform. For instance, “World product doubles 2025-30” tracks the likelihood of a substantial increase in world economic productivity by 2030.1 Prices represent a bet that a particular event will occur. If others believe the future price will be higher than the market indicates, they buy, and in so doing raise the consensus price. For instance, the odds for World GDP doubling were at 30-38% in 1999, moving to a more conservative 25-50% by 2000.

According to prediction market pioneer Robin Hanson, these markets can help swap our reliance on poorly incentivized experts for a market in ideas that encourages individuals to contribute their specialized knowledge: “if markets create a consensus about the value of an ownable item, such as the price, futures markets create an immediate consensus about future consensus.”

Plenty of variations are possible. Replication markets would be prediction markets that estimate the reproducibility of scientific research, reputation markets would trade in the estimated reputation of individuals or corporations and incentive markets would allow participants to incentivize actions by betting against them, making it lucrative for others to go after that very same action to prove the prediction wrong.

We can’t wave a magic wand and create a world free from interest groups trying to manipulate the information ecosystem in their favor. Instead, we can create a robust system against those forces. While we can’t assume that all problems are solvable, we should not underestimate the cleverness of games with novel payoffs for solving our problems. There will always be imperfections that an omniscient god could resolve. But the key feature of this system is that the overall direction of opinions results from the voluntary interaction of many individuals without anybody in charge. History showed that an alternative system of knowledge that could be commanded to operate in a particular way risks extremely destructive commands.

Listen to this group discussion on prediction markets on the problems they may solve.

New Rights to (Do) Things

When introducing property rights as one of the most basic institutions in the last chapter, we suggested that rights to things are really heuristics for rights to do things. But sometimes efficient rights to do things don’t correspond cleanly to tangible objects or geometric slices of the world. Our initial notion of property emerged from the game theory of trying to coordinate with each other.

Because the Earth’s crust moves slowly, we could force our notion of property onto this very naïve geometric notion corresponding to slices of the world. But there is no simple objective source of property boundaries in the physical world because physics itself has so many nonlocal interactions: If I move my arm so I am sending gravitons through your body, am I trespassing? As civilization and our desire to cooperate in complex ways grew, we advanced from geometrical slices of the world to require things like the rights of way so planes could fly over people's property, radio spectrum, pollution credits etc.

Space Property Rights

We should expect a future rich with property rights experimentation. For instance, take property in outer space: Plots of land do not necessarily generalize to bodies moving through space in ballistic trajectories. In our solar system, bodies orbit around each other and occlude each other with respect to sunlight and its energy. What if someone builds a Dyson Sphere between me and the Sun? Even if it didn't intersect my orbit, does it violate my property since I can no longer receive energy from the Sun? Determining rights in terms of orbits or radiation is clumsy and difficult.

Historical denotations of plots of land is not a bad starting point for outer space property rights. However, by starting with something physical as an initial endowment, we face substantial transaction costs when rearranging our endowments into something adapted to the activities we want to engage in.

Imagine that if you want to use a piece of radio spectrum as it passes through many plots of land, you would ultimately have to acquire it from every plot owner over that area. With property rights boundaries as vertical slices and the efficient property rights values as horizontal slices cutting through the vertical slices, this is a worst case misallocation of resources that maximizes transaction costs. Trying to get from the vertical regime to the horizontal regime by individual trades becomes extremely hard to do. For outer space property rights, we need to avoid outrageously high transaction costs.

If we discover a workable space property arrangement, it needs to be implemented with enough initial legitimacy that it isn’t immediately contested. Establishing legitimate resource claims relies on a secure title registry. We want to depend on this registry to do the right thing, not due to offered incentives, but because that is what its program requires. A permissionless blockchain might do the trick.

The blockchain, has two characteristics making it ideal for creating an incorruptible cooperation layer. First, it operates according to its stated specifications. Bitcoin is a virtual computer built out of agreement rather than physical hardware. A tremendous number of separate machines replicate the same computation and cross-check each other. While physical hardware may have security trap doors, if a quorum of many different algorithms that don’t trust each other agree on a transaction, this is a credible machine to run our computation on.

Blockchain’s second innovation is that it provides an agreed message order. A single piece of arbiter hardware can implement censorship by deciding to never see a message it doesn’t want to accept. In contrast, at least for proof of work, messages coming into the blockchain are coming into the memory pool, the transactions waiting room, and are visible by the blockchain’s miners. The miners compete to gather the messages into a block and publish them. This avoids the problem of double spending, which is fundamental to currency creation, something we'll discuss next.2

For registering space property rights, it is the combination of the blockchain’s ability to act according to the stated specifications and to reliably service requests, that makes it a candidate for creating incorruptible institutions. Knowing it’s the internal logic that makes it function reliably lets the rest of us incentive-align around it.

As long as there is no competing governmental title registry for the resources, claims on this title registry blockchain could be created without an opposing claim. There might be little interest in abolishing its legitimacy and much interest in upholding it. Suddenly, there are space property rights where previously there were none. Everyone is now either better or similarly off as in the absence of the registry. It's not impossible that establishing a blockchain-based system for space resources starts off seeming so unreal to people that it can grow to substantial legitimacy without competing government claims to the same resources.

Watch the seminar on Legal Considerations for Space Property Rights.

Reliable Money

The legacy monetary system comes with a plethora of pathologies: Robert Mugabe can print endless cash, inflating away the savings of Zimbabwe’s citizens. Vladimir Putin can freeze an NGO’s bank account, and refugees can get locked out of the banking system. Situations such as Operation Choke Point, when the US government used its control over banks to shut down unquestionably legal businesses, or when credit card companies blocked payments to WikiLeaks, show democracies aren’t insulated from these problems either.

Fortunately, Bitcoin happened. Built on the blockchain, Bitcoin injects internet properties such as programmability, interoperability, and composability into monetary value exchange, while guaranteeing its own scarcity. Bitcoin is a great example of a new layer of rules built on top of the internet’s permanence. We can think of its rules as a constitutional system for the digital jurisdiction, so that the web’s information economy can now operate within it. It directly threatens what offline jurisdictions consider their prerogative; minting money. Nevertheless, it succeeded at providing the world with a currency that is very costly to corrupt, even for governmental jurisdictions.

Because crypto offers a cross-jurisdictional, censorship-resistant exchange medium, it is not only attractive for people lacking access to institutions, but also for those who distrust them. It allows anyone to store, send, and receive money without asking permission and without proving one’s identity. Over half the world’s population live under an authoritarian regime, and could stand to benefit from this. In Nassim Taleb’s words, it offers an “insurance policy against an Orwellian Future”. Simple money is the most obvious high-leverage institution needing public non-corruptibility and our initial success is encouraging here.

Private Transactions

Bitcoin gives us monetary sovereignty. But Bitcoin, and almost all existing cryptocurrencies, put all transactions on an open public ledger, readable by anyone anywhere. Ethereum, and most blockchains that support smart contracts today, do all their contract execution on an equally public ledger. Although participation is said to be anonymous, in practice it is easy to do traffic analysis, a form of statistical reasoning, to correlate blockchain activity with players in the physical world. The tools for doing traffic analysis are quickly becoming a commodity. Because blockchains are immutable, malicious actors have lots of time to learn more about our activities.

Blockchains without strong privacy, i.e. most blockchains today, create dangerous new opportunities for real world crime. If real-world commerce moves onto such blockchains, private agreements would be near impossible. So-called rubber hose attacks and kidnapping come immediately to mind. Besides individual criminals, we should also worry about corrupt governments or interest groups engaged in human rights abuses or blackmailing.

Innovations like zero-knowledge proofs make it possible to encrypt transactions fully, while still allowing them to be verified via consensus. Blockchains with privacy would deny anyone the information needed to figure out who to attack. Currently, this level of privacy is available for cryptocurrency but not widely used. We should make it the default.

Check out this seminar on Zero-knowledge enabled Cooperation.

Complex Contracts

Complex play can emerge from simple rules. Through commerce we gradually figured out how to use contracts to bind ourselves to complex games. Right of contract states that voluntary arrangements between consenting adults should be allowed because they allow adults to find novel ways to cooperate.

To cooperate between mutually suspicious parties, we mostly rely on legal contracts. Contract enforcement is expensive and most disputed commercial relations are well below the threshold for going to court or arbitration. This limitation creates barriers. Unless we are rich enough to take everything to court, complex forms of cooperation are restricted to those we trust and have long-standing iterated relationships with.

Being members of the same jurisdiction somewhat reduces risk since operating under the same law lets us assume each other will behave well. The internet enabled the novel opportunity to create a website and cooperate with millions of total strangers. Still, on the internet, you might engage with someone who is completely anonymous. Some cyber attacks are committed in jurisdictions where the attackers aren't subject to punishment.

The result is that cooperation among strangers at a distance is restricted to simple interactions, such as giving away information for free. Even taking payment in exchange for something, a rather simple deal in itself, is often enabled by third parties such as credit card companies taking on the transaction risk. Our ability to cooperate richly with the majority of the world whom we don’t know still remains underdeveloped.

Smart contracts could change this. They were conceived by Nick Szabo in 1996 to automate contract execution by embedding it "in the hardware and software we deal with”. Smart contracts are contract-like arrangements expressed in program code, where the program’s behavior enforces the contract’s terms. They create automated arrangements to which we bind ourselves by placing rights into escrow with the contract. These rights are only released back to the contract participants according to the agreed contract terms.

Lawrence Lessig suggests we can achieve a particular end through different means, with legal contracts being one and computer code being another. Having only been able to build cooperative arrangements whose execution relies on human beings, we struggled to insulate them from human corruptibility. Smart contracts significantly increase the range of possibilities for what computer code can achieve. This makes cooperation reliable, credible, and trustworthy to an extent never achievable when institutions had to rely on human beings to function.

Ethereum, a general purpose blockchain, was the first real-world system to establish a sound, smart contract architecture. With Ethereum as a precedent, we now have a user-extensible rules system in which every contract is a new set of rules that cannot be shut down. Rather than being the last invention of this new level of the game, we should think of smart contracts as a key to unlock the next levels.

Reusable Templates

In the past, making attractive printed materials required going to a print shop. With the advent of laser printers and the Macintosh, suddenly regular people could create and print attractive designs at home. What followed was a lot of horrible-looking newsletters with tremendous overuse of fonts. People had yet to develop the skills to best use their new tools, so demand built to improve them. Early desktop publishing programs evolved with templates guiding users away from using Comic Sans for everything.

Similarly, our cooperative arrangements are currently crafted by highly specialized, highly paid lawyers. They are supposed to be good at writing contracts that don't fail in unexpected costly ways. Most of a legal contract is actually about how to deal with a variety of failure cases and pathological contingencies that a layperson never would have thought of. Early smart contracts will have effects no party would have wanted if they anticipated them. It takes time to develop tools and templates that embody expertise about unintended consequences and contingencies. Just as early, awkward design efforts evolved into today’s slick templates allowing anyone to create sophisticated publications, automated contracts will evolve from clumsy, costly proofs of concepts to ever more sophisticated plug-and-play templates.

While code for smart contract templates is expensive to create the first time, it can be reused indefinitely. The more smart contracts are used, the more they get play-tested with regard to unintended consequences so that we can rely on those that prove robust over time. As the ecosystem grows, so will the demand for improved skills. Those who improve first can create templates and tools to benefit the rest of us. JavaScript already delivers a secure and easy-to-use toolkit that empowers non-expert programmers to write simple smart contracts that can be understood by non-experts. Equipped with the ability to enforce rights and responsibilities in code at low cost, people across the world are empowered to exchange, trade, and cooperate in new ways.

Split Contracts

Not everything can or should be automated. We may never be able to flawlessly represent our intentions in legalese or code without risking an outcome not reflecting what we actually wanted. Dumb paper contracts lock in states without knowing what future participants will consider relevant. Lawyers try to freeze the next 10 years in hard-to-parse prose, only to litigate when the contract fails. When writing a contract, we are writing interaction rules for an unanticipated world. We face an alignment problem among current and future parties. An open contract is a plausible alignment strategy.

Rather than litigating and blaming when predetermined contracts fail, open contracts would let parties embrace change and improve the game’s next iteration. Often we will want to rely on a human at crucial steps to negotiate future parts of an agreement chain. They could take into account information from previous rounds and consider new circumstances when initiating the game’s next iteration. Preserving a human in the loop can be a useful feature. Sometimes we don’t just want the automated contract to be the only credible commitment. Instead we want the credible commitment to account for factors that require human judgment.

A Split Contract would divide the contract into two parts: the automated part directly enforced by software, and the prose humans interpret in the case of dispute. If either party decides to dispute the contract’s automated part, pre-chosen arbitrators judge the disputed terms and may overrule them. This has two advantages:

First, the contract terms are fine-grained; specifying the amount paid upon committing to the contract, the amount paid upon delivering the answer, and the amount paid when agreeing it satisfies her criteria. Dumb paper contracts leave the default situation determined by the asset’s default location. By escrowing money, the contract changes this default distribution. In Nick Szabo’s terms, from possession being 90% of the law, we move to behavior becoming 90% of the law. By rearranging the burden of initiating a dispute, both parties get skin in the game to comply.

The second innovative feature is the fine-grained negotiation process for the contract terms. Payment amounts, prose text, and arbitrator selection are left up to the discretion of the contractors. By co-negotiating those parameters upfront, they can make expensive dispute resolution less likely. We dispute a contract hoping we win, but it carries some expense and risk. The obvious cost is that human arbitrators are expensive, but a more subtle one is uncertainty. If we don't dispute, we are stuck with a disliked, but known, outcome. Yet, once subject to the arbitrator's judgment, the result is less predictable. If we could have predicted it, we could have written a more mechanical prediction into the contract in the first place. Reducing the odds of a human dispute in the first place is a great hack for reducing enforcement costs.3

Watch this seminar on Split Contracts and Decentralized Arbitration.

The Ancient Technology of Reputation

Mixed Contracts combine the best of contracts with reputation. To see the value of reputation, let’s compare our consumer relationships to an iterated prisoner's dilemma in which a producer is defecting on the arrangement by selling a worthless product. To retaliate in the next round, we must notice this defection and decide to warn others away. Since this process is expensive and we frequently find ourselves in non-iterated games, reputation agencies have evolved. Reports by consumers put the producer in an iterated situation with the community as a whole while trademarking establishes valuable long-term producer reputations.

Reputation, if paired with contracts, can help create entirely novel arrangements. Crucially, such a reputation system could also work if Carol maintains pseudonymity. If a negative reputation system is one where participants avoid entities with a bad reputation, a positive reputation system is one in which participants seek out those with a good reputation. Pseudonyms are problematic for negative reputation systems, because it is easy to let go of a negative reputation by switching between pseudonyms.

Positive reputation systems can still succeed as long as no one can claim another’s identity. Imagine pseudonymous Alice and Bob would like to draft up a contract that relies on legal enforcement. Let's say Alice also has a contract with Carol, so she is subject to legal enforcement that is invisible to Bob. Carol can assure Bob that she has enforcement powers, since she knows Alice’s real identity. If she is reputable enough, this may assure Bob even if the specifics are opaque to him.

If Carol wants to continue being consulted for arranging deals between the Alices and Bobs in this world, she will make sure that she can honor her part of the contract. This role is similar to the role credit card companies played in enabling today's e-commerce. The demand for reputation information may eventually provide a market for pseudonymous reputation services, analogous to credit rating agencies and consumer experiences.

Check out this seminar on Blockchain Governance.

A New Menu of Organizational Choices

When starting a new organization, today's laws constrain founders to choose from a menu of confusing options ranging from a for-profit to a classic non-profit 501(c)(3), with several organizational types in-between. The combination of court cases and legislative compromise that led to the particular combinations available in each state is so complicated that only a lawyer can untangle them. They solidified in a very different past from our modern world, dating back to before the World Wide Web.

Compare this to a Decentralized Autonomous Organization, aka DAO. It’s a network-native entity with no associated central management and its own asset pool. Smart contracts between the organization stakeholders may decide how to use the assets. Through DAOs, people can participate in many new organizational structures that distribute reputation or money according to value contributed or make decisions via the wisdom of the crowd. It is very costly for people within the organization to prevent the relevant set of immutable smart contracts from executing as specified. Because they run on a blockchain with international nodes, these contracts are more resistant to internal and external corruption. This makes them an excellent candidate for experimenting with previously unthinkable cooperative arrangements.

DAO templates, pol.is voting systems, and other emerging tools could equip the rising tide of internet communities with collective ways to pool assets and govern over them to pursue shared goals. Many experiments will fail but we only need a few successes to create an entirely new menu of organizational choices. After all, current institutions are also made of contracts.

Watch this group discussion on Decentralized Autonomous Organizations.

Collective Action Problems

The world is full of possibilities we all would like to see realized, but no individual has an incentive to contribute to. This is especially true if we think we can benefit without contributing; the classic free rider problem that is at the root of the world's nastiest collective action dilemmas. But what if, instead of worrying that a small contribution won’t make a difference, you could ensure that it does? Kickstarter allows you to pre-commit to fund the open source project of our choice if and only if enough others commit to do the same to reach a critical threshold. If we all want an outcome enough, we can often move there via assured agreement. If not, no action takes place and any committed funds are returned. Such Assurance Contracts are great tools for solving large, multiway group commitments.

Dominant Assurance Contracts

A problem with assurance contracts is that even with minor transaction costs there is too little incentive to contribute. Imagine an open source software improvement costs $800 to fund and is worth $100 to each of 10 people. If each person pays $80, they get the results and are all better off. But each person may choose not to donate because they may still benefit from the software bug fixed, even if they don’t contribute. If you think that no-one else will contribute, it’s rational for you not to contribute; however, if you don’t, then why would anyone else? Not contributing can quickly turn into a self-fulfilling prophecy.

A creative solution, introduced by Alex Tabarrok, is the Dominant Assurance Contract. It’s an assurance contract with the added condition that if the funding benchmark isn’t reached, the sponsor must pay a prize to the pledgers. Pledging now becomes a dominant strategy, or in Tabarrok’s words, “a no-lose proposition—if enough people pledge you get the public good and if not enough pledge you get the prize.”

Listen to this seminar on Dominant Assurance Contracts.

Quadratic Funding

Another potential solution to help with the free-rider problem of public goods is Quadratic Funding. By pledging even small amounts to desired goods, participants can direct the relative funding amount of external entities which agree to match the funds. The trick is that matched amounts are calculated by using the quadratic funding formula. This is where the amount a project receives is proportional to the square of the sum of the square roots of received contributions. Matching sums are based on the number of supporters, not the amount donated. This creates strong incentives to give even a little, ultimately countering incentives to free-ride.

Retroactive Public Goods Funding adds to this the layer that it’s often much easier to agree on what was useful rather than what will be useful. By rewarding public goods projects that were successful, it can incentivize their creation. Such experiments are at an early stage but, if successful, they can create what Vitalik Buterin calls “a general purpose infrastructure for funding public goods in the same way that money is a general purpose infrastructure for funding private goods.”

Listen to Glen Weyl’s talk on how to develop political and economic systems that take more of the real social and political life of people into account.

Chapter Summary

Various costs prevent us from cooperating creatively. It takes time, energy, and money to find each other, bargain an agreement, and enforce it. If only we could lower these costs, cooperation could be rich. The digital realm provides new arenas to reinvent cooperation. New media, property rights, and contract experiments are just the tip of the iceberg. Most of these experiments will fail but the few that succeed may usher in a new era of cooperation. It’s hard to see how to get there from here. In the next chapter, we’ll take the first steps.

Next chapter: GENETIC TAKEOVER | Cryptocommerce

3

AMIX, The American Information Exchange, shown below, was an early prototype of a Split Contract. Designed back in the 1980s before the web browser, it was a computer-mediated market for matching information buyers and sellers. Up to then, finding information on a topic involved fishing for relevant bits of knowledge from the sea of newspapers, TV, journals, and books. AMIX replaced this random walk with a targeted market on which you could ask questions and those with relevant expertise could sell their answers. Barbara, an early AMIX user, wanted to hire a consultant to figure out if it makes sense to build a co-working space in Palo Alto. She defines what would count as a satisfactory answer and uploads the request to the exchange. If a consultant delivers an answer and thirty days go by without Barbara taking any action, the automated contract would pay him the full amount. To stop the payment, Barbara would have to explicitly state his document does not meet her terms. The contract would then take those contradictory propositions to the arbitrators. They would only get to overrule the amounts paid on accepting the document. Meanwhile, with paper contracts, the consultant could have given Barbara a perfectly fine document. She could refuse to pay, leaving him with the expensive legal action to get what she owes him.

Having met our players, with their diversity of values, we can’t rely on aligning them on one grand strategy. Instead, to set civilization up well over the next rounds of play, we must build a playing field that can handle fundamental value differences. 

A Manual for Strengthening Civilization

Bob’s Preferences

Imagine a game of civilization played by Alice and Bob. A world in which different players have different goals can be described in terms of preferences among future states. The center dot is the current state of the world that players Alice and Bob are in. The axes are the world states, organized by Alice's preferences vertically and by Bob's preferences horizontally. Bob prefers the green worlds to the current world. 

Positive Sum & Negative Sum

If we could extrapolate utilities from Alice's preferences and Bob’s preferences, we could say their interactions can lead to outcomes that have greater overall utility or smaller overall utility. Meaningfully comparing utilities across players will become more problematic the more diverse their futures get. But for now, let’s assume everything to the upper right of the red line are “positive sum” outcomes, and everything to the left are “negative sum outcomes”.

Voluntary Cooperation

There is a problem with simply seeking positive sum outcomes. If Bob would be worse off than he currently is, he would fight any attempt to get there. Likewise, Alice would fight the positive sum outcomes she likes less than the status quo. But if Alice and Bob are either equally or better off than they currently are, both have good reason to cooperate. Together, they can move to Pareto-preferred worlds. Situation B is Pareto-preferred to situation A if anyone prefers B to A, and no one prefers A to B. Those worlds can be reached by voluntary cooperation. For human players, we could say these interactions are “freely” consented to; for non-human players, they are simply based on their “internal logic”. 

 Cooperation Across Humans                          

Human similarities also come with the tendency to compare oneself to others, including strong fairness intuitions and envy reactions. If Alice’s gain is perceived as too unfair, only she would be invested into bringing about that future, even if, all else equal, Bob would have consented to the deal. The all-too-human tendency to compare ourselves to others may lead Bob to reject a Pareto-preferred deal. It narrows the scope of what the world’s human players can achieve by voluntary cooperation. 

Cooperation Across Intelligences       

Traditionally, the definition of an agent with utility assumes a comparability that future intelligent systems don't necessarily have going forward. Without meaningful metrics on which to compare utility across very different mind architectures, the diagonal red line, indicating positive and negative sum, disappears.

As long as players have goals and act as though they make choices, they will have revealed preferences. Those revealed preferences may be all we have when designing systems for players to reach their goals. Upholding voluntary cooperation could remain a stable common goal for both Alice and Bob across many rounds of future games, regardless of their intelligence. It’s all they need to unlock Paretotropian worlds that are better for each by their standards.

The rest of this book is about how to set civilization up for this path of intelligent voluntary cooperation. This path contains three components:

1. Upholding Voluntarism

 Involuntary Positive Sum                                  

Imagine that Alice, seeking a positive sum arrangement that makes her vastly better off, explains to Bob: “I'll be more better off than you'll be worse off” and embarks on her way to the blue point in the diagram. Bob doesn't like this plan so we have a conflict. Not only do we have a conflict, but Alice expects that there will be this conflict and Bob expects that Alice will expect the conflict.

Hobbesian Traps

In expectation of Alice’s involuntary action, Bob may strike first. Alice, expecting this, will want to weaken Bob first. A cascade of mutually expected conflict can result in a Hobbesian Trap, where the mutual expectation of conflict creates a preemptive conflict. While cooperation is better for both sides, lack of trust or fear of defection can lead to first-strike instabilities, wars, and other terrible games. By reliably upholding voluntary interactions as Schelling Point, and signaling this to Bob, Alice can lessen her and Bob’s incentive to introduce and abuse precedents that could potentially spiral into Hobbesian Traps. 

2. Improving Cooperation

Single Anonymous Interaction                                            

Today’s world of cooperation is complex and involves anonymous situations where neither reputation nor contracts can grip. In situations where all players jointly prefer an outcome, but cannot get there by only interacting voluntarily, an obstacle is obstructing their path to Pareto-preferred worlds. To cross the obstacle, we need better tools for cooperation.

Imagine Bob gave Anonymous Alice 10 shekels for the promise of a Gourd. She is 10 shekels richer and he is 10 shekels poorer. Anonymous Alice would no longer have a reason to give Bob what he wants. Upon Bob inquiring where his Gourd is, Anonymous Alice may run off, quoting Hobbes, “For he that performed first has no assurance the other will perform after, because the bonds of words are too weak”. Bob, knowing this, would never give Alice the 10 shekels in the first place. What if Bob could lock up his 10 shekels in escrow that automatically pays Anonymous Alice when she proves that she has delivered the Gourd?

Improving Technologies of Cooperation    

A tremendous number of remaining problems are basically the same phenomenon writ large. If all Alices and Bobs in this world could find each other, they may be able to build bridges to Pareto-preferred worlds. In reality, collective action dilemmas are often phenomena in which 1000s, or millions of players interact simultaneously. Rather than being simple trades, they might be complex arrangements that unfold over time. Players can’t get to preferred worlds in a way that ensures all are better off at each step along the way. To tackle these problems, we need to innovate in technologies of cooperation and democratize their use.

^{3. Intelligizing Voluntary Cooperation}

Civilization as Diagram with 7 Billion Dimensions

When considering Alice and Bob, we need to remember that we’re actually looking at a 7+ billion dimensional diagram. Any particular interaction involves only a bounded set of aspects of the world and only a bounded number of participants. For each one of these interactions, there is a separate diagram, in which we can organize the possible states of the world. 

Voluntary Independence

Imagine Carol and Dave are in some other part of the world. They have never heard of Alice and Bob, who have never heard of Carol and Dave. Let's rotate Bob out of the diagram and rotate Carol in. In parallel to Alice’s and Bob’s interaction, Carol is cooperating with Dave to bring about a world state in which Carol is better off. Even though each of these individual transitions hug the edges of the Pareto-box, collectively, they are taking orthogonal steps into the Pareto-preferred area. 

In order for the right to choose to cooperate to be meaningful, we need the right to choose not to cooperate. Voluntary independence is actually most of the 7+ billion dimensional diagram. Most people don't know each other and most of their activities have no strong connection to most other activities in the world.

Thanks to the independence of the arrangements that form in a vast experimentation space, some arrangements can go forward when others get stuck. The system continuously selects for arrangements that create productive cooperation such that it is dominated by their beneficial results. While we should improve our arrangements, this process is about becoming better at aligning everyone’s expectations of increasing payoffs by actually increasing payoffs. 

 Strengthening Civilization’s Paretotropism

Plants have a phototropism, in that they grow toward the light. Civilization has no goals of its own. But it does have a dynamic, a paretotropism: In the same way that plants grow toward the light, civilization, emerging from our voluntary interactions with each other, progresses towards worlds that are generally better for everyone. By cooperating via voluntary choices, and otherwise leaving each other alone for voluntary independence, our civilization is climbing Pareto-preferred hills.

To continue on this trajectory in the future, we need to find to extend the architecture of voluntary cooperation that has emerged among humans to other intelligences which will soon dominate the playing field.

Intelligent voluntary cooperation relies on respecting the boundaries of other entities such that their reaction is based on their internal logic. For humans, we have long-evolved norms around respecting our corporeal boundaries. In today’s object-oriented programming, specialized computing entities are encapsulated so one object cannot tamper with another’s contents. Such existing examples of boundaries may provide useful guidance when designing the cooperation architectures for the next rounds ahead. Successfully integrated into our fabric of voluntary request making, new intelligences have the potential to turbocharge the paretotropism of our civilization.

Intelligent Voluntary Cooperation: The Real World

Having looked at a manual for intelligent voluntary cooperation, let’s see how the strategy works in the real world and which lessons to apply in the game ahead.

1. Upholding Voluntarism

The Game So Far: Voluntary Schelling Points Emerge

How has voluntarism shaped our civilizational game? Data on prehistoric societies suggests that our ancestors lived with extraordinarily awful rates of violence.1 Since then, violent deaths have decreased, first relative to population size, and more recently in absolute numbers. Major powers have not fought a physical war in a while, and while there are more civil conflicts, they bring fewer deaths.2 

The decline in violence hasn’t been perfect but it has gradually shifted the balance of our interactions from involuntary interactions towards voluntary ones. We increasingly have the freedom to say no to interactions without being forced into them. 

With cooperation becoming a better strategy than violence to achieve our goals, it became beneficial to understand other peoples' goals so we can offer them cooperation opportunities.3 If I need money, and want to sell you a product you want, it pays if I can imagine what kind of product you would enjoy. Effective cooperation, such as in commerce, involves creating situations in which others serving our goals also serves theirs. The instrumental goal of cooperating evolved into the felt goal of caring about others' goals, i.e. empathy. Global mass media may have helped extend our empathy by increasing our ability to put ourselves in the shoes of strangers. Seeing each other as fellow humans, rather than dangerous outsiders, further reduces the impetus for violence.4

Think of one of the many bloody territorial wars across history. Often, not even victors come out net positive.  All players have an interest in avoiding such costly struggles. All sides should want to find a point of mutual agreement without anyone feeling they’ve made too much of a concession. If they can agree a disputed territory has an obvious geographic marking, this common-knowledge landmark could be a good point to settle on. The river can serve as Schelling Point for coordination. 

Thomas Schelling pioneered the idea that even without communication, we can often cooperate by converging on a focal point which we mutually expect to have prominence for the other.5  This is why many territorial boundaries are a mountain range or a river. 

We find a similar situation in American politics today. Political factions are increasingly polarized, and neither side finds the U.S. Constitution to be an ideal document for advancing their cause. All parties have some interest in a Constitutional Convention to rewrite the Constitution to exactly serve their purposes. However, all parties are terrified of risking a Constitutional Convention, and we should expect strong lobbying against it. The potential outcome is sufficiently risky that, while the current framework is not ideal for any party, it is better than what they might end up being forced to accept. The U.S. Constitution is stable because all players recognize that the prospect of its instability is more frightening than living with what we now have. 

Just as the river serves as a territorial Schelling Point, and the Constitution serves as a Schelling Point for U.S. governance, voluntarism is increasingly becoming a Schelling Point for our civilization. We all tend to gain by cooperating with each other and the more we rely on these dynamics to achieve our goals, the more we create feedback that keeps pushing the rules in that direction. Even if you find it is in your occasional interest to engage in involuntary interactions, your overall interest may well be to uphold the voluntary system if you expect to benefit from it. Weaker norms might better enable you to cheat but they also better enable everyone else to do so. The overall system is less likely to serve your own interests.

To the extent that we make exceptions for ourselves to rely on involuntary means when interacting with others, we establish precedents for interactions that overpower some players by non-agreed upon means.6 When voluntary Schelling Points are lost, we can spiral into Hobbesian traps, in which the mutual expectation of involuntary action leads to pre-emptive conflict.

Since the first strike instability during the Cold War we know that such traps can bring us to the brink of destroying the world.7 World War I with its entrenched warfare situation was another trap in which both sides were suffering tremendously, but neither knew how to get out of the situation. We must avoid Hobbesian Traps at all costs. 

The Game Ahead: Multipolarity & Compensating Dynamics

Multipolarity: More Like Natural Language, Less Like Governments

The more players uphold voluntary interactions as Schelling Point, the more we can rely on it. Unipolar systems with a single dominant actor can arbitrarily breach voluntarism without fearing feedback. Power is diminished when divided, and so is the power to engage in involuntary interactions. Multipolar systems can keep each other in check.

Take natural language as the ideal example of a voluntary system emerging from multipolar interaction. Historically, there was no understanding of the hard difference between the concepts of heat and temperature, or mass and weight.8 Instead, there was a cloud of words that sort of meant the same vague cloud of meanings. When people tried to reason precisely about this heat and temperature cloud of meaning, they were wrapped up in intellectual knots.9 Distinguishing between heat and temperature, allowed people to pick words from these clouds to nucleate them into different sounds of a newly explained concept. Sometimes it goes the other way, and concepts thought unrelated suddenly become connected. For instance, “entropy” is a synthesis of signals from engineering and thermodynamics.  

The coherence of natural language does not emerge from the top-down decision of an authoritative governance structure. Instead, it is a continuous re-negotiation with words as Schelling Points for meaning.10 The evolution of words has both a drive to coherence, where we mean the same thing, and sudden switching events, as we discover new concepts, create new coinages, vocabulary and jargon. Language is a beautiful, spontaneous order that emerges from the interaction of many players under the incentive of wanting to be understood by each other.11  

If language is a good example of extreme decentralization, markets are in an intermediate category. They don’t have decision-making power for the overall system but neither do they emerge only from individuals contracting directly with each other. Corporations are the dominant entities. They have internal centralized decision-making powers coexisting with a lot of bottom-up activity, and are themselves embedded within a multipolar world of markets. While individual large corporations are often seen to have a degree of agency, their overall market activity is still emergent through many parties interacting to try to influence the process. 

Let’s go up one layer of centralization to governments. In the U.S., power is decentralized in multiple instances via the Constitution. First, the Founding Fathers left most power in the hands of individuals and didn't centralize it in government. Second, they left most of the remaining governmental power to the states, not the federal government. Finally, the federal government power was divided among the legislative, judicial, and executive branches, and restricted via the Bill of Rights constitutional amendments. The founders were not confident this balance would be stable and there is no guarantee it will be. Today, we see formerly-decentralized decisions being gradually transferred from the states to the federal level. 

If we want cooperative partners to bring their local knowledge to bear and keep each other in check, any arrangement that can be more decentralized should be. This is embodied in the subsidiarity principle; when seeking to come to joint decisions, we should do so at the smallest scale necessary for a decision. Subsidiarity would not only favor decentralizing decisions to the state level instead of national level. It also means strengthening the cities’ role where most of us directly experience decision consequences. Within cities, subsidiarity would encourage decisions at the fine-grained local community level with fluid processes and reciprocity.

A move away from dominant hierarchies can be very beneficial. Robert Sapolsky shows how, in a baboon troop in which dominant violent males died due to an accident, the lower-ranking baboons, rather than reassembling traditional hierarchical pathologies, remained more peaceful and cooperative over the long-run.12 If baboons can do it, human players should be able to. The more we use multipolar interaction architectures (like languages) and less unipolar ones (like governments), the more we can rely on each other to uphold systems of voluntary cooperation.13  

Compensating Dynamics: From Peace of Westphalia to Crypto Nations

History shows we can work against strong pressures to centralize. The Peace of Westphalia stopped the notoriously bloody thirty year religious war in Europe. Each warring state sought to involuntarily impose their values onto the other side.14 All parties paid huge costs for following their own perceived best course of action. Finally, in 1648, they negotiated a peace allowing each nation to determine religious practices on their side of the boundary with little interference. Spain and France were assigned the role of guarantor powers that were, by treaty, obligated to defend the constitution of the Holy Roman Empire. They could be called upon by anyone injured according to the Peace. Instead of just providing friction against power centralization, it introduced a feedback loop that actively corrected and pulled the system away from centralization. 

The Peace of Westphalia lasted for centuries and it is sometimes regarded as one of the first applications of the collective security principle in International Relations.15 In a collective security arrangement, “each state in the system accepts that the security of one is the concern of all, and therefore commits to a collective response to threats to, and breaches to peace”. When one state in a pact gains power that threatens the peace, other states cooperate collectively against them. The actions of the rest restore the balance and provide security for all.

What is preferred to withstanding centralization? Actively self-correcting for it. If we realize early that part of an arrangement becomes a threat, a coalition of powers can proactively cooperate to restore equilibrium. In a system in which participants understand it is in their interests to preserve peaceful multipolarity, self-correcting dynamics can be built in. This is more than a system of “robust multipolarity”, where multiple players keep each other in check by watching each other. Systems in which participants cooperate to compensate for power grabs can create an “antifragile multipolarity” that gets stronger under stress. 

Charter cities and seasteading ambitions seek to create meatspace alternatives to national players. Cyberspace offers another arena to compensate for state power through the creation of transnational communities that are entirely voluntary. It has an almost perfect realization of a neutral simple rules framework to experiment with.16 Some already use this to prototype new forms of self-government in virtual communities, which increasingly become the realm in which we uphold each other’s rights.17 One day, crypto nations and network states, such as 1729, may allow their members to collectively negotiate with existing legal jurisdictions, expanding compensating powers from the digital to the physical world.18

Balaji Srinivasan popularized the idea of actualizing digital communities into the physical world through network states. Watch his Intelligent Cooperation seminar.

2. Improving Cooperation

The Game So Far: Rights, Contracts, Prices, and Institutions

How did we get from a civilization based on voluntary interactions to today’s sophisticated ways of cooperating, including property rights, contracts, and prices? Civilization is made up of many players pursuing many goals. Many of these plans make use of resources which are scarce. To pursue our goals, we had to gradually learn how to coordinate around a limited amount of resources. Property rights solve this problem. They confer the rights to things, which really is a proxy for the rights to do things. At first, a simple theory of accounting sufficed. Alice could tell her neighbor, Bob: “this cow is mine if it is on my property. In exchange for some fruits, I make it yours by moving it to your property”. 

But human beings are linguistic creatures, so rights about physical goods could be repackaged to more abstract arrangements. The transferability of physical goods created a model in our minds of what it means for something to be property. Once we had this notion, we could take less literally physical items, declare them to be property, and create abstract arrangements such as contracts to manipulate them. Thousands of years of the evolution of human institutions have given us technologies of commitment, such as reputation and contracts, to bind ourselves to mutually preferred agreements.19 Alice could tell Bob: “give me your fruits now and I promise to give you my cow tomorrow”. 

Pairwise barter deals where one good is exchanged for another for mutual benefit are hard to find and leave lots of value on the table. Because continuing to be a participant in a particular contract is itself valuable, that right can be treated as property for other contracts again. Bob could tell his neighbor Carole: “give me your grains, and you’ll get Alice’s cow when I get it in exchange for my fruits”. 

Such large multi-way deals would yield the full benefit of trade, but are difficult to negotiate. Currency allows the equivalent of multi-way deals via separate pair-wise trades. Prices can represent a summary of individual valuations of a good, the demanders’ use values and the suppliers’ costs. Instead of running back and forth between Alice and Carole, Bob can pay for the promise of Alice’s cow and use that currency to buy Carole’s grains.20

Via contracts, institutions for manipulating rights evolved that allow more parties to cooperate creatively. Gradually, markets emerged around the transferability of rights. By recognizing the contract participation right as property other contracts can use, we enable networks of contracts composed of contracts. This dynamic of higher order composition gave rise to today’s complex cooperation ecosystem. These arrangements were possible only by building more abstract arrangements from within a working system. Now we can reinvent them.

The Game Ahead: Polycentrism and Out-Competing

Forging Polycentric Bonds: Open Source 

A minimal set of voluntarism can lay the groundwork for constructing and joining cooperative arrangements with very complex rule sets.21 For instance, for managing common streams in the Swiss Valais, everyone in the community has a right to take from the stream, in return for a few days of maintenance work each year. These Swiss communities are small with high reciprocity and access control in the form of mountains.22 In civilization at large, we cannot casually eject people, so we find small-scale commons within larger scale units, with arrangements cutting across both. Having multiple poles of decision making that are independent of each other is often called “polycentrism”.23 Elinor Ostrom pioneered the concept by showing that it can help solve various collective action problems. 

But new cooperative arrangements come with their own pathologies, incentivizing people in unforeseen ways. Take today’s corporations as an example. A corporation has few ways to create an internal project with the guarantee that it won’t be canceled by its management. There are almost no formal means within the corporate governing structure and, if promised informally, there is no way of holding the company to it. Management can change their minds, and they seldom can deny themselves the right to do so. In response, many projects form as a collaboration between employees in multiple companies who use contracts to make credible commitments. Each of them think it is now much more credible that the project will stick around.  

Open source is an extreme case; if the company kills it, you can leave the company and continue working on it. Prior to open source, talented technologists have repeatedly put some of their best intellectual effort into a proprietary project that was canceled. They realized that projects which they were willing to start under those conditions weren’t the ones worth starting. With open source, companies can make a credible promise to the employees that they can keep working on a project even if the company loses interest. In turn, talented employees pour their heart and soul into the project.24 Making binding promises is an important means of cooperating. Rather than imposing a simple top-down model that makes this difficult, cooperative arrangements like open source cut across different circles. 

By joining different hierarchies in a variety of cross-cutting circles, we can compensate against arbitrary power in any one of them.25 Being part of overlapping communities is more manageable for our tendency to compare ourselves to our peers: Bob may be less affected by the grandiosity of Alice’s Dyson Sphere, when his wildlife spotter fan club praises him for preserving a pristine Platypus specimen.26 If you can seek social worth through different roles in different circles, violence may be less common than if all of your self-worth depends on one pecking order. So even if it can’t solve all collective action problems, polycentrism has other benefits by making it easier to engage in binding commitments, and harder to engage in involuntary actions.

Out-Competing Sub-Optimal Institutions: Education 

Every sub-optimality in our civilization should be an invitation to innovate—without upsetting voluntary Schelling Boundaries. Let's take education as a collective action problem. Scott Alexander diagnoses that “students’ incentive is to go to the most prestigious college they can get into so employers will hire them—whether or not they learn anything. Employers’ incentive is to get students from the most prestigious college they can so that they can defend their decision to their boss if it goes wrong—whether or not the college provides value added. And colleges’ incentive is to do whatever it takes to get more prestige —whether or not it helps students.”27 

When noticing an apparently maladaptive feature which has been part of a society growing and increasing in wealth and knowledge, let’s first stop and consider alternative functions. This avoids tearing down Chesterton’s Fence, the canonical example of a feature with an important function not understood by those seeking to remove it.28

American education is undoubtedly in a sub-optimal state. Nevertheless, the U.S. university system is still where most of  the world is eager to send their students. Much of the U.S.' actual knowledge growth has made use of its system of teaching and research, even if not in the ways we commonly think. Part of good universities’ attraction is prestige and trademarking.29 Even if a good school teaches a student nothing, graduating from one whose entrance criteria are predictors of intelligence is a signal. Especially since students from good schools spend years socializing with other students who went through the same filter. They received some education and a network whether their knowledge was actually increased by their classes or not.

While these are not official university functions, at least trademarks are not completely random. Noticing this divergence between claimed and delivered benefits is an innovation opportunity. The Thiel Fellowship has convinced high-potential students at good universities to join a prestigious entrepreneurial program instead. Having been accepted to the university first, they can still use this as a signal. This works so well in Silicon Valley that “dropping out” is becoming a positive signal itself.30 

Rather than rolling back evolved institutions, newly created systems can out-compete existing ones. Until we innovate by deploying our insight in the real world, we don’t know in how many ways it is wrong. If we underestimated how good the existing institutions were for reasons we didn’t understand, then it is good that our attempts to supersede them using voluntary cooperation failed. We want those attempts to fail. 

Even if we don’t solve a particular collective action problem via voluntary means, at least we are making poor choices among Pareto-preferred paths. If instead we could destroy institutions we didn’t appreciate by something more forceful than voluntary cooperation, we risk destroying hard-gained value, perhaps irrevocably so, and open up the door to Hobbesian Traps. Given what’s at stake in civilization, our goal needs to be depessimizing, not optimizing.31 

Curious for more? Check out Price Discovery in Markets & Civilizational Progress.

3. Intelligizing Voluntary Cooperation

The Game So Far: Civilization Serves Our Interests

From our vantage point, it’s hard to wrap our head around the many wonderful developments that got us where we are today. Circa 1820, nine in ten people on the planet lived on less than $1.90 per day. In 2015, this was only around one in ten. As the population increased, at first, only the ratio of people in extreme poverty fell. But more recently, the actual absolute number of people in poverty is plummeting faster and faster while our overall population continues to grow.32 

We are making progress on so many fronts. Modern hygiene, better nutrition and medicinal advances means fewer children die of starvation and the elderly can rejoice in living healthier lives.33 We are rapidly becoming more educated; not only are more kids attending school for longer, but in most countries, almost the entire population is literate now.34 Countless other achievements could be added, such as greater personal autonomy and fun, or improved communication and travel.35 

Recently, obstacles to our ascent have become more apparent. Some worry that the rate of scientific progress is slowing down, and we are entering a “great stagnation”.36 Others worry that even as technology and productivity improve, prices are rising in areas such as healthcare and education.37 While - thanks to a mix of competition and digitalization - key air pollutants, such as sulfur dioxide, are in decline, pollution continues to harm health and livelihoods, displacing humans and other species.38 Technological progress coupled with increasingly global reach also means increasing the potential reach of technologies’ violent uses.39 Those are tricky problems. Nevertheless, in broad strokes, civilization has been rapidly getting better at serving our interests.

If you’re surprised to hear this, you’re not alone. A 2016 poll found only eight in a hundred U.S. residents knew that global poverty declined in the previous 20 years.40 No wonder; it’s hard to appreciate how incremental positive developments add up across history. You weren’t alive yet when, in 1900, almost half of US households had more than one occupant per room, only every fourth household had running water, and only six in one hundred kids graduated high school. Since then, U.S. living standards have increased at least five-fold.41 

But expectations adjust, and the remaining brutal aspects of everyday interactions stand out. Scott Alexander captures this sentiment well: “Fit companies—defined as those that make the customer want to buy from them—survive, expand, and inspire future efforts, and unfit companies—defined as those no one wants to buy from—go bankrupt and die out along with their company DNA. The reasons Nature is red in tooth and claw are the same reasons the market is ruthless and exploitative.”42

If this sounds intuitive, consider an alternative perspective. Zooming in on evolution shows the differences between today’s civilization and nature’s predatory principles: “To survive, animals must eat animals or plants; this happens most often through predation. Producers, unlike prey, will voluntarily seek out those who want to consume what they have, using advertising, distribution networks, and so forth. Consumers, less surprisingly, will seek out producers [...] by reading advertising, traveling to stores, and so forth. The symbiotic nature of this interaction is shown by the interest each side has in facilitating it.[…]In human markets (as in idealized markets) producers within an industry compete, but chains of symbiotic trade connect industry to industry.” 43

In today’s civilization, competitors compete at better cooperating with the rest of the world. While competition is the norm within industries, chains of symbiotic trade are the norm across them. But we tend to find uncommon phenomena interesting and common phenomena boring, and so focus on unusual predatory behavior and crime over boring symbiosis in trade and markets. 

This tendency, let’s call it an outlier inversion, is amplified by the media reporting on interesting cases of crime and embezzlement rather than on positive trends, such as the daily massive satisfactory turnover of goods. When pointing out how far we still have to go, we can start with appreciating that we’ve come a long way. Progress occurs even when unnoticed. But noticing it seems an important first step towards accelerating it.44

The crucial thing in common is that our achievements are attractive from many perspectives, regardless of what one values and which goals one pursues: Longer lives provide more time to pursue goals, improved health provides more energy to do so, and better education and more freedom improve one’s chances of achieving them.45 Gradually and imperfectly, civilization is climbing Pareto-preferred hills: “If pairwise barter amounts to Pareto-hill-climbing across a rough terrain with few available moves; trade in a system with currency and prices amounts to hill-climbing across a smoother terrain with many available moves”.46 Tyler Cowen compares our modern civilization with a Crusonia plant, a mythical automatically-growing crop that generates more output each period. The more crops, the more there is to go around for different players to pursue their goals. 

Listen to Tyler Cowen’s seminar on civilization progress and stumbling blocks.

The Game Ahead: Boundaries and Composition

Set Voluntary Boundaries: Institutions & Computers

The evolution of complex institutions starts with our need to create plans in ignorance of the plans other players are creating. Rather than incentivization alone, a main bottleneck for civilization is coordination around limitations of knowledge. Even if we all wanted to maximally cooperate with each other, we would still mostly be ignorant about how best to do this. Cooperation brings local knowledge to bear for intelligent problem-solving:

If you want to have a package delivered, you walk into a post office, hand the clerk on the other side of the counter a package, tell her where it should go, give her the money, and the package is delivered to its specified destination. The clerk does not need specialized knowledge that you are sending your father a birthday present. Likewise, the package delivery service knows about trucks, airplanes, schedules, and barcodes that you don’t need to know. A lot of specialized knowledge on each side is abstracted through the common interface of a package delivery service.47 

The package delivery service functions as an abstraction boundary over two variable factors; why someone wants to deliver a package and the multiple means to deliver it. It can convert specific relationships into abstract relationships, and each side’s local knowledge can be composed into more complex problem-solving abilities. In addition to traditional institutions, we rely heavily on these less formal day-to-day institutions to cooperate.48 When such successful templates for cooperation emerge, they reduce the costs of reusing working solutions. We can draw on ever wider range of knowledge without having to acquire it ourselves. 

The benefit of composing local knowledge into more intelligent systems is only too familiar to software engineers: In the early days of software engineering, they realized big problems need to be broken down into smaller ones and the plans to solve them. But if each plan can draw on any program resources, any memory address, any bit of data, etc., the result is a massive interference problem that creates bugs everywhere.49 Knowledge locality in the face of plan complexity created a reliance on specialized objects. By dividing resources into portions which can have separately held rights, we can formulate parts of the plan in confidence it can unfold without arbitrary interference from other plans using those same resources. 

Complicated programs are broken down into simpler objects that engineers know how to build. Networks of objects communicate to exchange information. The information carried by each object is encapsulated so one object cannot tamper with another’s contents. These boundaries create independence across objects but also allow for them to be composed into architectures of cooperation. When the request-maker sends its request, the recipient responds only according to its internal logic. Via such voluntary request-making, separate objects can solve a small component of a problem, and combine their knowledge into much greater problem-solving ability. 

Composability is key in this process. In foundational mathematics, a predicate operating on numbers is a first order predicate, while a predicate that operates on first order predicates is a second order predicate. Unlike in foundational mathematics, in programming, higher order predicates don’t have to have a particular place in the order of predicates. They can just operate on higher order predicates without restriction or stratification. This lets us build abstraction layers on top of abstraction layers, which can both manipulate objects and be manipulated by objects. It’s thanks to this generic paramaterizability that we can build increasingly complex ecosystems of objects manipulating other objects.

Compose Local Knowledge: Markets & APIs

Object-oriented programming and human cooperative systems have many parallels. Similar to rights that can get composed via contracts into ever richer cooperative agreements, networks of computational entities can be composed into richer computation. The package delivery service with a structure of ritualistic interaction is similar to an API in object-oriented programming. Its primary function is to enable us to coordinate, despite our ignorance of all the specialties taken into account by the other side of that counter. This is similar to how an API coordinates sub-programs with specialized knowledge and composes them into an overall system with better problem-solving abilities.

In both cases, the bottleneck is not conflict of interest but the locality of knowledge. Just as the package clerk and you want to cooperate but need the postal service abstraction boundary to find out how, in object-oriented programming, abstract interfaces (APIs) explain how concrete objects can make requests of other concrete objects across abstraction boundaries. Well-designed systems compose local knowledge. By creating boundaries between request-making and request-receiving sides, selection pressures can operate on both sides. As systems adapt and grow, local knowledge is composed in more intelligent ways. 

This applies to primitive computer systems that utilize more knowledge than any one of the sub-components could, to systems that we can start to call intelligent, to market processes, institutions, and large scale human organizations, up to our entire civilization. 

Increase Civilization’s Superintelligence: Economic Computation

Civilization is a network of entities with specialized knowledge, making requests of entities with different specializations. Just as object oriented programming creates an intelligent system by coordinating its member specialists, our institutions evolved to increase civilization’s adaptive intelligence by coordinating its member intelligences. We are essentially the objects within human society’s problem-solving machine, which takes into account vastly more knowledge than any one of us could possibly possess.50 

To illustrate this point, Leonard Read traces back a pencil’s development process, from tree through production line to coloring to final product. While no one person knows how to make a pencil and most of the thousands of people involved in producing it don’t know of each other, live in different countries, speak different languages, they nevertheless cooperate to produce a pencil, one of the thousands of items we take for granted in our lives. Friedrich Hayek summed up this dynamic as “civilization begins when the individual in the pursuit of his ends can make use of more knowledge than he has himself acquired.”51 

The shared parallels between human and computing systems was appreciated by influential computer programmers in the field; Bertrand Meyer relied on the concept of interfaces as contracts52, Alan Kay explained his approach to computing in terms of machines talking to each other,53 and Carl Hewitt was very much inspired by Karl Popper’s idea of how knowledge composition in the scientific community works.54 The Agoric papers sum up that “like all systems involving goals, resources, and actions, computation can be viewed in economic terms”. 

If computing approaches inspired by human cooperation helped unlock many of today’s software engineering successes, they are a good start for building future human and computer arrangements. Once we improve cooperation across individual humans, we can scale it further down to individual computational processes and objects. We should expect market responsiveness to move further down into the automated objects interacting with both each other and ourselves. It is hard to imagine what is possible when we redesign system rules from the perspective of being objects inside the system. We can already guess the next levels of the game will reproduce composability of human commerce. 

Chapter Summary

When we play the game of civilization within a framework of simple rules, interesting patterns emerge. Today’s market economy is such a pattern, emergent from rights and contracts constructing the rules of the game. The historical decline of violence shifts interactions to voluntary ones that tend to serve the player's goals. Over time, civilization is becoming more intelligent by developing abstraction boundaries—from APIs to institutions—that coordinate local knowledge into an astounding problem-solving ability. Civilization becomes increasingly superintelligent and aligned with our interests. By exploring and reinventing the underlying rules from within the game, we can unlock new levels of cooperation. 

Next chapter: IMPROVE COOPERATION | Info, Money, New Rights to Do Things

So far we suggested that civilization is an inherited game shaped by those before us, and we must choose among possible games to pass on to our future selves and generations. In this chapter, we explore the question of what constitutes good play. It is a more philosophical chapter than the remainder of the book, but we hope it allows you to appreciate why we suggest voluntary cooperation as a strategy to play this game. 

Improve Your Game: Striving for Coherence, Built to Be Conflicted

What does it mean to play our civilization game well? A scroll through Twitter shows answers differ widely. No wonder, according to a social intuitionist model for human values. Our actions are mostly based on intuitions, such as a sudden revulsion at an action. Only when prompted to explain them do we invent rationalizations for them. When it comes to reasoned judgment about right and wrong, we resemble “a lawyer trying to build a case rather than a judge searching for the truth.”1

Our intuitions are shaped by factors outside of our control. Evolution built us to care about things because that caring structure had survival benefits. So here we are, creatures that care about things and that have ethical reactions to each other. We are also creatures that try to abstractly theorize about those reactions. Our intuitions as to what is better and worse often conflict, and so do our theories. How do we go forward from here?

Perhaps the tendency to create abstract rationalizations about our intuitions isn’t all that bad. We are endowed with a caring structure but also with the ability to reflect on it. At least from an individual problem-solving perspective, this ability can help us create an overall narrative aligning our various wants now with those of our future selves. We react to situations and others as infants, before we can reason about our reactions. But as we wonder about the reasons for these reactions, gain more experience, and live through times of conflict and growth, many of our intuitions will come up for revision.

We could think of ethics as our internal negotiation between which intuitions we want to hold onto as values and which we want to dismiss as biases. We revise what turns out to be futile or incompatible with the rest. In this process, nothing is sacred; our intuitions and even our beliefs change with multiple consistent equilibria among them. Nevertheless, with continuous adjustment of our default caring structure, the emerging set may help us move more towards who we want to be.

The Reflective Equilibrium: Coherence and Conflict 

While some of our reflection occurs rather opaquely to us, explicit models can sometimes help us gain insight and make better choices. One such model is John Rawls’ Reflective Equilibrium.2

It has three steps: 

1. Intuitions: Explore your intuitions for a number of situations.

2. Abstraction: Abstract provisional rules of thumb that account for those intuitions. 

3. Reflection: When these abstractions don’t reliably recommend actions you find intuitive or when you encounter objections to them, revise either your intuitions or abstractions until you achieve a new equilibrium. 

Let’s see this process at work using the Trolley Dilemma to illustrate our striving for coherence, and superlongevity to illuminate our internal conflict.

Striving for Coherence: The Trolley Dilemma

Imagine you see a trolleybus heading in your direction. The driver is slumped over the controls, unconscious. In its path on the tracks in front are five people chatting, oblivious, soon to be mowed down. They are all going to die, but you can save them. You’re standing by the switch and if you simply pull a lever, you can divert the out-of-control trolleybus onto a different set of tracks where only one person is standing.

There is no time to warn anyone. Do you act and save five lives at the cost of only one? If your initial instincts are that the right answer is yes, you may want to think again.3  Imagine that, instead of pulling a lever, you have to push a bystander in front of the trolley to stop it from hitting the five people. Would you push? Using the Reflective Equilibrium, you may reason as follows:

1. Intuitions: I feel like redirecting the trolley with the switch, even if it kills one and saves five, is okay but pushing the man into the threat would be wrong.

2. Abstraction: I can explain my willingness to switch with a utilitarian value theory. According to act utilitarianism, I ought to choose the action that maximizes a utility measure, such as happiness. All else equal, more lives saved means more utility. My aversion to pushing the man is well encapsulated with a deontological rule, such as the Kantian categorical imperative, which prohibits actions that fail to respect the individual’s autonomy. The pushed person would be used as a clear means to stop the threat, which fails to respect his autonomy and is thus impermissible.

3. Reflection: So far, so simple. But as I dive into the vast literature on the Trolley Dilemma, the following inner Socratic dialogue ensues:

• Pro Utilitarianism: Some neuroscientists object that my application of the Kantian rule in the push case is actually just a post-rationalization of a dubious intuition. The physical act of pushing the man may activate an emotive cue against up-front personal harm which evolved in past small hunter gatherer groups and is not triggered by seeing a switch. Experiments show that an aversion to pushing was linked to brain areas associated with emotional processing while a willingness to switch was linked to brain areas associated with rational decision-making. Emotionally-laden cues should be morally irrelevant if I could rationally do more good otherwise.  Ergo: I should revise my intuition against pushing the man in favor of utilitarian answers to both cases.4

• Pro Deontology: Hold on! Perhaps those premises are shakier than I think. Some psychologists suggest that emotions have potential hidden value for rational decision-making.5 Somatic markers—emotional reactions with a somatic component based upon previous similar experiences—can serve as action heuristics that prescreen choice options, so I might act in time to save the five people without taking the time to have this internal dialogue. My aversion against pushing may be an intuitive application of such a heuristic that encapsulates valuable information. For instance, having the reliable expectation that in my society we don’t push people into threats could be a good heuristic to move me closer to the trusting society I want to live in.  

• Pro Deontology as Rule Utilitarianism: Not so fast! If I praise deontology for its value as a heuristic rather than for its intrinsic rightness, am I actually a rule-utilitarian who merely uses the maxim as a short-cut for maximizing overall utility? For instance, Scott Alexander suggests that “when Kant says not to act on maxims that would be self-defeating if universalized, what he means is ‘don’t do things that undermine the possibility to offer positive-sum bargains.’”6 Instead of calculating the utility for every action, perhaps I follow Kantian heuristics as a rule to maximize total long-term utility?

• Pro Deontology as Cooperative Strategy: Not necessarily! That I can sometimes be modeled according to a utility-maximizing agent in iterative game play, doesn’t automatically make me an aggregative utilitarian. It is tempting to jump from the homo economicus of game theory to anthropomorphizing society as an agent whose utility can be maximized. But this would be a fallacy. Extrapolating from individuals to an overall mystical body, composed of our preferences having a utility, ignores the separateness of persons. When I object to pushing people into threats, it is not necessarily because it serves the greater good long-term but because not upsetting the expectation of reciprocal altruism gives me a society in which I can better achieve my own goals. What does this mean for my Trolley answers?

• … The Socratic dialogue continues … 

Whether or not you find this particular dialogue convincing, it is one example of what it could look like to seek a more coherent caring structure.

Built to be Conflicted: Superlongevity

We may never fully “solve” our internal conflicts, but that may be just as well. Marvin Minsky suggested that, while we think of humans as entities with single agency, our minds are built to be conflicted. According to his multiple self view, our minds consist of many internal agents, each having simpler preferences. Our adaptive intelligence arises from these agents keeping each other in check. 

Robin Hanson observes that “if your mood changes every month, and if you die in any month where your mood turns to suicide, then to live 83 years you need to have one thousand months in a row where your mood doesn’t turn to suicide.” Thanks to our internal division, even if parts of our mind went suicidal at times, the others are there to keep them in check. So the next time we beat ourselves up when part of us wants this, part of us wants that, we may take solace that having some internal conflict, rather than perfect alignment, may be more a feature than a bug. 

Having trained using the notorious Trolley Dilemma, you are ready for a more radical thought experiment: superlongevity. Those wanting a very long life with extreme personal growth may have to re-evaluate what it means to be alive. Your reflection on the desire for growth versus identity continuity might go like this:

1. Intuitions: If faced with the possibility of a very long life, I would be excited to grow into an incomprehensibly larger cognition than what I am now. I care about my future cognition pursuing a great variety of goals, creating greater adaptive complexity, problem-solving ability, and intelligence. But I would only want to pursue growth if it is actually me who is growing. What I care about now has changed a lot since my childhood. If I compare my far-future self to my current self, it is hard to imagine it will care about the same things. Becoming something incomprehensibly great is also incomprehensibly different. Is this still me? 

2. Abstraction: I need a theory of what it means to be me. The most obvious one is "similarity identity": maintaining similarity of aspects at the core of my current identity. 

3. Reflection: The theory of similarity identity clearly conflicts with my desire to grow. Factors at the core of my current identity may change. If I want to hold onto my desire for growth, I need to revisit what it means to be myself. 

• Against Similarity Identity: Choosing similarity as a defining factor of identity may be a trap. That choice developed unchallenged by new technological possibilities. My caring to stay alive is the result of an evolutionary process that offered far fewer choices and philosophical challenges. To my ancestors, what staying alive meant was pretty unambiguous. It was continuing my corporeal body’s future timeline. While there were many choices as to what to do to stay alive, there was an unambiguous interpretation of what it meant to be alive. I am now in a position to rethink my drive to stay alive, so it is better co-adapted to my other desires, such as personal growth. 

• Toward Credit Theory of Identity: Can I come up with a new theory of identity that feels intuitive? How about a “credit theory of identity”: It is unclear to what extent Ancient Greek civilization died and I am part of a new civilization or to what extent I am still part of the Ancient Greek civilization. Civilizations are mushy and without clear boundaries. Similarly, once the intimacy of knowledge between brains can be close to the intimacy of knowledge within a brain, personal identities may be more fluid.  I can say the Greek civilization is still alive in me, in that I think back on it fondly and credit it for having been a crucial aspect of what I grew up to become. Projecting forward, I can similarly imagine an incomprehensibly greater future version of myself who looks back on me fondly as the beginning that grew into it. In line with my new theory of identity, I seek to transform my desire for survival into a desire to grow into something so great that I will value its fond memory of having been me. Evolution created me with a drive for survival, for the obvious selective reasons. "similarity" and "credit" are two alternative interpretive choices that I can then make. Though one is more obvious, neither is a more true interpretation of survival/longevity than the other. It is up to me to shape my interpretation. I choose the credit theory, to shape my overall caring structure to be more internally aligned.

• Pro Conflict: Wait, am I sure that, if technologically possible, I really want to intervene in my mind and “repair” parts that conflict with others? I should probably remember Minsky’s warning; when some parts of my mind remove conflicting parts, I risk destroying the richness from my mind’s natural conflict. 

• Conflict as Coherence: Wait, have I just eliminated potentially such valuable internal conflict myself by reaching coherence on the fact that I want to remain conflicted? 

• … The Socratic dialogue continues … 

Again, whether or not you agree with the details, this is one way of updating a caring structure. 

Play With Others: Epistemic Humility, Open Minds

We just saw that, as individuals, we often have little insight into our caring structure. Even if we manage to make progress towards a more coherent whole, we should expect residual internal conflict. Currently, 7 billion other players are already playing the game of civilization. They all start with different ethical intuitions and abstractions translating into different caring structures. With imperfect insight and conflict regarding our own caring structures, epistemic humility is advised when addressing theirs. 

Currently, other humans are still similar enough that we can sometimes model and shape them. Many of our socially rich interactions rely heavily on this cognitive similarity: We react to people and can use our reactions to them to model their reactions to us. By predicting how others will judge us, we learn to judge ourselves. Adam Smith calls this model the impartial spectator, which continually asks: “if I were in your shoes, seeing me doing what I'm doing, how would I react to me?”7 

According to Vernon Smith, this impartial spectator is a good shortcut to explain rich human cooperative behaviors that are anomalies with respect to the game theory of that particular interaction.8 We don’t always cheat, even if we could, and we sometimes punish cheating to no personal benefit. For instance, experiments by Dan Ariely suggest our dishonesty budget when no one is looking is determined by how much dishonesty we can allow ourselves while still maintaining a basically honest self-image.9 That’s because rather than wanting to act so as to gain praise and avoid blame, our impartial observer makes us want to act in a praiseworthy manner, even if no one sees it. 

Knowing about each others’ impartial spectator, and how it is shaped by others' reactions, generates a rich social account of values: We not only react to the culture around us, but influence everyone else's values by providing an inspirational example, and by praising and supporting projects and people we admire. All of us, by leading what we think of as good lives, help to form the overall evolution of values that will outlive us.10

This is only possible as long as we are similar enough. Just as our ancestors would be shocked by our levels of tolerance, our descendants’ worlds will seem very strange indeed to us and even their fellows.11 According to Robin Hanson, rates of social change have sped up with increased growth, competition, and technological change, so we should also expect accelerating value drift over time.12 With people living longer lives, our descendants may increasingly live with more fellow descendants that are very different to them. As the menu of environments to explore, experiences to have, and biological changes to make expands, our ethical intuitions and abstractions may increasingly diverge. The more diverse our civilization, the less we may be able to meaningfully model others and their reactions to us.

Why Value Diversity is Here to Stay

Is so much epistemic humility really necessary for interpersonal approaches to values? Earlier, we suggested that individuals can personally strive toward more coherence across their ethical intuitions and abstractions. Surely there must be a few core intuitions or abstractions that we can all agree to? Let’s see why we shouldn’t count on it. 

Abstractions: The Moral Philosophy Trenches

The long history of rigorous ethical debate invites skepticism for soon converging on one value theory. Kant’s Categorical Imperative and act utilitarianism will serve as placeholders for a complex philosophical theory landscape. 

Kantians’ disagreement on how to interpret the Categorical Imperative is well illustrated in the case of lying. Kant recommends that to determine if an action is permissible, formulate it into a maxim. Then check if the maxim treats humanity as an end in itself and could apply equally to every person without being self-defeating. Kant thinks lying is prohibited, because it robs us of the autonomy to rationally decide and because, if everyone lied, no one would believe anyone. In short, the maxim “telling a lie” both fails to treat humans as an end in itself and fails to be universalizable.

The Categorical Imperative as a method to generate rules is often praised for better handling rule-ambiguity than other rule-based value theories.13 Nevertheless, while some Kantians side with Kant that the maxim to be universalized is “telling a lie”, others disagree. They suggest more fine-grained maxims such as “lying in situation x” to allow lying to save an innocent friend from murder. Different people have different intuitions as to which situational aspects are morally relevant for rule construction.

Similar disagreements hold across utilitarians. They tend to agree that actions ought to maximize aggregate utility but disagree on how, when, and for whom to calculate ‘utilities’. Do non-human animals count? What about future, as yet unborn, generations? How much do they count?14 Are we trying to maximize pleasure? Or just minimize suffering? Should we cast a wider net instead to consider happiness, preference-satisfaction, virtue, or other definitions of the good life?15 

Depending on one’s answers, one may soon run into Robert Nozick’s utility monsters; people “who get enormously greater sums of utility from any sacrifice of others than these others lose.”16 Alternatively, Derek Parfit’s mere addition paradox awaits, in which a large population with low positive utility may be better than fewer happy people as long as the final utility comes out higher.17 Each specification of utilitarian theories comes with different costs that different people with different intuitions will trade off differently. These cherry-picked disagreements ignore disagreements across Kantians and Utilitarians. They don't even begin to address other value theories. 

If we can’t agree on the same value theories, can we at least agree on a few general principles to govern our collective lives that we endorse for different reasons? This may only postpone the problem such that we end up disagreeing on theories to handle value theories disagreement. For instance, John Rawls proposes there are some principles we should all consent to from behind a Veil of Ignorance. Such a veil would prevent humans “from knowing their own particular moral beliefs or the position they will occupy in society”.18 But getting global consensus on which factors one can take behind the Veil of Ignorance is tricky. Do you know when you live? Which species you are? Strip too much of your caring structure away and you can’t say what you want. Strip away too little, and you get a highly individualized answer. Different people have different intuitions as to what can and cannot be ignored.

With so much room for interpretation within theories, we should rely even less on picking just one theory to guide our civilizational game. The problem is not that there is no reasonable choice, but that there are more than one. A preference for one theory, or even an interpretation, depends on the same idiosyncratic factors that led to our differences in the first place. 

Intuitions: Evolved to Value Differently

One may hope that even if we cannot agree on the abstraction level, our intuitions should track the same underlying foundations. After all, humans share an evolutionary context and a similar social environment. For instance, Scott Alexander notes that in a society we all need to “decide how to act, what to do, what behaviors to incentivize, what behaviors to punish, what signals to send.” So even if our intuitions “crystallize” differently on the surface, can we reverse engineer them to a common human morality?19

To see how tricky it is to distill a meaningful common core from different individual intuitions, let’s revisit the Trolley dilemma. Suppose you revise your intuition against pushing the man into the threat because you now think such evolutionary-influenced intuitions should be irrelevant if you can otherwise benefit more people. Then suppose you learn that you prefer your family over creatures removed in space and time because this was evolutionarily adaptive. You decide to stop caring for your family more than strangers.

Can you convince others to follow you and flatten their caring structure? They may think caring structures evolved to guide action, and it is unhelpful to care about entities we cannot help. For those who continue with a caring structure after reflecting on it, that may be part of the valuing core. As long as you hold onto some evolutionary intuitions as values, you may want to think twice before judging those who make different choices.

If you learn that human love for nature evolved as a mere heuristic for finding food and water, are you prepared to cut down all forests to replace them with utility-maximizing objects?20 Different creatures come to different decisions about what they regard as bias and what, upon reflection, they hold onto as value. Ultimately, even “utilitarian” intuitions may be debunked as shortcuts for having other people help us when needed, i.e. for reciprocal altruism.

Can we at least agree on some generic strategies such as reciprocal altruism?21 Marco Del Giudice suggests that from a comfortable environment, understanding the advantages of pro-social trusting strategies is easy because we expect to play many rounds of games with each other. In a much harsher environment, a more short-term survival strategy might be more adaptive because there may not be time to benefit from cooperation in future games.22 Should you convince those for whom cooperating gets them killed to switch strategies? How?

There may be nowhere outside of humans’ individual idiosyncratic circumstances from which we could recommend one universal strategy. Even if there was, it would be very difficult to reliably communicate this to others who are the product of their own environments. Without much hope to reach interpersonal value agreement, either on the level of abstractions or on the level of underlying intuitions, value diversity is here to stay. 

The Future: From Value Diversity to Value Drift

As we get more diverse, the Silver Rule may provide a good practical heuristic for our interactions for a while. If the Golden Rule is “Do unto others as you would have them do unto you", the Silver Rule is "Don't do unto others as you would have them not do unto you." Currently, our models of what we want done to us may sometimes still work to figure out what others would like to have done unto them. In an increasingly diverse world, the Silver Rule is more appropriate as an epistemically humble heuristic. Figuring out how to avoid harming others is difficult enough without also trying to actively act on their behalf.

Heuristics can be useful but if we want a robust civilizational architecture across the next few rounds of play, we need more reliable frameworks for engagement with different cognitive architectures. Even if we don’t have to worry about meeting alien minds tomorrow, we are actively creating mind-architectures very different from us.23 Robin Hanson explores a potential future economy in which humans create human-brain emulations. These have reduced inclinations for art, sex, and parenting, can change speeds by changing hardware, and create temporary copies of themselves.24 

This scenario at least assumes humans as a source, but we are also making remarkable progress via software and hardware in AIs that function nothing like the human brain. While current “neural-network” architectures at best crudely mirror some of our brain’s functionality, it is naive to suppose they will mirror human caring structures for long.25

Future minds may not exhibit much of what we call “values” at all, but they could be better characterized as “goal-seeking” entities. Nevertheless, as long as they have goals and act as though they make choices, they will have revealed preferences. With less and less comprehension of other players' values, those revealed preferences may be all we have when designing systems for different players to reach their goals. But they may also be all we need to up a playing field that allows for good games, as judged by each player. How? Find out in the next chapter.

Watch Robin Hanson discuss Value Drift followed up by Mark Miller on Voluntary Cooperation strategies.

Next chapter: SKIM THE MANUAL | Intelligent Voluntary Cooperation

Who are you? 

You are the result of evolutionary games:

“Not one of your pertinent ancestors was squashed, devoured, drowned, starved, stranded, stuck fast, untimely wounded, or otherwise deflected from its life's quest of delivering a tiny charge of genetic material to the right partner at the right moment in order to perpetuate the only possible sequence of hereditary combinations that could result -- eventually, astoundingly, and all too briefly -- in you.”1

What awaits after a few more iterations of this game?  This depends, in part, on your decisions. 

You could make nihilistic outcomes more likely:

"In some remote corner of the universe, poured out and glittering in innumerable solar systems, there once was a star on which clever animals invented knowledge. That was the highest and most mendacious minute of "world history" — yet only a minute. After nature had drawn a few breaths the star grew cold, and the clever animals had to die.  There have been eternities when it did not exist; and when it is done for again, nothing will have happened." 2

Or you could make fantastic futures more likely: 

“Whether anyone else is out there or not, we are on our way. [...]  Expansion will proceed, if we survive, because we are part of a living system and life tends to spread. Pioneers will move outward into worlds without end. Others will remain behind, building settled cultures throughout the oases of space [...]  Where goals change and complexity rules, limits need not bind us. [...] New technologies will nurture new arts, and new arts will bring new standards. The world of brute matter offers room for great but limited growth. The world of mind and pattern, though, holds room for endless evolution and change. The possible seems room enough.”3

This book starts with high hopes for the future and the following conundrum: You’re in this game of civilization with players who have a diversity of values. Some values are complementary, some opposing, and some include the destruction of the playing field. As the game continues, rapid increase in technologies give everyone more power. If you hate civilization, your next move is easy. But if you love this game, and want to see it evolve over many iterations, you have a more complicated set of choices to make. 

Civilization is an inherited game shaped by those before you. If you’re happy your ancestors didn’t lock you into a future, should you leave everything up to future players? If only that was an option. Any move you make will affect the choices that future players have available to them. You must play your turn, and therefore, you must choose among games to pass on to future generations. By choosing strategies of intelligent voluntary cooperation, you can set the game of civilization upon a path of rapidly increasing intelligence serving a diversity of goals.

Let’s look at a high level overview of what we later cover in depth.

Where To Start

We begin by showing that goals differ across players of civilization. Each of us has subjective best guesses about the world. We disagree about what future state to move to and about how to get there. Some want to grow a pristine garden; others want to explore new worlds; others want to drive the pursuit of knowledge. These differences will become more apparent as players evolve. Some future players may have artificial minds. What they want may be very alien to human players.

In light of this ignorance, relying on voluntary interactions across players is a good heuristic to serve different goals. A voluntary action only depends on a player’s internal logic, leaving them “free” to engage or not engage in interactions. We tend to only consent to moves from which we expect benefit rather than harm. Such moves, which make at least one player better off without making anyone worse off, are called Pareto-preferred. As a rule of thumb, voluntary interactions gradually move civilization into Pareto-preferred directions, i.e., directions that tend to be better for everyone by their own standards.

We show that this principle has a good historic track record. Human civilization is growing less violent over time and many things players care about, from health to education, are improving. Voluntarism enables cooperation but it does not, by itself, bring it about. It took thousands of years of institutional evolution to create complex systems of prices, property, and institutions that help players cooperate better. Instead of arguing about dividing the pie, they get better at growing it. 

As the game continues, it becomes increasingly intelligent. One constraint on civilization’s intelligence is that each of its players plans mostly in ignorance of others’ plans. Institutions evolve to better coordinate across them by providing signals about what would be beneficial to do. Some players are humans, some are institutions themselves, and an increasing number will be software entities. Composed together by improving networks of voluntary cooperation, they increase the adaptive intelligence of civilization. In the pursuit of their highest values, they unlock new levels across the board.

What To Seek More Of

Approaching such futures requires progress in technologies of cooperation. Much of players' ability to benefit from each other is still limited. Perhaps the biggest problems arise when there is a state of the world that we would all prefer to jump to, but lack the coordination to do so. A look at how institutions evolved to deal with these factors shows how to further diminish them. With contracts, players can make binding commitments to particular future actions and cooperate for mutual benefit. 

Countless cooperative constellations are possible across the 7 billion players of civilization. We’ve unlocked many of them but we can do even better. The internet secures the right to information, cryptocurrency grants monetary sovereignty, blockchain makes institutions incorruptible, and smart contracts democratize the right of contract. Cryptography-based commerce, cryptocommerce, can create a base for amplifying and democratizing cooperation.

Some coordination problems will remain tricky. Drafting mechanisms for 7 billion people to find each other, speed up the bargaining process, and enforce an arrangement is extraordinarily difficult. But it could unlock previously inconceivable layers of civilization. We won’t jump to this world tomorrow, but we can gradually grow into it. 

What To Defend Against

As civilization evolves, players will unlock new capabilities. Biotechnology will deliver healthier lives, nanotechnology will provide wasteless manufacturing, and AI will accelerate unprecedented discoveries across the board. But the same technologies could be leveraged to cause unprecedented destruction. The economics of fighting wars could lead to pervasive robotic enforcement via lethal autonomous weapons and surveillance. 

When guarding against the downsides of powerful technologies, players must resist the temptations of solutions that create more problems than they solve. Statist solutions that centralize the capacity for violence without checks and balances are such a danger. Checks on U.S. power, the world’s leading military player, are decreasing, while checks on its rising rival, China, are near absent. Both have access to weapons that could nuke the playing field of civilization.

Decentralized defense systems that allow for multipolar monitoring and cross-checking can make their own dangers more visible. Cryptography can make them more privacy-preserving. Such systems are hard to envision but will emerge from today’s game. So called “black boxes” already provide indelible records for internal surveillance of automated systems. Smartphone cameras already democratize surveillance, making human enforcement more accountable.

Any desirable future will rely on computer security at every level, from hardware, to operating systems, to software, all the way to the user interface. Computer security is essential to de-risking cooperation that increasingly takes place virtually. It is also essential for preventing automated weaponizable technologies that make mass-killing trivially easy. Fortunately, there are promising candidates to address the problem; instead of adding security to a system last, they prevent insecurities from the very start. The seL4 microkernel is an excellent example of a provably secure system. 

To make computer security adoptable, we need a mixture of research and entrepreneurship to test it in the real world. The cryptocommerce ecosystem already serves as a test arena, where rogue actors compete to steal cryptocurrencies. It is hostile enough that insecure software dies quickly so that the ecosystem is populated by the survivors. This gives us a better chance at building a full secure software stack from the foundations to the user. Such play-tested systems can grow within, co-exist with, and eventually outcompete current insecurable software infrastructure.

What To Hope For

As civilization expands, it could act like a seed crystal dropped into a supersaturated solution, expanding its ordering principles in all directions. There is no law saying that the result will be the continually growing spontaneous order intelligence of civilization. It could be the outcome of a winner-takes-all arms race to expand first. 

If human players upgrade their tools to cooperate with artificial players newly arriving on the scene, we have much to look forward to. But even if all goes well, the universe will eventually no longer be able to sustain computation of any sort, especially not the complex computation required for intelligence. Even if we create a game which makes everything up to now seem like an insignificant speck, it is all temporary. Nevertheless, insofar as the in-between is shaped by what current players value, it’s on us to shape what happens between now and then.

The game is on and the stakes are high. Let’s play.

Curious for more? Head to the Intelligent Cooperation intro seminar.

Next chapter: MEET THE PLAYERS | Value Diversity

Civilization: A Superintelligence Aligned with Human Interests

Consider civilization as a problem-solving superintelligence. 

The graph below shows the global decline in extreme poverty from 1820 to 2015, prompting Steven Pinker’s quote,

“We have been doing something right, and it would be nice to know what, exactly, it is.” 1

After 1980, the rate of decline increases dramatically and continues.2 It would be good to know the dynamics that culminated in this dramatic decline that continued for the next 40 years. What did we get right? What did civilization learn?

It’s not possible to answer this directly, but we would like to open it up for discussion. We need a conceptual framework for thinking abstractly about civilization over long periods of time.

We like the game metaphor. Games are universal. Asking someone to imagine a game board gives an intuitive context for explaining the iterative behavior of multiple players and the consequent outcomes.

Chess has rules. Then play begins. The game is the properties that emerge from the interactions of separately-interested players in a framework of rules.

What changed from “nature red in tooth and claw” to our current civilization? As civilization has grown less violent, it is increasingly dominated by voluntary interactions. The nature of the rules in this less threatening world evolves in a context of willing participation. The emergent outcome of the game is more effective cooperation, creating better outcomes for everybody. 

This is something we got right.

Technologies for Voluntary Cooperation

Can we game the future such that more of the world can know the power of voluntary cooperation? Can our future display new forms of cooperation that are not possible with our current technologies?

Luckily, our recent history includes an illustrative case: the timeline of the deployment of modern cryptography.

Cryptography has a long fascinating history, but here we follow a particular thread beginning in August 1977 with the publication of Martin Gardner’s “Mathematical Games” column in Scientific American. Gardner wrote about a ground-breaking discovery: The RSA (Rivest–Shamir–Adleman) public key algorithm.

Mark Miller (co-author) has a personal history in the battles over cryptography. In 1976, Mark apprenticed himself to Ted Nelson, creator of the Xanadu hypertext system. They shared a vision of a global hypertext publishing system as a censorship-free liberating force. But they knew that, without the right architecture, any such system would be corrupted into a tool of 1984-style oppression. Before the invention of modern cryptography, they could not solve this puzzle.

Mark was an avid reader of Scientific American and always went straight to Gardner’s column. After reading about the RSA discovery, he called Ted at an ungodly hour and joyously proclaimed, “Ted! We can prevent the Ministry of Truth!” 

They immediately set out to get a copy of the RSA publication. In those days you could not simply download scientific publications. Instead, they mailed a request for a copy. And waited. Nothing arrived. The United States intelligence community had suppressed publication, to preserve their ability to spy on all conversations.

Mark decided to take action. His mission began with a trip to the MIT campus. As a 20-year-old computer geek he was welcome in the tribe and before long, he had the paper in hand. He knew it was critical to distribute it far and wide to counter the attempt at suppression. Fully aware of the risk, he gave copies to his most-trusted friends, telling them “If I disappear, make sure this gets out.”

While handling the paper only with gloves he started making copies, going to several different copy shops. He mailed copies anonymously to technology-minded hobbyist groups and magazines.

In 1978, U.S. intelligence dropped the attempt to suppress the RSA paper. Mark will never know if his personal actions made any difference. But this experience made the stakes clear. Technologies of freedom are worth fighting for, and building.

Mathematics and technology do not, on their own, lead to privacy of individuals in their digital lives.

Together with like-minded cypherpunks, the fighting and building continued. On one side, export controls, mandatory Clipper chip backdoors, and weak cyphers were approved for general use. On the other, Phil Zimmerman’s PGP, Matt Blaze’s hacking of Clipper, John Gilmore’s breaking of government approved cyphers, and the Electronic Frontier Foundation’s case overturning export controls as unconstitutional violations of free speech. Mostly, we won.

We can easily imagine an alternate history in which these fights had been lost. Our world would be much more totalitarian. In the analog era, all of our conversations could be spied upon. The digital era would have combined inescapable surveillance with modern computing, leading to an inversion of democracy: Those shielded by classified cryptography are unaccountable to us, while we, in every minute aspect of our lives, are accountable to them.

While the real world resembles this nightmare to an uncomfortable extent, the public growth of modern cryptography gives us the tools to fight back.

The HTTPS encryption protocol gives us secure transactions and communication – secure email, end-to-end encryption, and secure credit card transactions necessary for the growth of a Web economy. Human rights activists are less vulnerable due to secure messaging.

All over the world, corrupt powers destroy lives. They interfere with individuals’ plans to have a good life through voluntary trade with others. How can you plan with the uncertainty of arbitrary coercive interference? It’s hard for those in the rich world to appreciate the motivation and inventiveness of people in this situation. Modern cryptography, including Bitcoin, allows for a parallel economy that does not rely on the government for commerce and trade. Some are fighting for their lives and these tools give them a chance.

Blockchains cannot be corrupted. The decentralized cryptographically secured interactions are protected from coercive corruption by governments or criminals. Transactional history cannot be rewritten. Smart contracts give us a new technological base to create complex voluntary arrangements, to realize new forms of cooperation.

We are an information society. Starting with these hard-earned gains, we can build a solid foundation crucial to our future, where each layer is built on previous layers in the context of trust and secure cooperation. Although in its infancy, we expect the cryptocommerce space to be transformative.

Centralized Power

Centralized power — concentrated sources of authority and control — is incompatible with a voluntary, cooperative framework of civilization. 

We’re not advocating for revolutionary takeover or government regulations to combat centralization. First, it’s often not clear what or how to regulate. Secondly, such tactics often backfire, creating more centralization as a consequence. 

Let’s take a look at a couple of cases: bank regulations and Google’s Gmail.

The banking industry is susceptible to top-down manipulation. Operation Choke Point3 successfully shut down lawful businesses, denying them access to standard business services. This was not voluntary action by some banks, rather, all banks reacting to regulatory pressure. The targeted businesses had nowhere to turn.

Google’s Gmail is an interesting case. Before webmail, we had a world of decentralized email. There were several projects working to add crypto, which would have given us truly decentralized secure email. Instead, centralized webmail took over.

When first released in 2004 Gmail had 2 million users. Currently, it’s 1.5 billion. Google stores and has access to the plain text of users’ email. Why were people willing to entrust Google with all their correspondence? One reason was Google’s slogan at the time, “Don't be evil”, projecting an image of trustworthiness. 

Over time, Google accumulated a centralized trove of the contents of email communication of over a billion people. The US federal government, unable to resist the temptation, issued national security letters (1) demanding the handover of private email and (2) prohibiting Google from telling anyone. Whether Google wanted to resist or comply is a separate issue; the point is that a centralized vulnerability inevitably led to its corruption. 

There is a natural dynamic between centralizing and decentralizing forces. As in the Gmail case, at first, we often risk centralized vulnerability for convenience. 

Centralized vulnerabilities create temptations to corruption that cannot be resisted. As the resulting corruption becomes apparent, they raise the competitive advantage of decentralized competitors. We seek to create incorruptible decentralized systems that can outcompete centralized systems. 

By working within the constraints of voluntary competition, we are more protected from our own mistakes. If our dreams are misconceived, they are also less likely to outcompete. If, on the other hand, they are the genuine improvements that we think they are, they are also more likely to win these competitions in the long run.

The battle for decentralization can never have an ultimate decisive victory, but, in the absence of astute watchfulness, it can suffer an ultimate defeat.

Superintelligence

There are many views on AI dangers. One prominent perspective goes as follows: Once an AI exceeds human capacity, it can improve its own design much faster than human designers can. As it improves itself, it also improves its ability to improve itself, leading to an explosive chain reaction which can suddenly catapult this one breakthrough AI into a superintelligent capacity exceeding all the rest of human civilization combined.4

This is the “hard takeoff” scenario, leading to a “unipolar” outcome. It is a hard takeoff because it happens so suddenly that nothing else has a chance to adapt during the process. It is unipolar in the sense that this one superintelligent entity may be more powerful than everything else, and so in a position to rule everything else.

Starting from the notion that this unipolar takeover is inevitable—that we will necessarily be ruled by a permanent dictator of our own design— some conclude we should design a benevolent dictator: one that wants to serve our interests. This would raise two design questions: (1) how to construct the AI so that it wants to serve our interests? and (2) what are our interests anyway? The first question is hard, but the second question opens up philosophical problems that have eluded any general agreement over the last few millennia.

From our perspective, any best case scenario arising from this notion is a worst case scenario, the one we must prevent at all costs. Any unipolar takeover of the world is unlikely to be benevolent. Rather than hoping that this unprecedented power over the world will be shaped by “the right kinds of people”, history tells us that powerful positions attract those who want power. 

Instead of a centrally designed formula encoding the general good, we currently have a diverse pluralistic world of many different people making their own choices about what they want and how to achieve it. People formulate their goals using their idiosyncratic personal knowledge within a great variety of cultural and philosophical systems. There may be no general good beyond the revealed preferences of each of us making choices to pursue our goals in our own way.

If the problem of superintelligence were a purely new problem with no historical precedent, there is little relevant to learn from history. But human institutions are already non-human intelligences with which we cooperate and against which we defend ourselves. Depending on the nature of the institution, they can be well aligned or badly aligned with human interests. 

Until the last few centuries, most of our history was the history of tyranny. By contrast, with the invention of democracy, separation of powers, rule of law, due process, individual rights, and independent judiciaries, not only have we better aligned our institutions with our interests, we have also enabled the ecosystem of these institutions — our civilization as a whole — to rapidly grow in intelligence and benefit to its constituents. 

The superintelligence of civilization is already emergent from the interplay of human and machine intelligences. Within our multipolar civilization, as machines get more intelligent, they will contribute more to the overall intelligence of our civilization. 

We're already facing, and have faced now for over seven decades, the existential risk of nuclear war. We are in a multipolar world of multiple nations armed with nukes and willing to use them if they absolutely have to. Now that we're in that situation, our only options going forward are multipolar options. Anything that threatens a unipolar takeover risks provoking a nuclear war. So unipolar solutions may be off the table anyway.

We cannot survey the landscape of choices from an imagined position outside the game. One can imagine Karl Marx sitting at his desk, overlooking the factory floor below, appreciating the coordination and efficiency of the workers. It’s all right there in front of him. He cannot fathom why the economy outside needs more than this elemental structure.

We are in a different game. The people who can shape the game are in it. We’re in an iterated game. We are grateful that previous players set it up such that we can now play the next moves from inside the game. Whatever we do, the result determines what kind of game gets played in the future. We cannot not play the game.

But we can iterate and make sure the game emerges in a multipolar manner within the voluntary framework of civilization. This book explores technologies that can be useful on this path. 

In the future, most cognition will be non-human. The game dynamics we start now must be good enough for human and non-human interests, so that these future players have an interest in upholding the voluntary nature of the game. That is our ultimate protection. If we can accomplish this difficult task, future players can unlock currently unimaginable levels of this beautiful game.

Next chapter: OVERVIEW | What to Expect From This Game

Gaming the Future: Technologies for Intelligent Voluntary Cooperation

Welcome to this living book and book club about technologies for intelligent voluntary cooperation by Allison Duettmann, Mark S. Miller, and Christine Peterson, Foresight Institute.

Intro

Opportunities for bright futures enabled by bio, nano, and AI are now within our reach. But technological proliferation also brings risks that threaten the very existence of civilization. To help civilization navigate this abyss, this book addresses three questions:

1. How can we help civilization cooperate better?
2. How can we help civilization defend itself better?
3. How can we help civilization do both - cooperation and defense - in light of AI?

Explore strategies, tools, and technologies for enabling voluntary cooperation across a diversity of intelligences. Let’s unlock Paretotropian futures of high technology in which valuing entities can pursue their highest function through iterative play.

Short-cuts to chapters

Follow the links at the bottom of each page, or jump around between chapters:

1. FOREWORD | What's at Stake in This Game?

2. OVERVIEW | What to Expect From This Game

3. MEET THE PLAYERS | Value Diversity

4. SKIM THE MANUAL | Intelligent Voluntary Cooperation & Paretotropism

5. IMPROVE COOPERATION | Information, Money, Rights, Contracts, and Privacy

6. GENETIC TAKEOVER | Cryptocommerce

7. DEFEND AGAINST PHYSICAL THREATS | Multipolar Active Shields

8. DEFEND AGAINST CYBER THREATS | Computer Security

9. WELCOME NEW PLAYERS | Artificial Intelligences

10. ITERATE THE GAME | Racing Where?

Thank you

We would like to thank members of our Foresight’s Intelligent Cooperation Group for shaping this book into what it is through our 2021 seminars. The seminars are now incorporated into the chapters as deep dive seminars.

• Robin Hanson, George Mason University | Value Drift

• Balaji S. Srinivasan, 1729 | The Network State

• Vernon Smith, Chapman University | Theory of Price Discovery in Markets 

• Andrew McAfee, MIT | Civilizational Progress 

• Tyler Cowen, George Mason University | Stubborn Attachments

• Audrey Tang, Taiwan’s Digital Minister | Tools for Openness

• Chris Hibbert, Anthony Aguirre, Martin Koeppelmann, Paul Gebheim, and Robin Hanson | Prediction & Replication Markets

• Christine Lemmer-Webber | Re-Decentralizing Social Networks

• Kate Sills, Independent | NFTs and Engineering Property Rights

• Arthur Breitman, Tezos | Blockchain Governance 

• Marc Stiegler, Sci-Fi author | The Digital Path

• Chip Morningstar, Meng Weng, and Federico Ast | Split Contracts, Computational Law & Decentralized Arbitration 

• Matan Field, Esteban Ordano, Jazear Brooks, Tyler Golato, and Patrick Joyce | DAOs

• Glen Weyl, RadicalxChange | Social Technology for a Political Economy of Increasing Returns 

• Alex Tabarrok, George Mason University | Dominant Assurance Contracts 

• Zooko Wilcox and Howard Wu | Zero-knowledge-enabled Cooperation

• Jim Epstein, Primavera De Filippi, and Brewster Kahle | A Peaceful Transition into Cryptocommerce? 

• Daniel Ellsberg, DoomsDay Machine | Nuclear Risks: Doomsday (Still) Hiding in Plain Sight 

• David Brin, The Transparent Society | Transparent Society & Sousveillance 

• Gernot Heiser, University of New South Wales| SeL4: Formal Proofs for Real-World Cybersecurity

• David Krakauer, Santa Fe Institute | Collective Computing

• Gillian Hadfield, University of Toronto | Incomplete Contracts & AI Alignment

• Richard Craib, NumerAI | Techniques for Intelligence Coordination
  Peter Norvig, Google | AI: A Modern Approach

• Anders Sandberg, Oxford University | Game Theory of Cooperating with Alien Minds

• Robin Hanson, George Mason University | A Simple Model of Grabby Aliens

We would also like to thank our book club guests for discussing the book post its publication. Their talks are now also included at the end of each chapter.

• David Friedman, Author of Legal Systems Very Different from Ours

• Robin Hanson, George Mason University

• Kate Sills, Independent Software Engineer

• Paul Gebheim, Forecast Foundation

• Primavera De Filippi, Koala

• Arthur Breitman, Tezos

• David Brin, Author of Transparent Society

• Gernot Heiser, SeL4

• Juan Benet, Protocol labs

• Trent McConaghy, Ocean Protocol

• Stuart Armstrong, Future of Humanity Institute

Finally, we would like to thank Keith Mansfield, Tom Galloway, Terry Stanley, Chris Hibbert, Alan Karp, Jazear Brooks, David Manheim, Kate Sills, Chip Morningstar, Gillian Hadfield, Robin Hanson, David Friedman, Jim Bennett, Micah Zoltu, and Dan Finlay for extensive comments on the book draft. The key ideas of Paretotopia (at the time) were originally worked out by Mark S. Miller in collaboration with Eric Drexler. We learned a lot and all remaining errors are our own.

Join

We hope you find interest in critiquing and augmenting the ideas by commenting. This book, like a good game, is here to be iterated and improved for the next round. Follow Foresight Institute on Twitter.

Next: FOREWORD | What’s at Stake in This Game?

The Book

Gaming the Future: Technologies for Intelligent Voluntary Cooperation

Abstract

Have you ever played Civilization? In the game, you’re discovering technologies that unlock new levels, one capability at a time. But not all innovations are equal. Better technologies of cooperation could unlock new levels of progress across the board. Opportunities for bright futures enabled by bio, nano, and computing technologies are now within our reach. Their proliferation also comes with risks and authoritarian attempts at control. This book explores how technologies of intelligent voluntary cooperation can help us navigate the traps. Cryptocommerce enables decentralized, secure cooperation across human and computing entities. This unlocks a Paretotropian future of high technology and high freedom. 

Table of contents with hyperlinks to chapters

1. FOREWORD | What's at Stake in This Game?

2. OVERVIEW | What to Expect From This Game

3. MEET THE PLAYERS | Value Diversity

4. SKIM THE MANUAL | Intelligent Voluntary Cooperation

5. IMPROVE COOPERATION | New Info, Money, Rights, Contracts, Privacy

6. GENETIC TAKEOVER | Cryptocommerce

7. DEFEND AGAINST PHYSICAL THREATS | Multipolar Active Shields

8. DEFEND AGAINST CYBER THREATS | Computer Security

9. WELCOME NEW PLAYERS | Artificial Intelligences

10. ITERATE THE GAME | Racing Where?

We hope you find interest in critiquing and augmenting the ideas by commenting. This book, like a good game, is here to be iterated and improved for the next round.

Acknowledgements

We would like to thank members of our Foresight’s Intelligent Cooperation Group for shaping this book into what it is through our 2021 seminars:

• Robin Hanson, George Mason University | Value Drift

• Balaji S. Srinivasan | The Network State

• Dr. Vernon Smith | Theory of Price Discovery in Markets

• A. McAfee | Civilizational Progress

• Tyler Cowen, George Mason University | Stubborn Attachments
  Audrey Tang, Taiwan Digital Minister: Tools for Openness
  Prediction & Replication Markets, Augur, Metaculus

• Christine Lemmer-Webber | Randy Farmer | Re-Decentralizing Networked Communities

• Kate Sills, Agoric | NFTs and Engineering Property Rights
  Arthur Breitman, Tezos: Blockchain Governance

• Marc Stiegler, Agoric | The Digital Path

• Chip Morningstar, Meng Weng, Federico Ast | Split Contracts, Comp. Law & Decentralized Arbitration

• DAOstack, Decentraland, SifChain, ResearchHub, VitaDAO | DAOs

• Glen Weyl, RadicalxChange | Social Technology for a Political Economy of Increasing Returns

• Alex Tabarrok, George Mason University | Dominant Assurance Contracts

• Zooko Wilcox, ECC, Howard Wu, Aleo | Zero-knowledge-enabled Cooperation

• Jim Epstein, Primavera De Filippi, Brewster Kahle | Peaceful Transition into Cryptocommerce?

• Daniel Ellsberg, DoomsDay Machine | Nuclear Risks: Doomsday (Still) Hiding in Plain Sight

• David Brin, The Transparent Society | Transparent Society & Sousveillance

• Gernot Heiser | SeL4: Formal Proofs for Real-World Cybersecurity

• David Krakauer, Santa Fe Institute | Collective Computing

• Gillian Hadfield, University of Toronto | Incomplete Contracts & AI Alignment

• Richard Craib, NumerAI | Techniques for Intelligence Coordination
  Peter Norvig, Google | AI: A Modern Approach

• Anders Sandberg, Oxford University | Game Theory of Cooperating with Alien Minds

• Robin Hanson, George Mason University | A Simple Model of Grabby Aliens

• The seminars are now incorporated into the text as deep dives

We would especially like to thank Keith Mansfield, Tom Galloway, Terry Stanley, Chris Hibbert, Alan Karp, Jazear Brooks, David Manheim, Kate Sills, Chip Morningstar, Gillian Hadfield, Robin Hanson, David Friedman, Jim Bennett, Micah Zoltu, and Dan Field for extensive comments on the book draft. We learned a lot and all remaining errors are ours.

Join Us!

Follow Foresight Institute on Twitter and apply to join our Discord.

Next up: FOREWORD | What’s at Stake in This Game?

Subscribe now