For the purposes of AI governance it is important that we understand the strategic parameters relevant to building safe AI systems, including the viability, constraints, costs, and properties of scalably safe systems. What is the safety production function, which maps the impact of various inputs on safety? Plausible inputs are compute, money, talent, evaluation time, constraints on the actuators, speed, generality, or capability of the deployed system, and norms and institutions conducive to risk reporting. To what extent do we need to spend time or resources at various stages of development (such as early or late) in order to achieve safety? If the safety-performance trade-offs are modest, and political or economic returns to absolute and relative performance are relatively inelastic (marginal improvements in performance are not that important), then achieving safe AI systems is more likely to be manageable; the world will not have to resort to radical institutional innovation or other extreme steps to achieve beneficial AI. If, however, the safety-performance trade-off is steep, or political or economic returns are highly elastic in absolute or especially relative performance, then the governance problem will be much harder to solve, and may require more extreme solutions.

For the purposes of AI governance it is important that we understand the strategic parameters relevant to building safe AI systems, including the viability, constraints, costs, and properties of scalably safe systems. What is the safety production function, which maps the impact of various inputs on safety? Plausible inputs are compute, money, talent, evaluation time, constraints on the actuators, speed, generality, or capability of the deployed system, and norms and institutions conducive to risk reporting. To what extent do we need to spend time or resources at various stages of development (such as early or late) in order to achieve safety? If the safety-performance trade-offs are modest, and political or economic returns to absolute and relative performance are relatively inelastic (marginal improvements in performance are not that important), then achieving safe AI systems is more likely to be manageable; the world will not have to resort to radical institutional innovation or other extreme steps to achieve beneficial AI. If, however, the safety-performance trade-off is steep, or political or economic returns are highly elastic in absolute or especially relative performance, then the governance problem will be much harder to solve, and may require more extreme solutions.