EA Global 2016

MIRI is interested in reasoning about highly advanced AI systems before they exist, specifically for developing models of safety and control for such systems. Current agent models used in game theory and economics are insufficient for this purpose, because of their highly unrealistic assumption that a rational agent must immediately know all the logical consequences of its beliefs at any given time. Logical induction refers to the more realistic process by which an agent's beliefs should change over time as it thinks and realizes the consequences of its past observations. In this talk, we will examine (1) some criteria for "ideal" logical induction, (2) a new algorithm for logical induction that satisfies many desirable properties, and (3) some implications for what a very powerful AI system is able to learn. About 50% of the talk will be technical in nature, with adjustments made for audience composition and interest.