the author says “being driven to stay within your own viability envelope is crucial to the kind of intelligence that we know about—the kind of intelligence that we are. So I think that anything that looks like an artificially generally intelligent system is going to have that, too.” However, I take exception to the word “crucial”. Instead, staying viable is inherent in our intelligence because our intelligence evolved among the species members who successfully maintained viability. But it being inherent in our sort of intelligence does not make it crucial for any generally intelligent system. Other intelligent systems might have evolved under different pressures - the systems that survive to become AGI will do so by gaining human approval. If that is “staying viable” the expression becomes too broad to be useful. imho.
the author says “being driven to stay within your own viability envelope is crucial to the kind of intelligence that we know about—the kind of intelligence that we are. So I think that anything that looks like an artificially generally intelligent system is going to have that, too.” However, I take exception to the word “crucial”. Instead, staying viable is inherent in our intelligence because our intelligence evolved among the species members who successfully maintained viability. But it being inherent in our sort of intelligence does not make it crucial for any generally intelligent system. Other intelligent systems might have evolved under different pressures - the systems that survive to become AGI will do so by gaining human approval. If that is “staying viable” the expression becomes too broad to be useful. imho.