> I maintain that it isn't just hard, it is computationally impossible.
I further maintain that it's definitionally impossible. Before we find it computationally impossible, we will find that we can't write the a complete, detailed requirements specification defining what rational is.
(Of course, we recognize egregious irrationality when we see it; that's not what I mean; you can't just define rationality as the opposite of that.)
People can behave rationally (or not) with respect to some stated values that they have. But those can be arbitrary. So the requirement specification for rationality has to refer to a "configuration space", so to speak, where we program these values. This means that the output is dependent on it; we can't write some absolute test case for rationality that doesn't include this.
Problem is, people with different values look at each other's values and point to them and say, "those values are irrational; those people should adopt my values instead".
You can't say values are irrational -- they just are. If you really like paperclips, no amount of logic can tell you otherwise. What logic can tell you (and other people could), is that your values conflict with each other and you have to balance one against another. Turning whole universe into paperclips is counterproductive if you also value pins. If you literally have no value the other person is basing their arguments on, then they can't convince you to have it.
Luckily we get our values from bunch of heuristics developed through millions of years of biological and social evolution, so we mostly have the same ones, just with different relative weights.
Won't be true if we ever meet (or make) some other sentient critters.
(Values can be contradictory/inconsistent. E.g. you say you value self-preservation, but you also enjoy whacking your head with a hammer. That would be a kind of irrational. That's not what I'm referring to though.)
I think they make a category mistake when they do then. Values tell you where you want to be, rationality is a most accurate process to get where you want to go and maybe to check if you want to be there before actually getting there and checking out personally. (I think we basically agree btw btw, it is all those other people who are wrong :))
I further maintain that it's definitionally impossible. Before we find it computationally impossible, we will find that we can't write the a complete, detailed requirements specification defining what rational is.
(Of course, we recognize egregious irrationality when we see it; that's not what I mean; you can't just define rationality as the opposite of that.)
People can behave rationally (or not) with respect to some stated values that they have. But those can be arbitrary. So the requirement specification for rationality has to refer to a "configuration space", so to speak, where we program these values. This means that the output is dependent on it; we can't write some absolute test case for rationality that doesn't include this.
Problem is, people with different values look at each other's values and point to them and say, "those values are irrational; those people should adopt my values instead".