Motivational system

1) I agree that direct reward has to be in-built
(into brain / AI system).
2) I don't see why direct reward cannot be used for rewarding mental
achievements. I think that this "direct rewarding mechanism" is
preprogrammed in genes and cannot be used directly by mind.
This mechanism probably can be cheated to the certain extend by the
mind. For example mind can claim that there is mental achievement when
actually there is none.
That possibility of cheating with rewards is definitely a problem.
I think this problem is solved (in human brain) by using only small
dozes of "mental rewards".
For example, you can get small positive mental rewards by cheating your
mind to like finding solutions to "1+1=2" problem.
However, if you do it too often you'll eventually get hungry and would
get huge negative reward. This negative reward would not just stop you
doing "1+1=2" operation over and over, it would also re-setup your
judgement mechanism, so you will not consider "1+1=2" problem as an
achievement anymore.

Also, we all familiar with what "boring" is.
When you solve a problem once - it's boring to solve it again.
I guess that that is another genetically programmed mechanism with
prevents cheating with mental rewards.

3) Indirect rewarding mechanisms definitely work too, but they are not
sufficient for bootstrapping strong-AI capable system.
Consider a baby. She doesn't know why it's good to play (alone or with
others). Indirect reward from "childhood playing" will come years later
from professional success.
Baby cannot understand human language yet, so she cannot envision this
success.
AI system would face the same problem.

My conclusion: indirect reward mechanisms (as you described them) would not be
able to bootstrap strong-AI capable system.

Back to real baby: typically nobody explains to baby that it's good to play.
But somehow babies/children like to play.
My conclusion: there are direct reward mechanisms in humans even for
things which are not directly beneficial to the system (like mental
achievements, speech, physical activity).

(from AGI email list).

Richard Loosemore - Reward

Richard Loosemore (rpwl at lightlink.com):
All thinking systems do have a motivation system of some sort (what you
were talking about below as "rewards"), but people's ideas about the
design of that motivational system vary widely from the implicit and
confused to the detailed and convoluted (but not necessarily less
confused).
===

Reward