Hollywood’s theory that machines with evil minds will drive armies of killer robots is just silly. The real problem relates to the possibility that artificial intelligence(AI) may become extremely good at achieving something other than what we really want. In 1960 a well-known mathematician NorbertWiener, who founded the field of cybernetics, put it this way: “If we use, to achieve our purposes, a mechanical agency with whose operation we can not effectively interfere, we had better be quite sure that the purpose which we really desire.”
好莱坞的理论认为有着邪恶头脑的机器会成为杀手机器人大军，这太愚蠢了。这一可能性的真正问题在于，AI（人工智能）可能会变得非常擅长于实现某些东西，不仅是我们真正想要的。1960年，著名数学家诺伯特维纳创立了控制论领域, 提到：“如果我们为了达到我们的目的而使用一种我们无法有效干预其运作的机械装置，我们最好确定我们真正想要的目的。”

A machine with a specific purpose has another quality, one that we usually associate with living things: a wish to preserve its own existence. For the machine, this quality is not in-born, nor is it something introduced by humans; it is a logical consequence of the simple fact that the machine can not achieve its original purpose if it is dead. So if we send out a robot with the single instruction of fetching coffee, it will have a strong desire to secure success by disabling its own off switch or even killing anyone who might interfere with its task. If we are not careful, then, we could face a kind of global chess match against very determined, super intelligent machines whose objectives conflict with our own, with the real world as the chess board.
有特定目的的机器还有另一种特性，我们通常把它与生物联系在一起：希望保留自己的存在。对于机器来说，这种特性不是天生的，也不是由人类引入的；如果机器死了，就无法达到其原始目的，这就是这一简单事实的逻辑化结果。因此，如果我们给机器人发送一条取咖啡的简单指令，它就会有强烈的愿望，禁止关闭自己的开关，甚至杀死任何可能干扰其任务的人，来确保成功。如果我们不小心，那么，我们可能会面临一场全球象棋比赛，棋盘就是现实世界，对手是异常坚定，其目标与我们的目标冲突的超级智能机器。

The possibility of entering into and losing such a match should concentrating the minds of computer scientists. Some researchers argue that we can seal the machines inside a kind of firewall, using them to answer difficult questions but never allowing them to affect the real world. Unfortunately, that plan seems unlikely to work: we have yet to invent a firewall that is secure against ordinary humans, let alone super intelligent machines.
参加并输掉这样一场比赛的可能性应该会引起计算机科学家的注意。一些研究人员认为，我们可以将这些机器密封在一种防火墙内，用它们解决困难的问题，但决不允许它们影响现实世界。不幸的是，这一计划似乎不太可能实现：我们还没有发明一种防火墙来保护普通人，更不用说超级智能机器了。

Solving the safety problem well enough to move forward in AI seems to be possible but not easy. There are probably decades in which to plan for the arrival of super intelligent machines. But the problem should not be dismissed out of hand, as it has been by some AI researchers. Some argue that humans and machines can coexist as long as they work in teams—yet that is not possible unless machines share the goals of humans. Others say we can just “switch them off” as if super intelligent machines are too stupid to think of that possibility. Still others think that super intelligent AI will never happen. On September11, 1933, famous physicist Ernest Rutherford stated, with confidence, “Anyone who expects a source of power in the transformation of these atoms is talking moonshine.” However, on September 12, 1933, physicist Leo Szilard invented the neutron-induced nuclear chain reaction.
很好地解决安全问题以推动AI的发展似乎是可能的，但并不容易。几十年内为超级智能机器的到来做好计划是可能的，但这个问题不应该像一些AI研究人员所做的那样，随意被忽视。有些人认为，人和机器可以共存，只要他们在团队中工作，但这是不可能的，除非机器与人类的目标相同。也有人说我们可以“关掉它们”，好像超级智能机器很蠢，不会想到这种可能性。还有人认为超级智能AI永远不会发生。1933年9月11日，著名物理学家欧内斯特·卢瑟福满怀信心地说，“任何期望这些原子转化过程成为能量来源的人都在胡说八道。”然而，1933年9月12日，物理学家利奥·西拉德发明了中子诱导链式核反应。