The Credit Assignment Problem
信用分配問題
Associative learning comes with another problem: When an animal gets food, there is never a single predictive cue beforehand but rather a whole swath of cues. If you pair a tap to the side of a slug with a shock, how does a slug’s brain know to associate only the tap with the shock and not the many other sensory stimuli that were present, such as the surrounding temperature, the texture of the ground, or the diverse chemicals floating around the seawater? In machine learning, this is “called the credit assignment problem: When something happens, what previous cue do you give credit for predicting it? The ancient bilaterian brain, which was capable of only the simplest forms of learning, employed four tricks to solve the credit assignment problem. These tricks were both crude and clever, and they became foundational mechanisms for how neurons make associations in all their bilaterian descendants.
聯想學習也帶來另一個問題：當動物獲得食物時，事先不會有任何單一的預測線索，而是一整套線索。如果你將敲擊與震動配對在蛞蝓的一側，那麼蛞蝓的大腦如何知道只將敲擊與震動聯繫起來，而不將當時存在的許多其他感官刺激聯繫起來，例如周圍的溫度、地面的質地，還是漂浮在海水周圍的各種化學物質？在機器學習中，這被稱為信用分配問題：當某件事發生時，您之前的預測線索是什麼？古代兩側對稱動物的大腦只能進行最簡單的學習形式，它採用四種技巧來解決學分分配問題。這些技巧既簡單又聰明，它們成為神經元如何在所有兩側對稱動物後代中建立聯繫的基本機制。
The first trick used what are called eligibility traces. A slug will associate a tap with a subsequent shock only if the tap occurs one second before the shock. If the tap occurs two seconds or more before the shock, no association will be made. A stimulus like a tap creates a short eligibility trace that lasts for about a second. Only within this short time window can associations be made. This is clever, as it invokes a reasonable rule of thumb: stimuli that are useful for predicting things should occur right before the thing you are trying to predict.
第一個技巧使用了所謂的資格追蹤。只有當敲擊發生在衝擊之前一秒時，蛞蝓才會將敲擊與後續衝擊關聯起來。如果點擊發生在電擊之前兩秒或更長時間，則不會建立任何關聯。像輕擊這樣的刺激會產生持續約一秒鐘的短暫資格痕跡。只有在這麼短的時間窗口內才能建立關聯。這很聰明，因為它引用了一個合理的經驗法則：對預測事物有用的刺激應該發生在您試圖預測的事物之前。
The second trick was overshadowing. When animals have multiple predictive cues to use, their brains tend to pick the cues that are the strongest—strong cues overshadow weak cues. If a bright light and a weak odor are both present before an event, the bright light, not the weak odor, will be used as the predictive cue.
第二招是強因至上。當動物有多種預測線索可供使用時，它們的大腦往往會選擇最強的線索——強線索會掩蓋弱線索。如果在事件發生之前同時出現明亮的光線和微弱的氣味，則將使用明亮的光線而不是微弱的氣味作為預測線索。
The third trick was latent inhibition—stimuli that animals regularly experienced in the past are inhibited from making future associations. In other words, frequent stimuli are flagged as irrelevant background noise. Latent inhibition is a clever way to ask, “What was different this time?” If a slug has experienced the current texture of the ground and the current temperature a thousand times but has never experienced a tap before, then the tap is far more likely to be used as a predictive cue.
第三個技巧是潛在抑制──動物過去常經歷的刺激被抑制，無法與未來產生關聯。換句話說，頻繁的刺激被標記為不相關的背景噪音。潛在抑制是一種聰明的方式來詢問“這次有什麼不同？”如果蛞蝓已經經歷過當前地面紋理和當前溫度一千次，但之前從未經歷過敲擊，那麼敲擊更有可能被用作預測提示。
The fourth and final trick for navigating the credit assignment problem was blocking. Once an animal has established an association between a predictive cue and a response, all further cues that overlap with the predictive cue are blocked from association with that response. If a slug has learned that a tap leads to shock, then a new texture, temperature, or chemical will be blocked from being associated with the shock. Blocking is a way to stick to one predictive cue and avoid redundant associations.
解決信用分配問題的第四個也是最後一個技巧是噪音阻塞。一旦動物在預測線索和反應之間建立了關聯，所有與預測線索重疊的其他線索都將被阻止與該反應關聯。如果蛞蝓知道敲擊會導致電擊，那麼新的紋理、溫度或化學物質將被阻止與電擊有關。阻止是一種堅持一個預測線索並避免冗餘關聯的方法。
Eligibility traces, overshadowing, latent inhibition, and blocking are ubiquitous across Bilateria. Pavlov identified these in the conditional reflexes of his salivating dogs; they are found in the involuntary reflexes of humans; and they are seen in the associative learning of flatworms, nematodes, slugs, fish, lizards, birds, rats, and most every bilaterian in the animal kingdom. These tricks for navigating the credit assignment problem evolved as far back as the very first brains to make associative learning work.
資格追蹤、強因至上、潛在抑制和噪音阻塞在雙邊對稱中普遍存在。巴甫洛夫在他的流口水的狗的條件反射中發現了這些。它們存在於人類的無意識反射中；它們出現在扁蟲、線蟲、蛞蝓、魚、蜥蜴、鳥類、老鼠以及動物王國中幾乎所有兩側對稱動物的聯想學習中。這些解決學分分配問題的技巧可以追溯到最早進行聯想學習的大腦。
“These tricks are hardly perfect. In some circumstances, the best predictive cue may occur one minute before the event, not one second before. In other circumstances, the best predictive cue may be the weak cue, not the strong one. Over time, brains evolved more sophisticated strategies for solving the credit assignment problem (stay tuned for breakthrough #2 and breakthrough #3). But the remnants of the first solutions—eligibility traces, overshadowing, latent inhibition, and blocking—still exist in modern brains. They are seen in our involuntary reflexes and in our most ancient brain circuits. Indeed, rats with their entire brains removed, with nothing left but their neural circuits in their spinal cord, still show latent inhibition, blocking, and overshadowing. Along with acquisition, extinction, spontaneous recovery, and reacquisition, this portfolio of tricks make up the foundation of the neural mechanisms of associative learning, mechanisms that are embedded deep into the inner workings of neurons, neural circuits, and brains themselves.
這些技巧並不完美。在某些情況下，最佳的預測提示可能出現在事件發生前一分鐘，而不是事件發生前一秒。在其他情況下，最好的預測線索可能是弱線索，而不是強線索。隨著時間的推移，大腦進化出了更複雜的策略來解決學分分配問題（請繼續專注於突破 #2 和突破 #3）。但第一個解決方案的殘餘——資格追蹤、強因至上、潛在抑制和噪音阻塞——仍然存在於現代大腦中。它們存在於我們的無意識反射和最古老的大腦迴路。事實上，整個大腦被切除的老鼠，除了脊髓中的神經迴路之外什麼都沒有留下，仍然表現出潛在的潛在抑制、噪音阻塞和強因至上。除了習得、消退、自發性恢復和重新習得之外，這一系列技巧構成了聯想學習的神經機制的基礎，這些機制深深地嵌入到神經元、神經迴路和大腦本身的內部運作中。
Excerpt From
A Brief History of Intelligence
Max Bennett
This material may be protected by copyright.
