![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxoAAAQZCAIAAADNEeslAAAACXBIWXMAABYlAAAWJQFJUiTwAAAazklEQVR42u3dQWgcZcPA8clLii80igiVgBaL9FBbsxZMSU0MNXQbqVR6qCCRXqTtoSBK8ShUE1BvgmLpQaS3LRaKCAEpbYiSJiUYQZO2igpNUUG6UAWz1JKFvIf5vnHZTdvNJpnMzvx+J330fbt5dmfmv/PME4MAAIBlaFlYWDALAAAN+48pAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAHLKFAAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAAA5ZQoAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAIDkajUFQJ1KpVKxWFzS/2TDhg3r1683dYCcAtZYsVgslUpVgxMTE4v+y5OTk3eJnmKxeOHChab4qQcGBu7yT7ds2bJ58+ba8QcffHDr1q2qDohTy8LCglmAOHvo77//npmZif7RzZs3q8JodHT0jz/+qPp/aG9v7+vrq62Erq6uRf/Ejo6O+++//y4vKbGFsWg7Vvr999+vX7++6D8aHh6uHTx9+nTtYO187tu3r/Jvu7u71Rggp2AVVS57VcZQ5eW86iqez+c3bNhQz8Xb9TvmYrt74F65cmV6enrRDqvM2ccee+yRRx4J/3rTpk0mGeQUZFe5XP7tt98qL7GVF9fKQsrlctu2batto8owcllN68ejKsJ++eWXH3/8MfzrypuL0YckCq9oLXL9+vWVeQ3IKWi+y2G0YBTeT6p8oii6BEZP51Q+jqOQqFN0C7MyvKL7l1GaR7e7os9bWOTuUIKcgrUULuVUBVN0zyCqpe7u7oceeii6erlbwBr2fZRc4U6Cyr4PH8kPYysqe1kPcgpWxuzsbPD/jy5VNVN4BQqDyRWIZv9uEMVW1ec8fCAv/JyHmw/c0wI5BYsLF02uXr36119/hZeTcKEkvM8UPpUSNpM7TGTtuAhLK3zCL7qnVZlZ3d3djguQU2S6nKJNVQMDA7IJ6hEuIIaL3eEz8uHdrPDrR9RY7mOBnCJV8TQxMRGe9MN7Tvl8fuvWrV1dXeHO80cffbS11e+bheWanZ0Nb2WFj2eFh1v0RaWjo6O9vd23FJBTNMeX5qtXr05NTUXxNDAwED5j6+syJOH7THhIdnZ2bt261ZcZkFMkpZ8mJiYmJye/+uqr6enpXC733HPPhV+FH3/8cfEEyQys2mO2u7tbXYGcIiazs7NXr149d+5ceC7O5/O9vb3hN10b66BJj+ipqamxsbELFy7kcrkDBw50dnbu2LHDyiDIKVZMuVz+4Ycfvv7664mJidOnT4ffZZ9//nn9BOk72MP7zcPDw6Ojo0EQ9PX17du3r7u728GOnJJTNJ5QX3zxxYULF/L5/P79+3ft2mX9DrKjWCz+9NNPIyMjZ8+enZ6eHhgYOHjwoLtWyCm499nz/Pnzw8PD4V2oI0eO7Nq164knnvA4BTg5fPPNN+fOnTtz5kwQBEePHt29e3dXV5eTA3IK/s/MzMznn39+9uzZGzduhGfJ7du3uwsF3CmtPvvss/DWdXjLateuXc4YyCkyXVEnT558+OGHjxw5sm/fPo9HAPUrlUrffffdmTNnPvroo3w+f+zYsf7+fverkFNk5ZvlyZMno4p6+eWXPQkBLEe5XJ6cnDxx4sTp06dff/31w4cPd3R0mBbS5D+mgOh8Nz4+vmfPnlwuFwTB9PT0999//9prr2kpYJlaW1t7enoKhcLc3FxXV9fBgwefeuqpQqFQLpdNDung7hRBqVQ6derUu+++++STT77zzjueHgVW28zMzPvvvz86OvrWW2+9+uqrnqyi2bk7lfWQGhoaamtr+/nnny9dunT+/Pmenh4tBay2jo6OQqFw6dKlmzdvtrW1ffzxx+5UIadoPuVyuVAotLW1BUEwNzf34YcfeswciNmmTZuOHz9+48aNiYmJjRs3FgoFc0KTstiXRTMzM/39/X19fe+9956KApJgdnZ2//7927Zt++STT6z90XTcncqWcrk8NDTU39//6aefFgoFLQUkxKZNm7799tstW7Zs3rx5fHzchNBc3J3KVkvt3bt3w4YNvvwBiTU+Pv7ss89evHixp6fHbNAs3J3KimKxuHHjxt7e3kKhoKWAxOrp6bl27dpLL73kHhVNxN2pTCiXy08//fSBAweOHz9uNoDkm5mZyeVyN27c8KvvkFMkxRtvvFEsFu2aAZpIoVAYHh524kJOkQjFYjGXy/36669+oRTQFML/2N/27dvb2trm5+dbW1vL5fKZM2deeeUVk0Myub6m3/nz548ePaqlgGZx3333nThxIgiCvr6+ycnJBx54INyPbGZILI+ip9+pU6d2795tHoCm+aLf2looFA4ePBg+pdDf3z89Pf3CCy+YGRLLYl/KlcvldevWhXfLzQbQXD744IM333zz2rVrfkkecoo1zqnJyUm/vgVo0jPYn3/+aXMfcgoAIOUsAMUesC0tWfgxZXp23uskvN0OK1Ntqk312vIo+tp8etLNWxwZHBzM5/Pz8/PR5KzIyPz8fD6fj3NkcHAwejFVIw6rrB1WpjrmqR4cHGxvb5+bm6s8IdSOVJ00qg7klRqJThFVJ4TGRlL1/dnFL/4vHKmf8yz8jHXOQz6f//LLL6N9AENDQ2NjY8scCf/bi0EQxDbS29sb/T792pEkvN0OK1Od7qkulUpV/3Gw2pFyuVy55ahcLgdBkPCRdevWpefjtIA7Nz5Xq/ZeV37JO3ToUNXXvnpGjh07VvXVcOfOnXGOVN6XunXrVu1IEt7uyu/xlS8vaSN3utV38eLFe44k5LByBotzqqvOBhcvXrznyKL3thN7UKTpYuGyt2bnfT9jpuZhcHBw586dt27dqhqpOutVjcQmvGBv37696pb+oUOHqv6d2pFE5VRjyxkxz3Pl1WVubq69vb2ekeRc453BYnsZDazxhSNr8oIbWPVL08fJooxb5X7GVZ+Hxtb4YtPYGl808vbbbydnsa+xRYeYZ7vqj65nHadUKrW1tVnsy9pi39zc3FLX+BYdifPjvaQDME0fJ4+iw+pq+Hmp2F7hcloqGkmI1tbWqgtJPSMxv8KqkaqrY50jZEH4vo+Pj+/ZsyeskNqR8BM1NDRUNRLzWS48ZUUHV50jaXqz/KIESGJLjY2Nxfk609FSkEojIyOXL1++fft2dIoYGRlJzssrl8tV56vGRpqdRZnYZ9yt8iy91yuysy+Gs+FyWsrOvkz9jKZ6TV5Gytb4opE07eyz2Lc2B0m6eYtr7/oEQXD48OGqTqpnJAZRS/X29laVU9XIP//8UzuSqAmPFhSWNBLnVWfPnj2Vf/qdRsbHx6tG0nEG8yKXpIE1vnAk5te51DW+aCRNZ3uLfWsgC9/tvMuVpRKeQa5cuTI6Olo1MjY2dpeROC/ze/fu3bZtW9VdqKqRF198sXYkURMeLh9Ufk2vZyRRbt++ffny5ZGRkeg/tRmOOIPF9iKTcwZL8hrfoqeRDK7x/fuxsSgTf2q4VZ6197qx39UZ52JfA2t8dvY1PNv3XKOxs8/PGNxhZ189nx87+9aExT5YXcv5vefxWE5L2dnXwCu854idfVS+7/Wv8TXRzr6kLWGvwKHt8wpZbqkgCFLTUpBK1viagkWZ2GfcrfIsvddL3dlX+xs1Yz4t2tnnsDLVCZzqlK3xRSN29rHcg8TOvoxY0j6+crm8Y8eONWypdOzsi5Y8opFm3NlXO5KOM5gXuSQNrPElfGdf5cfbzj6Wy86+7Kh/Z1941+e///3vGt6XeuaZZ5p9Z1+tptvZV8vOvphfpDNYw6eRDK7x/fuxsSgTf2q4VZ6197qBNT47+xr+yFVNXTPu7Ft0r5+dfVn7GVtaWubn5xv4tCR/1S/6Wzv7gHo11lJ29jUsBTv76tnrRxYsdY0vsTv7atf47OwDUtVSgZ19wDJkfI0vYlEm9hl3qzxL7/Xdd/bdpaXs7PORS+bPaKrX5GWkbI0vGrGzj+UeJHb2ZcSS9vE1486+69evJ2rCF93Zd8+9fnFOsp19XmT9mn1n353W+OzsY2XY2Zcdd9rZV7uPL5k7+6pGqnb2FYvFnTt3Nt2b0nQ7+5zBYn6RzmANn0YyuMb378fGokz8qeFWedbe6wbW+FZ2ZPlPR9WOFIvFXC539OjRhO/sCxra69fw4sXqjSRkWcQZLM6XsVI7+xoYiecwsbMPqFe6Wyr5O/uChvb61Tmy1D96BUfIguXs7FvmSP2/mTNoaI3Pzj5ASyW3pYCVVbU4bh/fnViUiX3Gs7Eq73MVZOkJjCQs9plnU22qTbWcAgBoVp6dAgCQUwAAcgoAQE4BAGSRX5QAq8XOPlOdsnk21aY6lVMtpyDppqbSv3O2s7PFVGdnnoMgWJiaSvkXoc5OJ5CsfaqXz2IfAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAGdKysLBgFmBVjq6Wloz8pGt+GsnIVCfhdG2qTXX6plpOAQCsPYt9AAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByyhQAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAEGo1BZBBLS0tSXtJCwsL3hegWU+qTmEAAMthsQ8AQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFALBSWk0BrJRSqdTW1pa0V7WwsOCtAVhV/wPCgiy8nPnFYwAAAABJRU5ErkJggg==)
372 Chapter 10. 基于循环神经网络的模型 肖桐 朱靖波
表示
ˆ
h,使得它与目标语言位置 j 对应得最好?
那么,如何理解这个过程?注意力机制的本质又是什么呢?换一个角度来看,实
际上,目标语言位置 j 可以被看作是一个查询,我们希望从源语言端找到与之最匹
配的源语言位置,并返回相应的表示结果。为了描述这个问题,可以建立一个查询系
统。假设有一个库,里面包含若干个 key-value 单元,其中 key 代表这个单元的索引
关键字,value 代表这个单元的值。比如,对于学生信息系统,key 可以是学号,value
可以是学生的身高。当输入一个查询 query,我们希望这个系统返回与之最匹配的结
果。也就是,希望找到匹配的 key,并输出其对应的 value。比如,当查询某个学生的
身高信息时,可以输入学生的学号,之后在库中查询与这个学号相匹配的记录,并
把这个记录中的 value(即身高)作为结果返回。
图10.22展示了一个这样的查询系统。里面包含四个 key-value 单元,当输入查询
query,就把 query 与这四个 key 逐个进行匹配,如果完全匹配就返回相应的 value。
在图中的例子中,query 和 key
3
是完全匹配的(因为都是横纹),因此系统返回第三
个单元的值,即 value
3
。当然,如果库中没有与 query 匹配的 key,则返回一个空结
果。
value
1
value
2
value
3
value
4
key
1
key
2
key
3
key
4
query
匹配
返回结果
图 10.22 传统查询模型
也可以用这个系统描述翻译中的注意力问题,其中,query 即目标语言位置 j 的
某种表示,key 和 value 即源语言每个位置 i 上的 h
i
(这里 key 和 value 是相同的)。
但是,这样的系统在机器翻译问题上并不好用,因为目标语言的表示和源语言的表
示都在多维实数空间上,所以无法要求两个实数向量像字符串一样进行严格匹配,或
者说这种严格匹配的模型可能会导致 query 几乎不会命中任何的 key。既然无法严
格精确匹配,注意力机制就采用了一个“模糊”匹配的方法。这里定义每个 key
i
和
query 都有一个 0~1 之间的匹配度,这个匹配度描述了 key
i
和 query 之间的相关程
度,记为 α
i
。而查询的结果(记为 value)也不再是某一个单元的 value,而是所有
单元 value 用 α
i
的加权和,具体计算如下:
value =
X
i
α
i
·value
i
(10.21)
也就是说所有的 value
i
都会对查询结果有贡献,只是贡献度不同罢了。可以通过设
计 α
i
来捕捉 key 和 query 之间的相关性,以达到相关度越大的 key 所对应的 value
对结果的贡献越大。
重新回到神经机器翻译问题上来。这种基于模糊匹配的查询模型可以很好的满