![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxoAAAN2CAIAAADxFGgkAAAACXBIWXMAABYlAAAWJQFJUiTwAAAgAElEQVR42u3dcXBchX3g8d/iFXazgthmFkSwg+yDDE6w3I5LsC13Ek9lLjLM0aYcwfbVE0KZa4KISdSZDISzYzfnTLnQnkE006GUTGvLc26TwRmwOkQTmgFByZRyWRG5k3CxMjJl0baRHWsJWCvv/bGuUJBsJFm2drWfzzCM/NAuu7+3Wn/13tv3IgAAOAuJYrFoCgAAU3aBEQAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAICKkDQCgHKTz+dra2tn6v9eLBatApiUhB8bAICzYWcfAICcAgCYOY6dApg9zv6gK0eAwBQ4dgoA4KzY2QcAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAIDzIWkEAEA1SyQSZ3sPxWLRHAEApszOPgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAAOWUEAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAoCIkjQAAqk0ikTin918sFqtrntX2hAEAppedfQAAZ8XOPgDgtPL5fG1t7WRvZWcfAACTYGcfAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAFDlkkYAUCgUDh06tHz5cqOYRl1d8bOfGQPjmD8/brghkrOoQRLFYtF6Barc+vXrOzs7+/v70+m0aZy9fD6WLo2LLorbbjMMxvHii9HZGYcPR329nAKofIVCobW1NSIeeuihurq6F154oX7WvMHPnL1749vfjrvvNglOq7MzBgbikUdmydNx7BRQvfL5/OLFi6+//vrdu3dHxAsvvHDzzTe3t7ebzFn6zndi2TJj4EyuvTZef332PB3HTgFVLZPJjOzgq6+vf+mllwYGBowFkFMAE5JKpVKp1K+8JyaTDp+C8+PkyZNvvXUiImpqaubMmSOnAAAm5/jx4//4jy+Xvl6yZMmVV15Zuc/FsVMAwAw7fPjw0NCQnAIAmLrh4WE5BQBQpeQUAICcAgCQUwAAcgoAQE4BACCnAADkFMD5ViwWDQGQUwAAM8A1+4CZl0gkDCFsIQM5BXA2nnmm2kti3TpNCZXKzj4AADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQD4FbmcGYCcAmCqurvj0ktj0yaTADkFwJRcdFFExL590dtrGMgpAJi8+vpoaoqIWL3aXj/kFABMSXt71NVFNhsNDYoKOQUAk5dORybzTlEdPGgkyCkAykAiEYnEOEtGLywUJvRt+fy03fB0D+zSSyObjYjIZuPGG6O72wpETgFQCZLJCX1bKjVtN5yIurpYtszKQU4BMNOKxSgWx1ky7sIpf9t03XDHjlNf79gRfX0TzTWYbb/hGAEAU9PeHtu3n2qpbdvMg+pl6xQAU1EoRGurloIIW6eAMrFuXcIQKsuLL0Y2G3V1cd99hoGcmh26umLv3ujvt0aZAYlE3HNPNDaaxJQV33VIDpXg1389Nm6Me+91vBTMjpzq7Y3f/d348pejudkaZQb098fv/E488YSimtYf697XXnut0UjLWCoV7e3GALMmp775zfj0p2PtWquTmfGBD8T998e+fXJquhQKhdWrV2ez2cOHD9fX1xsIUOZmxaHohw7F+99vXTKTli51iY3pksvlVq5c+dhjj0XE6tWru50XEpBTABPX3d3d0NCwZ8+eDRs2REQmk/niF7/Y1tY2+5Jx586dVjfMGg4ghOnx9okTx/r7IyKZTC5YsCCR8Dm1qVi2bFlfX1/yP45tTqfT3/3ud/P5/Gx6joVCoampKZPJXHXVVZs2bbLSYRawdQqmx+Dx4z09PT09PZlM5qWXXjKQKf6Gl0wmx3xOLDW1K56Upd7e3pUrV37ta1+LiMcff7ytra1QKFjvIKeAMWk1ODg0NGQOvEsul1u9evXIrsyOjo6IaC2dChOo6F8FjQDOheHh4ZqaGnNgtHQ6/frrr7/z/ptMtrS0GAvMArZOAQDIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAFNw6aVx/LgxcCa5XEScnDVPx1nRAZhmra3xm78ZV10V6bRhMH5LffOb0dyclVMA51axWDSEClVfHwcOxJ/9WZw8GRExNHTizTd/aSyMcrK5ObtuXb+cAoDTamyMxsZTX/f3H+3p6TETZjE5BZytRCJRWQ/Ydi9ATgFl55nDz1TKQ123ZJ31dZ6lUilDYHbzyT4AznlOLViwwBw4gwULFsybN69yH7+tUwCccytWrHjrrbfMgdOp6JaSUwD4+xLOlp19AAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAABERkTQCAM6FfD6fy+XMgbFSqVQ6nZZTAHBahUKhtXXrQw/9+cbbNpgGY33vmR8sv/baJw48mUql5BQAjOPFF1/s+dEPBo8dTKV+zTQYL7iHW//ozx988H9t2/YVOQVwyrol6wyBEW1tD93+6f+spThtfCTnfOGe/3rf/ftnzzOyUoGzVCwWDQGoZj7ZB5SjXC7X3d1tDkBFsHUKKDuFQqGhoSGbzfb398+yj/8AI44dO/YP//APEVFbW3v11Ve///3vr9znYusUUF5yuVxzc/ODDz4YEQ0NDbZRwaw3ODj48ssvDw8PyymAadDd3d3Q0PCnf/qnmzZtiohMJvPFL36xra3NZGDWGxoaqtwHb2cfUEaWLl3a19eXTJ56a0qn0x0dHQMDAyYDyCmACRl7Tr9kMunwKaDM2dkHACCnAADkFACAnAIAkFMAAMgpAM6n9vb2XC5nDsgpAJiKXC63efPmhoaGQqFgGsgpAJi0BQsW1NXVZbPZXbt2mQZyCgAmLZlMPvbYYxGxffv2gwcPGghyCgAmbcOGDTt27IiIG2+8cefOnQZCNf5eYQQATFZvb+/zzz8fEaWLVW/bti2fzz/wwAPbt29/9tlnL7lkvhFRVWydAuA9HDx4sK2trb29fWTJk08+uXnz5s2bN48cgf7Zz3629EVnZ+e///u/GxpyCmDmFYtFQ5gp+Xx+9Cf1brzxxrvvvvvJJ58cWbJw4cLSF0eOHImIQqHwjW98o7SkqanpkksuMUPkFADVG1Lr16+vra3dv3//yMLPf/7zEbFv376RJTfffPPhw4eHhobq6+sjYteuXQ888EBEPPzww9/97ncTCX+5IKcAxkhUByt67ty5nZ2dETF6W9SuXbsGBwdHby9MpVL19fXJZDIiDh48uH379oh47rnnWlpazJAq5FB0YKIGnnl5dj/BBet+o9rWaT6fP3DgQGtr69/93d81NjZGRDKZLG2LuvXWW0fH0+nuoVAo3HHHHRGxY8eO0j2AnAKgipTOaR4RjzzyyEgM7d69exKRPTCQzWbr6uruu+8+86Rq2dkHUF0KhUI+ny99XV9fv3Hjxrq6umuuuWZq95ZOp/fu3ZvJZEo7/qA6efUDVJGDBw/ecccd69atGznrwaOPPjp37tyziaHSqaegmtk6BVBF7r333mw2u2/fvt7e3tKSVCplwxLIKQAmas+ePRHx8MMPL1q0yDRATkElKBSiqytGnQ4RzrPe3t7169ePnJNz+fLlg4ODLS0ttkiBnIIKsWVLrF0bTz9tEsyIrq6uJUuWdHZ2tra2jiw8w1kPADkFZbdZIEpnkf7whw2DGXH99dfX1dWZA5xrNvbCOZHI5WL16oiIpqaorzcQzqdCoVDal5dMJl944YXjx48vX77cWODcsXUKpt+8bLbmuusim426unjiCQPhfGpra1u5cuXIwVL19fVaCuQUVJjLOjtXbdx4wRtvRERkMjFynEoiEe+6JNzplkzvt03LDamclrr77rszmcyWLVtMA+QUVKQLjx5d9j//5zt/TqfNhPPpD//wDxsaGurq6u666y7TADkFFenE/PmvffKT7/x57CkSxi75j8t9vIcpn21hyvc/wRtSTpLJZGdnZyaTcTVikFNQwX5y992Zr33t1B+am9/JlGIxisUYfbKf0pLRn1ovLSkWf+UeT3fDcb9t7JIp37+P01eIrq6uyy+/fORgqXQ6nbZZFOQUVLqfr1p14nvfi4jo7AyHsHAudXd3r127NpvNNjc3mwbIKZhVTq5eHc89FxGxb1/8x8XRYNotW7asqakpIn7rt37LNGCmOO8UnDONjbFxY+zbFz09Tj3FuXoTTyafeOKJn/70p86GADPI1ik4lx59NB5+OD72MZNgerW3t48cLJVKpbQUyCmYvVKpaGlxTDfTa+fOnZs3b25ubi64ujbIKQAmq1AoPPvssxHxyiuvDAwMGAjIKQAmJ5lMdnR0bNy4MZPJOCEClMsPphEAlL/Sfr2RCxu3t7ebCZQPW6cAKqClmpubHSwFcgqAKdq1a1dnZ2dnZ2dra6tpgJwCYNLuu+++pqamurq6+++/3zSgDDl2CpioBet+wxBm5p06mezo6BgYGHDsOZQnW6eACSlWh/IZeKFQ2Llz58jBUslkUkuBnAKYdE9U83Nvbm7evn27w89BTgFM0datW1euXKkkXnnllbffftvrAeQUwOS0t7f39PRkMpldu3ZVZ1GVLmzc0NCQyWRSLlIE5f8zawRA+cjn83feeWdEdHR01NTULFy4cOXKlZ2dnVVy2NDoc3WmUqkf/vCHXhIgpwAmJ5fL3XXXXY2NjaU/trS0fOxjH/vxj39cDTmVy+WampouvfTSjo6OUlEBcgpg0urr6+vr60cvWb58eZU8961bt2YymYjYtWvXtm3bvBiggjh2CqAsPProo3V1dU1NTffdd59pQGWxdQqgLKRSqVdffXXu3Ln29EHFsXUKYMZ0dXWtX79+5NOLqVRKS4GcAmASLbV27drOzs7m5mbTADkFwKRdf/31DQ0NEXH77bebBlS0WbFV+cMfjr4+65KZ9Oyz+Q9+0BiY3PtvMtnZ2ZnNZqvn04sgp8rYH/1R1NZGRCxebI0yA/r6hv/2b//1kUdMgvfU29t78803v/TSS6U/ptNpFzYGOVUeUqkYHIyvfz0ymYh4+8SJwePHrVrOm/wHP3jkscdOzJ9vFJxZd3d3ae/eli1bTAPkVFkW1fbtpS+P9ff39PRYtVDphoaGZtkzWrZsWekyfNdcc431C7OJQ9GBcv1tb7acMmDkPAilg6WeeuopJz0HOVXuLr74YuuVGTd37tyqer6JCnH+Q6qtrW3x4sX5fL60JJ1Ob9iwwQ8IzLZf/2bfU5o3b97VV1/9k5/8xNplRlx44YVXXXXV+f+be8YVC8+Ue/Ml153n/+P+/fvvvvvuiLjzzjvb29v9dICcqiRXXHHFFVdcYe0CM+vWW2/9kz/5k/7+/nvvvdc0QE4B8N4KhcL+/fvnz59f2qNXOlhqwYIFLh0DcgqACVm5cmUmk6mrq+vr6ysllNNKQTXwyT6AafOlL32p9MWhQ4dMA+QUAO8hn8/v3LlzxYoVI0tuvfXWHTt29PX1uW4MyCkA3tuDDz64ffv2TCbT1dVVWpJMJrdt2+ZIKZBTAIyjUCh0dXVt3bp1ZElra2tE1NXVHTt2zHygmvkVCmBC9u/fv3nz5oj4gz/4g9K+vFQqlclk7NcDbJ0CGEdbW9v69esvv/zykSVr1qwpfdHX1zeyUEsBcgqoOvl8vr29vb29vbe3d2Th+vXrE4nEpk2bRpYsXLiws7Mzm83mcrnSkvr6+qeeempwcNBVYoB3sbMPqC65XK60z27v3r319fWlhaWzQ/3oRz8a+bYrr7yyoaHhIx/5yOjbCilATgGMr7Qj75prrhlZ0tjY+MMf/tBkADkF8G6LFi0aGhp617kMWlpaWlpaDAeQUwATeNdzUqhzb9mya37+86PmwBm89lpu+KScAoDT2LLl9lWrrlu48KI1q681DcZtqc/e9b83/7c7Z80zShSLxQp96L1He7/+j1/P5rNel8zkj1Ak7vnoPY2LG6t9DolEsfBMuT/I5LryecdLJCr47Xciuru7//Ivv5Hrz0bE0NDQm2++6e2CEcMn46abfnfZsmVz5swZWbhq1ap58+bJqfMqP5Sv3VX71U9+tWFxg9clM+inuZ9+9Ttf/d7vf2/5pVV9/iE5JafOoL+/v6enx9sFZ1bROVWpO/u+3/v92z562yeWf8Lrj5n1gfkfuOeGe/7in/+i7RNtpgFQnSr1NJ5H3zp67SK75CkLDYsbcvmcOcDppFIpQ+A9jd7xJ6cA4N05ddlll5kDZ3DZZZfV1NRU7uP3yT6YBifePtHf3x8Rc+fOvfjiixOJhJnAaMuWLbvqqquGh4eNgrHmzJlT0S0lp2B6HB88PnKk7YIFC1asWGEm8C41NTWV/lcmnI6dfTDNBgYGhoaGzAGgetg6BdNveHi4Cn8LTyTXWfWAnAKYouo5hRLAWHb2AeUol8t1d3ebA1ARbJ0Cyk6hUGhqaoqIzs7OdDptIECZs3UKKC/5fL65ufnOO+/MZDJNTU29vb1mAsgpgInq7u5es2bNF77whZaWlog4cODA6tWrDx48aDJAObOzDygjS5cuff7550euSVJfX9/X1zcwMGAygJwCmJCxF3dLJpMOnwLKnJ19AAByCgBATgEAyCkAADkFAICcAgCQUwBUtj174rLLIpHwj3/e/c+118ZTT822F7zzTgEwzbq6YseO2L076uoMg3fr7o6bborBwRhzmrkKZusUANOsvT02bNBSjG/58vj0p+PAgVn1pOQUANPs3/4tLrrIGDitK66Ybc9ITgEAnBXHTgEA59trr7126NAvIuKSSy655JJL5syZI6cApt/Q0JAhwGx17Ngv3njjjYh44403lixZcuWVV1b007GzDyjX3/aSft+DqnD48OGKf7+yFoGxEomEIUREsVg0BEBOAVP0zDPVXhLr1mlKYELs7AMAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAMxq+XysWBFdXSYBcgqAKfn+9yOTiVtuiULBMJBTADB5N9wQdXWRzcaWLYaBnAKAyUsm4+mnIyL27YudO22jQk4BwOQtXx5790ZEbN8ezc2RzxsJcgqAX5VIRCIxzpLRC8cuKRQmdMN8ftpueK4f2BlGsXlzPPVURERnZ9TW2kaFnAJgOiSTE/q2VGrabniuH9iZLV4cdXWnvh4YsP6pyp96IwA4nWJxKkum99vO/w0ndf9dXdHQEBFRVxdPPx3ptFcN1cjWKQCmKJeLtWsjIpqaoq8vli83EuQUAExGU9Opf3d0THRPIsgpADilqysymYiIJ57QUsgpAJi8D30oImLv3qkfGg+zhl8oAJiKdDoGB7UURNg6BcCUaSmQUwAAcgoAYKY5dgoY37p1CUMAkFPAFBXHPVU2AOOxsw+Yuq6urgp92AWX6gXkFDDj2tvbb7nllra2toprqVtuuaW1tdUaBOQUMJMOHjzY2tqazWYPHDjQ3t5eQS31uc99LpvN9vT07Ny50zYqQE4BM2PTpk179ux59dVXI6Kjo+PFF19cv359+T/snTt3fuUrX+ns7Cw97IULF65cuTKfz1uhwFlyKDowabt27aqvrz/1JpJM7t69O5fLFQqFZHlfuW3Lli3btm0bedgtLS2f+tSn5s6da4UCcgo430ZaakQ6nfawgaplZx8AgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAgHPLRWaA0+o92hsRqQtT6feduhhLfiify+cion5+fWlJ4WThyC+OREQ6lU7VpEoLc2/m8ify495w0cWLkhckR9//2BuOe/9jbzjlBzb6Oc7gA8sP5Sc7MaA82ToFvBMWW/9+a37ondpYsnvJkt1LtnZsHVly4F8OlBaOLDnyiyOlJQf+5cDIwq0dW093w1JJjL7/sTcc9/7H3nDKD2z0s56pB1b6TxOc2Na/39rd3+0lCnIKKHcPPv/gQy8+NPrv+Pc0NDRUic/0tv9z28w/iP8x0W988sdPPvTiQzf8zQ1eolC2bEAGTvnGP30jIlqfbt20fFNpyd5P7o2IK+dfOfI9axavKS089Q6STKZT6dKSNYvXjCy/66N33fShm8a9YTqVHll4uhuOflQj9z/2hmd4YKNvWOkP7PH/+3hEZAezXX1djYsbvVChDCWKxWIlPu727vbDcfgTyz9hFTLj/vXov+7ev/v+D90/smTVqlXz5s2rxB+rUgr4O7us5Ifyj7/8+MJfW7hm8ZqRA7DK3Kc+FVdfHU1N1h7j6+yMiENNTW+MLPn4xz9e0c/I1inglJGNUpSVVE2q5aMt5gDlzLFTAAByCgBATgEzbtO3Nm361g4ILAcAAASvSURBVKauvi6jKCu5N3NWDcgpoDLse2Xfvlf2/ezoz4yirORP5K0aKHMORQcoa6kLUxuv3Ri/ev4FQE5BVTj4k4N3fOeOF+54oVI+3F7cXrTWylD6fen232s3ByhndvbBOVE4WbjjO3dkB7M9uR7TAJBTwKRbqnlvc3YwW1dbd8N/cm0QgFnOzj6YZsPF4S9970udP+2MiKd//+nkBRXzU9Z7tDci0ql0qiZlPZaP/FC+dCHFCjorOsgpYOqODh3940N//M8D/xwRz33mueWXLq+gB79k95KI2PvJvU6PXlZy+dzmb28urRo5BXIKZrnh4vAd/3THz0/8vNRSjYsbEzsSpf80+ijv0sKJLJneb3vPGwIgp6CSFE4W3rUfMD+Un8hetrE3nKD3vP/DWw9HRDqVtnbKyqKLF1k1IKegWsxJzHnsNx8r7exb+1drn/vMc+Nu+xm7cCJLpvfbxr2hHUll+jZ9QdKqgTLnk30wnebXzH9g+QOfW/m5iFj7V2u7+7vNBEBOAZMzJzHnwaYHm5Y2RcQNf3ND4WTBTADkFDBpHZs76mrrsoPZp//f05XymBM7EokdifZuJ+AuL71He60akFNQjZIXJB/7L4/V1dZ9OP1h0wCY5e/5RgDnyIarN7ze+noFPWDX2S1P6VR67yf3RsSaxWtMA+QUUNZcZ7c8pWpSTqwKZc7OPgAAOQUAMHPs7ANOKX1wzHV2y03uzdzWjq0RcddH72pc3GggIKeA8uU6u+UpfyK/75V9EXHTh26SU1Ce7OwDADgrtk4Bp4x7IT9mXP38eqsGypytUwAAcgoAYObY2Qec0nu0NyLSqXSqJmUa5aNwsnDkF0esGihntk4BpyzZvWTJ7iUH/uWAUZSVI784YtWAnAIqyas/f9UQykrqQlukoNzZ2Qec8vnrP3/rR269/orrY9SpI3c3706/L136hk3f2hQRaxavafloS2lJV1/XIz94JEZd72/cc062/aDt+b7nx73h2Psfe8Nx7796Hlj6fenB+wYff/nxCroE8jXXxNGjfqQ4rUOH4rrrhuQUMAvt/sTuka9HTh2567d3jcRBaUlEjMTBz47+rLRwJA7GPefk833PlxaOveHY+x/3hmPvv6oeWKomNXIPFeH22+O66+Lyy6O+3g8W7/bKK/Htb8eddx6TU2Xh2C+PeVFSDt58+01DgNHq6+OJJ2LPnvjBDyIihoZOvPnmL42Fkvnzf7lvX++8ecOz6UklisWKPDtcV1/XZ578zF//979OXmADGzPsy9/68nUnr/vt9G+PLFm1atW8efNMBkr6+/t7enrMgTP4+Mc/XtGPv1JbpHFx44pLV9z0ZzetrF/pVcgMevWNV+PtuGfFPUYBULUqeNPO/t/bn3szlz+Rj4i333775ZdftjqZAUujbl6dMcAZpFI+nMiZ1NbWyqmZlH5funTI51tvvfX6vNe9IgHKM6cWLFgwMDBgFIz/a+nSpXIKAN7DihUrhoaGhoeHjYJ3qampmTNnjpwqC/PmzbvwwgtPnDjhdcmMq62tdRw6jPu3Zk1NjTkwK1XqJ/vGeuutt44cOTI0NGSlMrN/YSxatEhOAVSV/w9RPh8BlfR0RQAAAABJRU5ErkJggg==)
418 Chapter 12. 基于自注意力的模型 肖桐 朱靖波
在得到 Q,K 和 V 后,便可以进行注意力的运算,这个过程可以被形式化为:
Attention(Q,K,V) = Softmax(
QK
T
√
d
k
+ Mask)V (12.9)
首先,通过对 Q 和 K 的转置进行矩阵乘法操作,计算得到一个维度大小为 L ×L 的
相关性矩阵,即 QK
T
,它表示一个序列上任意两个位置的相关性。再通过系数 1/
√
d
k
进行放缩操作,放缩可以减少相关性矩阵的方差,具体体现在运算过程中实数矩阵
中的数值不会过大,有利于模型训练。
在此基础上,通过对相关性矩阵累加一个掩码矩阵 Mask,来屏蔽掉矩阵中的无
用信息。比如,在编码器端,如果需要对多个句子同时处理,由于这些句子长度不统
一,需要对句子补齐。再比如,在解码器端,训练的时候需要屏蔽掉当前目标语言位
置右侧的单词,因此这些单词在推断的时候是看不到的。
随后,使用 Softmax 函数对相关性矩阵在行的维度上进行归一化操作,这可以理
解为对第 i 行进行归一化,结果对应了 V 中不同位置上向量的注意力权重。对于 value
的加权求和,可以直接用相关性系数和 V 进行矩阵乘法得到,即 Softmax(
QK
T
√
d
k
+Mask)
和 V 进行矩阵乘。最终得到自注意力的输出,它和输入的 V 的大小是一模一样的。
图12.10展示了点乘注意力的计算过程。
MatMul
Q
K
Scale
Mask(opt.)
SoftMax
MatMul
V
自注意力机制的 Query
Key 和 Value 均来自同一句
子,编码-解码注意力机制
与前面讲的一样
Query 和 Key 的转置进
行点积, 得到句子内部
各个位置的相关性
相关性矩阵在训练中
方差变大,不利于训练
所以对其进行缩放
在编码器端,对句子补齐
填充的部分进行屏蔽
解码时看不到未来的信息
需要对未来的信息进行屏蔽
用归一化的相关性打分
对 Value 进行加权求和
图 12.10 点乘注意力的计算过程
下面举个简单的例子介绍点乘注意力的具体计算过程。如图12.11所示,用黄色、
蓝色和橙色的矩阵分别表示 Q、K 和 V。Q、K 和 V 中的每一个小格都对应一个单词
在模型中的表示(即一个向量)。首先,通过点乘、放缩、掩码等操作得到相关性矩
阵,即粉色部分。其次,将得到的中间结果矩阵(粉色)的每一行使用 Softmax 激活
函数进行归一化操作,得到最终的权重矩阵,也就是图中的红色矩阵。红色矩阵中
的每一行都对应一个注意力分布。最后,按行对 V 进行加权求和,便得到了每个单
词通过点乘注意力计算得到的表示。这里面,主要的计算消耗是两次矩阵乘法,即
Q 与 K
T
的乘法、相关性矩阵和 V 的乘法。这两个操作都可以在 GPU 上高效地完成,
因此可以一次性计算出序列中所有单词之间的注意力权重,并完成所有位置表示的