![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxoAAAQVCAIAAAC60yteAAAACXBIWXMAABYlAAAWJQFJUiTwAAAgAElEQVR42u3db2xd9X348U86+kekaSmq2ZX42zRlNE3uoIWlc0aziFvS0ZQUwcJcOiqmRVoochXxALWTCk47qj5A/RWvZBubqrGajBTKoDErEEQjbK8ZdLQ2BFRFsbMRKfJtWVbstpvd3d+D4x0fn3vt+L/vdV4v8cA4jnN9fO75vu/3fM+5EQAAzMGKSqViKwAAzNqbbAIAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAdeUMmwCA5Wl4OMrl/CfPOy/OMPYhp4DJjI7Ga6/F4cNx9Gj09NT4gq1bIyLWr4/Vq2Plyvp68AMDcfx4HDsW+/fX+NPm5jj77Ljwwrj44mhqqsfNnkg2+4UXxrnnRkRcdFED7Dblcpw4EX19ERFHjsSrr9b+sksuiTVrxn60+iyScjmef35s5y+X48CBU3x9qRRr18aGDdHcrLGYoxWVSsVWgMbW1xcHD8b990dv7wz+VqEQ27fH9u1x6aVLllblcjz9dHzzm6ce+XJaWuLTn44rrliytEoe+f798eyzceLEVF9ZLMbv/35s2bKUjzZneDgOHownn4zDh2e85XP7z5YtsXbtUlZj+rPs25f/RRQKsXnzhCJPe7dmbBWLsWNHbN3aGBGMnALmx8BAfO1rce+9NV5tr18fq1bFypXj43d61uPw4XjhhXjuuQnDSakUu3bF1Vcv0gv04eG4557Ys2d8/CsW4wMfiK1ba8zrpHM/x4/Hiy9GT0/s3ZsfAm+5ZZGKsFyOPXvikUcmlGu62bNef73GHEmxGHfcEdu2LU2/9vXFo4/mH3zSHM3NsXp1rF079slcT6S/gsOH4+TJOHQoyuUJv4WIaG2N7dtjw4bFm+Pp7o59+/L7/7ZtsWlTFArTKtfk5+rpiSNHJjwjklLctUtXMQMVoOF0dVWKxUrE2H+trZWursrQ0Iy/T29vpa1t/FsVCpW2ttl8nxn9iy0t44+8VKp0dlYGB2f8fQYHK+3tlVJp/Fu1tVX6+xf1kXd0TOuRj4xU+vsrHR0THm1Ly8I+2ty2amurFAoTHnx7e6W3tzIyMqfvnPxcLS0TvnlbW6W3dwF/nKGhCT9OstP298/1Z0m+c2dnpbV1/GcpFiudnfPwnTkNyClotJBKB5JisdLVNT/H+mTEzY6I8x5V/f3jPVEoVDo65uefGBqqdHSMb5OFyJT+/vHiLBbn9MiTCky/20JHVVfXhIabdXZPMzdzIdLVtSAhlf1xFqjbRkYmbLqk2EQVcgqWg2yOLNAwnEuTjo75GUKGhsYH2oUYZZPxr7NzPFNaW+et1RbokWfnF1ta5j9xst8/mWJZ0EnHXIik03jzNbszMlJpb1/Y3J/sSZcGXPIaQFQhp6BRjYyMH9OLxQU/STQyUunoGP/n5jgB0Nk5Php1di74tursHMvBZPCbr2+1EAmYjZ65P9rJQm3RTilWh0gaVYXCnPai7OzgooVUrqqzUbVAOwNyClhAg4PjI+4i5Eh2CEmHw9md6RgZGf8O7e2L97I+O5NRKs1m9B0ZGZ+UWuhHnnu0s1hGlt1VFnr+ci5RNYtJuOzGWYg5vFk/I1paTFMhp6BxpFM7SzWW9PaOzdDMdKRP/+IiTKdN3RYznRrJ9utirhZPH+3sJj/SCcVSqS5CKrczzOIlwVK9kDhlIKaPakFX3COngPmRnl+Yr9NAs35RPtORvqtrfA3TEr6Iz54kbW+f7iNPKnBJph/SmZhpPtpcii35rjL17yL96ar3iupNneb4kk9KTb1fLe0ejpwC8iNi7nidFsxczv7Mo3TyIzdgDw3lh5N0pKmTJSZpIeVGvpGR/DidVuASRslMey79+jmeKFwc6dRO7tHmlo6lk7JtbfX7s2QnbhUVcgqW3tBQJWL8dEbaUvU2QKYjd3aQKxYrra35lqq38yCDgzVGvuTeRdVD+JJXYDrblBune3vz/Zc27oxms5ZW9qVCelKyVKqUSnWa49P5WYrFBmhZ5BQsf4XCWJRkW6oOX/KmXZIuTm9tHY+StKXqcGjJbtgkSpJ+Sh5q2iV1MoTX3A1KpUpLy/gXNFB2VMs9+OQ84NDQ2AeLuWpt7r+p5KqFBnrMyClYtpLRpc5bKldUSf8lUZLenqc+W6p6LmFkpDI4ODYpmJ7jq6vhsHrmI91JkrSq8619Sulm7+qq9PdXIirf/naj/lBpHSoqOQUspWR2ZOPGxliKMTQ0PkeVDITXX98YA2F674ZkIycf1O0cT/pok/7r7a1EjL0v0KzvAVGfRfW9743fonP//gZ+Cjd04CKnYJnkVFJUR47MzxuQLah0jmrXrrFH3tRU+eEPK/399T7GZ6cA0yH829+u9PfX6UCY9t8vf9lI5TqjompqGv9dpP810IKwRP1P0LKQVlQqFe8DDUvswQfjppvyn2xvj9tuq9/HXC5HsRgnTtT4o1Ipnn66fh/56Gj8wR/EgQM1/qhQiCNHYuXKeny0GzdGd3dExLveFXffHe94RzQ3R1NTfT3aWdi9O+68M9773vjgB+OTn4yzzoqTJ+P11+Oyy2Ljxob8WepwL2LhySmojzT58pdjw4ZYvz5WrYo33oi+vti2rd6PyAMD8Z73xB/+YXzkI3HZZXHuuRERPT3R3BwXXVTXj3x0NM4/PzZujGIxLr881q6NiDh8ON75znocwqfov2Ixfvzj5bD/NzUtk+fy7t3x3HPxz/8cZ5zhwCanAKZneLhRX4U31iMfHo7m5ti5My64IK64IpqaYnh4rELMgsyr78Z35zqsjv66csZvzP2RfCI+4dfRQOQzMAeNO5Y31iNfuTI/C7VypZCqT/PSUjScN9kEAAByCgBATgEAyCkAADkFAICcWt6Gh+OJJ2J42JYAADnFrOzYER//eBw8aEsAgJxi5rq7Y+/eiBi7iTMAIKeYgXI5brghIqKlpd7fwQMA5BR1p7t77O1mi8V44AHbAwDqhzeZaQQPPhg33TT2cW9vvPnN8/9PeOtGAJgts1N1b3h4vKUAADnFjK1cGe3tNgMAyCnm4LbboqsrCoWIiFIpRkaiUpnn/wAAObXMbdwYvb1RKMSBA3HzzbYHAMgpZq6pKR5+OCJi794YGLA9AEBOMXMbN0ZLS0TE4cM2BgDIKWbl/vujszM2bbIlAKBOuO9Uo1m5Mq65xmYAgPphdgoAQE4BAMgpAAA5BQAgpwAAkFMAAItpRcX7tQEAzIHZKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAAOWUTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByCgAAOQUAIKcAABrQGTYBwIIbHo5y+dRf1tMzp39l/fpYteoUX3PRRX4bIKcAli6DDh+OkyfH//fIkXj11QlfXy7HgQON92OWStHUNOEzW7eOf3zWWbF2rSCDKayoVCq2AnAaGRgY++D48Th2bOzjQ4fGs+nZZ+PEibn+K4VCbN5c+48uuSTWrFnUHzn702W9/HL09s7Pz9jcHGefHRFx4YVx7rkREeedF2d4xY6cAmjQVHrjjejri4h4/fWx02ezmDRqaZk0gNJiSC2bCZvR0XjttfH/TbfkXKIz2ZJNTbFhQ8T/nZFsaoqVK+2wyCmApRvsk5NuaS1Nc2gvFuMDH5gwtMfE9UZOY82xwNK1X/v3T6tik/mt9NfR3BwrV+ZPO4KcApj98JyMzcnAfMpgSlMpPeuUdlJjnXUql2N4ePx/s2ckU8k2mcxcTuE1N49//NOfxn//9/j/vvWt8e53T2im6ffr6OjYt/rpT8d+KW+8Ea+9Fv/+71NlVjIpuH59FAoaCzkFMHk2JaeTknmmqScz0rE5WSWdnnGr8yml7ORNupI9nVeLhl29vvjWrYv162Pr1rFfvalE5BRwekmuj0tiYv/+qQKi+uxP1Pf0UnrpX5JHs16tlVu6ns6xZT9TLVmBlDyGZB4rudhwOv96zYXkUXW3hV/8Is48M/93c3Nm2TqMmSyrSq8ZPOusOHkyzjwzfvnL6O6e7ka7+OLYtCl+//dj/fpYvdpiLOQUsIwMDIwNt1OXU0vLWDYlY2o9L5pJTsAlOZjeHGHv3lP/xfQGBGmvZG86MIvl2GmYHj16ivm8tJaSfzr9dxd5DXj1dZTpBjxldV17bUTE+efHmWdGuRw//3m8+mocPjxVnP3u78a118bGjfH+97uuEDkFNIjkTFY6utccIHPrYFatqt/TNEk2ZddsTd1MSS2lc2nzfo+AJJ56esYSpOaDyW7e5AE01kVzucsLTjm7VizGmjXx278dlUqsWBEvvBAvvRT9/TW+8txz42Mfi2uuiSuvtPQKOQXUWXD85CdjM08146lUirVrY8OGel/gkpZTMn5PsXw7aaZkjif5oRZuIi3ZvC++OGmbtrScRgu0q1fX1fw1pUF5zjnR3x+HDsXBg/mvOfPMKJXik5+MrVulFXIKWHQDA3H4cLzwQu3ZkTSe6nx0z55/nKyckqXuSTYls2iLsGZreDiOHo2DB6Onp8bmTeLp8stj7drT6K6YAwOnCNZyOU6ciL6+OHQoDh+uMY9VKsWGDfGOd8SxY3HoUPzwhxP+9N3vjo99LP7oj2LLFicEkVPAwkgmSJ55Jp57Lj9QJdMAyaVVF19cv/GUNEoy3H7/+zXiKVtOzc2LfXasXI7nn48nn8w/tkbZvKfc+Mna/OrFUjHnyxizb4aTvSPGyEiUy/Gzn9XO5aRKf/GLeOqpePHFCX/0oQ/Fn/xJfOYzlrEjp4C5GR2NV16Jvr7Yvz8/QZIO8HV+5VT2HFn1HE8ST0t7gX2aUPv2TTiFVyrFtm1x2WWN10/ZqzWTu6XP6H5X2ZvOT8eMbsJeqcSHPxzHj0e5HN/73oS/WCzG5s1x4kQ8/3wcPTr++dWr4/bbdRVyCpjhWPijH8WLL8Zjj+VnC1paGqCfkgSc4hxZc3OsXh1r1y7l4q3R0Th0KJ55Jh55ZEJnJFu4ubmRzt+d8vxatlwj8w7K6f0d5nEKMF29nns3oSmmvkql+PWvY2BgwqL1zZvjfe+L3t74wQ/GP/lbvxXf+EZs2uQ8IHIKmDxBHn00P7oXi3H99XH55XHFFXU9QZKehcw9/rSf6mSOZ3g4Dh6Mr31twtBeKsUtt0Rzc8PcfDLd2jXXzFVfrVlXlxMODMQbb0R3dxw6FK+/Ho8/nv+CCy6IX/86jh8f/3Guuiq6u8dv6/C2t8VNN8VXvmLdOnIK+L+hZf/+/CxUklBXXRWXXlrXZzeSNKl5juzKK+Oqq+roHFm5HA89FPffP556hULs3BlXXRUbNjTAVMcUE35JPNXDhN80H/+MTju+/e0xNDT28W/+Znzwg/Gv/xo/+9nYZ5qb4ytfiY98xIEEOQWnn/Q005494xXSKAk12Sxassxo06b6ukPj8HA89lh89avjD7VYjB074sYbG2Nio2ZtR4Oc853i8SenHZP5s5wjR+Kll+I//zN+8IPxkMo5++w4++w4cmTsfy+4IP7hH0QVcgpOD+lczr33ThgXP/3pej+Rlz74b31rwtRIfSZUors7vvGN8UdbLMYdd8S2bQ2wljlZObdv34Q5v0Ihtm+PLVsaYFeZrGKTVwszmrAcHo5jx2Lfvvinf4q+vvjf/81/werV4yvW3/ve+Pu/j40bHWmQU7AcJXNRd901/gI9Oc103XUN8FYbybj4zW/mH3zdniMbHo5vfjP+4i/GQqSB5qJqLupqbW2YhEoMDMQXvjBesaVS7NoVmzbNT8V+5zvxj/8Yjz4ao6MTPv+Od8TPfz728caN8eij1lQhp2AZ6e6OffvG56KSF+jXXRfr1zfe0F7/XZIbyFtb40//tAE2dXVtJ5t606YGePCTbf/kR7jlloWaC3zyyfjqV+PZZyd88i1vif/5n7GPv/71uPVWV/8hp6CR5SZI0rmohhgdqxOw/md3sgN5oRD33NMYJ/XK5dizZ3z9XLKf3Hxzw1xdmN3hd+wYD6n77lukM26/+lV88YvR3h6/+tX4J3/jN+LXv46IWLs2vv9901SMqQANpL+/0tJSiRj7r7W10tXVGI98aKjS0VEpFMYeeaFQaW+vDA420gYvFhtma3d1TdhP2toqvb2Nus93do7tNku4/e+8s/K2t41vz+x/nZ0OS1QqFTkFDaKrq1IsjrdIR0dlaKhhQqqtbUICNsTQ3rghle4nxWKls7MyMtKo+/zQ0PivoL19iX+QkZHK//t/tYuqrc3xCTkFjTZANsq4XqlUBgfHh8NkOqohEnBkpNLe3tghVSo10n5SU2/v+KRUf38dFV6ppKiQU9BQ+vvHj90tLXU0qExn1ElnpJK5tEaZI+nqGhvFk4fdKPtJmq2NtZ+k2Z3T2TmeKXW45zzwQI2ieuABRyw5BSz1EJ4bM0ZGKq2t4zMNdTtA9vbWmHBKF7vUc0gNDua3anabt7Y25ERagy6QSmYuU2mI1/ME2+BgZePGfFE17gI15BQ0vKGh/ILW9DRHoVDvC12LxUpr64SfpX4Wu0ytra1SKEwYHZOJwEKhTuO1szM/Wvf3j53dq//9ZGqlUqVYnNBSdftbyLXsbbdNyKmzz26AqyuQU7BsFYuVlpb8S/PW1gY4QZY82kR6mqyuFrtMpqurEjH2ONNH3tJSv9u8VKqUSuP/29ExfnavUS5KmEzys6QniAuFRoqSO++cUFRXXul4JqeAJY2SkZHx4aRRzhokUdLbOz661/mkVCqZFGxvH/sRko/rWXJSb2RkfD9ZNlfp9/dXIiq7djVeS2WfBel/Dbd2jfngNp5QB554Ij7+8bj22nj88SgUore3kd7x4z3vieuvj0ceiYjo6mqkdzRbsSJKpbF7hdf/I+/ri2IxDh6ML30pDhxosP1kOntRRDQ1xfe+F2efHU1NDXCX1NzNaX/v98Y+3r49HnrIUe108yabAJbeyZMREY8/HhdcEHfeGT/5SQwM5N81rJ498ki8612xZ0/813/FwEAMDzfMI09a6s4749ixGBiIcrl+H+r73x8R8dnPxoEDUSrFkSPL8H7c5XJ86EPxnvfE298eu3c30iPfuDG6usY+/va3G+nJy3y9OjM7BUvvwQfjppvyn2xvj9tua5h5haxSKZ5+uhGOfyvynykU4j/+o07fiC3d1KtXx+/8TnziExERzc2N944xNSvqy1+O970vNm2KVavijTeiry8++tHG68VnnolSKSKitTW+/nUHNjkFLK7h4Xjssbjwwrj00njrW+O11yIizjuvMd5g9cEH46yz4ooroqkpBgYiomHO1HR3x7FjY0VSLsfwcKxcWb9DeLkc55wz9nGxGL29jVSup48Pfzj+7d/ipZfi4ottDDkFQF3237nnTpiOGh1tjOZuHN+N79bJI/lEfMKvo4F4HgI0iOrF8loK6oOl6AAAcgoAQE4BAMgpAAA5BQCAnFrmyuXYvbuR7joNAMudi2wbTakUvb2xZk186lM2BgDUA7NTDaW7e+xWyB/9qI0BAHKKmbdU8o7lbW3L8K1PAUBOsbCeeGKspUql+MIXbA8AqB/WTjWCT30q9u4d+/jAgXjzm+f/n/DWjQAwW2an6l65PN5SAICcYsaamqKz02YAADnFHFxzTQwORqEQEVEoxOBgVCrz/B8AIKeWuaam6O2NQiFOnIhSyfYAADnFbIsqInp7o7vb9gAAOcWsiqqtzWYAADnFHHzxi9HbGxs22BIAUCfcd6oBrV9vGwBA/TA7BQAgpwAA5BQAgJwCAJBTAADIKQCAxbSi4v3aAADmwOwUAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcsgkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkPfmY0IAABKwSURBVFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFACAnLIJAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQCA5eUMmwCWn4GBgVN+zRtvvNHX1ze773/o0KFyuby0P+Mll1yyZs2a2f3d5ubmU37NypUrm5qa7EvAdKyoVCq2Aiyacrk8PDyc++Thw4dPnjxZ/cX79++f7JscOHDglP9WsVj8wAc+MJ1HtXXr1jn+XGedddbatWsXbrv19PTM8Tu8/vrr0/wmzz777IkTJ075ZS0tLTU/39TUtGHDhurPX3jhheeee271F69cudLzAuQUnKYldPz48WPHjk1dP3v37q3+PoVCYfPmzdNsmikyxfTJUpli8q9msU1WctPcPar7bP369atWrdJkIKdgyYyOjr722ms1x78jR468+uqrU8xSlEqlbMGccpyLiPPOO++MM5xVZ1qGh4dzZ1FPWe3Ve2luVrK5ufnss8+uWecXXXSRbQ5yCsZk543SPMpOCeTOjuUmALLTQrkzMl7307jPhSleLVSfL86eu0yfEdn88lwAOUVDyk4jpaNCugg6Nx6k80bZGaPsYODUGEwme/qy5guSl19+ube3t7q90qmv9LnmiYacgiU4gqfXl6XH7mwnZaeR0lfM6Qk1B25YwvZKT0Rmwyu7ICytruTJK7mQUzBjyUmHtJbSEw3pUo80ldIr3tMTbXV+WqHm1Xm5AWYKc7nLQM3Fy3OUWxA2I9l1OVN8zWR/ZGHZ6VBdaWklq76yr5fSZV7JjpT2lh0DOcVpJD0Zlxwuk0rIHiuT16ZpLaWzSou/ADZ3KVZ19NS89ip3gmP6FTLZpfI51evW52KKV/zTuQ3VjEx2Z4dTbtJpbtua9yCovvIxF2rmPBr0AJLuTklvpftG+ooria3k5ZbfMnKKxn6VmW2m9HiXJEVaD8nwtnBzS9lpoVwSZS93qnlbpur6qR6ea/aNF8qLqfrStmnmb80+y2ZZrnGz1xwYoev5yJP89tPfeDq9nTyjs6VlsTxyino5eCXn5pIjVxolyYR8Mhqls/HzNcOUFlL2Jt3Z02G568CzVZS7L3YuhlwEbuYj/d/cLeCzO9gUV31mCyybX3atOsnuXGklp8KT32Dyu5NZyCkW/EiUTLAn40pyGEqyKWmUuR+G0mmGdGohXT6VG8DSKYTs6JVtI/NDLH6EZQsse4uB7PK1NO7Tsrfauh5eFsb/zaYnk9bZ41sym9Xc3OwXhJxixsNDEjT79+9POyYZBrZu3Zoc/WdxZElrKbfuIZtK6WCTLlhOX+I7lrE8XpZEratTs9VVfclF8lLBU2CpjoRJHCfT3rnGMo+FnGL8kNHT05Mc1rPHi7ScZnS8SF7nJcGUm1HP1lLuKuh6OCTlzvVUm85i6qnlbqGems7FblOr+eZuWU42NdZTMptc6XnG3PMonZpNYsu4vjgGBgaSxsrO07e0tCS/jvXr1xcKBdWLnDotjgWHDx8+evRoGk+lUmnt2rUzOhBkVyEkiZDOMKUvrJNgWugra075png1bzEwxb0DJnt3vKy5v/vvfJnOe/TO9IfNLS+LqhVmTqQurXSKK/nVT/0ETC718CtbhN9IT09P8rtInnGlUunKK69cs2ZNc3Oz7Y+cWj7P80OHDn3/+9/v7e1NZ56mE0/JC+XkVXJypMhdoJdMq8zXi+PsJVrZqaApVgFH1duNVd9xoOZbAnspX/NWWNVxdsq3e8teCpedZstGmLmxxXy9lD59pnjOJpnl97IIgfXcc88dOHAgydzkwLt69WrziMipepeevEv7KXmRdPnll69du3bqo2d6mV71YvNsNs3ilVb1GvPJ7omcbaPs2Owa9XoevKcu4Ml+v+kkX3ovKKP7Qg/w2bNUSRknw3wyJelE1YI+U44fP/7iiy/29PTs3bs3W1fvf//7zV3JKeplmuH5559/4YUXHnnkkWn2U1Jd6Vm/5PVr8gyf3WUs2du9xJT3Mq6+oblx9LRq/eTj6ndUnOLe9xZiL+gwn51+zq2eTJ6kzlUtxEH7Jz/5yTPPPJPMXRWLxeuvv/7yyy+/4oor7ORyiiVIqCeffHLfvn0RkbzQmeIkffLsTS7WS+Ipd8ScTtBMcXPO3M2I62qNOQ03wEdmIXa6fj+d5UpOL7oZ44JWb3ptb/Xh4uKLLzbkz/s+n5xSSI7n27dv37Jli7SSUyxSQk39lEv6KTe33NzcfNlll50ynpJVNeldWGrenDPqcpXrKa/OU3jL4FkwPDxc82aM1XdBM60yj+N9eq4qvYRlmqsImPtxftOmTY5acoq56uvre/TRR5MTea2trZMl1Ojo6CuvvHLw4MHqc/NTrHxMl0wmI1P2rN+83yN46rtOx+TXqVUvf140Nd8J7pSr3RXbUg352Xv0p1On2YvYV61aZeyfu+Hh4aNHjybrLPft25fU1bZt2y677LJLL73Uzj+/afWtb31r7969xWJxx44dmzZtWr9+vS0jp5hBeSTHqXvvvTd5Fm3durV6GBgeHv7Rj370zDPPpKumpj6iZS/0y94xJV2IOovBJr1ALF2JnK2i6rdFywXKNN8Fb/EbZbL3AK6++1TublK5Hzl7M4LsPQisv160AWl4eDh3cxCnrhaorvbv35+8lnO6auFeVw8ODu7cufO6667TVXKKaVVUqVTatWtX9RxvOgt1//339/b2trS0THF5SPaUXzKQpPE0zfv5plNK2Zvc5GaM0kJKr8XLztacziuIJ3sHkvTuA9WbcdHeHPp05tTVQm/e3Eqg7du3m7Wax1cIDz300P3335901c6dOzWrnGLCAeiBBx7Ys2fPOeec85WvfKW6ooaHhw8ePHjKWd8ktrIvE9NTflNfAl39rnzpLEtuyVQ6e2TUmfdJlNyNsHP3Zsy+pbTMmt/Jldzt2ZKJ3k2bNrlF0Nx37NzpqpoT7cxu2+7ZsycZNe64445t27bZV+XU6SuZjrr11lsHBwf//M///MYbb8wVT1JRX/va1w4cONDa2lrzRV7yTaZ/yi+9S8LJkyezK82T2RE3U67P2o5ab9GavpHi9C/PZDp1dfTo0XQZYnIR+1VXXWV+ZY7HuldeecXpqoXQ19f3t3/7t/fee29bW5vJKjl1Oh5c9u3bd/vtt59zzjn33Xffxo0bqwvpG9/4xt69e5OK2rBhQ65sBgYG9u/f/9hjjx04cCBJqK1bt9YMoOpTfrlhWDY16C40xQXtbr48jy3b09OTTPcmT7Tqlz3MdFolPV1V82Uks3sZcM899+zZs2fz5s133323F1dy6jQKqZo7ffqUWLdu3a5du66++upc6CQLEvfs2RMRySu86lVTuZfX6Sm/GS28neyG1znpaqrq691y0sXXYd5rIcf+3M2XkxZvbm52eJ2v+ZXk9MqOHTt0wNx312SRw7p16+66667cq0rmOL7cf//9XlDJqWWru7v7hhtuqBlSyZHlzjvvrDlhmz1NXvM4njvl19LS0tzcPMXij+ybTqTX4uWuSstehVf9zrjTl7v2Lbv+Or1benKScdbvY8Nk0wDZO+YnN9pwD5u5S1/VWLYyLwWQrHmIiDvuuGP79u2e/nPfpHffffedd97Z3t7+Z3/2Z7annFpu+/fNN9/88ssvV5/aS2ekdu7cefvtt+eOy93d3XfddddLL71U8yKO7BL1Uql0yy231JyHyN4VPV0slX1L1CW5j3n6Xn7J7FdSddm1XOntgpy3mpetffDgwSeffDK9+4bJlXl5gZSclG9ra6t+8jKLjfnss88+/PDDZqrm5Sn/yU9+MiIefPBBz/R6VGHmkntjtra2joyM5P6oo6Mj+aOhoaHcH3V1dRWLxWKx2NnZWf0Xu7q6SqVSRLS2tnZ2dub++sjISG9vb3t7ezLDVCgUWlpa2tvbu7q6+vv7q79bvRkaGurv7+/o6Ghra0t/hOQnHRwctEfNRbJvtLW1FQqFUqnU1dVV//tD/e+ubW1tEdHW1mZjzlFy3CuVSp7p86K9vT0iurq6bIp6I6dmc3QoFArVe/PIyEhra2uxWOzv78/90eDgYEtLS7FYrP5bQ0ND7e3thUIh+dPcsXtwcLC9vT3JrFKp1N7e3tvbWx1qjWhwcLCzs7O1tTX52dva2qq3GzPtqiTKC4VCR0eHDTL3qEr2T0PX3PfM5ChnS87XGKSo5NSy3Y9HRkZKpVJLS0vNaaeI6OjomGwqq7okknGxWCwmf9rb27u8XyL39/dnm9JuNvftmcyDKtR52Zg6YF4kk/rt7e02xXyNRJ7gcqqBX61O0VJtbW3Tz6/BwcHJBrzkoJOctTkNjxHJeczlMQO3tDo7O3XAvBgcHExe2NgU87Il7ZPzoqOjo1Qq2Q71401Wj03fF77whba2tuo1lU899VRTU9MXv/jF6pWYN9xwQ39/f+6vlMvl5F6CP/7xj7PLzEdHR3fv3n311Vf/3d/93dNPP30aLt7cuHHj5z//+be85S1vf/vbk/uJj46Ofu5zn8u9uTLTcfTo0YceeuiGG27o7u5O1rF+7nOfS64VYEb27Nnz7LPP7tmzJ90nd+/enby1JTP1L//yLzfccEP2YGibzGJN+u7du7dv3z44OJg8u8vl8u7du22ZpSWnpqtcLif3/6j+o89//vOf/exnqz9/6623PvXUU7nr8kZHR4vF4s6dO6vz6+abb3711VePHDlyzTXXnLbb+dZbb33nO9+5bt26u+++e2Bg4Pzzz08uBrQHztSNN974pS996corr7z11lv7+vrWrFmzZcsWFwTNwnXXXbd58+YtW7Y8+uijfX19559//po1a1z0NwulUmn//v1NTU3d3d2jo6N/+Zd/mSwMZUZWrly5Zs2a888//1Of+tS+ffueeOKJYrF43XXX2TJLzATd9M9DtbS0TDZ9PdmSi5oLCIrF4mTfx2VEiQceeODMM88855xzXA00R62trRGxdu1aW3IuRkZGPvKRj5x11lnFYtGWnMtmbG9vv/TSS3fu3JlcW+OIN5czp5dccsm6detKpZLVEU72NZJjx45l7wOeevrpp3fu3Fn9+Z6enpqfP3jw4I4dO6o//9BDD+3cudP92RJ//Md//JnPfOY73/mO2ZQ5uueee9atW/c3f/M3tuRcnHHGGQ8//PCb3/zm++67z5acy2a87bbbPv7xj+/du/fKK6+87bbbHPFmramp6a//+q9feumlBx980FxpPXAbT4BpGR0dNfzPy2b8q7/6K3f3tk/KKQAAxp3uVbtixYrT5CedYzfbUHY5W3JZbkbsk/ZJOWXPWNQn+eOVx5f9hrp2xbWL8K/YkrZkXW1GEWCfrLd9Uk4B0HhEAMyRK/sAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFAFDvVlQqldP651+x4jT5Sef4i7ah7HK25LLcjLakLVmHW7IR/X9mTgOP6spLggAAAABJRU5ErkJggg==)
408 Chapter 12. 基于自注意力的模型 肖桐 朱靖波
对序列的处理十分缓慢。
w
1
w
2
w
3
...
w
m−1
w
m
信息传递
图 12.1 循环神经网络中单词之间的依赖关系
那么能否摆脱这种顺序传递信息的方式,直接对不同位置单词之间的关系进行
建模,即将信息传递的距离拉近为 1?自注意力机制的提出便有效解决了这个问题
[532]
。
图12.2给出了自注意力机制对序列进行建模的示例。对于单词 w
m
,自注意力机制直
接建立它与前 m −1 个单词之间的关系。也就是说,w
m
与序列中所有其他单词的距
离都是 1。这种方式很好地解决了长距离依赖问题,同时由于单词之间的联系都是相
互独立的,因此也大大提高了模型的并行度。
w
1
w
2
w
3
...
w
m−1
w
m
信息传递
图 12.2 自注意力机制中单词之间的依赖关系
自注意力机制也可以被看作是一个序列表示模型。比如,对于每个目标位置 j,
都生成一个与之对应的源语言句子表示,它的形式如下:
C
j
=
X
i
α
i,j
h
i
(12.1)
其中,h
i
为源语言句子每个位置的表示结果,α
i,j
是目标位置 j 对 h
i
的注意力权重。
以源语言句子为例,自注意力机制将序列中每个位置的表示 h
i
看作 query(查询),
并且将所有位置的表示看作 key(键)和 value (值)。自注意力模型通过计算当前
位置与所有位置的匹配程度,也就是在注意力机制中提到的注意力权重,来对各个
位置的 value 进行加权求和。得到的结果可以被看作是在这个句子中当前位置的抽
象表示。这个过程,可以叠加多次,形成多层注意力模型,对输入序列中各个位置进
行更深层的表示。
h(他) h(什么) h(也) h(没) h(学)h(他)
α
1
α
2
α
3
α
4
α
5
图 12.3 自注意力机制的计算实例
举个例子,如图12.3所示,一个汉语句子包含 5 个词。这里,用 h(他) 表示“他”
当前的表示结果,其中 h(·) 是一个函数,用于返回输入单词所在位置对应的表示结
果(向量)。如果把“他”看作目标,这时 query 就是 h(他),key 和 value 是图中所
有位置的表示,即:h(他)、h(什么)、h(也)、h(没)、h(学)。在自注意力模型中,首