![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxoAAAIWCAIAAABfl2zKAAAACXBIWXMAABYlAAAWJQFJUiTwAAAgAElEQVR42u3df3SUhZno8QkMq27klqpDsJgaazzVQNJlUVFTtGiIArHYLuJmQLxiWQUjlGYrRVgFCijbGykI6oqtpxFipdxVthhqQGFrA6bVcppARJuWYUMukJFKl8T+YCD3D/a6rT3XVvNr5p3P5z9yPDm+zzx58807M++EQgAAdEJGR0eHKQAAfGR9jAAAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAgJwCAEBOAQDIKQAAOQUAIKcAAJBTAAByCgBATgEAyCkAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFACAnAIAQE4BAMgpAAA5BQAgpwAAkFMAAHIKAEBOAQDIKQAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByCgAAOQUAIKcAAOQUAICcAgBATgEAyCkAADkFACCnAACQUwAAcgoAQE4BAMgpAADkFACAnAIAkFMAdJdEIpFIJMwB5BQAH9GUKVPGjBmjqEBOAfBRLFq0KBQKbd26denSpYoK5BQAH0J7e/vo0aP37t1bWVl56ivDhw+Px+MmA90to6OjwxQAAiAWi7W0tBQWFoZCoYyMjI6OjoaGhlAolJ+fbzggpwD4kCf3DKd36Dme7AMAkFMAAHIKAEBOAQDIKQAAPqSwEQDpKRaLOcAUlZOTY4FJKt5JC6SdqqqqZcuWhUKhIUOGmEbK2bNnT2tra0VFRTQaNQ2ShKtTQHqJx+Pl5eUbNmw4dbtLUlEsFhs/fvz555/vQSRJuDoFpJdTH2l3//33G0VKq6qq2rRpU1VVlVEgpwB6WjQaXbp0qRffpLpEItGvXz+/wkgS3tkHQOoJh71YBTkFACCnAACQUwAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAD0skUgYAnIKoHdcfPHFlZWV5pDqampqSktLzYEkkdHR0WEKQPqIx+MFBQUbNmwoLCw0jRQVi8XGjx//6KOPehBJEmEjANJKJBKpqKiYMWNGKBQaMmSIgaScPXv2tLa2zps3T0uRPFydAtJULBYL6qFVVlYePXp05syZQT3AnJwcC4ycAqC71NbWzpgxo76+/oUXXhg7dqyBQA/wUnSA4KiqqpowYcLWrVtDodDy5ctnzZrlHXAgpwD4S9XW1jY1NTU3N0cikVAotHnz5osuuqimpsZkoLt5sg8giCf3DKd36DmuTgEAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAAiOsBEA6am9vT0ejwf4AGOxWCCPKzMzMxKJWGCSSkZHR4cpAGklFovdd99Xn3lmg1GkqKLrrlmwcElhYaFRIKcAekF7e/uZZ565rrJ8/OevyMw83UBSTiJxombLrjv+4ZGdO1/PyckxEOQUQE9btWrlz9/cumL5PxhFaj+Oqzf9/Jd9Vqx41CiQUwA9LVr6xbn3Xpufn2MUKa29/bdnDrjZrzCShHf2Aemmo3//M0wh1XmiFjkFACCnAACQUwAAcgoAQE4BwEeVSCTOPffcRCJhFMgpAPgoysvLKyoqwmGfmYacAoAPb9asWY2NjdFo1CjoXXIegFRVWVl59OjRjIyMU/8sKioaOXLk9OnTfUYyPczVKQBS1ZQpU4qKijo6Ojo6Oo4fP75mzZqzzjpr4MCBVVVVhoOcAoA/b8WKFXl5eafiKRwO5+TklJWVtba2Llu2rLq62nzoMZ7sAyCFVVRUZGdnjx8/PjMz89RXIpHIxo0bL7jggtbWVs/60TNcnQIgla8KhMMHDx58r6VOycnJWbhw4ZYtW8wHOQUAH9Gll166adMmc0BOAcBHlJeXZwjIKQAAOQUAIKcAAiejpeWIKaS6ePzXWVnetYecAugNk2+9Y9Vjm80h1T32ePWMGTPMgSThvlNAeikuvn7cuJKBkf5fmlrcv/8ZBpJyjh37zXPP73xgUVVbW5tpIKcAeuOsFw63tbVt3Pj8pNuWNjQ0GkjKycqKzJ8/v63tiffdawrkFEDPyczMjEYnRaOTgnqAo0eP3r17d3NzczjsJA9yCqCrnDwZam8P/FEmTpxY+o1vjLz88q1bt94wevQPnnsu3Ldv8B/cM84ICUfkFED3amoKvfpq6N13g32U8WPHipYvnzZy5F1XX/1AKDT1wguzzz9/55w5OeecE/yHuLAwNGSITUdOAXSPkydDL7+cDgfa/rvfbZwx4714il5++ehLLnnr8OG0yKna2tCFF4ZOP92+I6cAuqMy2tPkQP80myL9+0f690+XB/r4cTlFb3HfKQCATnF1CkhHtU1ND//w1RMZgT0Hjrvs6pue+m4gD61vR+L2y/7mhrxL0uJV9sgpgCSUOHFi8Nfuy/z4eXdOf/zsswcbSMo5cqSl4oUVU59eu+/rCzNPO81AkFMAPa1u377BOcMWLn7ZKFJUVlZOXl7hE0/MenLHq7NGXWMgJAOvnQLSy8M/fPWW6NfNIdWNHz97yWafvYicAugNJzLCnuMLgKysnPiv3zEH5BQAgJwCAJBTRgAAIKcAAOQUAICcAgCQUwAAyCkAADkFACCnAADkFAAAcgoAQE4BAMgpAAA5BQCAnAIAkFMAAHIKAEBOAQAgpwAA5BQAgJwCAJBTAAByCiAd9O1IHDnSYg6p7vDhWORjHzcH5BRAL/jK1Vc8W/VP5pDqtm2rnDdmjDmQJMJGAKSVERdc0BJbM+ue/DunP3722YMNJOUcOdLygxdWbH/le0+sXGkayCmA3jjr9e3b8tDSun37Hn5u7omMYJ4Df3f896FQ6LR+fxXIo8voOFl+WcGW0kfDffvaZ+QUQK8VVWFubmFubiCPLnHixPAlS1qPHWtavDjztNM83NADvHYKIFAtNWblymkjRx76z/+86dFH48eOmQnIKYBOy8xMkwONvf129te+dnthYdmoUaFQaMGNNxYtX97QkjZvY+zXz7LTWzzZBwT+z8Y+oWuvDb38cuAPNNK//x8+wVeYm7tjzpx3f//7tHiUL700dPrplh05BdBtcnNDn/pUqL092Ef5R1fh7rwzVFqa+b4vBtUZZ4TCfp0hpwC6W58+of790+uQ0+14/1hLi5u10oMnGCMAIHh27dpVUlJiDvSMjI6ODlMACNrJPSOtT+/xeHzgwIGtra2RSMQy0ANcnQIgUNrb26PR6Lp167QUcgoAPpx4PF5VVZWbmzt+/PhoNGog9BgvRYcuVl1d/eKLL27fvr2+vt40PtigQYNGjRpVUlIyceLEsHdmdbWMjIz03KidO3fm5ORYAHr0x81rp6AL/zKORqOtra1z5sy56qqrzjvvPInwwWKxWGNj49q1a7dt21ZTU5Ofn28mXdhSTu8gpyDFJBKJ7Ozs6dOn33fffSrqw6qurh43bty+fftcVJBTIKcgfc2aNevU6zaM4qOpqqpatmzZ66+/LkblFMgpSEeJRKJfv35tbW2ZafPxcN3hM5/5zNq1az3lJ6cg5XhnH3SBAwcOFBQUaKlOmjZtWkNDgzkAcgrS0Y4dOz73uc+ZQycNGzZs06ZN5gDIKUhTI0aMMIROGjx4sCEAcgoAQE4BACCnAAB6jPu70EPi8fhjj6164429gTy69vb2UCj0/e9vDOTRRSKRadPuDNj9C4K9kKFQ6Oabv1haektQF/IrX/lHd3xFTpF2Sktv2bbtpRl3Xn/j2GzTSDm/+lVbcfHnIpFBO3f+OBg3g4hGb9627eXp/3CDhUzRhbzyyuGRyKCf/vRnbvqKnCJdNDQ0NO557UDsW+FwX9NIUWV3l0Qnf2Pjxuej0Umpfiy1tbV7dv+0ed+3LWSqL+T69c8GYCEJBq+dots9+eSjc776Bb+6Ut3cOTc/9NDSABzI+vVrLWQA3D19XDAWEjkFf5F466GrrrzEHFJdfn5OQ0OjhSRJFBbmBWMhkVMAAMgpAAA5BQAgpwAA5BT8saqqKkMAekB1dXUikTAH5BQBVFdXp6iA7tbQ0LB8+XL386R32T+6S0VFRXZ2dsA+mQRIKvF4vLi4uL6+3iiQUwRHIpE4cODAjh07Tv3zgQceKCkpOW/wQJMBusPFF1987733btmy5dQ/8/PzP/WpTwXjo5BILZ7so8tCatWqVf369Zs2bVpTU5OBAD3sV7/61YMPPnjmmWfOmjXLS6noYa5O0TUtNWbMmFAo1NraGolE3vtidnZ2TU3Ng0vvNyKgO+zdu7egoKC+vv7UmaesrKyysrK8vDw7O7upqcllKnqMq1N0gfLy8kgksmXLlvda6tQXKyoqvHYK6D6RSKSmpiYajb53OSocDq9YsWLevHnTpk0zH+QUKSMej69fv37NmjXv+/qIESOi0aj5AN0qPz9/9uzZ7/viXXfdtWfPnurqavNBTpEafvKTn0ycOPFPL6prKaBnjB079n03SgiHw3PmzHnttdcMBzlFanjttddGjBhhDkBSyc/P37t3rzkgp0gNe/fuveqqq8wBSCr9+/c3BOQUAICcAgIkkThhCFhIkFP0josvKah8+mVzSHV1dW8WXXdNIBZy6KYXfuIBtZAgp0glU6bc/tgTP4jHf20Uqau9/bfT73l8wcLFgVjIO5Y89D0LGYiFXGIUJAl3Rafb5eTkzJv3QP6w2flDz4uc48WhqSce//XWl3+2cOH8ESOuCMZCTp9+T/6wL+cPzbaQKb2QhYWFpoGcIo2Ulc28664ZdXV1+/fvD+QBvvXWW7FYrLi4OJBHN2DAgOf/7ZogfV7H/fcvuO+++RYydRdy84vF77vRFMgp0mPVwuHCwsJA/jUZj8fLy8sPHTo0bdo0fy5byORZyPLycp/yBHKKVJZIhGKx0H/8R+AP9IW6uvuefPLFr3/9M9Om/d2NNz48fXr0uuuC//gOGBAqKAi5PJB8qqur586dW1NTU1BQUFxcXFFR4fMJQE6Rsl58MdTSEvxfXQ0N6+rqdnz5y5knT4ZCoQNLlpRv2NDU0HB/SUnwH+LGxlA0Gurj7SzJ1VJr167dsWPHqWdmm5uby8vLm5qa7r//fsMBOUWq+e1v06GlQqHQ2Pz8sX/wZEq4b98Vt9ySLo/yu++G2ttDbjydVAs5duzYsWP/eyHD4RUrVhgL9AB/WdINjh83AwDSh6tT9JzY2287wBSVc845AXy8YrGAL2RwDzAnJ8cZFTlF2tnUsPve579/qO3d/ILAvkz76qtLb3txdyAP7cD+nx07erBiwoRbL780GEdU/cILX5tzb8aJ3w658LygLmTpuKvvK7stkIe25xcHDr9z7OGKh6OTJju7IqdIF+2/+92Nqx556KEf5eW5iUCqOnw4tmDxuE+dNaAwNzfVjyUej0+detv//uZXC4fneWRTVMObscn3PnB+zgVuTUKS8Noput0/b3kpGl2opVJaVlbO5Nu+8fAPXw3AsTz26OoZt1yvpVJa/qdz5nzpC6sf+aZRIKdIFz+L/+eoUVPMIdUNG1b8r3WvBOBA9u7+2ZSbRnlAU93EMZ995tkN5oCcAlJJ375eG0ASCYf7GgJyCgBATgEAIKcAAOQUAICcAgCQUwAAcgqgN2VmZm7bts0cADkF8BFFIpFDhw61t7cbBSCnAD6iRx55ZNq0aV3+bWfNmtXQ0GC8gJwCgu+uu+7as2fPokWLEolEF37b+fPnFxcXd+33BJBTQDIKh8Nbt2595ZVXhg8fXltbG4/Hu+TbRiKRDRs2DB8+vKu+IcAfnbuMAEgqkUhky5Yt1dXVq1ev3rZt26FDh7rwmw8cOLB0wk2GDHQtV6eAZDR27NiqqqqDBw92dJEf/ehHBQUFra2tZgt0OVengOCLx+MTJkxobm4Oh530gK7n6hQQfIsXL66pqdFSQDdxcgGCb8WKFYYAdB9XpwAA5BQAgJwCAJBTAAByCgAAOQUAIKeApHPihM8PJokkEicMATlFGvlM5H9s21ZpDqlu166aL44YGYADuXhIQeXz2zygqa6mdlfpLRPMATlFuii7ZmRV1QONjbVGkboOH46t/c5Xv3L1FQE4lul3lz367IsNb8Y8rKkrduDw3IfX3n3Pl42CJOGu6HS7SP/+T99xx9JVk3/X96/PO/8zBpJymvc3HDv6f+aPuaEwNzcICxmJPLx8xaS5SzNO/HbIhed5fFPO7qbm1qNt8+fNLywsNA3kFGlk8uWXT7788tjbbwf1ALe9+WZ9S8usa68N6PENzTnnnCAdTzQ6KRqdFIvFgrqQGzdu3L9//8yZM4N6gDk5Oc6ryCnSVMB+Jb8n9vbb33zppfqWltGXXDI2P98D7Vdy76qtrf32t79dX19fVFQ0duxYDzT0AK+dohv065c+x1r14x9fuWzZ1tmzQ6HQ8q1bZz37bOKENxzRewtZVTVhwoStW7eGQqG5c+cuWrQokfCWTJBTpKLTTw8NHpwOB1rb1FS3b1/zQw9F+vcPhUKbZ868aODAx3/4w7R4lP/6r0OZmZY9uRaytraurq65uTkSiYRCoddff/2ss85av369yUB382Qf3eP660Nvvhk6fDjYR1mYm1t4ww3//eP06U+XffrTafH4DhgQyssL9fH3WJItZGHhH746OxwOl5WVGQvIKVJAJBJpbGx8/8tQwuHQkCGhIUPSaxaBfSk6AB/EH5d01ogRI1588UVzAJLKjh07Lr74YnNATpEaRo8evX79ei93BZLKU089demll5oDcorUEIlEJk6cWF5ebhRAkqiqqgqFQu4TQY/J6OjoMAU6KZFIZGdnjxo1as2aNZnp+m6vjAw/TVjIpDgdPf7440uWLKmvrz/1DkfoAV6KTlesUTjc3NxcXl6em5s7atSokpISM4Fed+oKTVqpq6vbvn37kCFDmpqaMt3Ig578A8bf03SheDz+1ltv7d+/Pw2PfdKkSX6aSKKTe0bGunXr0u2oBwwYkJeX5yNokFOQwr+9/DRhISE9eSk6AICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIASA8+ZIae097eHo/HA3yAsVgskMeVmZkZyM8+s5AWErqK2+bSQ6f15cv/eeXKx4wiRRVdd82ChUsKCwsDtJDLVq583CObokpL/+7uu2cHZiEJAFen6IlrABdccMG6yvK2o9/LzDzdQFJOInGiZsuuCRM+v3Pn6wH4NDQLaSGhy/VdsGCBKdCtNmxYf94nEvf+49/91V/J95TUp0+fiy76xCfO/fiGf90+Zsy4VD+cJ554/KIL+1nIVF/I00/r9+qP37zmmmsNhGTgyT66XbT0i3PvvTY/3x+Rqa29/bdnDrg5AGcMC2khoesr3wjofh39+59hCqkuQM+LWUgLCXIKAEBOAQDIKQAA5BQAgJwiaZ177rnBvuU0kAwSicS5557b3t5uFMgpAmjDhg3RaNQcgG5VXl5eUVGRmZlpFMgpAqiwsHDkyJGjR48+fvy4aQDdYdasWY2Njf5yo9e5KTBdpr29/amnntqxY8czzzzzh1//5CcHh0JfMB+gy1VWVh49ejQjI+PUP4uKikaOHDl9+nSfkUwPc3WKrlFbW5ubm/vzn/987ty5bW1tHR0dHR0dCxcuLCoquvyyvzEfoDtMmTKlqKjo1Ann+PHja9asOeusswYOHFhVVWU4yClSTCwWmzBhQk1NzYoVK/Lz80+9iKGhoeGVV17ZsmVLv379jAjoDitWrMjLyzsVT+FwOCcnp6ysrLW1tby8vLq62nzoMZ7so7MSicSVV175rW99Kz8//w+/XlxcXF9fbz5At6qoqMjOzp44cWI4/F+/0SKRyM6dO6+88sr6+nrP+tEzXJ2is954442hQ4eOHTv2fV8/ePCgExnQ7VcFwuGDBw++11Kn5OTkTJ8+fcuWLeaDnCI1NDQ0jB8/3hyApHLppZdu2rTJHJBTpIZNmzYNGzbMHICkkpeXZwjIKVLJ4MGDDQEAOQUAgJwiOWX0bWk5YgypLh7/dVZWMN5bkGEhLSTIKVLM5Mm3L/j6d80h1T32ePWMGdODsJC33rHqsc0e0EAs5AxzIEm47xTdrrj4+qlTb581+4kvTS3Oz88xkJQTix2ufPrlBxZVtba2BmMhx40rGRjpbyEtJMgpUmfJwuFf/GLfxo3PT7ptaUNDo4GknKysyIwZM9ra2k7d7z4AC9nW1rZx43OTbnvQQqboQs6fP7+t7YlgLCRyCv5SmZmZ0eikaHRSUA9w9OjRu3fvbm5uft+9BEnihZwcjU62kECX8Nop6JREIrFo0aK8vLxDhw6NGTMmkUiYCUmykFOmTLGQIKcg2cXj8eHDh5911lkVFRWhUOj222/Pzs6OxWImQzIsZElJiYWEnuE6MHx07e3ta9eufe+zn6PR6OjRo996662cnBzDoVcWcuPGje+tn4UEOQUp4E9/S0UiER/8jIWEdOPJPgCATnF1ih5SW1u7vqoyHtz7xEy4aVz05i8E9M+uPpOnTC2+/vogvU2strZ29cqHQydPWkgLCXKK1HDuoKzIgMyHym/Nu/Ay00g5LYePrPqX/zX1jtt/8Yt9AbjTTyKRyD5v8P9byGyPbyou5IIH/ykwC4mcgr/0MkBkQGb9v60wihSVc15W4fC8Rau++9S3v1V2z8xUP5y6ujoLmeoLueXbC2ctfiIYC0kweO0U3W71yocfKr/VHFLdlJtGLV6yOBgL+dgDd3pAU93s/zl+yZIl5oCcIm2cPOkplWBcEjh8OB6MhRycdbYHNAALeeiwz+xDTgEAyCkAAOQUAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQDIKQAAOQUAgJwCAJBTAAByCgBATgEAyCnovi3r03L4iDGkutiBw1lZEQtJ8izkoKyB5oCcIl3cPfMr0xf+izmkusrnt82fNz8IC3nP7AWrn/WABmAh582bZw4kibAR0N1GjBgRP9pe8PlZjz1w5+Cssw0k5bQcPrJ6XfUzL/ywre2JICzkFVfsbmoZPfWBBXffYiFTdCFXPfOD735/ezAWEjkFf9mShcPNB1rq6upWr3w4dPJkII/xeOL4iRMnTz/ttGA+hBkZk6ffW/n8S+Fw2EJayGRYyFvv/Men/3VLMBYSOQUf4hdYYWFhYWFhII8ukUiMGTNm9+7dTU1NmZmZHm4L2esLOXz48NbWVgsJPcZrp6Czv7rKy8vHjx9/6NChm266KR6Pmwm9u5BTpkyZNm2ahQQ5BakhFotlZ2ePGDGirKwsFAotWLCgoKCgtrbWZOjFhSwpKTm1kLNnzy4qKmpoaDAZ6G6e7IOPLhKJ1NfXRyL/dfuAwsLCpqYm1wNIkoUcO3bsNddcYyFBTpEajh07lp4HnpmZ+b7XpvzpV8BCQuB5so/Ouuqqq/793//dHICk0tLSYgjIKVLGsGHDNm7caA5AUtm1a1dJSYk50DMyOjo6TIHOSCQS2dnZNTU1+fn5af2zlOGnCQuZLOLx+MCBA1tbW997JRl0K1en6KxwOFxTU1NcXByLxUwD6HXt7e3RaHTdunVaCjlFKsnPz6+oqLjgggtWrVolqoDeEo/Hq6urc3Nzx48fH41GDYQe4+kJuvJEtnjx4u3bt9fX16fnBPw0kUQn94yMNDzqQYMGjRo1aunSpTk5OXYAOQUp+dvLTxMWEtKTJ/sAAOQUAICcAgCQUwAAcgoAADkFACCnAADkFACAnAIA4M8KGwFdor29feNzz4VOngzcXxx9xn/hC5mZmV01pe3bt7/zzjvBW4DPfvazSfWxHhbSQvqcGeQUKWbmPfc8smrVwjvvzM3ODtihNTU3n3nrrfeUla185JEP/i//7Ad6rFu3bvLkyVOnTv3bv/3bgE3p6NGj5eXlWVlZP/3pT8Ph3j+rzCorW7l6tYVM84UcOnToSy+95PxMz/ChTnRWPB4vyMtr3rQpGX6PdodEIpFdUlLf2BiJRDrzTfr16/fqq69mB+4X/HsHeOutty5durSwsLB3/09isdiVl10W7IU878YbG/bs6fxC1tfXn3322UGd0g033LBmzZpeX0jSRN8FCxaYAp1R8Y1vXPbJT4667LKgHmCfPn3e/c1vXnvjjWs+97mP/E02b978m9/8prS0NMBTOv/88x966KHbbrutd/9PVn7zm9cPHXpFQUHAF3Lv3k4uZHt7+8033xzgKV1yySVPPvlkgI+R5Fo5I6CT3njjjSnjxgX7GKeMG/fGG2905ju888471113XbCnNGjQoJdffjkZFrJk5MjgL2RjYycXcvTo0YFfSOdn5BQAgJwCAJBTAADIKQAAOQUAIKcAAOQUAAByCgBATgEAyCkAADkFAICcAgCQUwAAcgoAQE4BAMgpAADkFDkQbWAAAAIZSURBVACAnAIAkFMAAHIKAAA5BQAgpwAA5BQAgJyC/5+BkUjjL38Z7GNs/OUvB0YinfkOGRkZ+/btC/aU2trasrKykuH/pKW11UL+WemwkM7PyClSxsTS0q+tXh3sY5z76KMTS0s78x2Ki4srKioSiUSAp7R58+Z58+b1+v9G2cyZ05cts5Af7Prrr3/66aeDvZBVVVWf//znnaLpGRkdHR2mQCedO2jQxFGjvnTTTfm5uQE7tIampieff379tm0HDx3q5Lf6+7//+9///ve33XbbZZddFrApHTly5Dvf+U5FRUVbW1tmZmaSLOTsaDTnE58I3kI++NRTuw8cqN+9u5Pf6rrrrvvYxz4WyIVsbm7+3ve+lzwLSVrogE5ra2tb9/TTg5LjiZ6uNSgra93TT7e1tX3wBBYuXFhUVPTB/83x48c3bdo0dOjQ4E0pKytrwYIFf3ZKFrLz8ocMsZApt5CkA1enoLNqa2sXLFiwdevWhQsX3n///QZCkizkI488UlZWZiDQA+QUfHSJRKK8vLyxsbGqqmrgwIGlpaWhUGjNmjWeX8BCgpwC/tLLALt27Tp1ASAjI6Ojo6O6uvro0aPRaNRw6JWF3L9//6n1s5AgpyAFf5Yy/DRhISFNuVECAICcAgCQUwAAcgoAQE4BACCnAADkFACAnAIAkFMAAMgpAAA5BQAgpwAA5BQAAHIKAEBOAQAklf8Lz7G4xQsrmo4AAAAASUVORK5CYII=)
424 Chapter 12. 基于自注意力的模型 肖桐 朱靖波
12.6 前馈全连接网络子层
在 Transformer 的结构中,每一个编码层或者解码层中都包含一个前馈神经网络,
它在模型中的位置如图12.16中红色方框所示。
Self-Attention
Add & LayerNorm
Feed Forward Network
Add & LayerNorm
Embedding
+
Position
编码器输入: 我 很 好
编码器
Self-Attention
Add & LayerNorm
Encoder-Decoder Attention
Add & LayerNorm
Feed Forward Network
Add & LayerNorm
Output layer
Embedding
+
Position
解码器输入: <sos> I am fine
解码器
解码器输出: I am fine <eos>
图 12.16 前馈神经网络在模型中的位置
Transformer 使用了全连接网络。全连接网络的作用主要体现在将经过注意力计
算之后的表示映射到新的空间中,新的空间会有利于接下来的非线性变换等操作。实
验证明,去掉全连接网络会对模型的性能造成很大影响。Transformer 的全连接前馈神
经网络包含两次线性变换和一次非线性变换(ReLU 激活函数:ReLU(x) = max(0,x)),
每层的前馈神经网络参数不共享,具体计算如下:
FFN(x) = max(0,xW
1
+ b
1
)W
2
+ b
2
(12.14)
其中,W
1
、W
2
、b
1
和 b
2
为模型的参数。通常情况下,前馈神经网络的隐藏层维度
要比注意力部分的隐藏层维度大,而且研究人员发现这种设置对 Transformer 是至关
重要的。比如,注意力部分的隐藏层维度为 512,前馈神经网络部分的隐藏层维度为
2048。当然,继续增大前馈神经网络的隐藏层大小,比如设为 4096,甚至 8192,还
可以带来性能的增益,但是前馈部分的存储消耗较大,需要更大规模 GPU 设备的支
持。因此在具体实现时,往往需要在翻译准确性和存储/速度之间找到一个平衡。