Matches in SemOpenAlex for { <https://semopenalex.org/work/W4317463669> ?p ?o ?g. }
Showing items 1 to 67 of
67
with 100 items per page.
- W4317463669 abstract "Activation functions play critical roles in neural networks, yet current off-the-shelf neural networks pay little attention to the specific choice of activation functions used. Here we show that data-aware customization of activation functions can result in striking reductions in neural network error. We first give a simple linear algebraic explanation of the role of activation functions in neural networks; then, through connection with the Diaconis-Shahshahani Approximation Theorem, we propose a set of criteria for good activation functions. As a case study, we consider regression tasks with a partially exchangeable target function, emph{i.e.} $f(u,v,w)=f(v,u,w)$ for $u,vin mathbb{R}^d$ and $win mathbb{R}^k$, and prove that for such a target function, using an even activation function in at least one of the layers guarantees that the prediction preserves partial exchangeability for best performance. Since even activation functions are seldom used in practice, we designed the ``seagull'' even activation function $log(1+x^2)$ according to our criteria. Empirical testing on over two dozen 9-25 dimensional examples with different local smoothness, curvature, and degree of exchangeability revealed that a simple substitution with the ``seagull'' activation function in an already-refined neural network can lead to an order-of-magnitude reduction in error. This improvement was most pronounced when the activation function substitution was applied to the layer in which the exchangeable variables are connected for the first time. While the improvement is greatest for low-dimensional data, experiments on the CIFAR10 image classification dataset showed that use of ``seagull'' can reduce error even for high-dimensional cases. These results collectively highlight the potential of customizing activation functions as a general approach to improve neural network performance." @default.
- W4317463669 created "2023-01-20" @default.
- W4317463669 creator A5048701309 @default.
- W4317463669 creator A5056218552 @default.
- W4317463669 date "2023-01-16" @default.
- W4317463669 modified "2023-10-01" @default.
- W4317463669 title "Data-aware customization of activation functions reduces neural network error" @default.
- W4317463669 doi "https://doi.org/10.48550/arxiv.2301.06635" @default.
- W4317463669 hasPublicationYear "2023" @default.
- W4317463669 type Work @default.
- W4317463669 citedByCount "0" @default.
- W4317463669 crossrefType "posted-content" @default.
- W4317463669 hasAuthorship W4317463669A5048701309 @default.
- W4317463669 hasAuthorship W4317463669A5056218552 @default.
- W4317463669 hasBestOaLocation W43174636691 @default.
- W4317463669 hasConcept C102634674 @default.
- W4317463669 hasConcept C111335779 @default.
- W4317463669 hasConcept C111472728 @default.
- W4317463669 hasConcept C11413529 @default.
- W4317463669 hasConcept C114614502 @default.
- W4317463669 hasConcept C118615104 @default.
- W4317463669 hasConcept C134306372 @default.
- W4317463669 hasConcept C138885662 @default.
- W4317463669 hasConcept C14036430 @default.
- W4317463669 hasConcept C154945302 @default.
- W4317463669 hasConcept C2524010 @default.
- W4317463669 hasConcept C2780586882 @default.
- W4317463669 hasConcept C33923547 @default.
- W4317463669 hasConcept C38365724 @default.
- W4317463669 hasConcept C41008148 @default.
- W4317463669 hasConcept C50644808 @default.
- W4317463669 hasConcept C78458016 @default.
- W4317463669 hasConcept C86803240 @default.
- W4317463669 hasConceptScore W4317463669C102634674 @default.
- W4317463669 hasConceptScore W4317463669C111335779 @default.
- W4317463669 hasConceptScore W4317463669C111472728 @default.
- W4317463669 hasConceptScore W4317463669C11413529 @default.
- W4317463669 hasConceptScore W4317463669C114614502 @default.
- W4317463669 hasConceptScore W4317463669C118615104 @default.
- W4317463669 hasConceptScore W4317463669C134306372 @default.
- W4317463669 hasConceptScore W4317463669C138885662 @default.
- W4317463669 hasConceptScore W4317463669C14036430 @default.
- W4317463669 hasConceptScore W4317463669C154945302 @default.
- W4317463669 hasConceptScore W4317463669C2524010 @default.
- W4317463669 hasConceptScore W4317463669C2780586882 @default.
- W4317463669 hasConceptScore W4317463669C33923547 @default.
- W4317463669 hasConceptScore W4317463669C38365724 @default.
- W4317463669 hasConceptScore W4317463669C41008148 @default.
- W4317463669 hasConceptScore W4317463669C50644808 @default.
- W4317463669 hasConceptScore W4317463669C78458016 @default.
- W4317463669 hasConceptScore W4317463669C86803240 @default.
- W4317463669 hasLocation W43174636691 @default.
- W4317463669 hasOpenAccess W4317463669 @default.
- W4317463669 hasPrimaryLocation W43174636691 @default.
- W4317463669 hasRelatedWork W1533959244 @default.
- W4317463669 hasRelatedWork W1887191277 @default.
- W4317463669 hasRelatedWork W1978042415 @default.
- W4317463669 hasRelatedWork W2046841137 @default.
- W4317463669 hasRelatedWork W2385264872 @default.
- W4317463669 hasRelatedWork W2920039561 @default.
- W4317463669 hasRelatedWork W3005627584 @default.
- W4317463669 hasRelatedWork W3086542228 @default.
- W4317463669 hasRelatedWork W4252911295 @default.
- W4317463669 hasRelatedWork W4287869072 @default.
- W4317463669 isParatext "false" @default.
- W4317463669 isRetracted "false" @default.
- W4317463669 workType "article" @default.