Matches in SemOpenAlex for { <https://semopenalex.org/work/W3103622152> ?p ?o ?g. }
- W3103622152 abstract "We study the effect of mini-batching on the loss landscape of deep neural networks using spiked, field-dependent random matrix theory. We demonstrate that the magnitude of the extremal values of the batch Hessian are larger than those of the empirical Hessian. We also derive similar results for the Generalised Gauss-Newton matrix approximation of the Hessian. As a consequence of our theorems we derive an analytical expressions for the maximal learning rates as a function of batch size, informing practical optimisation schemes for both stochastic gradient descent (linear scaling) and adaptive algorithms such as Adam (square root scaling). Whilst the linear scaling for stochastic gradient descent has been derived under more restrictive conditions, which we generalise, the square root scaling rule for adaptive optimisers is, to our knowledge, completely novel. We validate our claims on the VGG/WideResNet architectures on the CIFAR-100 and ImageNet datasets." @default.
- W3103622152 created "2020-11-23" @default.
- W3103622152 creator A5038607215 @default.
- W3103622152 creator A5058617210 @default.
- W3103622152 creator A5090331439 @default.
- W3103622152 date "2020-06-16" @default.
- W3103622152 modified "2023-09-27" @default.
- W3103622152 title "Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training." @default.
- W3103622152 cites W112019301 @default.
- W3103622152 cites W1499021337 @default.
- W3103622152 cites W1516903196 @default.
- W3103622152 cites W1522301498 @default.
- W3103622152 cites W1598866093 @default.
- W3103622152 cites W1686810756 @default.
- W3103622152 cites W1873595945 @default.
- W3103622152 cites W1899249567 @default.
- W3103622152 cites W196761320 @default.
- W3103622152 cites W1970136394 @default.
- W3103622152 cites W1989821597 @default.
- W3103622152 cites W1990457366 @default.
- W3103622152 cites W2006903949 @default.
- W3103622152 cites W2017262500 @default.
- W3103622152 cites W2060581589 @default.
- W3103622152 cites W2067186137 @default.
- W3103622152 cites W2097428361 @default.
- W3103622152 cites W2099111195 @default.
- W3103622152 cites W2099551908 @default.
- W3103622152 cites W2106084579 @default.
- W3103622152 cites W2112269233 @default.
- W3103622152 cites W2124492078 @default.
- W3103622152 cites W2125426869 @default.
- W3103622152 cites W2145615204 @default.
- W3103622152 cites W2151695970 @default.
- W3103622152 cites W2187365590 @default.
- W3103622152 cites W2194775991 @default.
- W3103622152 cites W2287972981 @default.
- W3103622152 cites W2295465721 @default.
- W3103622152 cites W2296319761 @default.
- W3103622152 cites W2325424062 @default.
- W3103622152 cites W2401231614 @default.
- W3103622152 cites W2519766976 @default.
- W3103622152 cites W2523060838 @default.
- W3103622152 cites W2566079294 @default.
- W3103622152 cites W2601424732 @default.
- W3103622152 cites W2605372163 @default.
- W3103622152 cites W2610053348 @default.
- W3103622152 cites W2618381130 @default.
- W3103622152 cites W2622263826 @default.
- W3103622152 cites W2626325961 @default.
- W3103622152 cites W2729937365 @default.
- W3103622152 cites W2731468224 @default.
- W3103622152 cites W2741430497 @default.
- W3103622152 cites W2752366553 @default.
- W3103622152 cites W2764280570 @default.
- W3103622152 cites W2768267830 @default.
- W3103622152 cites W2777662428 @default.
- W3103622152 cites W2796196168 @default.
- W3103622152 cites W2889737445 @default.
- W3103622152 cites W2899771611 @default.
- W3103622152 cites W2900167092 @default.
- W3103622152 cites W2900531695 @default.
- W3103622152 cites W2900726521 @default.
- W3103622152 cites W2902280036 @default.
- W3103622152 cites W2904142648 @default.
- W3103622152 cites W2904243021 @default.
- W3103622152 cites W2947654433 @default.
- W3103622152 cites W2948330917 @default.
- W3103622152 cites W2949117887 @default.
- W3103622152 cites W2950541952 @default.
- W3103622152 cites W2950990113 @default.
- W3103622152 cites W2963069632 @default.
- W3103622152 cites W2963173418 @default.
- W3103622152 cites W2963190151 @default.
- W3103622152 cites W2963397933 @default.
- W3103622152 cites W2963433607 @default.
- W3103622152 cites W2963446085 @default.
- W3103622152 cites W2963586744 @default.
- W3103622152 cites W2963655672 @default.
- W3103622152 cites W2963702144 @default.
- W3103622152 cites W2963807877 @default.
- W3103622152 cites W2963975913 @default.
- W3103622152 cites W2964125128 @default.
- W3103622152 cites W2964309400 @default.
- W3103622152 cites W2965944281 @default.
- W3103622152 cites W2970154823 @default.
- W3103622152 cites W2970388773 @default.
- W3103622152 cites W2970550417 @default.
- W3103622152 cites W2980149079 @default.
- W3103622152 cites W3009686669 @default.
- W3103622152 cites W3028253330 @default.
- W3103622152 cites W3035443457 @default.
- W3103622152 cites W3086499488 @default.
- W3103622152 cites W3093329015 @default.
- W3103622152 cites W3104363895 @default.
- W3103622152 cites W3106405340 @default.
- W3103622152 cites W3116687755 @default.
- W3103622152 cites W3124979234 @default.
- W3103622152 cites W3140968660 @default.
- W3103622152 cites W3141595720 @default.
- W3103622152 cites W71081281 @default.