S*ReLU: Learning Piecewise Linear Activation Functions via Particle Swarm Optimization

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem Konferenzband

Abstract

Recently, it was shown that using a properly parametrized Leaky ReLU (LReLU) as activation function yields significantly better results for a variety of image classification tasks. However, such methods are not feasible in practice. Either the only parameter (i.e., the slope of the negative part) needs to be set manually (L*ReLU), or the approach is vulnerable due to the gradient-based optimization and, thus, highly dependent on a proper initialization (PReLU). In this paper, we exploit the benefits of piecewise linear functions, avoiding these problems. To this end, we propose a fully automatic approach to estimate the slope parameter for LReLU from the data. We realize this via Stochastic Optimization, namely Particle Swarm Optimization (PSO): S*ReLU. In this way, we can show that, compared to widely-used activation functions (including PReLU), we can obtain better results on seven different benchmark datasets, however, also drastically reducing the computational effort.

Originalspracheenglisch
TitelVISAPP
Redakteure/-innenGiovanni Maria Farinella, Petia Radeva, Jose Braz, Kadi Bouatouch
Seiten645-652
Seitenumfang8
ISBN (elektronisch)9789897584886
PublikationsstatusVeröffentlicht - 2021
Veranstaltung16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications: VISIGRAPP 2021 - Virtuell, Österreich
Dauer: 8 Feb 202110 Feb 2021

Publikationsreihe

NameVISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications
Band5

Konferenz

Konferenz16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications
LandÖsterreich
OrtVirtuell
Zeitraum8/02/2110/02/21

ASJC Scopus subject areas

  • !!Computer Vision and Pattern Recognition
  • !!Computer Science Applications
  • !!Computer Graphics and Computer-Aided Design

Fingerprint

Untersuchen Sie die Forschungsthemen von „S*ReLU: Learning Piecewise Linear Activation Functions via Particle Swarm Optimization“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren