S*ReLU: Learning Piecewise Linear Activation Functions via Particle Swarm Optimization

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

Abstract

Recently, it was shown that using a properly parametrized Leaky ReLU (LReLU) as activation function yields significantly better results for a variety of image classification tasks. However, such methods are not feasible in practice. Either the only parameter (i.e., the slope of the negative part) needs to be set manually (L*ReLU), or the approach is vulnerable due to the gradient-based optimization and, thus, highly dependent on a proper initialization (PReLU). In this paper, we exploit the benefits of piecewise linear functions, avoiding these problems. To this end, we propose a fully automatic approach to estimate the slope parameter for LReLU from the data. We realize this via Stochastic Optimization, namely Particle Swarm Optimization (PSO): S*ReLU. In this way, we can show that, compared to widely-used activation functions (including PReLU), we can obtain better results on seven different benchmark datasets, however, also drastically reducing the computational effort.

Original languageEnglish
Title of host publicationVISAPP
EditorsGiovanni Maria Farinella, Petia Radeva, Jose Braz, Kadi Bouatouch
Pages645-652
Number of pages8
ISBN (Electronic)9789897584886
Publication statusPublished - 2021
Event16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications: VISIGRAPP 2021 - Virtuell, Austria
Duration: 8 Feb 202110 Feb 2021

Publication series

NameVISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications
Volume5

Conference

Conference16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications
CountryAustria
CityVirtuell
Period8/02/2110/02/21

Keywords

  • Activation function
  • Deep Learning
  • Visual categorization
  • Activation functions
  • Particle swarm optimization

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'S*ReLU: Learning Piecewise Linear Activation Functions via Particle Swarm Optimization'. Together they form a unique fingerprint.

Cite this