End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning

Aleksander Colovic, Patrick Knöbelreiter, Alexander Shekhovtsov, Thomas Pock

Research output: Contribution to conferencePaperResearchpeer-review

Abstract

In this work we tackle the problem of semantic image segmentation with a combination of convolutional neural networks (CNNs) and conditional random fields (CRFs). The CRF takes contrast sensitive weights in a local neighborhood as input (pairwise interactions) to encourage consistency (smoothness) within the prediction and align our segmentation boundaries with visual edges. We model unary terms with a CNN which outperforms non data driven models. We approximate the CRF inference with a fixed number of iterations of a linear-programming relaxation based approach. We experiment with training the combined model end-to-end using a discriminative formulation (structured support vector machine) and applying stochastic subgradient descend to it.
Our proposed model achieves an intersection over union score of 62.4 in the test set of the cityscapes pixel-level semantic labeling task which is comparable to state-of-the-art models.
Original languageEnglish
Publication statusPublished - 6 Feb 2017
EventComputer Vision Winter Workshop - Retz, Retz, Austria
Duration: 6 Feb 20178 Feb 2017

Conference

ConferenceComputer Vision Winter Workshop
Abbreviated titleCVWW 2017
CountryAustria
CityRetz
Period6/02/178/02/17

Fingerprint

Semantics
Neural networks
Image segmentation
Linear programming
Labeling
Support vector machines
Pixels
Experiments

Cite this

Colovic, A., Knöbelreiter, P., Shekhovtsov, A., & Pock, T. (2017). End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning. Paper presented at Computer Vision Winter Workshop, Retz, Austria.

End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning. / Colovic, Aleksander; Knöbelreiter, Patrick; Shekhovtsov, Alexander; Pock, Thomas.

2017. Paper presented at Computer Vision Winter Workshop, Retz, Austria.

Research output: Contribution to conferencePaperResearchpeer-review

Colovic, A, Knöbelreiter, P, Shekhovtsov, A & Pock, T 2017, 'End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning' Paper presented at Computer Vision Winter Workshop, Retz, Austria, 6/02/17 - 8/02/17, .
Colovic A, Knöbelreiter P, Shekhovtsov A, Pock T. End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning. 2017. Paper presented at Computer Vision Winter Workshop, Retz, Austria.
Colovic, Aleksander ; Knöbelreiter, Patrick ; Shekhovtsov, Alexander ; Pock, Thomas. / End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning. Paper presented at Computer Vision Winter Workshop, Retz, Austria.
@conference{07d3f414446549bea9e20f2050200b14,
title = "End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning",
abstract = "In this work we tackle the problem of semantic image segmentation with a combination of convolutional neural networks (CNNs) and conditional random fields (CRFs). The CRF takes contrast sensitive weights in a local neighborhood as input (pairwise interactions) to encourage consistency (smoothness) within the prediction and align our segmentation boundaries with visual edges. We model unary terms with a CNN which outperforms non data driven models. We approximate the CRF inference with a fixed number of iterations of a linear-programming relaxation based approach. We experiment with training the combined model end-to-end using a discriminative formulation (structured support vector machine) and applying stochastic subgradient descend to it.Our proposed model achieves an intersection over union score of 62.4 in the test set of the cityscapes pixel-level semantic labeling task which is comparable to state-of-the-art models.",
author = "Aleksander Colovic and Patrick Kn{\"o}belreiter and Alexander Shekhovtsov and Thomas Pock",
year = "2017",
month = "2",
day = "6",
language = "English",
note = "Computer Vision Winter Workshop, CVWW 2017 ; Conference date: 06-02-2017 Through 08-02-2017",

}

TY - CONF

T1 - End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning

AU - Colovic, Aleksander

AU - Knöbelreiter, Patrick

AU - Shekhovtsov, Alexander

AU - Pock, Thomas

PY - 2017/2/6

Y1 - 2017/2/6

N2 - In this work we tackle the problem of semantic image segmentation with a combination of convolutional neural networks (CNNs) and conditional random fields (CRFs). The CRF takes contrast sensitive weights in a local neighborhood as input (pairwise interactions) to encourage consistency (smoothness) within the prediction and align our segmentation boundaries with visual edges. We model unary terms with a CNN which outperforms non data driven models. We approximate the CRF inference with a fixed number of iterations of a linear-programming relaxation based approach. We experiment with training the combined model end-to-end using a discriminative formulation (structured support vector machine) and applying stochastic subgradient descend to it.Our proposed model achieves an intersection over union score of 62.4 in the test set of the cityscapes pixel-level semantic labeling task which is comparable to state-of-the-art models.

AB - In this work we tackle the problem of semantic image segmentation with a combination of convolutional neural networks (CNNs) and conditional random fields (CRFs). The CRF takes contrast sensitive weights in a local neighborhood as input (pairwise interactions) to encourage consistency (smoothness) within the prediction and align our segmentation boundaries with visual edges. We model unary terms with a CNN which outperforms non data driven models. We approximate the CRF inference with a fixed number of iterations of a linear-programming relaxation based approach. We experiment with training the combined model end-to-end using a discriminative formulation (structured support vector machine) and applying stochastic subgradient descend to it.Our proposed model achieves an intersection over union score of 62.4 in the test set of the cityscapes pixel-level semantic labeling task which is comparable to state-of-the-art models.

M3 - Paper

ER -