Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks

Christian Payer, Darko Štern, Thomas Neff, Horst Bischof, Martin Urschler

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandForschungBegutachtung

Abstract

Different to semantic segmentation, instance segmentation assigns unique labels to each individual instance of the same class. In this work, we propose a novel recurrent fully convolutional network architecture for tracking such instance segmentations over time. The network architecture incorporates convolutional gated recurrent units (ConvGRU) into a stacked hourglass network to utilize temporal video information. Furthermore, we train the network with a novel embedding loss based on cosine similarities, such that the network predicts unique embeddings for every instance throughout videos. Afterwards, these embeddings are clustered among subsequent video frames to create the final tracked instance segmentations. We evaluate the recurrent hourglass network by segmenting left ventricles in MR videos of the heart, where it outperforms a network that does not incorporate video information. Furthermore, we show applicability of the cosine embedding loss for segmenting leaf instances on still images of plants. Finally, we evaluate the framework for instance segmentation and tracking on six datasets of the ISBI celltracking challenge, where it shows state-of-the-art performance.

Originalspracheenglisch
TitelMedical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings
Herausgeber (Verlag)Springer Verlag Heidelberg
Seiten3-11
Seitenumfang9
ISBN (Print)9783030009335
DOIs
PublikationsstatusVeröffentlicht - 16 Sep 2018
Veranstaltung21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018 - Granada, Spanien
Dauer: 16 Sep 201820 Sep 2018

Publikationsreihe

NameLecture Notes in Computer Science
Band11071
ISSN (Print)0302-9743
ISSN (elektronisch)1611-3349

Konferenz

Konferenz21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018
LandSpanien
OrtGranada
Zeitraum16/09/1820/09/18

Fingerprint

Recurrent Networks
Network architecture
Segmentation
Labels
Network Architecture
Semantics
Left Ventricle
Evaluate
Assign
Leaves
Predict
Unit

Schlagwörter

    ASJC Scopus subject areas

    • !!Theoretical Computer Science
    • !!Computer Science(all)

    Fields of Expertise

    • Information, Communication & Computing

    Kooperationen

    • BioTechMed-Graz

    Dies zitieren

    Payer, C., Štern, D., Neff, T., Bischof, H., & Urschler, M. (2018). Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings (S. 3-11). (Lecture Notes in Computer Science; Band 11071 ). Springer Verlag Heidelberg. https://doi.org/10.1007/978-3-030-00934-2_1

    Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. / Payer, Christian; Štern, Darko; Neff, Thomas; Bischof, Horst; Urschler, Martin.

    Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Springer Verlag Heidelberg, 2018. S. 3-11 (Lecture Notes in Computer Science; Band 11071 ).

    Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandForschungBegutachtung

    Payer, C, Štern, D, Neff, T, Bischof, H & Urschler, M 2018, Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Lecture Notes in Computer Science, Bd. 11071 , Springer Verlag Heidelberg, S. 3-11, Granada, Spanien, 16/09/18. https://doi.org/10.1007/978-3-030-00934-2_1
    Payer C, Štern D, Neff T, Bischof H, Urschler M. Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Springer Verlag Heidelberg. 2018. S. 3-11. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-030-00934-2_1
    Payer, Christian ; Štern, Darko ; Neff, Thomas ; Bischof, Horst ; Urschler, Martin. / Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Springer Verlag Heidelberg, 2018. S. 3-11 (Lecture Notes in Computer Science).
    @inproceedings{2f299da380de4b8096dc8db4204d9513,
    title = "Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks",
    abstract = "Different to semantic segmentation, instance segmentation assigns unique labels to each individual instance of the same class. In this work, we propose a novel recurrent fully convolutional network architecture for tracking such instance segmentations over time. The network architecture incorporates convolutional gated recurrent units (ConvGRU) into a stacked hourglass network to utilize temporal video information. Furthermore, we train the network with a novel embedding loss based on cosine similarities, such that the network predicts unique embeddings for every instance throughout videos. Afterwards, these embeddings are clustered among subsequent video frames to create the final tracked instance segmentations. We evaluate the recurrent hourglass network by segmenting left ventricles in MR videos of the heart, where it outperforms a network that does not incorporate video information. Furthermore, we show applicability of the cosine embedding loss for segmenting leaf instances on still images of plants. Finally, we evaluate the framework for instance segmentation and tracking on six datasets of the ISBI celltracking challenge, where it shows state-of-the-art performance.",
    keywords = "Cell, Embeddings, Instances, Recurrent, Segmentation, Tracking, Video",
    author = "Christian Payer and Darko Štern and Thomas Neff and Horst Bischof and Martin Urschler",
    year = "2018",
    month = "9",
    day = "16",
    doi = "10.1007/978-3-030-00934-2_1",
    language = "English",
    isbn = "9783030009335",
    series = "Lecture Notes in Computer Science",
    publisher = "Springer Verlag Heidelberg",
    pages = "3--11",
    booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings",

    }

    TY - GEN

    T1 - Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks

    AU - Payer, Christian

    AU - Štern, Darko

    AU - Neff, Thomas

    AU - Bischof, Horst

    AU - Urschler, Martin

    PY - 2018/9/16

    Y1 - 2018/9/16

    N2 - Different to semantic segmentation, instance segmentation assigns unique labels to each individual instance of the same class. In this work, we propose a novel recurrent fully convolutional network architecture for tracking such instance segmentations over time. The network architecture incorporates convolutional gated recurrent units (ConvGRU) into a stacked hourglass network to utilize temporal video information. Furthermore, we train the network with a novel embedding loss based on cosine similarities, such that the network predicts unique embeddings for every instance throughout videos. Afterwards, these embeddings are clustered among subsequent video frames to create the final tracked instance segmentations. We evaluate the recurrent hourglass network by segmenting left ventricles in MR videos of the heart, where it outperforms a network that does not incorporate video information. Furthermore, we show applicability of the cosine embedding loss for segmenting leaf instances on still images of plants. Finally, we evaluate the framework for instance segmentation and tracking on six datasets of the ISBI celltracking challenge, where it shows state-of-the-art performance.

    AB - Different to semantic segmentation, instance segmentation assigns unique labels to each individual instance of the same class. In this work, we propose a novel recurrent fully convolutional network architecture for tracking such instance segmentations over time. The network architecture incorporates convolutional gated recurrent units (ConvGRU) into a stacked hourglass network to utilize temporal video information. Furthermore, we train the network with a novel embedding loss based on cosine similarities, such that the network predicts unique embeddings for every instance throughout videos. Afterwards, these embeddings are clustered among subsequent video frames to create the final tracked instance segmentations. We evaluate the recurrent hourglass network by segmenting left ventricles in MR videos of the heart, where it outperforms a network that does not incorporate video information. Furthermore, we show applicability of the cosine embedding loss for segmenting leaf instances on still images of plants. Finally, we evaluate the framework for instance segmentation and tracking on six datasets of the ISBI celltracking challenge, where it shows state-of-the-art performance.

    KW - Cell

    KW - Embeddings

    KW - Instances

    KW - Recurrent

    KW - Segmentation

    KW - Tracking

    KW - Video

    UR - http://www.scopus.com/inward/record.url?scp=85054076332&partnerID=8YFLogxK

    U2 - 10.1007/978-3-030-00934-2_1

    DO - 10.1007/978-3-030-00934-2_1

    M3 - Conference contribution

    SN - 9783030009335

    T3 - Lecture Notes in Computer Science

    SP - 3

    EP - 11

    BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings

    PB - Springer Verlag Heidelberg

    ER -