Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks

Justinas Mišeikis, Patrick Knöbelreiter, Inka Brijacak, Saeed Yahyanejad, Kyrre Glette, Ole Jakob Elle, Jim Torresen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Many works in collaborative robotics and human-robot interaction focuses on identifying and predicting human behaviour while considering the information about the robot itself as given. This can be the case when sensors and the robot are calibrated in relation to each other and often the reconfiguration of the system is not possible, or extra manual work is required. We present a deep learning based approach to remove the constraint of having the need for the robot and the vision sensor to be fixed and calibrated in relation to each other. The system learns the visual cues of the robot body and is able to localise it, as well as estimate the position of robot joints in 3D space by just using a 2D color image. The method uses a cascaded convolutional neural network, and we present the structure of the network, describe our own collected dataset, explain the network training and achieved results. A fully trained system shows promising results in providing an accurate mask of where the robot is located and a good estimate of its joints positions in 3D. The accuracy is not good enough for visual servoing applications yet, however, it can be sufficient for general safety and some collaborative tasks not requiring very high precision. The main benefit of our method is the possibility of the vision sensor to move freely. This allows it to be mounted on moving objects, for example, a body of the person or a mobile robot working in the same environment as the robots are operating in.

LanguageEnglish
Title of host publicationAIM 2018 - IEEE/ASME International Conference on Advanced Intelligent Mechatronics
PublisherInstitute of Electrical and Electronics Engineers
Pages181-187
Number of pages7
Volume2018-July
ISBN (Print)9781538618547
DOIs
StatusPublished - 30 Aug 2018
Event2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, AIM 2018 - Auckland, New Zealand
Duration: 9 Jul 201812 Jul 2018

Conference

Conference2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, AIM 2018
CountryNew Zealand
CityAuckland
Period9/07/1812/07/18

Fingerprint

Cameras
Robots
Neural networks
Sensors
Visual servoing
Human robot interaction
Mobile robots
Masks
Robotics
Color

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Control and Systems Engineering
  • Computer Science Applications
  • Software

Cite this

Mišeikis, J., Knöbelreiter, P., Brijacak, I., Yahyanejad, S., Glette, K., Elle, O. J., & Torresen, J. (2018). Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks. In AIM 2018 - IEEE/ASME International Conference on Advanced Intelligent Mechatronics (Vol. 2018-July, pp. 181-187). [8452236] Institute of Electrical and Electronics Engineers. DOI: 10.1109/AIM.2018.8452236

Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks. / Mišeikis, Justinas; Knöbelreiter, Patrick; Brijacak, Inka; Yahyanejad, Saeed; Glette, Kyrre; Elle, Ole Jakob; Torresen, Jim.

AIM 2018 - IEEE/ASME International Conference on Advanced Intelligent Mechatronics. Vol. 2018-July Institute of Electrical and Electronics Engineers, 2018. p. 181-187 8452236.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Mišeikis, J, Knöbelreiter, P, Brijacak, I, Yahyanejad, S, Glette, K, Elle, OJ & Torresen, J 2018, Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks. in AIM 2018 - IEEE/ASME International Conference on Advanced Intelligent Mechatronics. vol. 2018-July, 8452236, Institute of Electrical and Electronics Engineers, pp. 181-187, 2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, AIM 2018, Auckland, New Zealand, 9/07/18. DOI: 10.1109/AIM.2018.8452236
Mišeikis J, Knöbelreiter P, Brijacak I, Yahyanejad S, Glette K, Elle OJ et al. Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks. In AIM 2018 - IEEE/ASME International Conference on Advanced Intelligent Mechatronics. Vol. 2018-July. Institute of Electrical and Electronics Engineers. 2018. p. 181-187. 8452236. Available from, DOI: 10.1109/AIM.2018.8452236
Mišeikis, Justinas ; Knöbelreiter, Patrick ; Brijacak, Inka ; Yahyanejad, Saeed ; Glette, Kyrre ; Elle, Ole Jakob ; Torresen, Jim. / Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks. AIM 2018 - IEEE/ASME International Conference on Advanced Intelligent Mechatronics. Vol. 2018-July Institute of Electrical and Electronics Engineers, 2018. pp. 181-187
@inproceedings{c0cccf401c6444408b13d0b0a8b2c411,
title = "Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks",
abstract = "Many works in collaborative robotics and human-robot interaction focuses on identifying and predicting human behaviour while considering the information about the robot itself as given. This can be the case when sensors and the robot are calibrated in relation to each other and often the reconfiguration of the system is not possible, or extra manual work is required. We present a deep learning based approach to remove the constraint of having the need for the robot and the vision sensor to be fixed and calibrated in relation to each other. The system learns the visual cues of the robot body and is able to localise it, as well as estimate the position of robot joints in 3D space by just using a 2D color image. The method uses a cascaded convolutional neural network, and we present the structure of the network, describe our own collected dataset, explain the network training and achieved results. A fully trained system shows promising results in providing an accurate mask of where the robot is located and a good estimate of its joints positions in 3D. The accuracy is not good enough for visual servoing applications yet, however, it can be sufficient for general safety and some collaborative tasks not requiring very high precision. The main benefit of our method is the possibility of the vision sensor to move freely. This allows it to be mounted on moving objects, for example, a body of the person or a mobile robot working in the same environment as the robots are operating in.",
author = "Justinas Mišeikis and Patrick Kn{\"o}belreiter and Inka Brijacak and Saeed Yahyanejad and Kyrre Glette and Elle, {Ole Jakob} and Jim Torresen",
year = "2018",
month = "8",
day = "30",
doi = "10.1109/AIM.2018.8452236",
language = "English",
isbn = "9781538618547",
volume = "2018-July",
pages = "181--187",
booktitle = "AIM 2018 - IEEE/ASME International Conference on Advanced Intelligent Mechatronics",
publisher = "Institute of Electrical and Electronics Engineers",
address = "United States",

}

TY - GEN

T1 - Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks

AU - Mišeikis,Justinas

AU - Knöbelreiter,Patrick

AU - Brijacak,Inka

AU - Yahyanejad,Saeed

AU - Glette,Kyrre

AU - Elle,Ole Jakob

AU - Torresen,Jim

PY - 2018/8/30

Y1 - 2018/8/30

N2 - Many works in collaborative robotics and human-robot interaction focuses on identifying and predicting human behaviour while considering the information about the robot itself as given. This can be the case when sensors and the robot are calibrated in relation to each other and often the reconfiguration of the system is not possible, or extra manual work is required. We present a deep learning based approach to remove the constraint of having the need for the robot and the vision sensor to be fixed and calibrated in relation to each other. The system learns the visual cues of the robot body and is able to localise it, as well as estimate the position of robot joints in 3D space by just using a 2D color image. The method uses a cascaded convolutional neural network, and we present the structure of the network, describe our own collected dataset, explain the network training and achieved results. A fully trained system shows promising results in providing an accurate mask of where the robot is located and a good estimate of its joints positions in 3D. The accuracy is not good enough for visual servoing applications yet, however, it can be sufficient for general safety and some collaborative tasks not requiring very high precision. The main benefit of our method is the possibility of the vision sensor to move freely. This allows it to be mounted on moving objects, for example, a body of the person or a mobile robot working in the same environment as the robots are operating in.

AB - Many works in collaborative robotics and human-robot interaction focuses on identifying and predicting human behaviour while considering the information about the robot itself as given. This can be the case when sensors and the robot are calibrated in relation to each other and often the reconfiguration of the system is not possible, or extra manual work is required. We present a deep learning based approach to remove the constraint of having the need for the robot and the vision sensor to be fixed and calibrated in relation to each other. The system learns the visual cues of the robot body and is able to localise it, as well as estimate the position of robot joints in 3D space by just using a 2D color image. The method uses a cascaded convolutional neural network, and we present the structure of the network, describe our own collected dataset, explain the network training and achieved results. A fully trained system shows promising results in providing an accurate mask of where the robot is located and a good estimate of its joints positions in 3D. The accuracy is not good enough for visual servoing applications yet, however, it can be sufficient for general safety and some collaborative tasks not requiring very high precision. The main benefit of our method is the possibility of the vision sensor to move freely. This allows it to be mounted on moving objects, for example, a body of the person or a mobile robot working in the same environment as the robots are operating in.

UR - http://www.scopus.com/inward/record.url?scp=85053932324&partnerID=8YFLogxK

U2 - 10.1109/AIM.2018.8452236

DO - 10.1109/AIM.2018.8452236

M3 - Conference contribution

SN - 9781538618547

VL - 2018-July

SP - 181

EP - 187

BT - AIM 2018 - IEEE/ASME International Conference on Advanced Intelligent Mechatronics

PB - Institute of Electrical and Electronics Engineers

ER -