Moving beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

Abstract

Performance comparisons of low-power wireless systems are often not substantiated by accurate and realistic evaluations, which raises the need of a proper benchmark. In a first attempt towards a rigorous comparison of protocol performance under the exact same settings, we have developed in 2016 a prototype benchmarking infrastructure called D-Cube, and used it to run the first of a series of competitions aiming to quantitatively assess the performance of low-power wireless protocols in specific scenarios. Given the success of the competition among both academia and industry, we have significantly extended the benchmarking infrastructure in the following two editions: D-Cube now also supports, among others, remote experimentation, multiple traffic patterns and loads, a custom description of how to derive performance metrics, and is further able to control the network density as well as the harshness of the RF environment. In this paper we perform a critical analysis of the current capabilities of D-Cube and argue that its main limiting factor is that the traffic patterns and node identities are manually embedded in the source code by developers and cannot be changed automatically. We show that we can overcome this limitation by utilizing a well-known data structure and by having developers describe its memory address using a configuration file that is passed to the benchmarking infrastructure. Following this concept, we extend D-Cube with the ability of building and applying patches to binary files and show that this allows not only to automatically change traffic patterns and node identities, but to also change user-defined protocol parameters. We believe that this extension is one of the last missing stepping stones to make D-Cube a full-fledged benchmarking infrastructure for low-power wireless systems.

LanguageEnglish
Title of host publicationProceedings - 2018 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018
PublisherInstitute of Electrical and Electronics Engineers
Pages30-35
Number of pages6
ISBN (Print)9781538667422
DOIs
StatusPublished - 7 Aug 2018
Event1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018 - Porto, Portugal
Duration: 10 Apr 2018 → …

Conference

Conference1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018
CountryPortugal
CityPorto
Period10/04/18 → …

Fingerprint

Benchmarking
Data structures
Data storage equipment
Industry

Keywords

  • benchmarking
  • competition
  • dependability
  • IoT
  • low power wireless
  • measurement
  • performance
  • testbeds

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Safety, Risk, Reliability and Quality

Fields of Expertise

  • Information, Communication & Computing

Cite this

Schuß, M., Boano, C. A., & Romer, K. (2018). Moving beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems. In Proceedings - 2018 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018 (pp. 30-35). [8429499] Institute of Electrical and Electronics Engineers. DOI: 10.1109/CPSBench.2018.00012

Moving beyond Competitions : Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems. / Schuß, Markus; Boano, Carlo Alberto; Romer, Kay.

Proceedings - 2018 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018. Institute of Electrical and Electronics Engineers, 2018. p. 30-35 8429499.

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

Schuß, M, Boano, CA & Romer, K 2018, Moving beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems. in Proceedings - 2018 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018., 8429499, Institute of Electrical and Electronics Engineers, pp. 30-35, 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018, Porto, Portugal, 10/04/18. DOI: 10.1109/CPSBench.2018.00012
Schuß M, Boano CA, Romer K. Moving beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems. In Proceedings - 2018 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018. Institute of Electrical and Electronics Engineers. 2018. p. 30-35. 8429499. Available from, DOI: 10.1109/CPSBench.2018.00012
Schuß, Markus ; Boano, Carlo Alberto ; Romer, Kay. / Moving beyond Competitions : Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems. Proceedings - 2018 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018. Institute of Electrical and Electronics Engineers, 2018. pp. 30-35
@inproceedings{7a64e9551cde4f099b3ce02e0c73022c,
title = "Moving beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems",
abstract = "Performance comparisons of low-power wireless systems are often not substantiated by accurate and realistic evaluations, which raises the need of a proper benchmark. In a first attempt towards a rigorous comparison of protocol performance under the exact same settings, we have developed in 2016 a prototype benchmarking infrastructure called D-Cube, and used it to run the first of a series of competitions aiming to quantitatively assess the performance of low-power wireless protocols in specific scenarios. Given the success of the competition among both academia and industry, we have significantly extended the benchmarking infrastructure in the following two editions: D-Cube now also supports, among others, remote experimentation, multiple traffic patterns and loads, a custom description of how to derive performance metrics, and is further able to control the network density as well as the harshness of the RF environment. In this paper we perform a critical analysis of the current capabilities of D-Cube and argue that its main limiting factor is that the traffic patterns and node identities are manually embedded in the source code by developers and cannot be changed automatically. We show that we can overcome this limitation by utilizing a well-known data structure and by having developers describe its memory address using a configuration file that is passed to the benchmarking infrastructure. Following this concept, we extend D-Cube with the ability of building and applying patches to binary files and show that this allows not only to automatically change traffic patterns and node identities, but to also change user-defined protocol parameters. We believe that this extension is one of the last missing stepping stones to make D-Cube a full-fledged benchmarking infrastructure for low-power wireless systems.",
keywords = "benchmarking, competition, dependability, IoT, low power wireless, measurement, performance, testbeds",
author = "Markus Schu{\ss} and Boano, {Carlo Alberto} and Kay Romer",
year = "2018",
month = "8",
day = "7",
doi = "10.1109/CPSBench.2018.00012",
language = "English",
isbn = "9781538667422",
pages = "30--35",
booktitle = "Proceedings - 2018 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018",
publisher = "Institute of Electrical and Electronics Engineers",
address = "United States",

}

TY - GEN

T1 - Moving beyond Competitions

T2 - Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems

AU - Schuß,Markus

AU - Boano,Carlo Alberto

AU - Romer,Kay

PY - 2018/8/7

Y1 - 2018/8/7

N2 - Performance comparisons of low-power wireless systems are often not substantiated by accurate and realistic evaluations, which raises the need of a proper benchmark. In a first attempt towards a rigorous comparison of protocol performance under the exact same settings, we have developed in 2016 a prototype benchmarking infrastructure called D-Cube, and used it to run the first of a series of competitions aiming to quantitatively assess the performance of low-power wireless protocols in specific scenarios. Given the success of the competition among both academia and industry, we have significantly extended the benchmarking infrastructure in the following two editions: D-Cube now also supports, among others, remote experimentation, multiple traffic patterns and loads, a custom description of how to derive performance metrics, and is further able to control the network density as well as the harshness of the RF environment. In this paper we perform a critical analysis of the current capabilities of D-Cube and argue that its main limiting factor is that the traffic patterns and node identities are manually embedded in the source code by developers and cannot be changed automatically. We show that we can overcome this limitation by utilizing a well-known data structure and by having developers describe its memory address using a configuration file that is passed to the benchmarking infrastructure. Following this concept, we extend D-Cube with the ability of building and applying patches to binary files and show that this allows not only to automatically change traffic patterns and node identities, but to also change user-defined protocol parameters. We believe that this extension is one of the last missing stepping stones to make D-Cube a full-fledged benchmarking infrastructure for low-power wireless systems.

AB - Performance comparisons of low-power wireless systems are often not substantiated by accurate and realistic evaluations, which raises the need of a proper benchmark. In a first attempt towards a rigorous comparison of protocol performance under the exact same settings, we have developed in 2016 a prototype benchmarking infrastructure called D-Cube, and used it to run the first of a series of competitions aiming to quantitatively assess the performance of low-power wireless protocols in specific scenarios. Given the success of the competition among both academia and industry, we have significantly extended the benchmarking infrastructure in the following two editions: D-Cube now also supports, among others, remote experimentation, multiple traffic patterns and loads, a custom description of how to derive performance metrics, and is further able to control the network density as well as the harshness of the RF environment. In this paper we perform a critical analysis of the current capabilities of D-Cube and argue that its main limiting factor is that the traffic patterns and node identities are manually embedded in the source code by developers and cannot be changed automatically. We show that we can overcome this limitation by utilizing a well-known data structure and by having developers describe its memory address using a configuration file that is passed to the benchmarking infrastructure. Following this concept, we extend D-Cube with the ability of building and applying patches to binary files and show that this allows not only to automatically change traffic patterns and node identities, but to also change user-defined protocol parameters. We believe that this extension is one of the last missing stepping stones to make D-Cube a full-fledged benchmarking infrastructure for low-power wireless systems.

KW - benchmarking

KW - competition

KW - dependability

KW - IoT

KW - low power wireless

KW - measurement

KW - performance

KW - testbeds

UR - http://www.scopus.com/inward/record.url?scp=85052538956&partnerID=8YFLogxK

U2 - 10.1109/CPSBench.2018.00012

DO - 10.1109/CPSBench.2018.00012

M3 - Conference contribution

SN - 9781538667422

SP - 30

EP - 35

BT - Proceedings - 2018 1st Workshop on Benchmarking Cyber-Physical Networks and Systems, CPSBench 2018

PB - Institute of Electrical and Electronics Engineers

ER -