MBBench: A WCET Benchmark Suite

Metin Kuzhan; Veysel Harun Şahin

doi:10.35377/saucis.03.01.705777

TR EN

MBBench: A WCET Benchmark Suite

Abstract

One of the important features of any real-time software is the worst-case execution time (WCET). To get an understanding of the timing behavior of real-time systems and to prove that the real-time software meets its deadlines, WCET analysis is performed. Today, researchers actively develop new WCET analysis methods and tools. Therefore, they need benchmark programs to evaluate and compare their work. To meet this need, in this study we present a new benchmark suite, called MBBench. MBBench includes a collection of C programs for Linux operating system and RTEMS real-time operating system. Its main aim is to help the evaluation and comparison of measurement-based WCET analysis methods/tools. MBBench has been published as open source. It can be obtained freely over the Internet.

Keywords

Real-Time Systems,Benchmark,WCET Analysis,Software Engineering

Thanks

The authors would like to acknowledge that this work is supported by the Real-Time Systems Research Laboratory at Sakarya University, Faculty of Computer and Information Sciences, Department of Software Engineering. The MBBench benchmark suite can be obtained from the Real-Time Systems Research Laboratory homepage freely [18].

References

[1] G. C. Buttazzo, Hard Real-Time Computing Systems, 3rd ed., vol. 24. Boston, MA: Springer US, 2011.
[2] H. Kopetz, Real-Time Systems, 2nd ed. Boston, MA: Springer US, 2011.
[3] F. J. Cazorla, L. Kosmidis, E. Mezzetti, C. Hernandez, J. Abella, and T. Vardanega, “Probabilistic Worst-Case Timing Analysis: Taxonomy and Comprehensive Survey,” ACM Computing Surveys, vol. 52, no. 1, pp. 1–35, 2019, doi: 10.1145/3301283.
[4] R. I. Davis and L. Cucu-Grosjean, “A Survey of Probabilistic Timing Analysis Techniques for Real-Time Systems,” Leibniz Transactions on Embedded Systems (LITES), vol. 6, no. 1, pp. 3:1–3:60, 2019, doi: 10.4230/LITES-v006-i001-a003.
[5] J. Abella, D. Hardy, I. Puaut, E. Quinones, and F. J. Cazorla, “On the comparison of deterministic and probabilistic WCET estimation techniques,” Proceedings - Euromicro Conference on Real-Time Systems, pp. 266–275, 2014, doi: 10.1109/ECRTS.2014.16.
[6] J. Abella et al., “WCET analysis methods: Pitfalls and challenges on their trustworthiness,” in 10th IEEE International Symposium on Industrial Embedded Systems (SIES), Siegen,Germany, pp. 1–10, 2015, doi: 10.1109/SIES.2015.7185039.
[7] R. Wilhelm et al., “The worst-case execution-time problem—overview of methods and survey of tools,” Transactions on Embedded Computing Systems, vol. 7, no. 3, pp. 1–45, Apr. 2008, doi: 10.1145/1347375.1347389.
[8] L. Cucu-Grosjean et al., “Measurement-based probabilistic timing analysis for multi-path programs,” Proceedings - Euromicro Conference on Real-Time Systems, pp. 91–101, 2012, doi: 10.1109/ECRTS.2012.31.

[9] F. J. Cazorla et al., “PROXIMA: Improving Measurement-Based Timing Analysis through Randomisation and Probabilistic Analysis,” Proceedings - 19th Euromicro Conference on Digital System Design, DSD 2016, pp. 276–285, 2016, doi: 10.1109/DSD.2016.22.
[10] X. Li, Y. Liang, T. Mitra, and A. Roychoudhury, “Chronos: A timing analyzer for embedded software,” Science of Computer Programming, vol. 69, no. 1–3, pp. 56–67, 2007, doi: 10.1016/j.scico.2007.01.014.
[11] Université of Toulouse, “OTAWA - WCET is coming...,” 2020. [Online]. Available: http://otawa.fr. [Accessed: 17-Mar-2019].
[12] D. Hardy, B. Rouxel, and I. Puaut, “The Heptane Static Worst-Case Execution Time Estimation Tool,” in 17th International Workshop on Worst-Case Execution Time Analysis (WCET 2017), Dagstuhl, Germany, vol. 57, pp. 8:1—-8:12, 2017, doi: 10.4230/OASIcs.WCET.2017.8.
[13] Inria, “Heptane static WCET estimation tool - PACAP,” 2020. [Online]. Available: https://team.inria.fr/pacap/software/heptane. [Accessed: 17-Mar-2019].
[14] Rapita Systems, “Rapita Systems | On-target software verification solutions,” 2020. [Online]. Available: https://www.rapitasystems.com. [Accessed: 17-Mar-2020].
[15] AbsInt, “AbsInt: Cutting-Edge Tools for Static Analysis of Safety-Critical Software,” 2020. [Online]. Available: https://www.absint.com. [Accessed: 17-Mar-2020].
[16] Linux Kernel Organization, “The Linux Kernel Archives,” 2020. [Online]. Available: https://www.kernel.org. [Accessed: 17-Mar-2020].
[17] The RTEMS Project, “RTEMS Real Time Operating System (RTOS),” 2020. [Online]. Available: https://www.rtems.org. [Accessed: 17-Mar-2020].
[18] Sakarya University, Faculty of Computer and Information Sciences, Department of Software Engineering, “Real-Time Systems Research Laboratory homepage,” 2020. [Online]. Available: http://rtsrlab.sakarya.edu.tr. [Accessed: 17-Mar-2020].
[19] Standard Performance Evaluation Corporation, “Spec - Standard Performance Evaluation Corporation,” 2020. [Online]. Available: https://www.spec.org. [Accessed: 17-Mar-2020].
[20] EEMBC, “Embedded Microprocessor Benchmark Consortium,” 2020. [Online]. Available: https://www.eembc.org. [Accessed: 17-Mar-2020].
[21] C. Bienia, S. Kumar, J. P. Singh, and K. Li, “The PARSEC benchmark suite,” in Proceedings of the 17th international conference on Parallel architectures and compilation techniques - PACT ’08, New York, New York, USA, 2008, p. 72, doi: 10.1145/1454115.1454128.
[22] X. Tian et al., “BigDataBench-S: An Open-Source Scientific Big Data Benchmark Suite,” presented at the 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Lake Buena Vista, FL, USA, 2017, pp. 1068–1077, doi: 10.1109/IPDPSW.2017.111.
[23] S. M. Blackburn et al., “The DaCapo benchmarks,” in Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications - OOPSLA ’06, New York, New York, USA, 2006, p. 169, doi: 10.1145/1167473.1167488.
[24] T. Kalibera, P. Parizek, G. Haddad, G. T. Leavens, and J. Vitek, “Challenge benchmarks for verification of real-time programs,” in Proceedings of the 4th ACM SIGPLAN workshop on Programming languages meets program verification - PLPV ’10, New York, New York, USA, 2010, p. 57, doi: 10.1145/1707790.1707800.
[25] G. Bollella et al., The Real-Time Specification for Java. Addison-Wesley, 2000.
[26] J. Gustafsson, A. Betts, A. Ermedahl, and B. Lisper, “The Mälardalen WCET Benchmarks: Past, Present and Future,” in 10th International Workshop on Worst-Case Execution Time Analysis (WCET 2010), Dagstuhl, Germany, 2010, vol. 15, pp. 136–146, doi: 10.4230/OASIcs.WCET.2010.136.
[27] Mälardalen Real-Time Research Center, “The Mälardalen WCET Benchmarks,” 2013. [Online]. Available: http://www.mrtc.mdh.se/projects/wcet/benchmarks.html. [Accessed: 17-Mar-2020].
[28] F. Nemer, H. Cassé, P. Sainrat, J.-P. Bahsoun, and M. D. Michiel, “PapaBench: a Free Real-Time Benchmark,” in 6th International Workshop on Worst-Case Execution Time Analysis (WCET’06), Dagstuhl, Germany, 2006, vol. 4, doi: 10.4230/OASIcs.WCET.2006.678.
[29] H. Falk et al., “TACLeBench: A Benchmark Collection to Support Worst-Case Execution Time Research,” in 16th International Workshop on Worst-Case Execution Time Analysis (WCET 2016), Dagstuhl, Germany, 2016, vol. 55, pp. 2:1—-2:10, doi: 10.4230/OASIcs.WCET.2016.2.
[30] S. Serttaş and V. H. Şahin, “PBench: A Parallel, Real-Time Benchmark Suite,” in Academic Perspective Procedia, Alanya, Antalya, Turkey, 2018, vol. 1, pp. 178–186, doi: 10.33793/acperpro.01.01.37.
[31] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Algoritmalara Giriş (Üçüncü Baskıdan Çeviri). Palme Yayıncılık, 2017.
[32] M. M. Mano, Bilgisayar Sistemleri Mimarisi (3. Basımdan Çeviri). Literatür Yayınları, 2015.
[33] Linux Foundation, “Zephyr Project homepage,” 2020. [Online]. Available: https://www.zephyrproject.org. [Accessed: 17-Mar-2020].

Details

Primary Language

English

Subjects

Software Engineering

Journal Section

Research Article

Authors

Metin Kuzhan ^*
0000-0001-6420-8327
Türkiye

Veysel Harun Şahin
0000-0002-3381-1702
Türkiye

Publication Date

April 30, 2020

Submission Date

March 18, 2020

Acceptance Date

April 21, 2020

Published in Issue

Year 2020 Volume: 3 Number: 1

DOI

https://doi.org/10.35377/saucis.03.01.705777

IZ

https://izlik.org/JA36PE28MB

Cite

RIS / Bibtex

APA

Kuzhan, M., & Şahin, V. H. (2020). MBBench: A WCET Benchmark Suite. Sakarya University Journal of Computer and Information Sciences, 3(1), 40-50. https://doi.org/10.35377/saucis.03.01.705777

AMA

1.Kuzhan M, Şahin VH. MBBench: A WCET Benchmark Suite. SAUCIS. 2020;3(1):40-50. doi:10.35377/saucis.03.01.705777

Chicago

Kuzhan, Metin, and Veysel Harun Şahin. 2020. “MBBench: A WCET Benchmark Suite”. Sakarya University Journal of Computer and Information Sciences 3 (1): 40-50. https://doi.org/10.35377/saucis.03.01.705777.

EndNote

Kuzhan M, Şahin VH (April 1, 2020) MBBench: A WCET Benchmark Suite. Sakarya University Journal of Computer and Information Sciences 3 1 40–50.

IEEE

[1]M. Kuzhan and V. H. Şahin, “MBBench: A WCET Benchmark Suite”, SAUCIS, vol. 3, no. 1, pp. 40–50, Apr. 2020, doi: 10.35377/saucis.03.01.705777.

ISNAD

Kuzhan, Metin - Şahin, Veysel Harun. “MBBench: A WCET Benchmark Suite”. Sakarya University Journal of Computer and Information Sciences 3/1 (April 1, 2020): 40-50. https://doi.org/10.35377/saucis.03.01.705777.

JAMA

1.Kuzhan M, Şahin VH. MBBench: A WCET Benchmark Suite. SAUCIS. 2020;3:40–50.

MLA

Kuzhan, Metin, and Veysel Harun Şahin. “MBBench: A WCET Benchmark Suite”. Sakarya University Journal of Computer and Information Sciences, vol. 3, no. 1, Apr. 2020, pp. 40-50, doi:10.35377/saucis.03.01.705777.

Vancouver

1.Metin Kuzhan, Veysel Harun Şahin. MBBench: A WCET Benchmark Suite. SAUCIS. 2020 Apr. 1;3(1):40-5. doi:10.35377/saucis.03.01.705777