Research Article
BibTex RIS Cite

MBBench: A WCET Benchmark Suite

Year 2020, Volume: 3 Issue: 1, 40 - 50, 30.04.2020

Metin Kuzhan , Veysel Harun Şahin

https://doi.org/10.35377/saucis.03.01.705777
Cited By: 3

Abstract

One of the important features of any real-time software is the worst-case execution time (WCET). To get an understanding of the timing behavior of real-time systems and to prove that the real-time software meets its deadlines, WCET analysis is performed. Today, researchers actively develop new WCET analysis methods and tools. Therefore, they need benchmark programs to evaluate and compare their work. To meet this need, in this study we present a new benchmark suite, called MBBench. MBBench includes a collection of C programs for Linux operating system and RTEMS real-time operating system. Its main aim is to help the evaluation and comparison of measurement-based WCET analysis methods/tools. MBBench has been published as open source. It can be obtained freely over the Internet.

Keywords

Real-Time Systems Benchmark WCET Analysis Software Engineering

Thanks

The authors would like to acknowledge that this work is supported by the Real-Time Systems Research Laboratory at Sakarya University, Faculty of Computer and Information Sciences, Department of Software Engineering. The MBBench benchmark suite can be obtained from the Real-Time Systems Research Laboratory homepage freely [18].

References

  • [1] G. C. Buttazzo, Hard Real-Time Computing Systems, 3rd ed., vol. 24. Boston, MA: Springer US, 2011.
  • [2] H. Kopetz, Real-Time Systems, 2nd ed. Boston, MA: Springer US, 2011.
  • [3] F. J. Cazorla, L. Kosmidis, E. Mezzetti, C. Hernandez, J. Abella, and T. Vardanega, “Probabilistic Worst-Case Timing Analysis: Taxonomy and Comprehensive Survey,” ACM Computing Surveys, vol. 52, no. 1, pp. 1–35, 2019, doi: 10.1145/3301283.
  • [4] R. I. Davis and L. Cucu-Grosjean, “A Survey of Probabilistic Timing Analysis Techniques for Real-Time Systems,” Leibniz Transactions on Embedded Systems (LITES), vol. 6, no. 1, pp. 3:1–3:60, 2019, doi: 10.4230/LITES-v006-i001-a003.
  • [5] J. Abella, D. Hardy, I. Puaut, E. Quinones, and F. J. Cazorla, “On the comparison of deterministic and probabilistic WCET estimation techniques,” Proceedings - Euromicro Conference on Real-Time Systems, pp. 266–275, 2014, doi: 10.1109/ECRTS.2014.16.
  • [6] J. Abella et al., “WCET analysis methods: Pitfalls and challenges on their trustworthiness,” in 10th IEEE International Symposium on Industrial Embedded Systems (SIES), Siegen,Germany, pp. 1–10, 2015, doi: 10.1109/SIES.2015.7185039.
  • [7] R. Wilhelm et al., “The worst-case execution-time problem—overview of methods and survey of tools,” Transactions on Embedded Computing Systems, vol. 7, no. 3, pp. 1–45, Apr. 2008, doi: 10.1145/1347375.1347389.
  • [8] L. Cucu-Grosjean et al., “Measurement-based probabilistic timing analysis for multi-path programs,” Proceedings - Euromicro Conference on Real-Time Systems, pp. 91–101, 2012, doi: 10.1109/ECRTS.2012.31.
  • [9] F. J. Cazorla et al., “PROXIMA: Improving Measurement-Based Timing Analysis through Randomisation and Probabilistic Analysis,” Proceedings - 19th Euromicro Conference on Digital System Design, DSD 2016, pp. 276–285, 2016, doi: 10.1109/DSD.2016.22.
  • [10] X. Li, Y. Liang, T. Mitra, and A. Roychoudhury, “Chronos: A timing analyzer for embedded software,” Science of Computer Programming, vol. 69, no. 1–3, pp. 56–67, 2007, doi: 10.1016/j.scico.2007.01.014.
  • [11] Université of Toulouse, “OTAWA - WCET is coming...,” 2020. [Online]. Available: http://otawa.fr. [Accessed: 17-Mar-2019].
  • [12] D. Hardy, B. Rouxel, and I. Puaut, “The Heptane Static Worst-Case Execution Time Estimation Tool,” in 17th International Workshop on Worst-Case Execution Time Analysis (WCET 2017), Dagstuhl, Germany, vol. 57, pp. 8:1—-8:12, 2017, doi: 10.4230/OASIcs.WCET.2017.8.
  • [13] Inria, “Heptane static WCET estimation tool - PACAP,” 2020. [Online]. Available: https://team.inria.fr/pacap/software/heptane. [Accessed: 17-Mar-2019].
  • [14] Rapita Systems, “Rapita Systems | On-target software verification solutions,” 2020. [Online]. Available: https://www.rapitasystems.com. [Accessed: 17-Mar-2020].
  • [15] AbsInt, “AbsInt: Cutting-Edge Tools for Static Analysis of Safety-Critical Software,” 2020. [Online]. Available: https://www.absint.com. [Accessed: 17-Mar-2020].
  • [16] Linux Kernel Organization, “The Linux Kernel Archives,” 2020. [Online]. Available: https://www.kernel.org. [Accessed: 17-Mar-2020].
  • [17] The RTEMS Project, “RTEMS Real Time Operating System (RTOS),” 2020. [Online]. Available: https://www.rtems.org. [Accessed: 17-Mar-2020].
  • [18] Sakarya University, Faculty of Computer and Information Sciences, Department of Software Engineering, “Real-Time Systems Research Laboratory homepage,” 2020. [Online]. Available: http://rtsrlab.sakarya.edu.tr. [Accessed: 17-Mar-2020].
  • [19] Standard Performance Evaluation Corporation, “Spec - Standard Performance Evaluation Corporation,” 2020. [Online]. Available: https://www.spec.org. [Accessed: 17-Mar-2020].
  • [20] EEMBC, “Embedded Microprocessor Benchmark Consortium,” 2020. [Online]. Available: https://www.eembc.org. [Accessed: 17-Mar-2020].
  • [21] C. Bienia, S. Kumar, J. P. Singh, and K. Li, “The PARSEC benchmark suite,” in Proceedings of the 17th international conference on Parallel architectures and compilation techniques - PACT ’08, New York, New York, USA, 2008, p. 72, doi: 10.1145/1454115.1454128.
  • [22] X. Tian et al., “BigDataBench-S: An Open-Source Scientific Big Data Benchmark Suite,” presented at the 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Lake Buena Vista, FL, USA, 2017, pp. 1068–1077, doi: 10.1109/IPDPSW.2017.111.
  • [23] S. M. Blackburn et al., “The DaCapo benchmarks,” in Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications - OOPSLA ’06, New York, New York, USA, 2006, p. 169, doi: 10.1145/1167473.1167488.
  • [24] T. Kalibera, P. Parizek, G. Haddad, G. T. Leavens, and J. Vitek, “Challenge benchmarks for verification of real-time programs,” in Proceedings of the 4th ACM SIGPLAN workshop on Programming languages meets program verification - PLPV ’10, New York, New York, USA, 2010, p. 57, doi: 10.1145/1707790.1707800.
  • [25] G. Bollella et al., The Real-Time Specification for Java. Addison-Wesley, 2000.
  • [26] J. Gustafsson, A. Betts, A. Ermedahl, and B. Lisper, “The Mälardalen WCET Benchmarks: Past, Present and Future,” in 10th International Workshop on Worst-Case Execution Time Analysis (WCET 2010), Dagstuhl, Germany, 2010, vol. 15, pp. 136–146, doi: 10.4230/OASIcs.WCET.2010.136.
  • [27] Mälardalen Real-Time Research Center, “The Mälardalen WCET Benchmarks,” 2013. [Online]. Available: http://www.mrtc.mdh.se/projects/wcet/benchmarks.html. [Accessed: 17-Mar-2020].
  • [28] F. Nemer, H. Cassé, P. Sainrat, J.-P. Bahsoun, and M. D. Michiel, “PapaBench: a Free Real-Time Benchmark,” in 6th International Workshop on Worst-Case Execution Time Analysis (WCET’06), Dagstuhl, Germany, 2006, vol. 4, doi: 10.4230/OASIcs.WCET.2006.678.
  • [29] H. Falk et al., “TACLeBench: A Benchmark Collection to Support Worst-Case Execution Time Research,” in 16th International Workshop on Worst-Case Execution Time Analysis (WCET 2016), Dagstuhl, Germany, 2016, vol. 55, pp. 2:1—-2:10, doi: 10.4230/OASIcs.WCET.2016.2.
  • [30] S. Serttaş and V. H. Şahin, “PBench: A Parallel, Real-Time Benchmark Suite,” in Academic Perspective Procedia, Alanya, Antalya, Turkey, 2018, vol. 1, pp. 178–186, doi: 10.33793/acperpro.01.01.37.
  • [31] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Algoritmalara Giriş (Üçüncü Baskıdan Çeviri). Palme Yayıncılık, 2017.
  • [32] M. M. Mano, Bilgisayar Sistemleri Mimarisi (3. Basımdan Çeviri). Literatür Yayınları, 2015.
  • [33] Linux Foundation, “Zephyr Project homepage,” 2020. [Online]. Available: https://www.zephyrproject.org. [Accessed: 17-Mar-2020].

MBBench: WCET Kıyaslama Kümesi

Year 2020, Volume: 3 Issue: 1, 40 - 50, 30.04.2020

Metin Kuzhan , Veysel Harun Şahin

https://doi.org/10.35377/saucis.03.01.705777
Cited By: 3

Abstract

Herhangi bir gerçek zamanlı yazılımın en önemli özelliklerinden birisi, en kötü durum yürütme süresidir (WCET). Gerçek zamanlı sistemlerin zamanlama davranışını anlamak ve gerçek zamanlı yazılımın son teslim tarihlerini karşıladığını kanıtlamak için WCET analizi yapılır. Günümüzde araştırmacılar aktif olarak yeni WCET analiz yöntemleri ve araçları geliştirmektedir. Dolayısıyla, çalışmalarını değerlendirmek ve karşılaştırmak için kıyaslama programlarına ihtiyaç duymaktadırlar. Bu çalışmada, bu ihtiyacı karşılamaya yardımcı olmak amacıyla MBBench isminde yeni bir kıyaslama kümesi sunuyoruz. MBBench, Linux işletim sistemi ve RTEMS gerçek zamanlı işletim sistemi için C programları koleksiyonu içermektedir. Kıyaslama kümesinin temel amacı, ölçüm tabanlı WCET analizi yöntemlerinin/araçlarının değerlendirilmesine ve karşılaştırılmasına yardımcı olmaktır. MBBench, açık kaynak kodlu olarak yayınlanmıştır ve İnternet üzerinden ücretsiz olarak edinilebilir.

Keywords

Gerçek Zamanlı Sistemler Kıyaslama WCET analizi Yazılım Mühendisliği

References

  • [1] G. C. Buttazzo, Hard Real-Time Computing Systems, 3rd ed., vol. 24. Boston, MA: Springer US, 2011.
  • [2] H. Kopetz, Real-Time Systems, 2nd ed. Boston, MA: Springer US, 2011.
  • [3] F. J. Cazorla, L. Kosmidis, E. Mezzetti, C. Hernandez, J. Abella, and T. Vardanega, “Probabilistic Worst-Case Timing Analysis: Taxonomy and Comprehensive Survey,” ACM Computing Surveys, vol. 52, no. 1, pp. 1–35, 2019, doi: 10.1145/3301283.
  • [4] R. I. Davis and L. Cucu-Grosjean, “A Survey of Probabilistic Timing Analysis Techniques for Real-Time Systems,” Leibniz Transactions on Embedded Systems (LITES), vol. 6, no. 1, pp. 3:1–3:60, 2019, doi: 10.4230/LITES-v006-i001-a003.
  • [5] J. Abella, D. Hardy, I. Puaut, E. Quinones, and F. J. Cazorla, “On the comparison of deterministic and probabilistic WCET estimation techniques,” Proceedings - Euromicro Conference on Real-Time Systems, pp. 266–275, 2014, doi: 10.1109/ECRTS.2014.16.
  • [6] J. Abella et al., “WCET analysis methods: Pitfalls and challenges on their trustworthiness,” in 10th IEEE International Symposium on Industrial Embedded Systems (SIES), Siegen,Germany, pp. 1–10, 2015, doi: 10.1109/SIES.2015.7185039.
  • [7] R. Wilhelm et al., “The worst-case execution-time problem—overview of methods and survey of tools,” Transactions on Embedded Computing Systems, vol. 7, no. 3, pp. 1–45, Apr. 2008, doi: 10.1145/1347375.1347389.
  • [8] L. Cucu-Grosjean et al., “Measurement-based probabilistic timing analysis for multi-path programs,” Proceedings - Euromicro Conference on Real-Time Systems, pp. 91–101, 2012, doi: 10.1109/ECRTS.2012.31.
  • [9] F. J. Cazorla et al., “PROXIMA: Improving Measurement-Based Timing Analysis through Randomisation and Probabilistic Analysis,” Proceedings - 19th Euromicro Conference on Digital System Design, DSD 2016, pp. 276–285, 2016, doi: 10.1109/DSD.2016.22.
  • [10] X. Li, Y. Liang, T. Mitra, and A. Roychoudhury, “Chronos: A timing analyzer for embedded software,” Science of Computer Programming, vol. 69, no. 1–3, pp. 56–67, 2007, doi: 10.1016/j.scico.2007.01.014.
  • [11] Université of Toulouse, “OTAWA - WCET is coming...,” 2020. [Online]. Available: http://otawa.fr. [Accessed: 17-Mar-2019].
  • [12] D. Hardy, B. Rouxel, and I. Puaut, “The Heptane Static Worst-Case Execution Time Estimation Tool,” in 17th International Workshop on Worst-Case Execution Time Analysis (WCET 2017), Dagstuhl, Germany, vol. 57, pp. 8:1—-8:12, 2017, doi: 10.4230/OASIcs.WCET.2017.8.
  • [13] Inria, “Heptane static WCET estimation tool - PACAP,” 2020. [Online]. Available: https://team.inria.fr/pacap/software/heptane. [Accessed: 17-Mar-2019].
  • [14] Rapita Systems, “Rapita Systems | On-target software verification solutions,” 2020. [Online]. Available: https://www.rapitasystems.com. [Accessed: 17-Mar-2020].
  • [15] AbsInt, “AbsInt: Cutting-Edge Tools for Static Analysis of Safety-Critical Software,” 2020. [Online]. Available: https://www.absint.com. [Accessed: 17-Mar-2020].
  • [16] Linux Kernel Organization, “The Linux Kernel Archives,” 2020. [Online]. Available: https://www.kernel.org. [Accessed: 17-Mar-2020].
  • [17] The RTEMS Project, “RTEMS Real Time Operating System (RTOS),” 2020. [Online]. Available: https://www.rtems.org. [Accessed: 17-Mar-2020].
  • [18] Sakarya University, Faculty of Computer and Information Sciences, Department of Software Engineering, “Real-Time Systems Research Laboratory homepage,” 2020. [Online]. Available: http://rtsrlab.sakarya.edu.tr. [Accessed: 17-Mar-2020].
  • [19] Standard Performance Evaluation Corporation, “Spec - Standard Performance Evaluation Corporation,” 2020. [Online]. Available: https://www.spec.org. [Accessed: 17-Mar-2020].
  • [20] EEMBC, “Embedded Microprocessor Benchmark Consortium,” 2020. [Online]. Available: https://www.eembc.org. [Accessed: 17-Mar-2020].
  • [21] C. Bienia, S. Kumar, J. P. Singh, and K. Li, “The PARSEC benchmark suite,” in Proceedings of the 17th international conference on Parallel architectures and compilation techniques - PACT ’08, New York, New York, USA, 2008, p. 72, doi: 10.1145/1454115.1454128.
  • [22] X. Tian et al., “BigDataBench-S: An Open-Source Scientific Big Data Benchmark Suite,” presented at the 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Lake Buena Vista, FL, USA, 2017, pp. 1068–1077, doi: 10.1109/IPDPSW.2017.111.
  • [23] S. M. Blackburn et al., “The DaCapo benchmarks,” in Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications - OOPSLA ’06, New York, New York, USA, 2006, p. 169, doi: 10.1145/1167473.1167488.
  • [24] T. Kalibera, P. Parizek, G. Haddad, G. T. Leavens, and J. Vitek, “Challenge benchmarks for verification of real-time programs,” in Proceedings of the 4th ACM SIGPLAN workshop on Programming languages meets program verification - PLPV ’10, New York, New York, USA, 2010, p. 57, doi: 10.1145/1707790.1707800.
  • [25] G. Bollella et al., The Real-Time Specification for Java. Addison-Wesley, 2000.
  • [26] J. Gustafsson, A. Betts, A. Ermedahl, and B. Lisper, “The Mälardalen WCET Benchmarks: Past, Present and Future,” in 10th International Workshop on Worst-Case Execution Time Analysis (WCET 2010), Dagstuhl, Germany, 2010, vol. 15, pp. 136–146, doi: 10.4230/OASIcs.WCET.2010.136.
  • [27] Mälardalen Real-Time Research Center, “The Mälardalen WCET Benchmarks,” 2013. [Online]. Available: http://www.mrtc.mdh.se/projects/wcet/benchmarks.html. [Accessed: 17-Mar-2020].
  • [28] F. Nemer, H. Cassé, P. Sainrat, J.-P. Bahsoun, and M. D. Michiel, “PapaBench: a Free Real-Time Benchmark,” in 6th International Workshop on Worst-Case Execution Time Analysis (WCET’06), Dagstuhl, Germany, 2006, vol. 4, doi: 10.4230/OASIcs.WCET.2006.678.
  • [29] H. Falk et al., “TACLeBench: A Benchmark Collection to Support Worst-Case Execution Time Research,” in 16th International Workshop on Worst-Case Execution Time Analysis (WCET 2016), Dagstuhl, Germany, 2016, vol. 55, pp. 2:1—-2:10, doi: 10.4230/OASIcs.WCET.2016.2.
  • [30] S. Serttaş and V. H. Şahin, “PBench: A Parallel, Real-Time Benchmark Suite,” in Academic Perspective Procedia, Alanya, Antalya, Turkey, 2018, vol. 1, pp. 178–186, doi: 10.33793/acperpro.01.01.37.
  • [31] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Algoritmalara Giriş (Üçüncü Baskıdan Çeviri). Palme Yayıncılık, 2017.
  • [32] M. M. Mano, Bilgisayar Sistemleri Mimarisi (3. Basımdan Çeviri). Literatür Yayınları, 2015.
  • [33] Linux Foundation, “Zephyr Project homepage,” 2020. [Online]. Available: https://www.zephyrproject.org. [Accessed: 17-Mar-2020].
There are 33 citations in total.

Details

Primary Language English
Subjects Software Engineering
Journal Section Articles
Authors

Metin Kuzhan 0000-0001-6420-8327

Veysel Harun Şahin 0000-0002-3381-1702

Publication Date April 30, 2020
Submission Date March 18, 2020
Acceptance Date April 21, 2020
Published in Issue Year 2020Volume: 3 Issue: 1

Cite

IEEE M. Kuzhan and V. H. Şahin, “MBBench: A WCET Benchmark Suite”, SAUCIS, vol. 3, no. 1, pp. 40–50, 2020, doi: 10.35377/saucis.03.01.705777.

Cited By

BOOM-Explorer: RISC-V BOOM Microarchitecture Design Space Exploration
ACM Transactions on Design Automation of Electronic Systems
https://doi.org/10.1145/3630013
ABench2020: A WCET Benchmark Suite in Ada Programming Language
Academic Perspective Procedia
https://doi.org/10.33793/acperpro.03.01.99
https://doi.org/
 Download Cover Image

29070    The papers in this journal are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License