Research Article
BibTex RIS Cite

Year 2024, , 92 - 102, 30.04.2024

Zeynep Karaca , Bihter Daş

https://doi.org/10.35377/saucis...1339931

Abstract

References

  • [1] T. Alqahtani et al., The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research," Research in Social and Administrative Pharmacy, vol. 19, no. 8, pp. 1236–1242, Aug. 2023, doi: 10.1016/j.sapharm.2023.05.016.
  • [2] J. J. Cavallo, I. de Oliveira Santo, J. L. Mezrich, and H. P. Forman, Clinical Implementation of a Combined Artificial Intelligence and Natural Language Processing Quality Assurance Program for Pulmonary Nodule Detection in the Emergency Department Setting,Journal of the American College of Radiology, vol. 20, no. 4, pp. 438–445, Apr. 2023, doi: 10.1016/j.jacr.2022.12.016.
  • [3] J. Doe and A. Smith, "Recent Advances in Image Captioning: A Comprehensive Survey," IEEE Transactions on Artificial Intelligence, vol. 7, no. 3, pp. 210-225, 2022.
  • [4] M. Johnson, B. Brown, and C. Wilson, "Innovative Approaches for Image Caption Generation using Attention Mechanisms," Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, 2021, pp. 750-760.
  • [5] S. Kim and E. Lee, "Enhancing Image Captioning Performance through Multimodal Fusion Techniques," IEEE Transactions on Multimedia, vol. 25, no. 6, pp. 1350-1365, 2020.
  • [6] L. Wang, H. Chen, and X. Zhang, "Leveraging Transformers for Improved Image Captioning," Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 2018, pp. 240-255.
  • [7] R. Patel and S. Gupta, "Attention Is All You Need: Exploring Self-Attention Mechanisms in Image Captioning," Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 2019, pp. 1800-1810.
  • [8] C. Yue, W. Hu, H. Song, and W. Kang, Thangka image caption method based on attention mechanism and encoder-decoder architecture in 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP), Nis. 2022, pp. 1752-1756. doi: 10.1109/ICSP54964.2022.9778737.
  • [9] S. R. Chandaran, S. Natesan, G. Muthusamy, P. K. Sivakumar, P. Mohanraj, and R. J. Gnanaprakasam, Image Captioning Using Deep Learning Techniques for Partially Impaired Peoplein 2023 International Conference on Computer Communication and Informatics (ICCCI), Oca. 2023, pp. 1-6. doi: 10.1109/ICCCI56745.2023.10128287.
  • [10] S. S. Bhadauria, D. Bisht, T. Poongodi, and S. A. Yadav, Assertive Vision Using Deep Learning and LSTMin 2022 2nd International Conference on Innovative Practices in Technology and Management (ICIPTM), Şub. 2022, pp. 761-764. doi: 10.1109/ICIPTM54933.2022.9754057.
  • [11] M. K. Shaikh and M. V. Joshi, Recursive Network with Explicit Neighbor Connection for Image Captioningin 2018 International Conference on Signal Processing and Communications (SPCOM), Tem. 2018, pp. 392-396. doi: 10.1109/SPCOM.2018.8724400.
  • [12] Z. Xue, L. Wang, and P. Guo, Slot based Image Captioning with WGAN in 2019 IEEE/ACIS 18th International Conference on Computer and Information Science (ICIS), Haz. 2019, pp. 241-246. doi: 10.1109/ICIS46139.2019.8940218.
  • [13] A. Singh et al., Image Captioning Using Python in 2023 International Conference on Power, Instrumentation, Energy and Control (PIECON), Şub. 2023, pp. 1-5. doi: 10.1109/PIECON56912.2023.10085724.
  • [14] S.-H. Han and H.-J. Choi, Explainable Image Caption Generator Using Attention and Bayesian Inference in 2018 International Conference on Computational Science and Computational Intelligence (CSCI), Ara. 2018, pp. 478-481. doi: 10.1109/CSCI46756.2018.00098.
  • [15] B. Wang et al., Cross-Lingual Image Caption Generation Based on Visual Attention Model IEEE Access, vol. 8, pp. 104543-104554, 2020, doi: 10.1109/ACCESS.2020.2999568.
  • [16] H. Chen et al., Captioning Transformer With Scene Graph Guiding in 2021 IEEE International Conference on Image Processing (ICIP), Eyl. 2021, pp. 2538-2542. doi: 10.1109/ICIP42928.2021.9506193.
  • [17] S. Rafi and R. Das, A Linear Sub-Structure with Co-Variance Shift for Image Captioning in 2021 8th International Conference on Soft Computing & Machine Intelligence (ISCMI), Kas. 2021, pp. 242-246. doi: 10.1109/ISCMI53840.2021.9654828.
  • [18] D. Sharma et al., ghtweight Transformer with GRU Integrated Decoder for Image Captioning in 2022 16th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), Eki. 2022, pp. 434-438. doi: 10.1109/SITIS57111.2022.00072.
  • [19] L. Panigrahi et al., Hybrid Image Captioning Model in 2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development (OTCON), Şub. 2023, pp. 1-6. doi: 10.1109/OTCON56053.2023.10113957.
  • [20] Q. Sun et al., Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Tem. 2017, pp. 7215-7223. doi: 10.1109/CVPR.2017.763.
  • [21] Y. Keneshloo et al., Deep Reinforcement Learning for Sequence-to-Sequence Models IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 7, pp. 2469-2489, Tem. 2020, doi: 10.1109/TNNLS.2019.2929141.
  • [22] M. Sahrial Alam et al., arison of Different CNN Model used as Encoders for Image Captioning in 2021 International Conference on Data Analytics for Business and Industry (ICDABI), Eki. 2021, pp. 523-526. doi: 10.1109/ICDABI53623.2021.9655846.
  • [23] Y. Zhou et al., Triple Sequence Generative Adversarial Nets for Unsupervised Image Captioning in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Haz. 2021, pp. 7598-7602. doi: 10.1109/ICASSP39728.2021.9414335.
  • [24] R. Kushwaha and A. Biswas, Hybrid Feature and Sequence Extractor based Deep Learning Model for Image Caption Generation in 2021 12th International Conference on Computing Communication and Networking Technologies (ICCCNT), Tem. 2021, pp. 1-6. doi: 10.1109/ICCCNT51525.2021.9579897.
  • [25] H. Liu et al., Vocabulary-Wide Credit Assignment for Training Image Captioning Models IEEE Transactions on Image Processing, vol. 30, pp. 2450-2460, 2021, doi: 10.1109/TIP.2021.3051476.
  • [26] Y. Zheng et al., Divcon: Learning Concept Sequences for Semantically Diverse Image Captioning in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Haz. 2023, pp. 1-5. doi: 10.1109/ICASSP49357.2023.10094565.
  • [27] B. Birmingham and A. Muscat, KENGIC: KEyword-driven and N-Gram Graph based Image Captioning in 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Nov. 2022, pp. 1-8. doi: 10.1109/DICTA56598.2022.10034584.
  • [28] P. Dwivedi and A. Upadhyaya, A Novel Deep Learning Model for Accurate Prediction of Image Captions in Fashion Industry in 2022 12th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Oca. 2022, pp. 207-212. doi: 10.1109/Confluence52989.2022.9734171.
  • [29] F.Akalin and N. Yumusak, "Detection and classification of white blood cells with an improved deep learning-based approach,"Turkish Journal of Electrical Engineering and Computer Sciences, vol. 30, no. 7, article 16. https://doi.org/10.55730/1300-0632.3965
  • [30] F. Akalin, and N.Yumusak. "Classification of ALL, AML and MLL leukaemia types on microarray dataset using LSTM neural network Approach" , Journal of Faculty of Engineering and Archıtecture Of Gazı Unıversıty vol. 38, no. 3, 2023, pp. 1299-1306.
  • [31] I. Sutskever, O. Vinyals, and Q. V. Le, Sequence to Sequence Learning with Neural Networks in Advances in Neural Information Processing Systems (pp. 3104-3112), 2014.
  • [32] P. Anderson et al., Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 6, pp. 1416-1432, 2018.
  • [33] J. Xie et al., A multimodal fusion emotion recognition method based on multitask learning and attention mechanism Neurocomputing, p. 126649, Aug. 2023, doi: 10.1016/j.neucom.2023.126649.
  • [34] K. Yang et al., A multi-sensor mapping Bi-LSTM model of bridge monitoring data based on spatial-temporal attention mechanism Measurement, vol. 217, p. 113053, Aug. 2023, doi: 10.1016/j.measurement.2023.113053.
  • [35] H. Won, B. Kim, I.-Y. Kwak, ve C. Lim, Using various pre-trained models for audio feature extraction in automated audio captioningExpert Systems with Applications, c. 231, s. 120664, Kas. 2023, doi: 10.1016/j.eswa.2023.120664.

The Role of Attention Mechanism in Generating Image Captions: An Innovative Approach with Neural Network-Based Seq2seq Model

Year 2024, , 92 - 102, 30.04.2024

Zeynep Karaca , Bihter Daş

https://doi.org/10.35377/saucis...1339931

Abstract

Image-to-text generation contributes significantly across various domains such as entertainment, communication, commerce, security, and education by establishing a connection between visual and textual content through the creation of explanations. This process aims to transform image data into meaningful text, enhancing content accessibility, comprehensibility, and processability. Hence, advancements and studies in this field hold paramount importance. This study focuses on how the fusion of the Sequence-to-Sequence (Seq2seq) model and attention mechanism enhances the generation of more meaningful captions from images. Experiments conducted on the Flickr8k dataset highlight the Seq2seq model's capacity to produce captions in alignment with reference sentences. Leveraging the dynamic focus of the attention mechanism, the model effectively captures detailed aspects of images.

Keywords

Attention mechanism Seq2seg model Image Capturing Deep learning Image-to-text

References

  • [1] T. Alqahtani et al., The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research," Research in Social and Administrative Pharmacy, vol. 19, no. 8, pp. 1236–1242, Aug. 2023, doi: 10.1016/j.sapharm.2023.05.016.
  • [2] J. J. Cavallo, I. de Oliveira Santo, J. L. Mezrich, and H. P. Forman, Clinical Implementation of a Combined Artificial Intelligence and Natural Language Processing Quality Assurance Program for Pulmonary Nodule Detection in the Emergency Department Setting,Journal of the American College of Radiology, vol. 20, no. 4, pp. 438–445, Apr. 2023, doi: 10.1016/j.jacr.2022.12.016.
  • [3] J. Doe and A. Smith, "Recent Advances in Image Captioning: A Comprehensive Survey," IEEE Transactions on Artificial Intelligence, vol. 7, no. 3, pp. 210-225, 2022.
  • [4] M. Johnson, B. Brown, and C. Wilson, "Innovative Approaches for Image Caption Generation using Attention Mechanisms," Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, 2021, pp. 750-760.
  • [5] S. Kim and E. Lee, "Enhancing Image Captioning Performance through Multimodal Fusion Techniques," IEEE Transactions on Multimedia, vol. 25, no. 6, pp. 1350-1365, 2020.
  • [6] L. Wang, H. Chen, and X. Zhang, "Leveraging Transformers for Improved Image Captioning," Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 2018, pp. 240-255.
  • [7] R. Patel and S. Gupta, "Attention Is All You Need: Exploring Self-Attention Mechanisms in Image Captioning," Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 2019, pp. 1800-1810.
  • [8] C. Yue, W. Hu, H. Song, and W. Kang, Thangka image caption method based on attention mechanism and encoder-decoder architecture in 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP), Nis. 2022, pp. 1752-1756. doi: 10.1109/ICSP54964.2022.9778737.
  • [9] S. R. Chandaran, S. Natesan, G. Muthusamy, P. K. Sivakumar, P. Mohanraj, and R. J. Gnanaprakasam, Image Captioning Using Deep Learning Techniques for Partially Impaired Peoplein 2023 International Conference on Computer Communication and Informatics (ICCCI), Oca. 2023, pp. 1-6. doi: 10.1109/ICCCI56745.2023.10128287.
  • [10] S. S. Bhadauria, D. Bisht, T. Poongodi, and S. A. Yadav, Assertive Vision Using Deep Learning and LSTMin 2022 2nd International Conference on Innovative Practices in Technology and Management (ICIPTM), Şub. 2022, pp. 761-764. doi: 10.1109/ICIPTM54933.2022.9754057.
  • [11] M. K. Shaikh and M. V. Joshi, Recursive Network with Explicit Neighbor Connection for Image Captioningin 2018 International Conference on Signal Processing and Communications (SPCOM), Tem. 2018, pp. 392-396. doi: 10.1109/SPCOM.2018.8724400.
  • [12] Z. Xue, L. Wang, and P. Guo, Slot based Image Captioning with WGAN in 2019 IEEE/ACIS 18th International Conference on Computer and Information Science (ICIS), Haz. 2019, pp. 241-246. doi: 10.1109/ICIS46139.2019.8940218.
  • [13] A. Singh et al., Image Captioning Using Python in 2023 International Conference on Power, Instrumentation, Energy and Control (PIECON), Şub. 2023, pp. 1-5. doi: 10.1109/PIECON56912.2023.10085724.
  • [14] S.-H. Han and H.-J. Choi, Explainable Image Caption Generator Using Attention and Bayesian Inference in 2018 International Conference on Computational Science and Computational Intelligence (CSCI), Ara. 2018, pp. 478-481. doi: 10.1109/CSCI46756.2018.00098.
  • [15] B. Wang et al., Cross-Lingual Image Caption Generation Based on Visual Attention Model IEEE Access, vol. 8, pp. 104543-104554, 2020, doi: 10.1109/ACCESS.2020.2999568.
  • [16] H. Chen et al., Captioning Transformer With Scene Graph Guiding in 2021 IEEE International Conference on Image Processing (ICIP), Eyl. 2021, pp. 2538-2542. doi: 10.1109/ICIP42928.2021.9506193.
  • [17] S. Rafi and R. Das, A Linear Sub-Structure with Co-Variance Shift for Image Captioning in 2021 8th International Conference on Soft Computing & Machine Intelligence (ISCMI), Kas. 2021, pp. 242-246. doi: 10.1109/ISCMI53840.2021.9654828.
  • [18] D. Sharma et al., ghtweight Transformer with GRU Integrated Decoder for Image Captioning in 2022 16th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), Eki. 2022, pp. 434-438. doi: 10.1109/SITIS57111.2022.00072.
  • [19] L. Panigrahi et al., Hybrid Image Captioning Model in 2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development (OTCON), Şub. 2023, pp. 1-6. doi: 10.1109/OTCON56053.2023.10113957.
  • [20] Q. Sun et al., Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Tem. 2017, pp. 7215-7223. doi: 10.1109/CVPR.2017.763.
  • [21] Y. Keneshloo et al., Deep Reinforcement Learning for Sequence-to-Sequence Models IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 7, pp. 2469-2489, Tem. 2020, doi: 10.1109/TNNLS.2019.2929141.
  • [22] M. Sahrial Alam et al., arison of Different CNN Model used as Encoders for Image Captioning in 2021 International Conference on Data Analytics for Business and Industry (ICDABI), Eki. 2021, pp. 523-526. doi: 10.1109/ICDABI53623.2021.9655846.
  • [23] Y. Zhou et al., Triple Sequence Generative Adversarial Nets for Unsupervised Image Captioning in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Haz. 2021, pp. 7598-7602. doi: 10.1109/ICASSP39728.2021.9414335.
  • [24] R. Kushwaha and A. Biswas, Hybrid Feature and Sequence Extractor based Deep Learning Model for Image Caption Generation in 2021 12th International Conference on Computing Communication and Networking Technologies (ICCCNT), Tem. 2021, pp. 1-6. doi: 10.1109/ICCCNT51525.2021.9579897.
  • [25] H. Liu et al., Vocabulary-Wide Credit Assignment for Training Image Captioning Models IEEE Transactions on Image Processing, vol. 30, pp. 2450-2460, 2021, doi: 10.1109/TIP.2021.3051476.
  • [26] Y. Zheng et al., Divcon: Learning Concept Sequences for Semantically Diverse Image Captioning in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Haz. 2023, pp. 1-5. doi: 10.1109/ICASSP49357.2023.10094565.
  • [27] B. Birmingham and A. Muscat, KENGIC: KEyword-driven and N-Gram Graph based Image Captioning in 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Nov. 2022, pp. 1-8. doi: 10.1109/DICTA56598.2022.10034584.
  • [28] P. Dwivedi and A. Upadhyaya, A Novel Deep Learning Model for Accurate Prediction of Image Captions in Fashion Industry in 2022 12th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Oca. 2022, pp. 207-212. doi: 10.1109/Confluence52989.2022.9734171.
  • [29] F.Akalin and N. Yumusak, "Detection and classification of white blood cells with an improved deep learning-based approach,"Turkish Journal of Electrical Engineering and Computer Sciences, vol. 30, no. 7, article 16. https://doi.org/10.55730/1300-0632.3965
  • [30] F. Akalin, and N.Yumusak. "Classification of ALL, AML and MLL leukaemia types on microarray dataset using LSTM neural network Approach" , Journal of Faculty of Engineering and Archıtecture Of Gazı Unıversıty vol. 38, no. 3, 2023, pp. 1299-1306.
  • [31] I. Sutskever, O. Vinyals, and Q. V. Le, Sequence to Sequence Learning with Neural Networks in Advances in Neural Information Processing Systems (pp. 3104-3112), 2014.
  • [32] P. Anderson et al., Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 6, pp. 1416-1432, 2018.
  • [33] J. Xie et al., A multimodal fusion emotion recognition method based on multitask learning and attention mechanism Neurocomputing, p. 126649, Aug. 2023, doi: 10.1016/j.neucom.2023.126649.
  • [34] K. Yang et al., A multi-sensor mapping Bi-LSTM model of bridge monitoring data based on spatial-temporal attention mechanism Measurement, vol. 217, p. 113053, Aug. 2023, doi: 10.1016/j.measurement.2023.113053.
  • [35] H. Won, B. Kim, I.-Y. Kwak, ve C. Lim, Using various pre-trained models for audio feature extraction in automated audio captioningExpert Systems with Applications, c. 231, s. 120664, Kas. 2023, doi: 10.1016/j.eswa.2023.120664.
There are 35 citations in total.

Details

Primary Language English
Subjects Software Engineering (Other)
Journal Section Articles
Authors

Zeynep Karaca 0000-0002-7751-8567

Bihter Daş 0000-0002-2498-3297

Early Pub Date April 27, 2024
Publication Date April 30, 2024
Submission Date August 9, 2023
Acceptance Date March 26, 2024
Published in Issue Year 2024

Cite

IEEE Z. Karaca and B. Daş, “The Role of Attention Mechanism in Generating Image Captions: An Innovative Approach with Neural Network-Based Seq2seq Model”, SAUCIS, vol. 7, no. 1, pp. 92–102, 2024, doi: 10.35377/saucis...1339931.

29070    The papers in this journal are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License