Empowering Language Models Through Advanced Prompt Engineering: A Comprehensive Bibliometric Review

Izzul Fatawi, Muhammad Asy'ari, H. Hunaepi, Taufik Samsuri, Muhammad Roil Bilad

Abstract


This review examines the transformative impact of prompt engineering on language models in artificial intelligence. Language models have advanced from simple probabilistic frameworks to sophisticated neural networks like the Generative Pre-trained Transformer series, enhancing machine understanding and generating human-like language. Prompt engineering customizes these models for specific tasks, improving human-AI interaction. This technique has expanded the capabilities of language models, making them useful in healthcare for better diagnostics and in education for interactive learning. The review, based on publications from January 2022 to February 2024 from the Scopus database, highlights significant advancements in prompt engineering. Network visualization with VOSviewer underscores its central role in AI's progress. The findings advocate integrating prompt engineering with quantum computing and updating educational programs to include advanced AI technologies. Establishing ethical standards and promoting open access to research are also recommended to foster innovation and responsible AI usage.

Keywords


Artificial intelligence; Bibliometric analysis; Ethics; Generative pre-trained; Language models; Transformers

Full Text:

PDF

References


Alam, M. M., Rony, M. R. A. H., Nayyeri, M., Mohiuddin, K., Akter, M. S. T. M., Vahdati, S., and Lehmann, J. (2022). Language model guided knowledge graph embeddings. IEEE Access, 10, 76008–76020.

Ameen, Z. J. M., and Kadhim, A. A. (2023). Deep learning methods for arabic autoencoder speech recognition system for electro-larynx device. Advances in Human-Computer Interaction, 2023, 1–11.

Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., Jacob, A. P., Komeili, M., Konath, K., Kwon, M., Lerer, A., Lewis, M., Miller, A. H., Mitts, S., Renduchintala, A., Roller, S., … Zijlstra, M. (2022). Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067–1074.

Barić, D., Fumić, P., Horvatić, D., and Lipic, T. (2021). Benchmarking attention-based interpretability of deep learning in multivariate time series predictions. Entropy, 23(2), 143.

Belcastro, L., Cantini, R., Marozzo, F., Talia, D., and Trunfio, P. (2020). Learning political polarization on social media using neural networks. IEEE Access, 8, 47177–47187.

Bendre, N., Desai, K., and Najafirad, P. (2021). Generalized zero-shot learning using multimodal variational auto-encoder with semantic concepts. 2021 IEEE International Conference on Image Processing (ICIP), 2021, 1284–1288.

Berquand, A., Darm, P., and Riccardi, A. (2021). SpaceTransformers: Language modeling for space systems. IEEE Access, 9, 133111–133122.

Bilad, M. R., Yaqin, L. N., and Zubaidah, S. (2023). Recent progress in the use of artificial intelligence tools in education. Jurnal Penelitian Dan Pengkajian Ilmu Pendidikan: E-Saintika, 7(3), 279–315.

Booton, S. A., Hodgkiss, A., and Murphy, V. A. (2023). The impact of mobile application features on children’s language and literacy learning: A systematic review. Computer Assisted Language Learning, 36(3), 400–429.

Chiang, T.-R., Ye, H.-T., and Chen, Y.-N. (2020). An empirical study of content understanding in conversational question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 7578–7585.

Coronado, E., Behravesh, R., Subramanya, T., Fernandez-Fernandez, A., Siddiqui, M. S., Costa-Perez, X., and Riggio, R. (2022). Zero touch management: A survey of network automation solutions for 5G and 6G networks. IEEE Communications Surveys and Tutorials, 24(4), 2535–2578.

Das, A., and Verma, R. M. (2020). Can machines tell stories? A comparative study of deep neural language models and metrics. IEEE Access, 8, 181258–181292.

Ding, L., and Ye, S. (2023). Using bidirectional prompt learning in nlp few shot tasks. Frontiers in Computing and Intelligent Systems, 3(1), 167–172.

Farahani, M., Gharachorloo, M., Farahani, M., and Manthouri, M. (2021). ParsBERT: Transformer-based model for persian language understanding. Neural Processing Letters, 53(6), 3831–3847.

Farhan, M., and Wang, Q. (2023). Efficient maintenance of highway cover labelling for distance queries on large dynamic graphs. World Wide Web, 26(5), 2427-2452.

Fleischmann, A., and Stary, C. (2012). Whom to talk to? A stakeholder perspective on business process development. Universal Access in the Information Society, 11(2), 125–150.

Frey, N. C., Soklaski, R., Axelrod, S., Samsi, S., Gómez-Bombarelli, R., Coley, C. W., and Gadepally, V. (2023). Neural scaling of deep chemical models. Nature Machine Intelligence, 5(11), 1297–1305.

Gadekallu, T. R., Alazab, M., Kaluri, R., Maddikunta, P. K. R., Bhattacharya, S., Lakshmanna, K., and M, P. (2021). Hand gesture classification using a novel CNN-crow search algorithm. Complex and Intelligent Systems, 7(4), 1855–1868.

Iglesias, C. A., and Moreno, A. (2019). Sentiment analysis for social media. Applied Sciences, 9(23), 5037.

Jiang, Z., Xu, F. F., Araki, J., and Neubig, G. (2020). How can we know what language models know?. Transactions of the Association for Computational Linguistics, 8, 423-438.

Khemasuwan, D., Sorensen, J. S., and Colt, H. G. (2020). Artificial intelligence in pulmonary medicine: Computer vision, predictive model and COVID-19. European Respiratory Review, 29(157), 200181.

Lee, K.-S., Kim, J. Y., Jeon, E., Choi, W. S., Kim, N. H., and Lee, K. Y. (2020). Evaluation of scalability and degree of fine-tuning of deep convolutional neural networks for covid-19 screening on chest x-ray images using explainable deep-learning algorithm. Journal of Personalized Medicine, 10(4), 213.

Liu, L., Yu, B., and Wu, W. (2019). The formation and effects of exploitative dynamic capabilities and explorative dynamic capabilities: An empirical study. Sustainability, 11(9), 2581.

Liu, Z., Roberts, R. A., Lal-Nag, M., Chen, X., Huang, R., and Tong, W. (2021). AI-based language models powering drug discovery and development. Drug Discovery Today, 26(11), 2593–2607.

Lund, B. (2023). The prompt engineering librarian. Library Hi Tech News, 40(8), 6–8.

Mäntylä, M. V., Graziotin, D., and Kuutila, M. (2018). The evolution of sentiment analysis—A review of research topics, venues, and top cited papers. Computer Science Review, 27, 16–32.

Montazeri, M., Galavi, Z., and Ahmadian, L. (2024). What are the applications of ChatGPT in healthcare: Gain or loss?. Health Science Reports, 7(2), e1878.

Moon, J., Ke, F., and Sokolikj, Z. (2020). Automatic assessment of cognitive and emotional states in virtual reality‐based flexibility training for four adolescents with autism. British Journal of Educational Technology, 51(5), 1766–1784.

Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., and Kasai, J. (2024). Large language models as tax attorneys: A case study in legal capabilities emergence. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270), 20230159.

Pokrivcakova, S. (2019). Preparing teachers for the application of AI-powered technologies in foreign language education. Journal of Language and Cultural Education, 7(3), 135–153.

Polverini, G., and Gregorcic, B. (2024). How understanding large language models can inform the use of ChatGPT in physics education. European Journal of Physics, 45(2), 025701.

Rahardjanto, A., and Husamah, H. (2024). Bioindicators for forest area condition: A systematic literature review. Jurnal Penelitian Dan Pengkajian Ilmu Pendidikan: E-Saintika, 8(1), 138–164.

Ramírez, J. G. C. (2024). Constructing executing and overcoming challenges in distributed ai systems: A study of federated learning framework. Journal of Artificial Intelligence General Science (JAIGS) ISSN:3006-4023, 3(1), 27–46.

Rodić‐Trmčić, B., Labus, A., Barać, D., Popović, S., and Radenković, B. (2018). Designing a course for smart healthcare engineering education. Computer Applications in Engineering Education, 26(3), 484–499.

Salathé, M., and Khandelwal, S. (2011). Assessing vaccination sentiments with online social media: Implications for infectious disease dynamics and control. PLoS Computational Biology, 7(10), e1002199.

Sarker, I. H., Furhad, M. H., and Nowrozy, R. (2021). AI-driven cybersecurity: An overview, security intelligence modeling and research directions. SN Computer Science, 2(3), 173.

Schlichtkrull, M., Kipf, T. N., Bloem, P., Van Den Berg, R., Titov, I., and Welling, M. (2018). Modeling Relational Data with Graph Convolutional Networks. In A. Gangemi, R. Navigli, M.-E. Vidal, P. Hitzler, R. Troncy, L. Hollink, A. Tordai, and M. Alam (Eds.), The Semantic Web, 10843, 593–607.

Shahin, M. A. (2016). State-of-the-art review of some artificial intelligence applications in pile foundations. Geoscience Frontiers, 7(1), 33–44.

Shang, C., Tang, Y., Huang, J., Bi, J., He, X., and Zhou, B. (2019). End-to-end structure-aware convolutional networks for knowledge base completion. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 3060–3067.

Siddhant, A., Johnson, M., Tsai, H., Ari, N., Riesa, J., Bapna, A., Firat, O., and Raman, K. (2020). Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 8854–8861.

Sorensen, T., Robinson, J., Rytting, C. M., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., and Wingate, D. (2022). An information-theoretic approach to prompt engineering without ground truth labels. In Muresan S., Nakov P., and Villavicencio A. (Eds.), Proc. Annu. Meet. Association for Computational Linguistics, 1, 819–862.

Suleiman, T. A., and Adinoyi, A. (2023). Telemedicine and smart healthcare—the role of artificial intelligence, 5G, cloud services, and other enabling technologies. International Journal of Communications, Network and System Sciences, 16(03), 31–51.

Syed, B., Verma, G., Srinivasan, B. V., Natarajan, A., and Varma, V. (2020). Adapting language models for non-parallel author-stylized rewriting. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 9008–9015.

Tao, D., Yang, P., and Feng, H. (2020). Utilization of text mining as a big data analysis tool for food science and nutrition. Comprehensive Reviews in Food Science and Food Safety, 19(2), 875–894.

Van De Guchte, M., Braaksma, M., Rijlaarsdam, G., and Bimmel, P. (2015). Learning new grammatical structures in task‐based language learning: The effects of recasts and prompts. The Modern Language Journal, 99(2), 246–262.

Van Noort, G., Himelboim, I., Martin, J., and Collinger, T. (2020). Introducing a model of automated brand-generated content in an era of computational advertising. Journal of Advertising, 49(4), 411–427.

VanGessel, F. G., Perry, E., Mohan, S., Barham, O. M., and Cavolowsky, M. (2023). Natural language processing for knowledge discovery and information extraction from energetics corpora. Propellants, Explosives, Pyrotechnics, 48(11), e202300109.

Wang, T., Lu, K., Chow, K. P., and Zhu, Q. (2020). COVID-19 sensing: Negative sentiment analysis on social media in China via BERT model. IEEE Access, 8, 138162–138169.

Wang, Y., Si, S., Li, D., Lukasik, M., Yu, F., Hsieh, C. J., and Kumar, S. (2022). Two-stage LLM fine-tuning with less specialization and more generalization. arXiv preprint arXiv:2211.00635, 2022, 1-19.

Wirzal, M. D. H., Nordin, N. A. H. M., Bustam, M. A., and Joselevich, M. (2022). Bibliometric analysis of research on scientific literacy between 2018 and 2022: Science education subject. International Journal of Essential Competencies in Education, 1(2), 69-83.

Wu, T., Khan, A., Yong, M., Qi, G., and Wang, M. (2022). Efficiently embedding dynamic knowledge graphs. Knowledge-Based Systems, 250, 109124.

Yang, X., Bian, J., Hogan, W. R., and Wu, Y. (2020). Clinical concept extraction using transformers. Journal of the American Medical Informatics Association, 27(12), 1935–1942.

Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., Zhang, Y., Magoc, T., Harle, C. A., Lipori, G., Mitchell, D. A., Hogan, W. R., Shenkman, E. A., Bian, J., and Wu, Y. (2022). A large language model for electronic health records. Npj Digital Medicine, 5(1), 194.

Yong, G., Jeon, K., Gil, D., and Lee, G. (2023). Prompt engineering for zero-shot and few-shot defect detection and classification using a visual-language pretrained model. Computer-Aided Civil and Infrastructure Engineering, 38(11), 1536–1554.

Yuan, Z., Zhang, L., and Duan, L. (2018). A novel fusion diagnosis method for rotor system fault based on deep learning and multi-sourced heterogeneous monitoring data. Measurement Science and Technology, 29(11), 115005.

Zhang, J., Budhdeo, S., William, W., Cerrato, P., Shuaib, H., Sood, H., Ashrafian, H., Halamka, J., and Teo, J. T. (2022). Moving towards vertically integrated artificial intelligence development. Npj Digital Medicine, 5(1), 143.

Zhang, L., Huang, Y., Yang, X., Yu, S., and Zhuang, F. (2022). An automatic short-answer grading model for semi-open-ended questions. Interactive Learning Environments, 30(1), 177–190.

Zhou, Z., Tam, V. W. L., and Lam, E. Y. (2021). SignBERT: A BERT-based deep learning framework for continuous sign language recognition. IEEE Access, 9, 161669–161682.




DOI: https://doi.org/10.17509/ijost.v9i2.71481

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 Universitas Pendidikan Indonesia

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Indonesian Journal of Science and Technology is published by UPI.
StatCounter - Free Web Tracker and Counter
View My Stats