Advancing a Speech Act-Based Model to Improve Future Quality of Information Security Policies Using Large Language Models

Fredrik Karlsson; Shang Gao; John Krogstie; Leila Aro-Sati

doi:10.7250/csimq.2026-46.03

Advancing a Speech Act-Based Model to Improve Future Quality of Information Security Policies Using Large Language Models

Authors

Fredrik Karlsson Department of Informatics, Örebro University, Örebro, 701 82 Örebro, Sweden https://orcid.org/0000-0002-3265-7627
Shang Gao Department of Informatics, Örebro University, Örebro, 701 82 Örebro, Sweden https://orcid.org/0000-0002-3722-6797
John Krogstie Department of Computer Science, Norwegian University of Science and Technology, Trondheim, NO-7491, Norway https://orcid.org/0000-0003-4830-1876
Leila Aro-Sati Department of Informatics, Örebro University, Örebro, 701 82 Örebro, Sweden https://orcid.org/0009-0001-9207-3236

DOI:

https://doi.org/10.7250/csimq.2026-46.03

Keywords:

Information Security Policy, Cybersecurity Policy, Speech Act, Large Language Model

Abstract

Employee compliance with Information Security Policies (ISPs) depends on communicating clear and comprehensible content. However, existing research has shown that many ISPs are of poor communicative quality. Large Language Models (LLMs) could enhance ISPs if fine-tuned on high-quality data, but to do such fine-tuning requires a conceptual model for classifying the data and evaluating the resulting text. Therefore, as a step in this direction, the aim of this article is to develop a conceptual model of ISPs using Speech Act Theory as a theoretical lens to enable assessments of the communicative quality of ISPs. We used conceptual modeling and document analysis to develop the model based on 600 ISP statements from ten British National Health Service ISPs. We used selected parts from the SEQUAL framework to evaluate the model. The evaluation pointed to potential areas for improving the model’s semantic, empirical, physical, and deontic qualities. By incorporating these improvements, the final class diagram contains 21 classes, six of which address ISP statement quality as speech acts.

References

B. Kör and B. Metin, “Understanding human aspects for an effective information security management implementation,” International Journal of Applied Decision Sciences, vol. 14, no. 2, pp. 105–122, 2021. Available: https://doi.org/10.1504/IJADS.2021.113532

M. J. Culnan and C. C. Williams, “How Ethics Can Enhance Organizational Privacy: Lessons from the Choicepoint and TJX Data Breaches,” MIS Quarterly, vol. 33, no. 4, pp. 673–687, 2009. Available: https://doi.org/10.2307/20650322

E. Kolkowska, F. Karlsson, and K. Hedström, “Towards analysing the rationale of information security noncompliance: Devising a Value-Based Compliance analysis method,” Journal of Strategic Information Systems, vol. 26, no. 1, pp. 39–57, 2017. Available: https://doi.org/10.1016/j.jsis.2016.08.005

G. Dhillon, Information Security – Text & Cases. Edition 2.0 ed. Burlington, USA: Prospect Press, 2017.

Truesec, “Threat Intelligence Report 2023,” Truesec, Stockholm, Sweden, 2023.

Crowdstrike, “2025 Global Threat Report,” Crowdstrike Inc, 2025.

S. Chatterjee, X. Gao, S. Sarkar, and C. Uzmanoglu, “Reacting to the scope of a data breach: The differential role of fear and anger,” Journal of Business Research, vol. 101, pp. 183–193, 2019. Available: https://doi.org/10.1016/j.jbusres.2019.04.024

K. D. Loch, H. H. Carr, and M. E. Warkentin, “Threats to information systems: today’s reality, yesterday’s understanding,” MIS Quarterly, vol. 16, no. 2, pp. 173–186, 1992. Available: https://doi.org/10.2307/249574

N. H. Chowdhury, M. T. Adam, and G. Skinner, “The impact of time pressure on cybersecurity behaviour: a systematic literature review,” Behav. Inf. Technol., vol. 38, no. 12, pp. 1290–1308, 2019. Available: https://doi.org/10.1080/0144929X.2019.1583769

K. Höne and J. H. P. Eloff, “Information security policy – what do international information security standards say?” Computers & Security, vol. 21, no. 5, pp. 402–409, 2002. Available: https://doi.org/10.1016/S0167-4048(02)00504-7

S. Goel and I. N. Chengalur-Smith, “Metrics for characterizing the form of security policies,” Journal of Strategic Information Systems, vol. 19, no. 4, pp. 281–295, 2010. Available: https://doi.org/10.1016/j.jsis.2010.10.002

R. Baskerville and M. Siponen, “An information security meta-policy for emergent organizations,” Logistics Information Management, vol. 15, no. 5/6, pp. 337–346, 2002. Available: https://doi.org/10.1108/09576050210447019

W. A. Cram, J. G. Proudfoot, and J. D’Arcy, “Organizational information security policies: a review and research framework,” European Journal of Information Systems, vol. 26, no. 6, pp. 605–641, 2017. Available: https://doi.org/10.1057/s41303-017-0059-9

M. E. Whitman, “Security Policy – From Design to Maintenance,” in Information Security – Policy, Processes, and Practices, 2008, pp. 123–151.

S. V. Flowerday and T. Tuyikeze, “Information security policy development and implementation: The what, how and who,” Computers & Security, vol. 61, pp. 169–183, 2016. Available: https://doi.org/10.1016/j.cose.2016.06.002

M. Siponen and A. Vance, “Neutralization: New Insights into the Problem of Employee Information Systems Security Policy Violations,” MIS Quarterly, vol. 34, no. 3, pp. 487–502, 2010. Available: https://doi.org/10.2307/25750688

Ponemon, “Cost of insider threats global report,” Ponemon Institute, North Traverse City, 2020. Available: https://www.ibm.com/downloads/documents/us-en/107a02e94cc8f836

E. Rostami, F. Karlsson, E. Kolkowska, and S. Gao, “Towards software for tailoring information security policies to organisations’ different target groups,” Computers & Security, vol. 159, Article 104687, 2025. Available: https://doi.org/10.1016/j.cose.2025.104687

K. Höne and J. H. P. Eloff, “What makes an effective information security policy?” Network Security, vol. 6, no. 1, pp. 14–16, 2002. Available: https://doi.org/10.1016/S1353-4858(02)06011-7

I. Lopes and P. Oliveira, “Applying Action Research in the Formulation of Information Security Policies,” in New Contributions in Information Systems and Technologies. Advances in Intelligent Systems and Computing, vol. 353, pp. 513–522, 2015. Available: https://doi.org/10.1007/978-3-319-16486-1_50

F. Karlsson, K. Hedström, and G. Goldkuhl, “Practice-based discourse analysis of information security policies,” Computers & Security, vol. 67, pp. 267–279, 2017. Available: https://doi.org/10.1016/j.cose.2016.12.012

E. Rostami and F. Karlsson, “Qualitative Content Analysis of Actionable Advice in Information Security Policies – Introducing the Keyword Loss of Specificity Metric,” Information & Computer Security, vol. 32, no. 4, pp. 492–508, 2024. Available: https://doi.org/10.1108/ICS-10-2023-0187

B. C. Stahl, N. F. Doherty, and M. Shaw, “Information security policies in the UK healthcare sector: a critical evaluation,” Information Systems Journal, vol. 22, pp. 77–94, 2012. Available: https://doi.org/10.1111/j.1365-2575.2011.00378.x

ISO, “ISO/IEC 27002:2022 Information security, cybersecurity and privacy protection – Information security controls,” International Organization for Standardization (ISO), 2022.

C. Sundt, “Information security and the law,” Information Security Technical Report, vol. 11, no. 1, pp. 2–9, 2006. Available: https://doi.org/10.1016/j.istr.2005.11.003

I. Trummer, “From BERT to GPT-3 codex: harnessing the potential of very large language models for data management,” in Proceedings of the VLDB Endowment, vol. 15, no. 12, 2022, pp. 3770–3773. Available: https://doi.org/10.14778/3554821.3554896

M. Abdullah, A. Madain, and Y. Jararweh, “ChatGPT: Fundamentals, Applications and Social Impacts,” 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS), 2022, pp. 1–8. Available: https://doi.org/10.1109/SNAMS58071.2022.10062688

F. Karlsson, S. Gao, J. Krogstie, and L. Aro-Sati, “Towards a Speech Act-Based Model to Enable Future Quality Improvements of Information Security Policies Using Large Language Models,” Perspectives in Business Informatics Research. BIR 2025. Lecture Notes in Business Information Processing, vol. 562, 2025, pp. 349–364. Available: https://doi.org/10.1007/978-3-032-04375-7_22

J. R. Searle, “A Classification of Illocutionary Acts,” Language in Society, vol. 5, no. 1, pp. 1–23, 1976. Available: https://doi.org/10.1017/S0047404500006837

M. Alshaikh, S. B. Maynard, A. Ahmad, and S. Chang, “Information Security Policy: A Management Practice Perspective,” Australasian Conference on Information Systems, 2015.

N. Doherty and H. Fulford, “Aligning the information security policy with the strategic information systems plan,” Computer & Security, vol. 25, no. 1, pp. 55–63, 2006. Available: https://doi.org/10.1016/j.cose.2005.09.009

E. Rostami, F. Karlsson, and G. Shang, “Policy components – a conceptual model for modularizing and tailoring of information security policies,” Information & Computer Security, vol. 31, no. 3, pp. 331–352, 2023. Available: https://doi.org/10.1108/ICS-10-2022-0160

E. Rostami, M. Hanif, F. Karlsson, and S. Gao, “Defining Actionable Advice in Information Security Policies - Guiding Employees to Strengthen Digital Sovereignty of Organizations,” Procedia Computer Science, vol. 254, pp. 30–38, 2025. Available: https://doi.org/10.1016/j.procs.2025.02.061

E. Rostami, F. Karlsson, and S. Gao, “Requirements for computerized tools to design information security policies,” Computers & Security, vol. 99, Article 102063, 2020. Available: https://doi.org/10.1016/j.cose.2020.102063

S. Diver, “Information Security Policy – A Development Guide for Large and Small Companies,” SANS Institute, 2021.

NIST, “Information Security Handbook: A Guide for Managers,” National Institute of Standards and Technology, Gaithersburg, USA, 2006.

T. R. Peltier, Information Security Policies and Procedures – A Practitioner’s Reference. Second Edition, Boca Raton, 2004. Available: https://doi.org/10.1201/9780203488737

C. R. Smith, “The Definitive Guide to Writing Effective Information Security Policies and Procedures,” Createspace, 2010.

Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly,” High-Confidence Computing, vol. 4, no. 2, Article 100211, 2024. Available: https://doi.org/10.1016/j.hcc.2024.100211

J. Yang et al., “Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond,” ACM Transactions on Knowledge Discovery from Data, vol. 18, no. 6, pp. 1–32, 2024. Available: https://doi.org/10.1145/3649506

L. Yun, S. Yun, and H. Xue, “Improving citizen-government interactions with generative artificial intelligence: Novel human-computer interaction strategies for policy understanding through large language models,” PLoS ONE, vol. 19, no. 12, 2024. Available: https://doi.org/10.1371/journal.pone.0311410

S. Lawal, X. Zhao, A. Rios, R. Krishnan, and D. Ferraiolo, “Translating Natural Language Specifications into Access Control Policies by Leveraging Large Language Models,” 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), pp. 361–370, 2024. Available: https://doi.org/10.1109/TPS-ISA62245.2024.00048

E. Quevedo et al., “Creation and Analysis of a Natural Language Understanding Dataset for DoD Cybersecurity Policies (CSIAC-DoDIN V1. 0),” 2023 International Conference on Computational Science and Computational Intelligence (CSCI), pp. 91–98, 2023. Available: https://doi.org/10.1109/CSCI62032.2023.00021

S. Deldari et al., “AuditNet: A Conversational AI-based Security Assistant,” Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction (MobileHCI ’24 Adjunct), pp. 1–4, 2024. Available: https://doi.org/10.1145/3640471.3680444

D. Najafali, J. M. Camacho, E. Reiche, L. G. Galbraith, S. D. Morrison, and A. H. Dorafshar, “Truth or lies? The pitfalls and limitations of ChatGPT in systematic review creation,” Aesthetic Surgery Journal, vol. 43, no. 8, pp. NP654-NP655, 2023. Available: https://doi.org/10.1093/asj/sjad108

G. Goldkuhl and E. Braf, “Organisational Ability: Constituents and Congruencies,” in Knowledge Management in the SocioTechnical World, pp. 30–42, 2002. Available: https://doi.org/10.1007/978-1-4471-0187-1_4

J. L. Austin, How to do Things with Words. Cambridge: Oxford University Press, 1962.

J. R. Searle, Speech Acts: An Essay in the Philosophy of Language. Cambridge: Cambridge University Press, 1969. Available: https://doi.org/10.1017/CBO9781139173438

J. R. Searle and D. Vanderveken, “Speech acts and illocutionary logic,” in Logic, Thought and Action. Logic, Epistemology, and the Unity of Science, vol. 2, pp. 109–132, 1985. Available: https://doi.org/10.1007/1-4020-3167-X_5

T. Holtgraves, “The production and perception of implicit performatives,” Journal of Pragmatics, vol. 37, no. 12, pp. 2024–2043, 2005. Available: https://doi.org/10.1016/j.pragma.2005.03.005

J. R. Searle, Expression and Meaning: Studies in the Theory of Speech Acts. Cambridge: Cambridge University Press, 1979. Available: https://doi.org/10.1017/CBO9780511609213

R. Gasparatou, “How to do things with words: Speech acts in education,” Educational Philosophy and Theory, vol. 50, no. 5, pp. 510–518, 2018. Available: https://doi.org/10.1080/00131857.2017.1382353

J. V. Schmidt, “Can Artificial Agents be Authors?” Philosophy & Technology, vol. 38, no. 1, pp. 1–25, 2025. Available: https://doi.org/10.1007/s13347-025-00839-y

O. I. Lindland, G. Sindre, and A. Solvberg, “Understanding quality in conceptual modeling,” IEEE Software, vol. 11, no. 2, pp. 42–49, 1994. Available: https://doi.org/10.1109/52.268955

J. Krogstie, Quality in Business Process Modeling. Springer, 2016. Available: https://doi.org/10.1007/978-3-319-42512-2

J. Krogstie, Model-Based Development and Evolution of Information Systems: A Quality Approach. Springer, 2012. Available: https://doi.org/10.1007/978-1-4471-2936-3

B. Thalheim, “The Science of Conceptual Modelling,” Database and Expert Systems Applications. DEXA 2011. Lecture Notes in Computer Science, vol. 6860, 2011, pp. 12–26. Available: https://doi.org/10.1007/978-3-642-23088-2_2

G. A. Bowen, “Document Analysis as a Qualitative Research Method,” Qualitative Research Journal, vol. 9, no. 2, pp. 27–40, 2009. Available: https://doi.org/10.3316/QRJ0902027

M. E. Duffy, “Methodological triangulation: a vehicle for merging quantitative and qualitative research methods,” Image: The Journal of Nursing Scholarship, vol. 19, no. 3, pp. 130–133, 1987. Available: https://doi.org/10.1111/j.1547-5069.1987.tb00609.x

E. Rostami, “Tailoring information security policies - computerized tool and a design theory,” Ph.D. dissertation, Department of Informatics, Örebro University, Örebro, 2023.

E. G. Guba and Y. S. Lincoln, Fourth Generation Evaluation. SAGE Publications, 1989.

Y. S. Lincoln and E. G. Guba, Naturalistic Inquiry. Sage Publications, 1985.

D. S. Collingridge and E. E. Gantt, “The Quality of Qualitative Research,” American Journal of Medical Quality, vol. 23, no. 5, pp. 389–395, 2008, Available: https://doi.org/10.1177/1062860608320646