Graphical Abstract Figure
Abstract
Intelligent systems have been rapidly evolving and play a pivotal role in assisting individuals across diverse domains, from healthcare to transportation. Understanding the dynamics of human–artificial intelligence (AI) partnering, particularly how humans trust and collaborate with intelligent systems, is becoming increasingly critical to design effective systems. This paper presents an experimental analysis to assess the impact of AI design attributes on users’ trust, workload, and performance when solving classification problems supported by an AI assistant. Specifically, we study the effect of transparency, fairness, and robustness in the design of an AI assistant and analyze the role of participants’ gender and education background on the outcomes. The experiment is conducted with 47 students in undergraduate, master’s, and Ph.D. programs using a drawing game application where the users are asked to recognize incomplete sketches revealed progressively while receiving recommendations from multiple versions of an AI assistant. The results show that when collaborating with the AI, participants achieve a higher performance than their individual performance or the performance of the AI. The results also show that gender does not have an impact on users’ trust and performance when collaborating with different versions of the AI system, whereas education level has a significant impact on the participants’ performance but not on trust. Finally, the impact of design attributes on participants’ trust and performance highly depends on the accuracy of the AI recommendations, and improvements in participants’ performance and trust in some cases come at the expense of increased workload.
References
1.
Asan
, O.
, Bayrak
, A. E.
, and Choudhury
, A.
, 2020
, “Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians
,” J. Med. Int. Res.
, 22
(6
), p. e15154
. 2.
Bayrak
, A. E.
, McComb
, C.
, Cagan
, J.
, and Kotovsky
, K.
, 2021
, “A Strategic Decision-Making Architecture Toward Hybrid Teams for Dynamic Competitive Problems
,” Decis. Supp. Syst.
, 144
, p. 113490
. 3.
Saadi
, J. I.
, and Yang
, M. C.
, 2023
, “Generative Design: Reframing the Role of the Designer in Early-Stage Design Process
,” ASME J. Mech. Des.
, 145
(4
), p. 041411
. 4.
Chaudhari
, A. M.
, and Selva
, D.
, 2023
, “Evaluating Designer Learning and Performance in Interactive Deep Generative Design
,” ASME J. Mech. Des.
, 145
(5
), p. 051403
. 5.
Song
, B.
, Soria Zurita
, N. F.
, Nolte
, H.
, Singh
, H.
, Cagan
, J.
, and McComb
, C.
, 2022
, “When Faced With Increasing Complexity: The Effectiveness of Artificial Intelligence Assistance for Drone Design
,” ASME J. Mech. Des.
, 144
(2
), p. 021701
. 6.
Gyory
, J. T.
, Stump
, G.
, Nolte
, H.
, and Cagan
, J.
, 2024
, “Adaptation Through Communication: Assessing Human-Artificial Intelligence Partnership for the Design of Complex Engineering Systems
,” ASME J. Mech. Des.
, 146
(8
), p. 081401
. 7.
Parasuraman
, R.
, and Riley
, V.
, 1997
, “Humans and Automation: Use, Misuse, Disuse, Abuse
,” Human Fact.
, 39
(2
), pp. 230
–253
. 8.
Lee
, J. D.
, and Moray
, N.
, 1994
, “Trust, Self-Confidence, and Operators’ Adaptation to Automation
,” Int. J. Hum-Comput. Stud.
, 40
(1
), pp. 153
–184
. 9.
Deng
, M.
, Chen
, J.
, Wu
, Y.
, Ma
, S.
, Li
, H.
, Yang
, Z.
, and Shen
, Y.
, 2024
, “Using Voice Recognition to Measure Trust During Interactions With Automated Vehicles
,” Appl. Ergon.
, 116
, p. 104184
. 10.
Lotfalian Saremi
, M.
, and Bayrak
, A. E.
, 2021
, “A Survey of Important Factors in Human- Artificial Intelligence Trust for Engineering Design
,” The ASME International Design Engineering Technical Conferences & Computers and Information in Engineering Conference. Virtual, Paper No: DETC2021-70550.11.
Wright
, J. L.
, Chen
, J. Y.
, and Lakhmani
, S. G.
, 2019
, “Agent Transparency and Reliability in Human-Robot Interaction: The Influence on User Confidence and Perceived Reliability
,” IEEE Trans. Human-Mach. Syst.
, 50
(3
), pp. 254
–263
. 12.
Mehrabi
, N.
, Morstatter
, F.
, Saxena
, N.
, Lerman
, K.
, and Galstyan
, A.
, 2019
, “A Survey on Bias and Fairness in Machine Learning
,” ACM Comput. Surv. (CSUR)
, 54
(6
), p. 115
. 13.
Guo
, A.
, Kamar
, E.
, Vaughan
, J. W.
, Wallach
, H.
, and Morris
, M. R.
, 2019
, “Toward Fairness in AI for People with Disabilities: A Research Roadmap
,” ACM SIGACCESS Accessibility and Computing
, 125
, p. 2
. 14.
Venkatesh
, V.
, Morris
, M. G.
, and Ackerman
, P. L.
, 2000
, “A Longitudinal Field Investigation of Gender Differences in Individual Technology Adoption Decision-Making Processes
,” Organ. Behav. Human Decis. Process.
, 83
(1
), pp. 33
–60
. 15.
Li
, S.
, Glass
, R.
, and Records
, H.
, 2008
, “The Influence of Gender on New Technology Adoption and Use-Mobile Commerce
,” J. Inter. Comm.
, 7
(2
), pp. 270
–289
. 16.
Felzmann
, H.
, Fosch-Villaronga
, E.
, Lutz
, C.
, and Tamo-Larrieux
, A.
, 2019
, “Robots and Transparency: The Multiple Dimensions of Transparency in the Context of Robot Technologies
,” IEEE Rob. Autom. Mag.
, 26
(2
), pp. 71
–78
. 17.
Marangunić
, N.
, and Granić
, A.
, 2015
, “Technology Acceptance Model: A Literature Review From 1986 to 2013
,” Univ. Access Infor. Soc.
, 14
, pp. 81
–95
. 18.
Liu
, K.
, Or
, C. K.
, So
, M.
, Cheung
, B.
, Chan
, B.
, Tiwari
, A.
, and Tan
, J.
, 2022
, “A Longitudinal Examination of Tablet Self-Management Technology Acceptance by Patients With Chronic Diseases: Integrating Perceived Hand Function, Perceived Visual Function, and Perceived Home Space Adequacy With the TAM and TPB
,” Appl. Ergon.
, 100
, p. 103667
. 19.
Goodhue
, D. L.
, and Thompson
, R. L.
, 1995
, “Task-Technology Fit and Individual Performance
,” MIS Q.
, 19
(2
), pp. 213
–236
. 20.
Dishaw
, M. T.
, and Strong
, D. M.
, 1999
, “Extending the Technology Acceptance Model With Task–Technology Fit Constructs
,” Inf. Manage.
, 36
(1
), pp. 9
–21
. 21.
Ajzen
, I.
, 1991
, “The Theory of Planned Behavior
,” Organ. Behav. Human Decis. Process.
, 50
(2
), pp. 179
–211
. 22.
Conner
, M.
, 2020
, “Theory of Planned Behavior,” Handbook of Sport Psychology
, 4th ed., G.
Tenenbaum
, and R. C.
Eklund
, eds., John Wiley & Sons, Ltd
, Hoboken, NJ
, pp. 1
–18
. 23.
McKnight
, D. H.
, and Chervany
, N. L.
, 2000
, “What Is Trust? A Conceptual Analysis and an Interdisciplinary Model
,” Americas Conference on Information Systems (AMCIS)
, Long Beach, CA
, Aug. 10–13
, pp. 827
–833
.24.
Lewis
, J. D.
, and Weigert
, A.
, 1985
, “Trust as a Social Reality
,” Soc. Forces
, 63
(4
), pp. 967
–985
. 25.
Shapiro
, S. P.
, 1987
, “The Social Control of Impersonal Trust
,” Am. J. Sociol.
, 93
(3
), pp. 623
–658
. 26.
Mayer
, R. C.
, Davis
, J. H.
, and Schoorman
, F. D.
, 1995
, “An Integrative Model of Organizational Trust
,” Acad. Manage. Rev.
, 20
(3
), pp. 709
–734
. 27.
Mcknight
, D. H.
, Carter
, M.
, Thatcher
, J. B.
, and Clay
, P. F.
, 2011
, “Trust in a Specific Technology: An Investigation of Its Components and Measures
,” ACM Trans. Manage. Inf. Syst. (TMIS)
, 2
(2
), pp. 1
–25
. 28.
Aljazzaf
, Z. M.
, Perry
, M.
, and Capretz
, M. A.
, 2010
, “Online Trust: Definition and Principles
,” Fifth International Multi-conference on Computing in the Global Information Technology
, Valencia, Spain
, Sept. 20–25
, IEEE
, pp. 163
–168
.29.
Ghazizadeh
, M.
, Lee
, J. D.
, and Boyle
, L. N.
, 2012
, “Extending the Technology Acceptance Model to Assess Automation
,” Cog. Technol. Work
, 14
(1
), pp. 39
–49
. 30.
Hoff
, K. A.
, and Bashir
, M.
, 2015
, “Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
,” Human Fact.
, 57
(3
), pp. 407
–434
. 31.
Riedl
, R.
, and Javor
, A.
, 2012
, “The Biology of Trust: Integrating Evidence From Genetics, Endocrinology, and Functional Brain Imaging.
,” J. Neurosci. Psychol. Econ.
, 5
(2
), p. 63
. 32.
Madhavan
, P.
, and Wiegmann
, D. A.
, 2004
, “A New Look at the Dynamics of Human-Automation Trust: Is Trust in Humans Comparable to Trust in Machines?
” Proc. Hum. Factors Ergon. Soc. Annu. Meet.
, 48
(3
), pp. 581
–585
. 33.
Siau
, K.
, and Wang
, W.
, 2018
, “Building Trust in Artificial Intelligence, Machine Learning, and Robotics
,” Cutter Bus. Technol. J.
, 31
(2
), pp. 47
–53
.34.
McClumpha
, A.
, James
, M.
, Green
, R.
, and Belyavin
, A.
, 1991
, “Pilots’ Attitudes to Cockpit Automation
,” Proc. Hum. Factors Ergon. Soc. Annu. Meet.
, 35
(2
), pp. 107
–111
. 35.
Araujo
, T.
, Helberger
, N.
, Kruikemeier
, S.
, and De Vreese
, C. H.
, 2020
, “In AI We Trust? Perceptions About Automated Decision-Making by Artificial Intelligence
,” AI Soc.
, 35
, pp. 1
–13
. 36.
Lotfalian Saremi
, M.
, and Bayrak
, A. E.
, 2022
, “Agent-Based Simulation of Optimal Trust in a Decision Support System in One-on-One Collaboration
,” The ASME International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, St. Louis, MO
, Paper No. DETC2021-87388.37.
Chong
, L.
, Raina
, A.
, Goucher-Lambert
, K.
, Kotovsky
, K.
, and Cagan
, J.
, 2023
, “The Evolution and Impact of Human Confidence in Artificial Intelligence and in Themselves on AI-Assisted Decision-Making in Design
,” ASME J. Mech. Des.
, 145
(3
), p. 031401
. 38.
Chong
, L.
, Kotovsky
, K.
, and Cagan
, J.
, 2024
, “Human Designers’ Dynamic Confidence and Decision-Making When Working With More Than One AI
,” ASME J. Mech. Des.
, 146
(8
), p. 081402
. 39.
Beller
, J.
, Heesen
, M.
, and Vollrath
, M.
, 2013
, “Improving the Driver–Automation Interaction: An Approach Using Automation Uncertainty
,” Human Fact.
, 55
(6
), pp. 1130
–1141
. 40.
Brown
, T. M.
, and Miller
, C. E.
, 2000
, “Communication Networks in Task-Performing Groups: Effects of Task Complexity, Time Pressure, and Interpersonal Dominance
,” Small Group Res.
, 31
(2
), pp. 131
–157
. 41.
Williams
, T. M.
, 1999
, “The Need for New Paradigms for Complex Projects
,” Int. J. Proj. Manage.
, 17
(5
), pp. 269
–273
. 42.
Lee
, J. D.
, and See
, K. A.
, 2004
, “Trust in Automation: Designing for Appropriate Reliance
,” Human Fact.
, 46
(1
), pp. 50
–80
. 43.
Rempel
, J. K.
, Holmes
, J. G.
, and Zanna
, M. P.
, 1985
, “Trust in Close Relationships
,” J. Person. Soc. Psychol.
, 49
(1
), p. 95
. 44.
Biros
, D. P.
, Daly
, M.
, and Gunsch
, G.
, 2004
, “The Influence of Task Load and Automation Trust on Deception Detection
,” Group Decis. Negot.
, 13
(2
), pp. 173
–189
. 45.
Schmidt
, P.
, and Biessmann
, F.
, 2020
, “Calibrating Human-AI Collaboration: Impact of Risk, Ambiguity and Transparency on Algorithmic Bias
,” 4th International Cross-Domain Conference for Machine Learning and Knowledge Extraction (CD-MAKE)
, Dublin, Ireland
, Aug. 25–28
, Springer, pp. 431–449
.46.
Zhang
, G.
, Raina
, A.
, Brownell
, E.
, and Cagan
, J.
, 2023
, “Artificial Intelligence Impersonating a Human: The Impact of Design Facilitator Identity on Human Designers
,” ASME J. Mech. Des.
, 145
(5
), p. 051404
. 47.
Gutman
, D.
, Olatunji
, S. A.
, Markfeld
, N.
, Givati
, S.
, Sarne-Fleischmann
, V.
, Oron-Gilad
, T.
, and Edan
, Y.
, 2023
, “Evaluating Levels of Automation With Different Feedback Modes in an Assistive Robotic Table Clearing Task for Eldercare
,” Appl. Ergon.
, 106
, p. 103859
. 48.
Felzmann
, H.
, Fosch-Villaronga
, E.
, Lutz
, C.
, and Tamò-Larrieux
, A.
, 2020
, “Towards Transparency by Design for Artificial Intelligence
,” Sci. Eng. Ethics
, 26
(6
), pp. 3333
–3361
. 49.
Meijer
, A.
, 2014
, “Transparency,” The Oxford Handbook of Public Accountability
, 1st ed., M.
Bovens
, R.
Goodin
, and T.
Schillemans
, eds., Oxford University Press
, Oxford, UK
, p. 507
.50.
Miller
, T.
, 2019
, “Explanation in Artificial Intelligence: Insights From the Social Sciences
,” Artif. Intell.
, 267
, pp. 1
–38
. 51.
Castelnovo
, A.
, Crupi
, R.
, Greco
, G.
, Regoli
, D.
, Penco
, I. G.
, and Cosentini
, A. C.
, 2022
, “A Clarification of the Nuances in the Fairness Metrics Landscape
,” Sci. Rep.
, 12
(1
), p. 4209
. 52.
Mourelatos
, Z. P.
, and Liang
, J.
, 2005
, “A Methodology for Trading-Off Performance and Robustness Under Uncertainty
,” ASME J. Mech. Des.
, 128
(4
), pp. 856
–863
. 53.
Rauber
, J.
, Zimmermann
, R.
, Bethge
, M.
, and Brendel
, W.
, 2020
, “Foolbox Native: Fast Adversarial Attacks to Benchmark the Robustness of Machine Learning Models in PyTorch, Tensorflow, and JAX
,” J. Open Sourc. Softw.
, 5
(53
), p. 2607
. 54.
Taori
, R.
, Dave
, A.
, Shankar
, V.
, Carlini
, N.
, Recht
, B.
, and Schmidt
, L.
, 2020
, “Measuring Robustness to Natural Distribution Shifts in Image Classification
,” Adv. Neural Inf. Process. Syst.
, 33
, pp. 18583
–18599
.55.
Papenmeier
, A.
, Englebienne
, G.
, and Seifert
, C.
, 2019
, “How Model Accuracy and Explanation Fidelity Influence User Trust in AI
,” IJCAI Workshop on Explainable Artificial Intelligence (XAI)
, Macau, China
, Aug. 11
, pp. 94
–100
.56.
McGuirl
, J. M.
, and Sarter
, N. B.
, 2006
, “Supporting Trust Calibration and the Effective Use of Decision Aids by Presenting Dynamic System Confidence Information
,” Human Fact.
, 48
(4
), pp. 656
–665
. 57.
Schmidt
, P.
, Biessmann
, F.
, and Teubner
, T.
, 2020
, “Transparency and Trust in Artificial Intelligence Systems
,” J. Decis. Syst.
, 29
(4
), pp. 260
–278
. 58.
Zhou
, J.
, Verma
, S.
, Mittal
, M.
, and Chen
, F.
, 2021
, “Understanding Relations Between Perception of Fairness and Trust in Algorithmic Decision Making
,” 8th International Conference on Behavioral and Social Computing (BESC)
, Doha, Qatar
, Oct. 29–31
, IEEE, pp. 1
–5
.59.
Angerschmid
, A.
, Zhou
, J.
, Theuermann
, K.
, Chen
, F.
, and Holzinger
, A.
, 2022
, “Fairness and Explanation in AI-Informed Decision Making
,” Mach. Learn. Knowl. Extract.
, 4
(2
), pp. 556
–579
. 60.
Lee
, M. K.
, and Rich
, K.
, 2021
, “Who Is Included in Human Perceptions of AI?: Trust and Perceived Fairness Around Healthcare AI and Cultural Mistrust
,” Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
, Yokohama, Japan
, May 8–13
, pp. 1
–14
.61.
Wang
, B.
, Zhou
, J.
, Li
, Y.
, and Chen
, F.
, 2023
, “Impact of Fidelity and Robustness of Machine Learning Explanations on User Trust
,” 36th Australasian Joint Conference on Artificial Intelligence
, Brisbane, Australia
, Nov. 28–Dec. 1
, Springer, pp. 209
–220
.62.
Li
, B.
, Qi
, P.
, Liu
, B.
, Di
, S.
, Liu
, J.
, Pei
, J.
, Yi
, J.
, and Zhou
, B.
, 2023
, “Trustworthy AI: From Principles to Practices
,” ACM Comput. Surv.
, 55
(9
), pp. 1
–46
. 63.
Plotly, 2012. Dash-Plotly.
64.
Google, 2018. Google Quickdraw Dataset.
65.
Springer
, A.
, and Whittaker
, S.
, 2019
, “Progressive Disclosure: Empirically Motivated Approaches to Designing Effective Transparency
,” 24th International Conference on Intelligent User Interfaces
, Marina del Ray, CA
, Mar. 17–20
, pp. 107
–120
.66.
Gill
, H. S.
, and Khehra
, B. S.
, 2022
, “An Integrated Approach Using CNN-RNN-LSTM for Classification of Fruit Images
,” Mater. Today: Proc.
, 51
(1
), pp. 591
–595
. 67.
Muhuri
, P. S.
, Chatterjee
, P.
, Yuan
, X.
, Roy
, K.
, and Esterline
, A.
, 2020
, “Using a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) to Classify Network Attacks
,” Information
, 11
(5
), p. 243
. 68.
Sakinah
, N.
, Tahir
, M.
, Badriyah
, T.
, and Syarif
, I.
, 2019
, “LSTM With ADAM Optimization-Powered High Accuracy Preeclampsia Classification
,” International Electronics Symposium (IES)
, Surabaya, Indonesia
, Sept. 27–28
, IEEE, pp. 314–319
.69.
Hart
, S. G.
, and Staveland
, L. E.
, 1988
, “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research,” Human Mental Workload
, P. A.
Hancock
, and N.
Meshkati
, eds., Elsevier
, North-Holland
, pp. 139
–183
. 
