• Complex
  • Title
  • Keyword
  • Abstract
  • Scholars
  • Journal
  • ISSN
  • Conference
成果搜索

author:

Ke, Yusong (Ke, Yusong.) [1] | Lin, Hongru (Lin, Hongru.) [2] | Ruan, Yuting (Ruan, Yuting.) [3] | Tang, Junya (Tang, Junya.) [4] | Li, Li (Li, Li.) [5]

Indexed by:

SCIE

Abstract:

Large language models (LLMs) are increasingly adopted in medical question answering (QA) scenarios. However, LLMs have been proven to generate hallucinations and nonfactual information, undermining their trustworthiness in high-stakes medical tasks. Conformal Prediction (CP) is now recognized as a robust framework within the broader domain of machine learning, offering statistically rigorous guarantees of marginal (average) coverage for prediction sets. However, the applicability of CP in medical QA remains to be explored. To address this limitation, this study proposes an enhanced CP framework for medical multiple-choice question answering (MCQA) tasks. The enhanced CP framework associates the non-conformance score with the frequency score of the correct option. The framework generates multiple outputs for the same medical query by leveraging self-consistency theory. The proposed framework calculates the frequency score of each option to address the issue of limited access to the model's internal information. Furthermore, a risk control framework is incorporated into the enhanced CP framework to manage task-specific metrics through a monotonically decreasing loss function. The enhanced CP framework is evaluated on three popular MCQA datasets using off-the-shelf LLMs. Empirical results demonstrate that the enhanced CP framework achieves user-specified average (or marginal) error rates on the test set. Moreover, the results show that the test set's average prediction set size (APSS) decreases as the risk level increases. It is concluded that it is a promising evaluation metric for the uncertainty of LLMs.

Keyword:

average prediction set size conformal prediction large language models medical multiple-choice question answering

Community:

  • [ 1 ] [Ke, Yusong]Tongji Univ, Sch Elect & Informat Engn, Shanghai 201804, Peoples R China
  • [ 2 ] [Li, Li]Tongji Univ, Sch Elect & Informat Engn, Shanghai 201804, Peoples R China
  • [ 3 ] [Lin, Hongru]Hunan Univ, Sch Elect & Informat Engn, Changsha 410082, Peoples R China
  • [ 4 ] [Ruan, Yuting]Fuzhou Univ, Sch Econ & Management, Fuzhou 350100, Peoples R China
  • [ 5 ] [Tang, Junya]Tongji Univ, Sch Comp Sci & Technol, Shanghai 200070, Peoples R China

Reprint 's Address:

  • [Tang, Junya]Tongji Univ, Sch Comp Sci & Technol, Shanghai 200070, Peoples R China

Show more details

Related Keywords:

Source :

MATHEMATICS

Year: 2025

Issue: 9

Volume: 13

2 . 3 0 0

JCR@2023

CAS Journal Grade:4

Cited Count:

WoS CC Cited Count:

SCOPUS Cited Count:

ESI Highly Cited Papers on the List: 0 Unfold All

WanFang Cited Count:

Chinese Cited Count:

30 Days PV: 2

Online/Total:99/10105325
Address:FZU Library(No.2 Xuyuan Road, Fuzhou, Fujian, PRC Post Code:350116) Contact Us:0591-22865326
Copyright:FZU Library Technical Support:Beijing Aegean Software Co., Ltd. 闽ICP备05005463号-1