• Complex
  • Title
  • Keyword
  • Abstract
  • Scholars
  • Journal
  • ISSN
  • Conference
成果搜索

author:

Wei, Yifan (Wei, Yifan.) [1] | Su, Yisong (Su, Yisong.) [2] | Ma, Huanhuan (Ma, Huanhuan.) [3] | Yu, Xiaoyan (Yu, Xiaoyan.) [4] | Lei, Fangyu (Lei, Fangyu.) [5] | Zhang, Yuanzhe (Zhang, Yuanzhe.) [6] | Zhao, Jun (Zhao, Jun.) [7] | Liu, Kang (Liu, Kang.) [8]

Indexed by:

EI

Abstract:

Large language models (LLMs) have shown nearly saturated performance on many natural language processing (NLP) tasks. As a result, it is natural for people to believe that LLMs have also mastered abilities such as time understanding and reasoning. However, research on the temporal sensitivity of LLMs has been insufficiently emphasized. To fill this gap, this paper constructs Multiple Sensitive Factors Time QA (MenatQA), which encompasses three temporal factors (scope factor, order factor, counterfactual factor) with total 2,853 samples for evaluating the time comprehension and reasoning abilities of LLMs. This paper tests current mainstream LLMs with different parameter sizes, ranging from billions to hundreds of billions. The results show most LLMs fall behind smaller temporal reasoning models with different degree on these factors. In specific, LLMs show a significant vulnerability to temporal biases and depend heavily on the temporal information provided in questions. Furthermore, this paper undertakes a preliminary investigation into potential improvement strategies by devising specific prompts and leveraging external tools. These approaches serve as valuable baselines or references for future research endeavors. © 2023 Association for Computational Linguistics.

Keyword:

Computational linguistics Natural language processing systems Statistical tests

Community:

  • [ 1 ] [Wei, Yifan]The Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA, China
  • [ 2 ] [Wei, Yifan]University of Chinese Academy of Sciences, China
  • [ 3 ] [Su, Yisong]The Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA, China
  • [ 4 ] [Su, Yisong]Fuzhou University, China
  • [ 5 ] [Ma, Huanhuan]University of Chinese Academy of Sciences, China
  • [ 6 ] [Yu, Xiaoyan]The Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA, China
  • [ 7 ] [Yu, Xiaoyan]Beijing Institute of Technology, China
  • [ 8 ] [Lei, Fangyu]The Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA, China
  • [ 9 ] [Lei, Fangyu]University of Chinese Academy of Sciences, China
  • [ 10 ] [Zhang, Yuanzhe]The Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA, China
  • [ 11 ] [Zhang, Yuanzhe]University of Chinese Academy of Sciences, China
  • [ 12 ] [Zhao, Jun]The Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA, China
  • [ 13 ] [Zhao, Jun]University of Chinese Academy of Sciences, China
  • [ 14 ] [Liu, Kang]The Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA, China
  • [ 15 ] [Liu, Kang]University of Chinese Academy of Sciences, China
  • [ 16 ] [Liu, Kang]Shanghai Artificial Intelligence Laboratory, China

Reprint 's Address:

Email:

Show more details

Related Keywords:

Related Article:

Source :

Year: 2023

Page: 1434-1447

Language: English

Cited Count:

WoS CC Cited Count:

SCOPUS Cited Count:

ESI Highly Cited Papers on the List: 0 Unfold All

WanFang Cited Count:

Chinese Cited Count:

30 Days PV: 2

Online/Total:100/10109299
Address:FZU Library(No.2 Xuyuan Road, Fuzhou, Fujian, PRC Post Code:350116) Contact Us:0591-22865326
Copyright:FZU Library Technical Support:Beijing Aegean Software Co., Ltd. 闽ICP备05005463号-1