What causes SSD write timeout over 120 seconds?

Question

system:CentOS Linux release 7.6.1810 (Core)

kernel:Linux l-db11.tj.cn5 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

SSD:Dera

cpu: 24 core

memory: 128G

The operating system can only write at a very small rate。

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme0n1           0.00     0.00    2.00  130.00     0.03     0.91    14.67     0.00    0.03    0.00    0.03   0.03   0.40
nvme0n1           0.00     0.00    0.00  122.00     0.00     0.84    14.05     0.00    0.03    0.00    0.03   0.03   0.40
nvme0n1           0.00     0.00   16.00  201.00     0.25     3.83    38.53     0.01    0.03    0.06    0.03   0.03   0.60
nvme0n1           0.00     0.00    8.00  204.00     0.12     1.12    12.03     0.01    0.03    0.12    0.03   0.03   0.70
nvme0n1           0.00     0.00    0.00  154.00     0.00     1.05    13.95     0.00    0.01    0.00    0.01   0.01   0.20
nvme0n1           0.00     0.00    1.00  162.00     0.02     1.07    13.65     0.01    0.04    0.00    0.04   0.04   0.70

memory:

free -g
              total        used        free      shared  buff/cache   available
Mem:            125          58          50           3          16          62
Swap:            63           0          63

dmesg:

[Mon Mar 25 14:40:24 2024] INFO: task mysqld:25801 blocked for more than 120 seconds.
[Mon Mar 25 14:40:24 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:24 2024] mysqld          D ffff8d94ebbdb0c0     0 25801  23578 0x00000080
[Mon Mar 25 14:40:24 2024] Call Trace:
[Mon Mar 25 14:40:24 2024]  [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:24 2024]  [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:24 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:24 2024]  [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:24 2024]  [<ffffffffc05778e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
[Mon Mar 25 14:40:24 2024]  [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:24 2024]  [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:25 2024]  [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:25 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:25 2024] INFO: task mysqld:25802 blocked for more than 120 seconds.
[Mon Mar 25 14:40:25 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:25 2024] mysqld          D ffff8d94f0b85140     0 25802  23578 0x00000080
[Mon Mar 25 14:40:25 2024] Call Trace:
[Mon Mar 25 14:40:25 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:25 2024]  [<ffffffffa536a27d>] rwsem_down_read_failed+0x10d/0x1a0
[Mon Mar 25 14:40:25 2024]  [<ffffffffc05695d0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4f87078>] call_rwsem_down_read_failed+0x18/0x30
[Mon Mar 25 14:40:25 2024]  [<ffffffffa5367f00>] down_read+0x20/0x40
[Mon Mar 25 14:40:25 2024]  [<ffffffffc056955c>] xfs_ilock+0xdc/0x120 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc05695d0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc054f9ca>] __xfs_get_blocks+0x9a/0x6d0 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4c79912>] ? get_user_pages_fast+0x122/0x1a0
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0550e24>] xfs_get_blocks_direct+0x14/0x20 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e848f6>] do_blockdev_direct_IO+0xf96/0x20a0
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e85a55>] __blockdev_direct_IO+0x55/0x60
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0550e30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055c759>] xfs_file_dio_aio_write+0x179/0x340 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc0550e30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:26 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:26 2024] INFO: task mysqld:25803 blocked for more than 120 seconds.
[Mon Mar 25 14:40:26 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:26 2024] mysqld          D ffff8d94f0b830c0     0 25803  23578 0x00000080
[Mon Mar 25 14:40:26 2024] Call Trace:
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:26 2024]  [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:26 2024]  [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:26 2024]  [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:26 2024]  [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:26 2024]  [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:26 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:26 2024] INFO: task mysqld:25804 blocked for more than 120 seconds.
[Mon Mar 25 14:40:26 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:26 2024] mysqld          D ffff8d94ea39e180     0 25804  23578 0x00000080
[Mon Mar 25 14:40:27 2024] Call Trace:
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:27 2024]  [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:27 2024]  [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:27 2024]  [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:27 2024]  [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:27 2024] INFO: task mysqld:25809 blocked for more than 120 seconds.
[Mon Mar 25 14:40:27 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:27 2024] mysqld          D ffff8d94b53b0000     0 25809  23578 0x00000080
[Mon Mar 25 14:40:27 2024] Call Trace:
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:27 2024]  [<ffffffffa536a27d>] rwsem_down_read_failed+0x10d/0x1a0
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055b1b4>] ? xfs_file_fsync+0xc4/0x1c0 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4f87078>] call_rwsem_down_read_failed+0x18/0x30
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055b0f0>] ? xfs_file_readdir+0x40/0x40 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5367f00>] down_read+0x20/0x40
[Mon Mar 25 14:40:27 2024]  [<ffffffffc056955c>] xfs_ilock+0xdc/0x120 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055b1b4>] xfs_file_fsync+0xc4/0x1c0 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e762e7>] do_fsync+0x67/0xb0
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e765d0>] SyS_fsync+0x10/0x20
[Mon Mar 25 14:40:28 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27

Is there something wrong with the OS or XFS?