4

Currently troubleshooting a MySQL server spending a lot of time in futex syscalls. All databases are InnoDB and seeing high load on my 16 core machine, and slightly more than half of CPU time is in system.

Running strace on a process shows it spending a lot of time in futex

root@db1:~# strace -c -p 5026
Process 5026 attached
^CProcess 5026 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.95    9.583778        3651      2625       897 futex
  0.02    0.001987          86        23           poll
  0.01    0.001349         337         4           pwrite
  0.01    0.000702         351         2           access
  0.01    0.000561           5       121           sendto
  0.00    0.000410           3       125        23 recvfrom
  0.00    0.000082          41         2           setsockopt
  0.00    0.000071          36         2           getpeername
  0.00    0.000000           0        74           read
  0.00    0.000000           0        18           write
  0.00    0.000000           0         4           open
  0.00    0.000000           0         6           close
  0.00    0.000000           0        14         1 lstat
  0.00    0.000000           0        45           lseek
  0.00    0.000000           0         7           pread
  0.00    0.000000           0         5           sched_yield
  0.00    0.000000           0         2           shutdown
  0.00    0.000000           0         2           unlink
  0.00    0.000000           0         3         3 readlink
------ ----------- ----------- --------- --------- ----------------
100.00    9.588940                  3084       924 total

The part of my config relevant to InnoDB is as follows, innodb_thread_concurrency is set to 16 which matches the CPU count so I'm surprised it's waiting for futex as much as it is:

innodb_buffer_pool_size = 26000M
innodb_log_file_size = 1250M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_flush_neighbors = 0
innodb_log_files_in_group = 3
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 16
innodb_file_io_threads = 4
innodb_lock_wait_timeout = 180
innodb_file_per_table
innodb_max_dirty_pages_pct = 90

What other sort of thing could cause long waits for futex? Is it possible there are issues with our host which could be causing this?

Improve this question
1
  • Futex is a locking mechanism used for threads to be able to either lock or synchronise state. If they are waiting on futexes, then some thread or threads are locking said futex and taking a long time to do whatever they are doing before unlocking and letting other threads do their thing. On a DB this usually means contention on disk or (if replicating to other nodes) network. Check basics to find the bottleneck first, such as iotop, iostat, etc.
    – Robert Glonek
    Commented Jun 23 at 23:40

1 Answer 1

Reset to default
0 
[mysqld]
innodb_buffer_pool_size = 26000M
innodb_log_file_size = 1250M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_flush_neighbors = 0
innodb_log_files_in_group = 3
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 0
innodb_file_io_threads = 8
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_buffer_pool_instances = 16
innodb_spin_wait_delay = 96
innodb_io_capacity = 2000
innodb_lock_wait_timeout = 180
innodb_file_per_table
innodb_max_dirty_pages_pct = 90
Improve this answer
New contributor
warashi nguyen is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .