One of those exciting Bugs that hang up your database and leave you stranded. At 6:30 and the workload hang up on the databases, every query local or remote hangs up with no outcome.
Examination of MySQL error log reveled the following cool error messages:
InnoDB: Warning: a long semaphore wait:
--Thread 140178284349184 has waited at dict0dict.c line 747 for 262.00 seconds the semaphore:
Mutex at 0x2c6dc9e8 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 140178284054272 has waited at dict0dict.c line 747 for 257.00 seconds the semaphore:
Mutex at 0x2c6dc9e8 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 140178283759360 has waited at dict0dict.c line 747 for 256.00 seconds the semaphore:
Mutex at 0x2c6dc9e8 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 140178283464448 has waited at buf0buf.c line 1837 for 256.00 seconds the semaphore:
Mutex at 0x3ed5558 '&buf_pool->LRU_list_mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 140178283169536 has waited at dict0dict.c line 747 for 255.00 seconds the semaphore:
Mutex at 0x2c6dc9e8 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 140178282874624 has waited at dict0dict.c line 747 for 255.00 seconds the semaphore:
Mutex at 0x2c6dc9e8 '&dict_sys->mutex', lock var 1
waiters flag 1
Turns know bug in MySQL 5.5.24 and 5.5.25 to fix the issue upgrade your databases to MySQL 5.5.27 the same is true for the MariaDB.
This is related to the BUG https://bugs.launchpad.net/percona-server/+bug/1026926