环境说明
OS操作系统：SUSE Linux 11
数据库版 本：MYSQL 5.6
故障问题描述
客户反映MYSQL数据库因为内存不足被重启。
故障分析
1、根据MYSQL报警日志确认问题：
2016-10-24 12:28:49 29226 [Note] Plugin
‘FEDERATED’ is disabled.
2016-10-24 12:28:49 29226 [Note] InnoDB:
Using atomics to ref count buffer pool pages
2016-10-24 12:28:49 29226 [Note] InnoDB: The
InnoDB memory heap is disabled
2016-10-24 12:28:49 29226 [Note] InnoDB:
Mutexes and rw_locks use GCC atomic builtins
2016-10-24 12:28:49 29226 [Note] InnoDB:
Compressed tables use zlib 1.2.3
2016-10-24 12:28:49 29226 [Note] InnoDB:
Using Linux native AIO
2016-10-24 12:28:49 29226 [Note] InnoDB:
Using CPU crc32 instructions
2016-10-24 12:28:49 29226 [Note] InnoDB:
Initializing buffer pool, size = 2.0G
2016-10-24 12:28:50 29226 [Note] InnoDB:
Completed initialization of buffer pool
2016-10-24 12:28:50 29226 [Note] InnoDB:
Highest supported file format is Barracuda.
2016-10-24 12:28:50 29226 [Note] InnoDB: Log
scan progressed past the checkpoint lsn 27852365485
2016-10-24 12:28:50 29226 [Note] InnoDB:
Database was not shutdown normally!
2016-10-24 12:28:50 29226 [Note] InnoDB:
Starting crash recovery.
2016-10-24 12:28:50 29226 [Note] InnoDB:
Reading tablespace information from the .ibd files…
2016-10-24 12:28:50 29226 [Note] InnoDB:
Restoring possible half-written data pages
2016-10-24 12:28:50 29226 [Note] InnoDB:
from the doublewrite buffer…
InnoDB: Doing recovery: scanned up to log
sequence number 27852370215
2016-10-24 12:28:51 29226 [Note] InnoDB:
Starting an apply batch of log records to the database…
InnoDB: Progress in percent: 13 14 15 16 17 18
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96
97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0
401322767, file name TSPTONLDB01-it_mgmad-bin.000023
2016-10-24 12:28:52 29226 [Note] InnoDB: 128
rollback segment(s) are active.
2016-10-24 12:28:52 29226 [Note] InnoDB:
5.6.19 started; log sequence number 27852370215
2016-10-24 12:28:52 29226 [Note] Recovering
after a crash using /dbdata/it_mgmad/binlog/it_mgmad/TSPTONLDB01-it_mgmad-bin
2016-10-24 12:29:00 29226 [Note] Starting
crash recovery…
2016-10-24 12:29:00 29226 [Note] Crash
recovery finished.
2016-10-24 12:29:06 29226 [Note] Server
hostname (bind-address): ‘*’; port: 60002
2016-10-24 12:29:06 29226 [Note] IPv6 is
available.
2016-10-24 12:29:06 29226 [Note] – ‘::’ resolves to ‘::’;
2016-10-24 12:29:06 29226 [Note] Server
socket created on IP: ‘::’.
2016-10-24 12:29:06 29226 [Warning]
‘proxies_priv’ entry ‘@ root@tsptonldb01’ ignored in –skip-name-resolve mode.
2016-10-24 12:29:06 29226 [Note] Event
Scheduler: Loaded 0 events
2016-10-24 12:29:06 29226 [Note]
/usr/local/mysql/bin/mysqld: ready for connections.
Version: ‘5.6.19-upsql-0.1.1-log’ socket: ‘/etc/mysock/it_mgmad.sock’ port: 60002 UpSQL Server
2016-10-24 12:30:21 29226 [ERROR]
/usr/local/mysql/bin/mysqld: Out of memory (Needed 8160 bytes)
2016-10-24 12:30:21 29226 [ERROR]
Out of memory; check if mysqld or some other process uses all available memory;
if not, you may have to use ‘ulimit’ to allow mysqld to use more memory or you
can add more swap space
161024 12:31:35 mysqld_safe Number of
processes running now: 0
161024 12:31:35 mysqld_safe mysqld restarted
2016-10-24 12:31:37 29659
[Warning] Buffered warning: Could not increase number of max_open_files to more
than 10240 (request: 25000)
从上面的信息，我们看到， MSYQL报内存不足的错误无法启动。
2. 操作系统日志提示错误
Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996482]
Mem-Info:
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996483] Node 0 DMA per-cpu:
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996485] CPU 0: hi: 0, btch: 1 usd: 0
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996486] CPU 1: hi: 0, btch: 1 usd: 0
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996487] Node 0 DMA32 per-cpu:
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996488] CPU 0: hi: 186, btch: 31 usd: 0
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996489] CPU 1: hi: 186, btch: 31 usd: 5
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996490] Node 0 Normal per-cpu:
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996491] CPU 0: hi: 186, btch: 31 usd: 30
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996492] CPU 1: hi: 186, btch: 31 usd: 144
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996495] active_anon:658762 inactive_anon:260562 isolated_anon:0
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996496] active_file:353
inactive_file:1716 isolated_file:0
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996496] unevictable:0 dirty:6
writeback:5 unstable:0
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996497] free:21800
slab_reclaimable:2214 slab_unreclaimable:10742
Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996498] mapped:765 shmem:456 pagetables:6664 bounce:0
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996499] Node 0 DMA free:15880kB min:256kB low:320kB high:384kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996505] lowmem_reserve[]: 0 3000 4010 4010
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996507] Node 0 DMA32 free:54356kB min:50368kB low:62960kB
high:75552kB active_anon:2275764kB inactive_anon:569032kB active_file:12kB
inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:12kB
slab_reclaimable:1896kB slab_unreclaimable:7060kB kernel_stack:2904kB
pagetables:6588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:611
all_unreclaimable? yes
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996513] lowmem_reserve[]: 0 0 1010 1010
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996515] Node 0 Normal free:16964kB min:16956kB low:21192kB
high:25432kB active_anon:359284kB inactive_anon:473216kB active_file:1400kB
inactive_file:6776kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:1034240kB mlocked:0kB dirty:24kB writeback:20kB mapped:3028kB
shmem:1812kB slab_reclaimable:6960kB slab_unreclaimable:35908kB
kernel_stack:3624kB pagetables:20068kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:12878 all_unreclaimable? yes
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996520] lowmem_reserve[]: 0 0 0 0
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996522] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB
0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996544] Node 0 DMA32: 1308*4kB 973*8kB 448*16kB 181*32kB 91*64kB
43*128kB 11*256kB 8*512kB 8*1024kB 1*2048kB 0*4096kB = 54456kB
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996549] Node 0 Normal: 439*4kB 303*8kB 169*16kB 83*32kB 18*64kB
5*128kB 4*256kB 3*512kB 3*1024kB 0*2048kB 0*4096kB = 16964kB
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996553] 38343 total pagecache pages
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996554] 35726 pages in swap cache
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996556] Swap cache stats: add 5506519, delete 5470793, find
22920240/23085658
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996557] Free swap = 0kB
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317509.996557] Total swap = 2097148kB
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006955] 1048560 pages RAM
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006957] 67010 pages reserved
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006958] 1978 pages shared
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006959] 953768 pages non-shared
Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006960]
[ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006974] [ 571] 0 571 4223 1 1 -17 -1000 udevd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006981] [ 1689] 0 1689 998 1 0 0 0 acpid
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006983] [ 1704] 100 1704 5444 117 0 0 0 dbus-daemon
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006986] [ 1722] 0 1722 4824 62 0 0 0 syslog-ng
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006988] [ 1725] 0 1725 1065 0 0 0 0 klogd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006991] [10038] 0
10038 31835 168 1 0 0 vmtoolsd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006994] [10096] 0
10096 15193 1 0 0 0 VGAuthService
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006996] [10185] 101
10185 8608 146 0 0 0 hald
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.006999] [10188] 0
10188 27278 0 0 0 0 console-kit-dae
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007001] [10189] 0
10189 4505 2 0 0 0 hald-runner
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007004] [10284] 0
10284 8690 0 0 0 0 hald-addon-inpu
Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007007]
[10308] 0 10308 8690 17 0 0 0 hald-addon-stor
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007009] [10312] 101
10312 9129 0 1 0 0 hald-addon-acpi
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007012] [10313] 0
10313 8690 16 0 0 0 hald-addon-stor
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007014] [12046] 0
12046 7409 8 0 -17 -1000 auditd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007017] [12048] 0
12048 3606 0 0 0 0 audispd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007020] [12081] 0
12081 4723 16 0 0 0 rpcbind
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007022] [12083] 0
12083 2088 17 0 0 0 haveged
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007025] [12140] 0
12140 4216 1 1 -17 -1000 udevd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007027] [12141] 0
12141 4216 1 0 -17 -1000 udevd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007029] [12224] 0
12224 6759 1 1 0 0 mcelog
Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007031]
[12701] 0 12701 17296 28 0 -17 -1000 sshd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007034] [12715] 0
12715 17933 0 1 0 0 cupsd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007036] [12730] 0
12730 59557 71 0 0 0 nscd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007038] [12798] 0
12798 9552 28 0 0 0 master
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007041] [12825] 51
12825 10052 31 1 0 0 qmgr
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007043] [12864] 0
12864 16711 47 1 0 0 gdm
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007046] [12878] 0 12878 27377 121 0 0 0 X
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007048] [12885] 0
12885 3697 15 0 0 0 cron
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007051] [12924] 0
12924 1121 2 1 0 0 mingetty
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007055] [12925] 0
12925 1121 2 1 0 0 mingetty
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007058] [12926] 0
12926 1121 2 0 0 0 mingetty
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007061] [12927] 0
12927 1121 2 1 0 0 mingetty
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007064] [12928] 0
12928 1121 2 1 0 0 mingetty
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007067] [12929] 0
12929 1121 2 0 0 0 mingetty
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007071] [13064] 107
13064 5548 0 1 0 0 dbus-launch
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007074] [13065] 107
13065 3362 0 1 0 0 dbus-daemon
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007077] [13066] 107
13066 39551 82 0 0 0 gnome-session
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007080] [13079] 107
13079 9863 67 0 0 0 gconfd-2
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007083] [13083] 107
13083 31861 0 1 0 0 at-spi-registry
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007086] [13084] 107
13084 70831 2912 0 0 0 gnome-settings-
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007090] [13086] 107
13086 20207 0 1 0 0 bonobo-activati
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007092] [13094] 107
13094 57536 135 0 0 0 gnome-power-man
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007095] [13096] 107
13096 10854 0 0 0 0 gvfsd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007098] [13097] 107
13097 54246 0 0 0 0 metacity
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007101] [21349] 1018
21349 2857 2 1 0 0 mysqld_safe
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007105] [12777] 1016
12777 2857 2 0 0 0 mysqld_safe
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007109] [18450] 0
18450 13384 49 1 0 0 gdm-simple-slav
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007112] [18452] 0
18452 66809 13462 1 0 0 X
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007115] [18489] 107
18489 5548 0 1 0 0 dbus-launch
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007118] [18518] 0
18518 24613 0 1 0 0 gdm-session-wor
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007121] [18540] 0
18540 39584 89 1 0 0 gnome-session
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007124] [18613] 0
18613 4178 19 0 0 0 gpg-agent
Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007126]
[18619] 0 18619 5548 0 1 0 0 dbus-launch
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007129] [18620] 0
18620 3429 0 0 0 0 dbus-daemon
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007132] [18623] 0
18623 9887 123 1 0 0 gconfd-2
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007135] [18627] 0
18627 91696 332 0 0 0 gnome-settings-
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007138] [18631] 0
18631 23814 1 1 0 0 gnome-keyring-d
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007141] [18635] 0
18635 10908 0 1 0 0 gvfsd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007144] [18639] 0
18639 16822 19 0 0 0 gvfs-fuse-daemo
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007147] [18645] 0
18645 51782 400 1 0 0 metacity
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007149] [18650] 0
18650 93659 621 1 0 0 gnome-panel
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007152] [18651] 0
18651 135905 13563 1 0 0 nautilus
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007155] [18653] 0
18653 36624 0 0 0 0 bonobo-activati
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007160] [18659] 0
18659 82105 455 1 0 0 main-menu
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007171] [18662] 0
18662 12083 0 1 0 0 gvfsd-trash
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007174] [18666] 0
18666 13609 1 0 0 0 gvfs-hal-volume
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007178] [18668] 0
18668 15774 1 1 0 0 gvfs-gphoto2-vo
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007181] [18677] 0
18677 75456 1 1 0 0 gnome-volume-co
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007184] [18681] 0
18681 56947 1 0 0 0 python
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007187] [18686] 0
18686 39298 113 0 0 0 vmtoolsd
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007189] [18688] 0
18688 57564 125 0 0 0 gnome-power-man
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007192] [18692] 0
18692 38020 114 1 0 0 gnome-screensav
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007195] [18693] 0
18693 48186 0 1 0 0 pulseaudio
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007198] [18718] 0
18718 10962 0 0 0 0 gvfsd-burn
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007201] [21417] 0
21417 59168 142 1 0 0 application-bro
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007204] [21420] 0
21420 57129 199 0 0 0 baobab
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007208] [21463] 0
21463 35883 1 0 0 0 gnomesu
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007211] [21464] 0
21464 13445 4 0 0 0 gnomesu-pam-bac
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007214] [21471] 0
21471 3221 1 0 0 0 yast2
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007217] [21501] 0
21501 118060 114 0 0 0 y2base
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007221] [21621] 0
21621 55974 123 0 0 0 gnome-terminal
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007224] [21622] 0
21622 1616 0 0 0 0 gnome-pty-helpe
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007227] [21623] 0
21623 3505 1 0 0 0 bash
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007230] [21649] 0
21649 9244 1 0 0 0 gvfsd-metadata
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007233] [31142] 1016
31142 1100222 219744 1 0 0 mysqld
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007236] [31181] 1018
31181 1039509 163992 1 0 0 mysqld
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007239] [ 699] 1017 699 2856 98 0 0 0 mysqld_safe
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007242] [ 1625] 1017 1625 1076677 414298 1 0 0 mysqld
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007246] [ 1861] 1019 1861 641372 48223 1 0 0 java
Oct 25 13:41:07 TSPTONLDB01 kernel:
[14317510.007284] [14971] 51
14971 10014 128 0 0 0 pickup
Oct 25 13:41:07 TSPTONLDB01
kernel: [14317510.007287] Out of memory: Kill process 31142 (mysqld) score 283
or sacrifice child
Oct 25 13:41:07 TSPTONLDB01
kernel: [14317510.007358] Killed process 31142 (mysqld) total-vm:4400888kB,
anon-rss:878976kB, file-rss:0kB
Oct 25 13:45:45 TSPTONLDB01
kernel: [14317788.351765] vmtoolsd invoked oom-killer: gfp_mask=0x201da,
order=0, oom_adj=0, oom_score_adj=0
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351772] vmtoolsd cpuset=/ mems_allowed=0
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351777] Pid: 10038, comm: vmtoolsd Tainted: G X 3.0.13-0.27-default #1
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351780] Call Trace:
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351803] [<ffffffff810048b5>] dump_trace+0x75/0x300
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351814] [<ffffffff8143ea0f>] dump_stack+0x69/0x6f
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351825] [<ffffffff810f2b6f>] dump_header+0x8f/0x1f0
Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351834] [<ffffffff810f3235>]
oom_kill_process+0x95/0x180
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351841] [<ffffffff810f346c>] out_of_memory+0x14c/0x230
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351848] [<ffffffff810f8efb>] __alloc_pages_slowpath+0x7ab/0x7f0
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351854] [<ffffffff810f90f1>] __alloc_pages_nodemask+0x1b1/0x1c0
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351863] [<ffffffff811312b5>] alloc_pages_current+0xa5/0x120
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351871] [<ffffffff810f08f2>] filemap_fault+0x172/0x3d0
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351878] [<ffffffff8111916a>] __do_fault+0x7a/0x5d0
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351884] [<ffffffff8111975d>]
handle_pte_fault+0x9d/0x230
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351893] [<ffffffff81445327>] do_page_fault+0x207/0x4c0
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.351900] [<ffffffff81442065>] page_fault+0x25/0x30
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.353040] DWARF2 unwinder stuck at page_fault+0x25/0x30
Oct 25 13:45:45 TSPTONLDB01 kernel:
[14317788.353042]
通过日志发现发现在10月24日到11月份出现多次内存不足，操作系统 KILL 掉MYSQLD进程的情况。
在该服务器上有3个MYSQL实例，每个MYSQL实例的 INNODB 缓冲池设置为2GB，但操作系统的物理内存
只有4GB，另外在服务器上还有一个数据同步的 JAVA进程。结合日志分析，MYSQL 内存分配不合理，建议把服务器内存扩展到10GB。
建议MYSQL INNODB池计算方法： INNODB 缓冲池内存= 操作系统物理内存 – （预留给操作系统的内存1-2GB）
– （JAVA进程的内存）–（MYSQL最大连接数*2MB）
