霸王硬上弓's Blog

朋友有台云主机（概念…），今天打电话给我说数据库出错。内容如下：
1030 Got error 28 from storage engine
in:
[select banners_id, banners_title, banners_image, banners_html_text, banners_open_new_windows, banners_url from banners where status = 1 and ( banners_group = 'BannerTwo') order by rand()]


根据搜索表述，是目录满了。
使用df -h命令查看，发现/根目录的剩余空间为0。总共系统盘容量才20G。
root@10-4-29-105:/lib/modules# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 19G 0 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 991M 4.0K 991M 1% /dev
tmpfs 201M 784K 200M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1001M 0 1001M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vdb 20G 8.9G 9.7G 48% /data

那么是什么情况导致空间无缘无故的缩水?使用 du -h -x –max-depth=1 命令可以查看每个目录下文件 文件夹的占用体积。
root@10-4-29-105:/# du -h -x –max-depth=1
4.0K ./mnt
11M ./sbin
5.9M ./etc
5.9G ./var
4.0K ./srv
950M ./boot
4.0K ./lib64
8.0K ./media
4.0K ./boot_ucloud
4.0K ./opt
36K ./root
5.5G ./usr
16K ./lost+found
396K ./tmp
9.7M ./bin
6.2G ./lib
68K ./home
19G .

归类了一下，有如下3个文件夹占用最多。
5.5G ./usr
6.2G ./lib
5.9G ./var

首先需要关注的是lib。正常情况下lib并不会占用这么多空间。那么瞧瞧吧。依旧使用 du -h -x –max-depth=1 命令 最后发现modules文件夹占用几乎全部。
root@10-4-29-105:/lib/modules# du -h -x –max-depth=1
185M ./3.13.0-85-generic
185M ./3.13.0-105-generic
185M ./3.13.0-98-generic
185M ./3.13.0-96-generic
184M ./3.13.0-77-generic
184M ./3.13.0-79-generic
184M ./3.13.0-63-generic
184M ./3.13.0-58-generic
184M ./3.13.0-62-generic
184M ./3.13.0-57-generic
184M ./3.13.0-76-generic
184M ./3.13.0-71-generic
184M ./3.13.0-67-generic
185M ./3.13.0-88-generic
185M ./3.13.0-100-generic
4.0K ./3.13.0-24-generic
184M ./3.13.0-61-generic
185M ./3.13.0-91-generic
185M ./3.13.0-86-generic
185M ./3.13.0-95-generic
185M ./3.13.0-65-generic
185M ./3.13.0-93-generic
184M ./3.13.0-59-generic
184M ./3.13.0-49-generic
185M ./3.13.0-106-generic
184M ./3.13.0-68-generic
184M ./3.13.0-74-generic
185M ./3.13.0-83-generic
185M ./3.13.0-87-generic
184M ./3.13.0-66-generic
185M ./3.13.0-92-generic
185M ./3.13.0-103-generic
185M ./3.13.0-101-generic
184M ./3.13.0-55-generic
6.0G .

可以看出基本都是内核文件。既然这么多内核文件，到底使用的是哪个？ cat /proc/version or uname -a会告知你。

Linux 10-4-29-105 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

使用 dpkg –get-selections|grep linux-image 命令 告诉你已安装了哪些内核。
root@10-4-29-105:/lib/modules# dpkg --get-selections|grep linux-image
linux-image-3.13.0-100-generic install
linux-image-3.13.0-101-generic install
linux-image-3.13.0-103-generic install
linux-image-3.13.0-105-generic install
linux-image-3.13.0-106-generic install
linux-image-3.13.0-49-generic install
linux-image-3.13.0-55-generic install
linux-image-3.13.0-57-generic install
linux-image-3.13.0-58-generic install
linux-image-3.13.0-59-generic install
linux-image-3.13.0-61-generic install
linux-image-3.13.0-62-generic install
linux-image-3.13.0-63-generic install
linux-image-3.13.0-65-generic install
linux-image-3.13.0-66-generic install
linux-image-3.13.0-67-generic install
linux-image-3.13.0-68-generic install
linux-image-3.13.0-71-generic install
linux-image-3.13.0-74-generic install
linux-image-3.13.0-76-generic install
linux-image-3.13.0-77-generic install
linux-image-3.13.0-79-generic install
linux-image-3.13.0-83-generic install
linux-image-3.13.0-85-generic install
linux-image-3.13.0-86-generic install
linux-image-3.13.0-87-generic install
linux-image-3.13.0-88-generic install
linux-image-3.13.0-91-generic install
linux-image-3.13.0-92-generic install
linux-image-3.13.0-93-generic install
linux-image-3.13.0-95-generic install
linux-image-3.13.0-96-generic install
linux-image-3.13.0-98-generic install
linux-image-extra-3.13.0-100-generic install
linux-image-extra-3.13.0-101-generic install
linux-image-extra-3.13.0-103-generic install
linux-image-extra-3.13.0-105-generic install
linux-image-extra-3.13.0-106-generic install
linux-image-extra-3.13.0-49-generic install
linux-image-extra-3.13.0-55-generic install
linux-image-extra-3.13.0-57-generic install
linux-image-extra-3.13.0-58-generic install
linux-image-extra-3.13.0-59-generic install
linux-image-extra-3.13.0-61-generic install
linux-image-extra-3.13.0-62-generic install
linux-image-extra-3.13.0-63-generic install
linux-image-extra-3.13.0-65-generic install
linux-image-extra-3.13.0-66-generic install
linux-image-extra-3.13.0-67-generic install
linux-image-extra-3.13.0-68-generic install
linux-image-extra-3.13.0-71-generic install
linux-image-extra-3.13.0-74-generic install
linux-image-extra-3.13.0-76-generic install
linux-image-extra-3.13.0-77-generic install
linux-image-extra-3.13.0-79-generic install
linux-image-extra-3.13.0-83-generic install
linux-image-extra-3.13.0-85-generic install
linux-image-extra-3.13.0-86-generic install
linux-image-extra-3.13.0-87-generic install
linux-image-extra-3.13.0-88-generic install
linux-image-extra-3.13.0-91-generic install
linux-image-extra-3.13.0-92-generic install
linux-image-extra-3.13.0-93-generic install
linux-image-extra-3.13.0-95-generic install
linux-image-extra-3.13.0-96-generic install
linux-image-extra-3.13.0-98-generic install
linux-image-generic install

那么这么多内核，不用的就全部清理掉。
摘抄：
Ubuntu 的内核映像文件主要包括以下类型的包：

linux-image-<版本号> ：内核映像文件
linux-image-extra-<版本号> ：扩展内核模块
linux-headers-<版本号> ：内核头文件

确定好要删除的内核版本，删之~
sudo apt-get purge linux-image-3.13.0-55
sudo apt-get purge linux-headers-3.13.0-55

自动全删掉。。。记住别把正在使用的内核删了。。同时也要留下最近最新的一个，之后更新启动内核为最新的。
sudo apt-get purge -y linux-image-3.13.0-100 linux-headers-3.13.0-100 linux-image-3.13.0-101 linux-headers-3.13.0-101 linux-image-3.13.0-103 linux-headers-3.13.0-103 linux-image-3.13.0-105 linux-headers-3.13.0-105 linux-image-3.13.0-49 linux-headers-3.13.0-49 linux-image-3.13.0-55 linux-headers-3.13.0-55 linux-image-3.13.0-57 linux-headers-3.13.0-57 linux-image-3.13.0-58 linux-headers-3.13.0-58 linux-image-3.13.0-59 linux-headers-3.13.0-59 linux-image-3.13.0-61 linux-headers-3.13.0-61 linux-image-3.13.0-62 linux-headers-3.13.0-62 linux-image-3.13.0-63 linux-headers-3.13.0-63 linux-image-3.13.0-65 linux-headers-3.13.0-65 linux-image-3.13.0-66 linux-headers-3.13.0-66 linux-image-3.13.0-67 linux-headers-3.13.0-67 linux-image-3.13.0-68 linux-headers-3.13.0-68 linux-image-3.13.0-71 linux-headers-3.13.0-71 linux-image-3.13.0-76 linux-headers-3.13.0-76 linux-image-3.13.0-77 linux-headers-3.13.0-77 linux-image-3.13.0-79 linux-headers-3.13.0-79 linux-image-3.13.0-83 linux-headers-3.13.0-83 linux-image-3.13.0-85 linux-headers-3.13.0-85 linux-image-3.13.0-86 linux-headers-3.13.0-86 linux-image-3.13.0-87 linux-headers-3.13.0-87 linux-image-3.13.0-88 linux-headers-3.13.0-88 linux-image-3.13.0-91 linux-headers-3.13.0-91 linux-image-3.13.0-92 linux-headers-3.13.0-92 linux-image-3.13.0-93 linux-headers-3.13.0-93 linux-image-3.13.0-95 linux-headers-3.13.0-95 linux-image-3.13.0-96 linux-headers-3.13.0-96 linux-image-3.13.0-98 linux-headers-3.13.0-98

…………………………

查看 目录，只剩下了正在用的和最新更新的内核。
root@10-4-29-105:/lib/modules# du -h -x –max-depth=1
4.0K ./3.13.0-24-generic
185M ./3.13.0-106-generic
184M ./3.13.0-74-generic
369M .

此时再查看剩余空间，居然腾出了一大半。
root@10-4-29-105:/lib/modules# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 9.2G 9.5G 50% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 991M 4.0K 991M 1% /dev
tmpfs 201M 784K 200M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1001M 0 1001M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vdb 20G 8.9G 9.7G 48% /data
root@10-4-29-105:/lib/modules#

卸载完成后更新下GRUB配置。sudo update-grub2

root@10-4-29-105:/lib/modules# sudo update-grub2
Generating grub configuration file …
Found linux image: /boot/vmlinuz-3.13.0-106-generic
Found initrd image: /boot/initrd.img-3.13.0-106-generic
Found linux image: /boot/vmlinuz-3.13.0-74-generic
Found initrd image: /boot/initrd.img-3.13.0-74-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
root@10-4-29-105:/lib/modules#

回到根目录，再来看看各个目录的占用情况：
root@10-4-29-105:/# du -h -x –max-depth=1
4.0K ./mnt
11M ./sbin
5.9M ./etc
5.8G ./var
4.0K ./srv
65M ./boot
4.0K ./lib64
8.0K ./media
4.0K ./boot_ucloud
4.0K ./opt
36K ./root
2.3G ./usr
16K ./lost+found
396K ./tmp
9.7M ./bin
520M ./lib
68K ./home
9.1G .

发现var目录还没瘦身。var存放cache和log等文件。查看下，log目录存放了不少内容，足足有5.2G之多。
root@10-4-29-105:/var# du -h -x –max-depth=1
391M ./cache
4.0K ./crash
4.0K ./local
4.0K ./opt
5.2G ./log
2.1M ./backups
32K ./spool
4.0K ./tmp
4.0K ./mail
244M ./lib
5.8G .

使用 ls -lh 查看。
root@10-4-29-105:/var/log# ls -lh
total 5.0G
-rw-r–r– 1 root root 0 Dec 1 06:47 alternatives.log
-rw-r–r– 1 root root 5.2K Nov 30 06:39 alternatives.log.1
-rw-r–r– 1 root root 818 Jul 4 2015 alternatives.log.10.gz
-rw-r–r– 1 root root 2.2K Apr 18 2015 alternatives.log.11.gz
-rw-r–r– 1 root root 135 Nov 22 06:54 alternatives.log.2.gz
-rw-r–r– 1 root root 108 Aug 16 06:41 alternatives.log.3.gz
-rw-r–r– 1 root root 108 May 12 2016 alternatives.log.4.gz
-rw-r–r– 1 root root 134 Mar 4 2016 alternatives.log.5.gz
-rw-r–r– 1 root root 132 Feb 24 2016 alternatives.log.6.gz
-rw-r–r– 1 root root 109 Jan 19 2016 alternatives.log.7.gz
-rw-r–r– 1 root root 126 Aug 20 2015 alternatives.log.8.gz
-rw-r–r– 1 root root 297 Jul 4 2015 alternatives.log.9.gz
-rw-r—– 1 root adm 0 Aug 23 2015 apport.log
-rw-r—– 1 root adm 2.3K Aug 22 2015 apport.log.1
-rw-r—– 1 root adm 222 Jul 4 2015 apport.log.2.gz
drwxr-xr-x 2 root root 4.0K Dec 1 06:47 apt
-rw-r—– 1 syslog adm 107K Dec 26 01:39 auth.log
-rw-r—– 1 syslog adm 20M Dec 25 06:50 auth.log.1
-rw-r—– 1 syslog adm 2.8M Dec 18 06:37 auth.log.2.gz
-rw-r—– 1 syslog adm 3.9M Dec 12 06:26 auth.log.3.gz
-rw-r—– 1 syslog adm 1.8M Dec 4 06:41 auth.log.4.gz
-rw-r–r– 1 root root 3.7K Jan 13 2016 boot.log
-rw-r–r– 1 root root 61K Apr 17 2014 bootstrap.log
-rw-rw—- 1 root utmp 498M Dec 26 01:39 btmp
-rw-rw—- 1 root utmp 218M Dec 1 06:22 btmp.1
-rw-rw—- 1 root utmp 182M Mar 1 2016 btmp.10
-rw-rw—- 1 root utmp 170M Feb 1 2016 btmp.11
-rw-rw—- 1 root utmp 338M Jan 1 2016 btmp.12
-rw-rw—- 1 root utmp 271M Dec 1 2015 btmp.13
-rw-rw—- 1 root utmp 237M Nov 1 2015 btmp.14
-rw-rw—- 1 root utmp 513M Oct 1 2015 btmp.15
-rw-rw—- 1 root utmp 736M Sep 1 2015 btmp.16
-rw-rw—- 1 root utmp 298M Aug 1 2015 btmp.17
-rw-rw—- 1 root utmp 2.4M Jul 4 2015 btmp.18
-rw-rw—- 1 root utmp 5.1M Apr 18 2015 btmp.19
-rw-rw—- 1 root utmp 209M Nov 1 06:34 btmp.2
-rw-rw—- 1 root utmp 173M Oct 1 02:35 btmp.3
-rw-rw—- 1 root utmp 197M Sep 1 06:33 btmp.4
-rw-rw—- 1 root utmp 210M Aug 1 04:14 btmp.5
-rw-rw—- 1 root utmp 261M Jul 1 06:35 btmp.6
-rw-rw—- 1 root utmp 198M Jun 1 2016 btmp.7
-rw-rw—- 1 root utmp 162M May 1 2016 btmp.8
-rw-rw—- 1 root utmp 173M Apr 1 2016 btmp.9
drwxr-xr-x 2 root root 4.0K Apr 12 2014 dist-upgrade
-rw-r—– 1 root adm 32K Jan 13 2016 dmesg
-rw-r—– 1 root adm 31K Jul 4 2015 dmesg.0
-rw-r—– 1 root adm 9.8K Jul 3 2015 dmesg.1.gz
-rw-r—– 1 root adm 9.9K Jul 3 2015 dmesg.2.gz
-rw-r—– 1 root adm 9.8K Jul 3 2015 dmesg.3.gz
-rw-r—– 1 root adm 9.7K Apr 21 2015 dmesg.4.gz
-rw-r–r– 1 root root 135K Dec 26 01:34 dpkg.log
-rw-r–r– 1 root root 47K Dec 1 06:47 dpkg.log.1
-rw-r–r– 1 root root 2.7K Feb 26 2016 dpkg.log.10.gz
-rw-r–r– 1 root root 2.5K Jan 30 2016 dpkg.log.11.gz
-rw-r–r– 1 root root 2.5K Dec 21 2015 dpkg.log.12.gz
-rw-r–r– 1 root root 1.5K Oct 22 06:54 dpkg.log.2.gz
-rw-r–r– 1 root root 1.7K Sep 28 06:26 dpkg.log.3.gz
-rw-r–r– 1 root root 2.6K Aug 30 06:48 dpkg.log.4.gz
-rw-r–r– 1 root root 1.1K Jul 15 06:40 dpkg.log.5.gz
-rw-r–r– 1 root root 1.5K Jun 28 06:42 dpkg.log.6.gz
-rw-r–r– 1 root root 3.9K Jun 1 2016 dpkg.log.7.gz
-rw-r–r– 1 root root 1.6K Apr 30 2016 dpkg.log.8.gz
-rw-r–r– 1 root root 3.0K Apr 1 2016 dpkg.log.9.gz
-rw-r–r– 1 root root 32K Jul 4 2015 faillog
-rw-r–r– 1 root root 643 Aug 18 06:42 fontconfig.log
drwxr-xr-x 2 root root 4.0K Apr 19 2014 fsck
drwxr-xr-x 3 root root 4.0K Apr 19 2014 installer
-rw-r—– 1 syslog adm 0 Oct 6 06:54 kern.log
-rw-r—– 1 syslog adm 211 Oct 6 06:54 kern.log.1
-rw-r—– 1 syslog adm 502 Jan 22 2016 kern.log.2.gz
-rw-r—– 1 syslog adm 298 Jan 15 2016 kern.log.3.gz
-rw-r—– 1 syslog adm 11K Jan 13 2016 kern.log.4.gz
drwxr-xr-x 2 landscape root 4.0K Apr 19 2014 landscape
-rw-rw-r– 1 root utmp 287K Dec 25 22:45 lastlog
drwxr-xr-x 2 ntp ntp 40K Dec 25 08:00 ntpstats
-rw-r—– 1 syslog adm 0 Dec 26 01:17 syslog
-rw-r—– 1 syslog adm 2.5K Dec 20 06:25 syslog.1
-rw-r—– 1 syslog adm 358 Dec 19 06:25 syslog.2.gz
-rw-r—– 1 syslog adm 343 Dec 18 06:25 syslog.3.gz
-rw-r—– 1 syslog adm 335 Dec 17 06:25 syslog.4.gz
-rw-r—– 1 syslog adm 338 Dec 16 06:25 syslog.5.gz
-rw-r—– 1 syslog adm 334 Dec 15 06:25 syslog.6.gz
-rw-r—– 1 syslog adm 328 Dec 14 06:25 syslog.7.gz
-rw-r–r– 1 root root 166K Jan 13 2016 udev
drwxr-xr-x 2 root root 12K Dec 20 06:51 unattended-upgrades
drwxr-xr-x 2 root root 4.0K Jan 15 2016 upstart
-rw-rw-r– 1 root utmp 1.9K Dec 25 22:46 wtmp
-rw-rw-r– 1 root utmp 0 Nov 1 06:34 wtmp.1
-rw-rw-r– 1 root utmp 0 Feb 1 2016 wtmp.10
-rw-rw-r– 1 root utmp 11K Jan 18 2016 wtmp.11
-rw-rw-r– 1 root utmp 768 Dec 22 2015 wtmp.12
-rw-rw-r– 1 root utmp 768 Nov 3 2015 wtmp.13
-rw-rw-r– 1 root utmp 3.0K Oct 10 2015 wtmp.14
-rw-rw-r– 1 root utmp 768 Sep 16 2015 wtmp.15
-rw-rw-r– 1 root utmp 3.0K Aug 19 2015 wtmp.16
-rw-rw-r– 1 root utmp 15K Jul 30 2015 wtmp.17
-rw-rw-r– 1 root utmp 22K Jul 4 2015 wtmp.18
-rw-rw-r– 1 root utmp 9.4K Apr 17 2015 wtmp.19
-rw-rw-r– 1 root utmp 0 Oct 1 06:28 wtmp.2
-rw-rw-r– 1 root utmp 0 Sep 1 06:37 wtmp.3
-rw-rw-r– 1 root utmp 0 Aug 1 06:28 wtmp.4
-rw-rw-r– 1 root utmp 0 Jul 1 06:35 wtmp.5
-rw-rw-r– 1 root utmp 0 Jun 1 2016 wtmp.6
-rw-rw-r– 1 root utmp 3.8K May 14 2016 wtmp.7
-rw-rw-r– 1 root utmp 3.0K May 1 2016 wtmp.8
-rw-rw-r– 1 root utmp 0 Mar 1 2016 wtmp.9

分析后可以看出，btmp命名的文件占用了大量的空间。
root@10-4-29-105:/var/log# ls btm* -lh
-rw-rw—- 1 root utmp 498M Dec 26 01:42 btmp
-rw-rw—- 1 root utmp 218M Dec 1 06:22 btmp.1
-rw-rw—- 1 root utmp 182M Mar 1 2016 btmp.10
-rw-rw—- 1 root utmp 170M Feb 1 2016 btmp.11
-rw-rw—- 1 root utmp 338M Jan 1 2016 btmp.12
-rw-rw—- 1 root utmp 271M Dec 1 2015 btmp.13
-rw-rw—- 1 root utmp 237M Nov 1 2015 btmp.14
-rw-rw—- 1 root utmp 513M Oct 1 2015 btmp.15
-rw-rw—- 1 root utmp 736M Sep 1 2015 btmp.16
-rw-rw—- 1 root utmp 298M Aug 1 2015 btmp.17
-rw-rw—- 1 root utmp 2.4M Jul 4 2015 btmp.18
-rw-rw—- 1 root utmp 5.1M Apr 18 2015 btmp.19
-rw-rw—- 1 root utmp 209M Nov 1 06:34 btmp.2
-rw-rw—- 1 root utmp 173M Oct 1 02:35 btmp.3
-rw-rw—- 1 root utmp 197M Sep 1 06:33 btmp.4
-rw-rw—- 1 root utmp 210M Aug 1 04:14 btmp.5
-rw-rw—- 1 root utmp 261M Jul 1 06:35 btmp.6
-rw-rw—- 1 root utmp 198M Jun 1 2016 btmp.7
-rw-rw—- 1 root utmp 162M May 1 2016 btmp.8
-rw-rw—- 1 root utmp 173M Apr 1 2016 btmp.9

btmp文件是记录ssh错误登陆信息的文件，朋友的主机的ssh端口并未修改，所以这是个安全问题。也发现有大量的设备在扫描并且通过字典的方式尝试登陆。全删之~。

root@10-4-29-105:/var/log# rm -rf btm*
root@10-4-29-105:/var/log# touch btm*

重新查看磁盘剩余空间，从刚才的9.XG变成剩余15G，使用率为23%。
root@10-4-29-105:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 4.2G 15G 23% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 991M 4.0K 991M 1% /dev
tmpfs 201M 784K 200M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1001M 0 1001M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vdb 20G 8.9G 9.7G 48% /data

各个目录大小
root@10-4-29-105:/# du -h -x –max-depth=1
4.0K ./mnt
11M ./sbin
5.9M ./etc
840M ./var
4.0K ./srv
65M ./boot
4.0K ./lib64
8.0K ./media
4.0K ./boot_ucloud
4.0K ./opt
36K ./root
2.3G ./usr
16K ./lost+found
396K ./tmp
9.7M ./bin
520M ./lib
68K ./home
4.2G .

无需重启MYSQL服务，服务直接恢复正常。问题解决。