Gao Jie 发布的文章

今天发现内网一台服务器的samba速度在10~30MB/s之间波动,原以为是smb.conf配置的问题,调整了半天也没找到原因。测试用scpnetperf发现都能达到100MB左右的满速度,非常奇怪。

找了半天也没找到有用的信息,而且这台机器的虚拟机的samba也同样速度不佳,但是在netperf压测的同时,从samba拷贝文件的速度居然能突然提上去,都是些非常奇怪的现象。后来怀疑是内核参数有不兼容变化导致tcp性能出现问题, 想起来最近刚升级了linux内核到5.0.6.arch1-1, 干脆降级到4.17.4-1-ARCH, 发现一切正常了。仔细想想,当时用netperf测试的时候,64字节小包只能到800+Mb,被忽略了。

最近4.10+以来,已经发生过3次网络问题,1次显卡驱动问题了, archlinux的更新速度注定了系统的不稳定。。。

发现内网的电脑的hostname出现了问题, macbook的hostname全部变成了bogon,不少统计网络连接的服务会把内网的ip都解析为bogon。其原因是dns反向解析的问题。
比如:

dig -x 192.168.11.12

; <<>> DiG 9.13.4 <<>> -x 192.168.11.12
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44486
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;12.11.168.192.in-addr.arpa.    IN    PTR

;; ANSWER SECTION:
12.11.168.192.in-addr.arpa. 38934 IN    PTR    bogon.

;; Query time: 3 msec
;; SERVER: 192.168.10.1#53(192.168.10.1)
;; WHEN: 一 1月 07 16:46:11 CST 2019
;; MSG SIZE  rcvd: 74

仔细研究一下, 发现223.5.5.5119.29.29.29这两个dns公共服务都会解析出bogon, 但是8.8.8.8没有这个问题。

最后的解决办法,就是在网关的dnsmasq上加入:

bogus-priv

即可。
官方文档的说明:

-b, --bogus-priv
Bogus private reverse lookups. All reverse lookups for private IP ranges (ie 192.168.x.x, etc) which are not found in /etc/hosts or the DHCP leases file are answered with "no such domain" rather than being forwarded upstream. The set of prefixes affected is the list given in RFC6303, for IPv4 and IPv6.

再查询一次:

dig -x 192.168.11.12

; <<>> DiG 9.13.4 <<>> -x 192.168.11.12
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 33874
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;12.11.168.192.in-addr.arpa.    IN    PTR

;; Query time: 2 msec
;; SERVER: 192.168.10.1#53(192.168.10.1)
;; WHEN: 一 1月 07 18:11:25 CST 2019
;; MSG SIZE  rcvd: 55

我使用archlinux的机器最近发生了两次内存占用过多导致卡死的问题。第一次我以为是chrome开太多窗口造成的,没有在意。但是第二次出现的时候,有来得及kill掉进程, 所以顺带看了一下free,发现和预想不一样的地方。

free
              total        used        free      shared  buff/cache   available
Mem:           7666        5616        1305         228         743        1561
Swap:           958         448         509

我在已经kill掉chrome的情况下,是不可能占用5G以上内存的啊?,只能说:一定是什么地方出了问题。。。
htop检查得知所有进程占用不到几百M。而清理/proc/sys/vm/drop_caches依然没有改善。 于是怀疑还是slap的问题,检查/proc/meminfo信息:

cat /proc/meminfo |grep ^S
SwapCached:          500 kB
SwapTotal:        981276 kB
SwapFree:         513248 kB
Shmem:            227892 kB
Slab:            5145568 kB
SReclaimable:      44640 kB
SUnreclaim:      5100928 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB

如上,SUnreclaim居然占了5个G,不科学。 再联想到只有最近才出现,且连续出现了两次内存问题, 怀疑是升级的新内核版本有内存泄露的bug。
我的版本是:

Linux NAS-Arch 4.18.12-arch1-1-ARCH #1 SMP PREEMPT Thu Oct 4 01:01:27 UTC 2018 x86_64 GNU/Linux

稍微搜了一下,好像的确是内存泄露的原因。
Huge memory leak on linux kernel 4.18
可以用Kernel Memory Leak Detector来检测,
好像也有一些猜测的原因:

I have updated the gist with the output of kmemleak after clearing and scanning (done after about 45 mins of runtime)
https://gist.github.com/coolsidd/d8a1d5addafd6a2367b68e6a6b243dc4/revisions

As for the amdgpu I don't believe that it is the cause (of atleast the major part) of the leak. It was the first module I removed while checking so I can confirm that the leak persists after removing amdgpu.

As for the rtl8723be (my network driver) , the lts version is very different doesn't properly work (it does not have the antenna select option). However I have been using this module since a year and it is also present of 4.17.x. Were there any changes in this version (their github page does not show any major changes since last 6 months). I will build for 4.17.x tomorrow to confirm whether the leak is due to the rtl drivers.
--
There were a few commits for rtlwifi between 4.17.14 and 4.18.6
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/drivers/net/wireless/realtek/rtlwifi?h=v4.17.1&qt=range&q=v4.17.14..v4.18.6

没仔细看, 我先升级到最新的内核,如果再有问题,就回退到4.17.11

前段时间发现sql经常不按照预期的索引走,而且最严重的有次发生在重启的mysql机器上。
后来通过万能的google发现是tokudb的bug导致的,虽然percona的文档说新版本已经fix,但是mariadb最新版上看起来仍然没有完全修复。
具体现象就是执行show index from [table_name]返回的Cardinality为0,或者非常低的数值。
准确的修复步骤忘了,按照回忆整理如下:

INSTALL PLUGIN tokudb_background_job_status SONAME 'ha_tokudb.so';
SELECT @@tokudb_version;

SET SESSION TOKUDB_ANALYZE_IN_BACKGROUND=1;
SET SESSION TOKUDB_ANALYZE_MODE=TOKUDB_ANALYZE_RECOUNT_ROWS;
ANALYZE TABLE feed;
select * from INFORMATION_SCHEMA.TOKUDB_BACKGROUND_JOB_STATUS;

然后等待后台执行完ANALYZE,即可恢复。

下载kernel源文件,
https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.16.9.tar.xz
解压,然后到对应目录执行make

$cd /opt/dev/workspace/kernel/linux-4.16.9/drivers/gpu/drm/i915
$make -C /usr/lib/modules/`uname -r`/build/ M=`pwd` modules    -j 3

然后发现有错误:

In file included from /opt/dev/workspace/kernel/linux-4.16.9/drivers/gpu/drm/i915/i915_trace.h:1006:0,
                 from /opt/dev/workspace/kernel/linux-4.16.9/drivers/gpu/drm/i915/i915_trace_points.c:13:
./include/trace/define_trace.h:89:42: 致命错误:../../drivers/gpu/drm/i915/i915_trace.h:没有那个文件或目录
 #include TRACE_INCLUDE(TRACE_INCLUDE_FILE)
                                          ^
编译中断

按照经验是需要修改TRACE_INCLUDE_PATH就可以了。
但是发现改为绝对路径以后仍然不能。 卡了很久。
后来一检查:

配置的是:
#define TRACE_INCLUDE_PATH /opt/dev/workspace/kernel/linux-4.16.9/drivers/gpu/drm/i915

错误是:
In file included from /opt/dev/workspace/kernel/linux-4.16.9/drivers/gpu/drm/i915/i915_trace.h:1006:0,
                 from /opt/dev/workspace/kernel/linux-4.16.9/drivers/gpu/drm/i915/i915_trace_points.c:13:
./include/trace/define_trace.h:89:42: 致命错误:/opt/dev/workspace/kernel/1-4.16.9/drivers/gpu/drm/i915/i915_trace.h:没有那个文件或目录
 #include TRACE_INCLUDE(TRACE_INCLUDE_FILE)
                                          ^

错误的是我的目录有-号: linux-4.16.9 VS 1-4.16.9

神奇。。。。

于是换了一个目录名搞定。