一次关于Redis内存诡异增长的排查过程实战记录_Redis_

一、现象

实例名：r-bp1cxxxxxxxxxd04(主从)

问题：一分钟内存上涨了2G，如下图所示：

键值规模：6000万左右

内存一分钟增长2G.png

二、Redis内存分析

1. 内存组成

上图中的内存统计的是Redis的info memory命令中的used_memory属性，例如：

 redis>infomemory#Memoryused_memory:9195978072used_memory_human:8.56Gused_memory_rss:9358786560used_memory_peak:10190212744used_memory_peak_human:9.49Gused_memory_lua:38912mem_fragmentation_ratio:1.02mem_allocator:jemalloc-3.6.0 

每个属性的详细说明

计算公式如下：

used_memory = 自身内存+对象内存+缓冲内存+lua内存used_rss = used_memory + 内存碎片

如下图所示：

2. 内存分析

(1) 自身内存：一个空的Redis占用很小，可以忽略不计

(2) kv内存：key对象 + value对象

(3) 缓冲区：客户端缓冲区(普通 + slave伪装 + pubsub)以及aof缓冲区(比较固定，一般没问题)

(4) Lua：Lua引擎所消耗的内存

3. 内存突增常见问题

(1) kv内存：bigkey、大量写入

(2) 客户端缓冲区：一般常见的有普通客户端缓冲区(例如monitor命令)或者pubsub客户端缓冲区

三、问题排查

(1) bigkey ? 经扫描未发现bigkey

 Sampled 67234427 keys in the keyspace! Total key length in bytes is 1574032382 (avg len 23.41) Biggest string found 'CCARD_DEVICE_CARD_REF_MAP_KEY_016817000004209' has 20862 bytes Biggest list found 'CCARD_VALID_DEVICE_TRAIN_QUEUE_KEY' has 51 items Biggest hash found 'CCARD_VALID_DEVICE_TRAIN_MAP_KEY' has 51 fields 67234359 strings with 71767890 bytes (100.00% of keys, avg size 1.07) 67 lists with 151 items (00.00% of keys, avg size 2.25) 0 sets with 0 members (00.00% of keys, avg size 0.00) 1 hashs with 51 fields (00.00% of keys, avg size 51.00) 0 zsets with 0 members (00.00% of keys, avg size 0.00)

(2) 键值个数增加？未发现键值有明显变化

(3) 客户端缓冲区

由于内存增上去后，长时间没下落，如果是因为缓冲区问题，会从info clients找到明显问题，执行后发现：

 redis> info clients # Clients connected_clients:43 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 admin_clients:6 rejected_vpc_conn_count:0 close_idle_unknown_conn_count:0

执行client中也没有明显的omem大于0的情况

id=80207addr=10.xx.0.4:63920fd=46name=age=624idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80215addr=10.xx.0.23:43489fd=36name=age=591idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80366addr=10.xx.0.8:59785fd=18name=age=84idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=delread=0write=0type=user
id=80356addr=10.xx.0.33:32117fd=13name=age=114idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80064addr=10.xx.59.4:53446fd=38name=age=1070idle=1070flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80276addr=10.xx.0.23:48511fd=8name=age=387idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80188addr=10.xx.0.33:16265fd=42name=age=681idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80326addr=10.xx.0.32:59779fd=16name=age=209idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80065addr=10.xx.59.4:53447fd=45name=age=1070idle=1070flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=79936addr=10.xx.0.22:10607fd=30name=age=1480idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80174addr=10.xx.0.5:60914fd=6name=age=722idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80300addr=10.xx.0.22:22757fd=48name=age=298idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80037addr=10.xx.0.5:55189fd=15name=age=1143idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80330addr=10.xx.0.8:48533fd=17name=age=199idle=10flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79896addr=10.xx.0.30:26814fd=11name=age=1616idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80299addr=10.xx.0.24:11227fd=44name=age=303idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80086addr=10.xx.0.32:52526fd=40name=age=1002idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80202addr=10.xx.0.33:16658fd=26name=age=636idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80256addr=10.xx.0.24:60496fd=19name=age=448idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79908addr=10.xx.0.29:18975fd=12name=age=1583idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80365addr=10.xx.0.29:46429fd=14name=age=85idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79869addr=10.xx.27.4:48455fd=35name=age=1700idle=1700flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80334addr=10.xx.0.23:50012fd=39name=age=189idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80041addr=10.xx.0.32:51107fd=33name=age=1132idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79992addr=10.xx.0.22:12068fd=28name=age=1289idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80251addr=10.xx.0.30:44213fd=23name=age=468idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80006addr=10.xx.0.2:45895fd=31name=age=1242idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80321addr=10.xx.0.30:48048fd=5name=age=224idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80381addr=10.xx.0.8:13360fd=22name=age=24idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=delread=0write=0type=user
id=80200addr=10.xx.0.24:59183fd=24name=age=640idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80113addr=10.xx.0.2:52492fd=21name=age=915idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=174addr=11.216.117.242:53027fd=9name=age=281390idle=0flags=S db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=replconf read=0write=0type=admin
id=79991addr=10.xx.0.4:48412fd=25name=age=1296idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80301addr=127.0.0.1:47869fd=49name=age=291idle=261flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=strlen read=0write=0type=admin
id=80047addr=10.xx.59.4:53184fd=41name=age=1114idle=1114flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80236addr=10.xx.0.5:62546fd=47name=age=516idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80364addr=10.xx.0.4:18794fd=7name=age=85idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80175addr=10.xx.0.4:62245fd=29name=age=718idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80336addr=10.xx.0.29:45701fd=50name=age=180idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80050addr=10.xx.59.4:53188fd=43name=age=1114idle=1114flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=79765addr=10.xx.0.2:33832fd=37name=age=2027idle=177flags=N db=0sub=0psub=0multi=-

提示： 本文由神整理自网络，如有侵权请联系本站删除！
本站声明：
1、本站所有资源均来源于互联网，不保证100%完整、不提供任何技术支持；
2、本站所发布的文章以及附件仅限用于学习和研究目的;不得将用于商业或者非法用途；否则由此产生的法律后果，本站概不负责！

上一篇：Windows环境下Redis Cluster环境搭建(图文)_Redis_

下一篇：为啥懒 Redis 是更好的 Redis_Redis_