Redis内存淘汰和过期删除策略原理分析
384 Redis是一个内存键值对数据库，所以对于内存的管理尤为重要。Redis内部对于内存的管理主要包含两个方向，过期删除策略和数据淘汰策略。思考：什么是数据淘汰？数据过期和数据淘汰都是删除数据，两者有什么区别？实际使用场景是多样化的，如何选择合适的淘汰策略？淘汰策略原理所谓数据淘汰是指在Redis内存使用达到一定阈值的时候，执行某种策略释放内存空间，以便于接收新的数据。内存可使用空间由配置参数maxmemory决定(单位mb/GB)。故又叫"最大内存删除策略"，也叫"缓存删除策略"。maxmemory配置# 客户端命令方式配置和查看内存大小127.0.0.1:6379> config get maxmemory"maxmemory""0"127.0.0.1:6379> config set maxmemory 100mbOK127.0.0.1:6379> config get maxmemory"maxmemory""104857600"#通过redis.conf 配置文件配置127.0.0.1:6379> info# Server#...# 配置文件路径config_file:/opt/homebrew/etc/redis.conf#...# 修改内存大小> vim /opt/homebrew/etc/redis.conf############################## MEMORY MANAGEMENT ################################# Set a memory usage limit to the specified amount of bytes.# When the memory limit is reached Redis will try to remove keys# according to the eviction policy selected (see maxmemory-policy).##...maxmemory 100mb#...注：若`maxmemory=0`则表示不做内存限制，但是对于windows系统来说，32位系统默认可使用空间是3G，因为整个系统内存是4G，需要留1G给系统运行。且淘汰策略会自动设置为noeviction，即不开启淘汰策略，当使用空间达到3G的时候，新的内存请求会报错。淘汰策略分类淘汰策略配置maxmemory-policy，表示当内存达到maxmemory时，将执行配置的淘汰策略，由redis.c/freeMemoryIfNeeded 函数实现数据淘汰逻辑。maxmemory-policy配置# 命令行配置方式127.0.0.1:6379> CONFIG GET maxmemory-policy"maxmemory-policy""noeviction"127.0.0.1:6379> CONFIG SET maxmemory-policy volatile-lruOK127.0.0.1:6379> CONFIG GET maxmemory-policy"maxmemory-policy""volatile-lru"#redis.conf文件配置方式# MAXMEMORY OLICY: how Redis will select what to remove when maxmemory# is reached. You can select one from the following behaviors:## volatile-lru -> Evict using approximated LRU, only keys with an expire set.# allkeys-lru -> Evict any key using approximated LRU.# volatile-lfu -> Evict using approximated LFU, only keys with an expire set.# allkeys-lfu -> Evict any key using approximated LFU.# volatile-random -> Remove a random key having an expire set.# allkeys-random -> Remove a random key, any key.# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)# noeviction -> Don't evict anything, just return an error on write operations.## LRU means Least Recently Used# LFU means Least Frequently Used## Both LRU, LFU and volatile-ttl are implemented using approximated# randomized algorithms.# The default is:# ...maxmemory-policy noeviction freeMemoryIfNeeded逻辑处理int freeMemoryIfNeeded(void) { size_t mem_used, mem_tofree, mem_freed; int slaves = listLength(server.slaves); /* Remove the size of slaves output buffers and AOF buffer from the count of used memory.*/ // 计算出 Redis 目前占用的内存总数，但有两个方面的内存不会计算在内： // 1）从服务器的输出缓冲区的内存 // 2）AOF 缓冲区的内存 mem_used = zmalloc_used_memory(); if (slaves) { listIter li; listNode *ln; listRewind(server.slaves,&li); while((ln = listNext(&li))) { redisClient *slave = listNodeValue(ln); unsigned long obuf_bytes = getClientOutputBufferMemoryUsage(slave); if (obuf_bytes > mem_used) mem_used = 0; else mem_used -= obuf_bytes; } } if (server.aof_state != REDIS_AOF_OFF) { mem_used -= sdslen(server.aof_buf); mem_used -= aofRewriteBufferSize(); } /* Check if we are over the memory limit. */ // 如果目前使用的内存大小比设置的 maxmemory 要小，那么无须执行进一步操作 if (mem_used eviction_pool; while(bestkey == NULL) { evictionPoolPopulate(dict, db->dict, db->eviction_pool); /* Go backward from best to worst element to evict. */ for (k = REDIS_EVICTION_POOL_SIZE-1; k >= 0; k--) { if (pool[k].key == NULL) continue; de = dictFind(dict,pool[k].key); /* Remove the entry from the pool. */ sdsfree(pool[k].key); /* Shift all elements on its right to left. */ memmove(pool+k,pool+k+1, sizeof(pool[0])*(REDIS_EVICTION_POOL_SIZE-k-1)); /* Clear the element on the right which is empty since we shifted one position to the left. */ pool[REDIS_EVICTION_POOL_SIZE-1].key = NULL; pool[REDIS_EVICTION_POOL_SIZE-1].idle = 0; /* If the key exists, is our pick. Otherwise it is a ghost and we need to try the next element. */ if (de) { bestkey = dictGetKey(de); break; } else { /* Ghost... */ continue; } } } } /* volatile-ttl */ // 策略为 volatile-ttl ，从一集 sample 键中选出过期时间距离当前时间最接近的键 else if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_TTL) { for (k = 0; k id); decrRefCount(keyobj); keys_freed++; /* When the memory to free starts to be big enough, we may */ /* start spending so much time here that is impossible to */ /* deliver data to the slaves fast enough, so we force the */ /* transmission here inside the loop. */ if (slaves) flushSlavesOutputBuffers(); } } if (!keys_freed) return REDIS_ERR; /* nothing to free... */ } return REDIS_OK;}8种淘汰策略Redis定义的策略常量(version 对配置了过期时间的key，淘汰最近使用频率最少的数据。allkeys-lfu(REDIS_MAXMEMORY_ALLKEYS_LFU): Evict any key using approximated LFU -> 对所有key，淘汰最近使用频率最少的数据。volatile-lru( REDIS_MAXMEMORY_VOLATILE_LRU): Evict using approximated LRU, only keys with an expire set -> 内存不足时，对所有配置了过期时间的key，淘汰最近最少使用的数据。allkeys-lru(REDIS_MAXMEMORY_ALLKEYS_LRU): Evict any key using approximated LRU -> 内存不足时，对所有key，淘汰最近最少使用的数据。volatile-random( REDIS_MAXMEMORY_VOLATILE_RANDOM): Remove a random key having an expire set -> 内存不足时，对所有配置了过期时间的key，淘汰随机数据。allkeys-random(REDIS_MAXMEMORY_ALLKEYS_RANDOM): Remove a random key, any key -> 内存不足时，对所有key，淘汰随机数据。volatile-ttl( REDIS_MAXMEMORY_VOLATILE_TTL): Remove the key with the nearest expire time (minor TTL) -> 内存不足时，对所有配置了过期时间的key，淘汰最近将要过期的数据。noeviction( REDIS_MAXMEMORY_NO_EVICTION): Don't evict anything, just return an error on write operations -> 不开启淘汰策略，在不配置淘汰策略的情况下，maxmemory-policy默认等于该值。内存不足时，会抛出异常，写操作不可用。不同系统存在差异性-具体见⇑淘汰策略的选择存在冷热数据区别，即意味着访问频率存在较大差异，4.0及以上版本建议选择allkeys-lfu策略，但要设置lfu-decay-time 计数衰减值，一般默认1，这样可避免缓存污染现象；3.0及以下版本建议选择allkeys-lru策略。LFU访问计数衰减配置# The counter decay time is the time, in minutes, that must elapse in order# for the key counter to be divided by two (or decremented if it has a value# less
|手机版|心飞设计-版权所有：微度网络信息技术服务中心
( 鲁ICP备17032091号-12 )|网站地图
发表于 2024-10-12 21:56:19
OLICY: how Redis will select what to remove when maxmemory# is reached. You can select one from the following behaviors:## volatile-lru -> Evict using approximated LRU, only keys with an expire set.# allkeys-lru -> Evict any key using approximated LRU.# volatile-lfu -> Evict using approximated LFU, only keys with an expire set.# allkeys-lfu -> Evict any key using approximated LFU.# volatile-random -> Remove a random key having an expire set.# allkeys-random -> Remove a random key, any key.# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)# noeviction -> Don't evict anything, just return an error on write operations.## LRU means Least Recently Used# LFU means Least Frequently Used## Both LRU, LFU and volatile-ttl are implemented using approximated# randomized algorithms.# The default is:# ...maxmemory-policy noeviction freeMemoryIfNeeded逻辑处理int freeMemoryIfNeeded(void) { size_t mem_used, mem_tofree, mem_freed; int slaves = listLength(server.slaves); /* Remove the size of slaves output buffers and AOF buffer from the count of used memory.*/ // 计算出 Redis 目前占用的内存总数，但有两个方面的内存不会计算在内： // 1）从服务器的输出缓冲区的内存 // 2）AOF 缓冲区的内存 mem_used = zmalloc_used_memory(); if (slaves) { listIter li; listNode *ln; listRewind(server.slaves,&li); while((ln = listNext(&li))) { redisClient *slave = listNodeValue(ln); unsigned long obuf_bytes = getClientOutputBufferMemoryUsage(slave); if (obuf_bytes > mem_used) mem_used = 0; else mem_used -= obuf_bytes; } } if (server.aof_state != REDIS_AOF_OFF) { mem_used -= sdslen(server.aof_buf); mem_used -= aofRewriteBufferSize(); } /* Check if we are over the memory limit. */ // 如果目前使用的内存大小比设置的 maxmemory 要小，那么无须执行进一步操作 if (mem_used eviction_pool; while(bestkey == NULL) { evictionPoolPopulate(dict, db->dict, db->eviction_pool); /* Go backward from best to worst element to evict. */ for (k = REDIS_EVICTION_POOL_SIZE-1; k >= 0; k--) { if (pool[k].key == NULL) continue; de = dictFind(dict,pool[k].key); /* Remove the entry from the pool. */ sdsfree(pool[k].key); /* Shift all elements on its right to left. */ memmove(pool+k,pool+k+1, sizeof(pool[0])*(REDIS_EVICTION_POOL_SIZE-k-1)); /* Clear the element on the right which is empty since we shifted one position to the left. */ pool[REDIS_EVICTION_POOL_SIZE-1].key = NULL; pool[REDIS_EVICTION_POOL_SIZE-1].idle = 0; /* If the key exists, is our pick. Otherwise it is a ghost and we need to try the next element. */ if (de) { bestkey = dictGetKey(de); break; } else { /* Ghost... */ continue; } } } } /* volatile-ttl */ // 策略为 volatile-ttl ，从一集 sample 键中选出过期时间距离当前时间最接近的键 else if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_TTL) { for (k = 0; k id); decrRefCount(keyobj); keys_freed++; /* When the memory to free starts to be big enough, we may */ /* start spending so much time here that is impossible to */ /* deliver data to the slaves fast enough, so we force the */ /* transmission here inside the loop. */ if (slaves) flushSlavesOutputBuffers(); } } if (!keys_freed) return REDIS_ERR; /* nothing to free... */ } return REDIS_OK;}8种淘汰策略Redis定义的策略常量(version 对配置了过期时间的key，淘汰最近使用频率最少的数据。allkeys-lfu(REDIS_MAXMEMORY_ALLKEYS_LFU): Evict any key using approximated LFU -> 对所有key，淘汰最近使用频率最少的数据。volatile-lru( REDIS_MAXMEMORY_VOLATILE_LRU): Evict using approximated LRU, only keys with an expire set -> 内存不足时，对所有配置了过期时间的key，淘汰最近最少使用的数据。allkeys-lru(REDIS_MAXMEMORY_ALLKEYS_LRU): Evict any key using approximated LRU -> 内存不足时，对所有key，淘汰最近最少使用的数据。volatile-random( REDIS_MAXMEMORY_VOLATILE_RANDOM): Remove a random key having an expire set -> 内存不足时，对所有配置了过期时间的key，淘汰随机数据。allkeys-random(REDIS_MAXMEMORY_ALLKEYS_RANDOM): Remove a random key, any key -> 内存不足时，对所有key，淘汰随机数据。volatile-ttl( REDIS_MAXMEMORY_VOLATILE_TTL): Remove the key with the nearest expire time (minor TTL) -> 内存不足时，对所有配置了过期时间的key，淘汰最近将要过期的数据。noeviction( REDIS_MAXMEMORY_NO_EVICTION): Don't evict anything, just return an error on write operations -> 不开启淘汰策略，在不配置淘汰策略的情况下，maxmemory-policy默认等于该值。内存不足时，会抛出异常，写操作不可用。不同系统存在差异性-具体见⇑淘汰策略的选择存在冷热数据区别，即意味着访问频率存在较大差异，4.0及以上版本建议选择allkeys-lfu策略，但要设置lfu-decay-time 计数衰减值，一般默认1，这样可避免缓存污染现象；3.0及以下版本建议选择allkeys-lru策略。LFU访问计数衰减配置# The counter decay time is the time, in minutes, that must elapse in order# for the key counter to be divided by two (or decremented if it has a value# less