问题现象
生产上运行一段时间出现错误如下:2023-04-24 13:45:19.458 ERROR Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is io.lettuce.core.RedisException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 159383552 byte(s) of direct memory (used: 205520896, max: 358088704)] with root cause io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 159383552 byte(s) of direct memory (used: 205520896, max: 358088704) at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:656) at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:610) at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) at io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:754) at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:260) at io.netty.buffer.PoolArena.allocate(PoolArena.java:231) at io.netty.buffer.PoolArena.reallocate(PoolArena.java:397) at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118) at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:299) at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:278) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1103) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1096) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1087) at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:554) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)我们引入redis依赖是 在压测的收产生堆外内存溢出:OutOfDirectMemoryError的原因:
1)、Springboot2.0以后默认使用Lettuce作为操作redis的客户端。它使用netty进行网络通信。
2), Lettuce的bug导致netty堆外内存溢出 -Xmx300m; netty如果没有指定堆外内存,默认使用Xmx300m
可以通过-Dio.netty.maxDirectMemory进行设置
解决方案:
不能使用-Dio.netty.maxDirectMemory只是去调大堆外内存。1)、可以升级Lettuce客户端。 2)可以切换使用jedis
这里我们采用的是切换到jedis,所以需要修改对应的将默认的lettuce排除调,添加jedis依赖。