Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

References:

①Use mtrace to track JVM off-heap memory leaks

②A case study on high Java memory usage

1. Troubleshooting ideas

Get the memory leak location (see reference ① for detailed commands)

1. Use jna to enable mtrace in Java code to track memory allocation

2. Use the mtrace command to analyze the log file and obtain the memory leak call point

3. arthas checks the call stack of the memory leak call point (you can use the call point address directly without the function name)

Get the memory leak content (see reference ② for detailed commands)

1. Execute the pmap command for the first time to obtain the memory usage of the Java process
2. After a period of time, the memory usage increases. Execute the pmap command again to obtain the memory usage of the Java process.
3. Use a text comparison tool to compare the output of the two pmap commands.
4. Transcode to view the data content of the newly added memory

2. Case Study

When stress testing a service offline, when the QPS is high, the memory occupied by the nio.directbuffer area of ​​the machine will increase suddenly, causing OOM in the container.

Using the above troubleshooting ideas, we finally located that our RPC framework generated a large amount of data in the directbuffer area. In fact, when the QPS is high, the number of selector threads is small and the consumption capacity is insufficient, resulting in a backlog of request and response data in the directbuffer area.