[Bug Rewards] Remember a CMS GC caused by irregular thread usage

Hits: 0

problem background

In the morning, I received feedback from the site that the performance of the host in the production environment dropped significantly after running for a period of time, and a large number of CMS GCs appeared .
CMS GC: CMS GC occurs in the old age, which is a precursor to Full GC. If Full GC starts to occur, OOM is likely to occur.

log analysis

According to the analysis of the dump file captured on site with MAT, it is found that there are 7490 instances of thread objects (objects of the java.lang.Thread class) that have not been released, occupying a large amount of memory (push memory, total 4.2G, thread object 3.8 G, accounting for 91.35%)
MAT use detailed explanation

Knowledge points:
Leak Suspects:
        Automatically analyze the main causes of current memory leaks through MAT;
Shallow Size:
        is the size of the memory occupied by the object itself, excluding the objects it references.
        Shallow Size for regular objects (non-arrays) is determined by the number and type of its member variables,
        The ShallowSize of the array is determined by the array type and array length, which is the sum of the array element sizes;
Retained Size:
        = the current object size + the sum of the sizes of the objects to which the current object can be directly or indirectly referenced.
        (The meaning of indirect reference: A->B->C, C is an indirect reference), and exclude objects directly or indirectly referenced by GC Roots;

Big question mark?

It is confusing: why are there so many thread objects that are not released in time? ? ?
You must know that the thread object itself occupies a very small memory, as shown in the figure below:
the corresponding Shallow Size is only 120,
while the direct reference Retatined heap has 15,009,569, and
more seriously, there are 60,330,872 objects in the indirect application
. As can be seen from the figure, the indirect application objects are business-related, and the memory of these objects cannot be released.

Business log analysis

Analyze the thread name: for example: pool-7700-thread-1

    1. The thread at the beginning of the pool indicates that the thread is created through the thread pool
    1. pool-xxx (sequence) is the name of the thread pool, which indicates that the system has created a large number of thread pools (search results: 7445, this number basically matches the analysis of the MAT tool)
    1. The thread name ends with thread-1, which means that a thread pool creates a thread

So the question is: why does each thread pool create a thread, and the thread created at the same time will not be closed?

Reasonably suspect single- [threaded] and not closed

At this time, you should be able to guess a possible reason.
It is reasonable to suspect that the thread pool created by Executors.newFixedThreadPool(1) is not closed, resulting in the creation of threads not being closed.

Executors .newFixedThreadPool ( nThreads ) method to create a thread pool,
    The input parameter ( nThreads ) is the fixed number of threads, both the number of core threads and the maximum number of threads,
    There is no idle thread, the core thread will not be released and closed

Following this suspicion, I searched the code and found that the problematic code is as follows: the
problem is here! This code tells us that it is risky to manually create a thread pool , and it is easy to forget to write and close the thread pool (forget to write pool.shutdown())

little question

The problem is located, but the questions about the use of the thread pool are still:

    1. How should the code of the thread pool be written to avoid such risks and negligence?
    1. At the same time, is it necessary to create a thread pool every time Executors.newFixedThreadPool(1) is to create a thread to work (to process things asynchronously)? (Creating thread pools and threads will consume and occupy computer resources)
    1. When is it necessary to manually create a thread pool, and how should it be created?
      So what is the correct posture for using threads?

thread specification

Next, the specifications for thread pool usage are summarized in combination with specific scenarios:

Scenario 1: Asynchronous Processing

In business processing, we want to start another thread to process asynchronous tasks, such as the above problem scenario.

    1. Consider using a global public thread pool to reuse resources to avoid the situation where the thread pool is not closed.
    1. The business code exception in the thread needs to capture the print log, and do not directly throw the business exception . Because the thread pool will remove the thread after the thread is abnormal, create a new thread, and consume resources.

Scenario 2: Batch Data Processing

When encountering the need for multi-threaded batch processing, it is better to manually create a thread pool.
According to the actual needs of the business, specify the appropriate parameters (of course, it is best not to hardcode the parameters here, but to make them configurable, which is convenient to adjust according to the actual situation). The most important thing is to note that the thread pool must be closed
after the task is completed ! ! !

ExecutorService executorService = new ThreadPoolExecutor(CORE_POOL_SIZE, MAXIMUM_POOL_SIZE,
                    KEEP_ALIVE_TIME, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>());

If you think this article is a little helpful to you, please like it.
If you have other opinions, please share in the comment area!

You may also like...

Leave a Reply

Your email address will not be published.