Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Overview of performance collection and analysis tools

There are many tools for collecting, analyzing, and displaying mobile application performance data, which can be roughly divided into the following categories. For example, mobile performance tools that can collect multiple performance indicators include perfdog and Solopi, of which Solopi is open source and pefdog is a commercial tool. Tools that can perform crash analysis, such as the commercial Firebase Crashlytics and the open source Sentry tool. If you need to analyze network-related data, you can use Charles or Wireshark. In addition, there are comprehensive performance management tools, such as Dynatrace. In general, commercial tools are definitely better than open source tools in terms of capabilities, accuracy, and user experience. As a personal study and research, I only start from open source tools. Therefore, in the writing of this blog, I will also use open source tools as examples.

Mobile performance testing tools：
PerfDog: A tool for monitoring application performance, CPU, memory, power consumption and other data. It supports performance testing and real-time monitoring of mobile applications.
Solopi: Focuses on testing and monitoring mobile application performance, including real-time monitoring of indicators such as frame rate, CPU, and memory.

Crash analysis tool：
Firebase Crashlytics: A powerful crash reporting and analysis tool provided by Google that can monitor app crashes in real time.
Sentry: An open source error tracking tool that supports multiple platforms, including error reporting and performance monitoring for mobile applications.

Network performance testing tools：
Charles Proxy: A network debugging proxy tool that can capture and analyze network request and response data of mobile applications.
Wireshark: An open source network analysis tool that supports capturing and analyzing network packets of mobile applications.

APM Tools(Application Performance Management):
Dynatrace: A cloud-native full-stack performance monitoring solution that supports mobile application performance monitoring and user experience analysis.

What performance metrics to collect and how to collect them

  Before performing performance analysis, you must first understand the meaning of each performance indicator. Taking the performance indicators collected by solopi as an example, let's take a look at the common mobile application performance indicators and how to collect these indicators.

Frame rate

The formula for calculating the frame rate is:Frame rate = number of frames drawn / time period,Most application and game developers target a frame rate of60FPS(Frames per second), why is the target frame rate 60FPS? Because the screen refresh rate of most modern smartphones and tablets is 60Hz, so as long as the frame rate reaches 60FPS, that is, the number of frames drawn per second is 60, the user's visual effect will be smoother and there will be no sense of lag. The minimum cannot be lower than 30FPS, because the display repeatedly displays the image of the previous frame while waiting for the new frame, resulting in an unsmooth picture. The highest can reach 90FPS (for example, for devices with a high refresh rate).

How to collect frame rate information

  There are two ways to collect frame rate. One is to use the Choreographer class to count frame rate. This requires writing code inside the application and can only collect frame rate information for the application. The other way is to usegfxinfo tool. Taking Solopi as an example, the timeout frame time within 1 second is calculated through the gfxinfo tool information, and the actual frame rate is deduced. Therefore, in the case of near-stillness, some frame rates may be displayed incorrectly. It is recommended to perform frame rate tests in dynamic scenes such as sliding or page switching. The gfxinfo method can collect frame rate information of all applications, which is very suitable for performance data collection tools. Therefore, here, we also focus on how to collect frame rate information through gfxinfo.

There are two main steps to collect frame rate information.step one: Get gfxinfo information through adb command,Step 2: Analyze the information and calculate the frame rate information. The gfxinfo information content is roughly as follows:

Stats since: Statistics start time, in nanoseconds.
Total frames rendered: Total number of rendered frames.
Janky frames: Janky frames, that is, the number of frames that exceed 16ms (the refresh interval of 60 frames per second).
Janky frames percentage: The ratio of janky frames to the total number of frames.
90th percentile: The 90th percentile, meaning that 90% of the frames take less time to render than this value (in milliseconds).
95th percentile: The 95th percentile, meaning that 95% of the frames take less time to render than this value (in milliseconds).
99th percentile: The 99th percentile, meaning that 99% of the frames take less time to render than this value (in milliseconds).
Number Missed Vsync: The number of times vertical synchronization is not synchronized, that is, the number of frame drops caused by failure to render in time.
Number High input latency: The number of high input latency times, which indicates the number of delays caused by the long processing time of input events.
Number Slow UI thread: The number of times the UI thread responds slowly, that is, the number of times the UI thread processing time is too long.

Taking Solopi as an example, the frame rate is inferred through the timeout frame time in the gfxinfo information. The code of the specific project can be viewed in the solopi source code.

Jam rate/jam times

Number of freezes: The number of time periods during the test period when the frame rate was detected to be lower than the threshold. Each time the frame rate is lower than the threshold and lasts for a period of time, it is counted as a jam. The threshold here can be customized, for example, if it is lower than the target frame rate of 60FPS, it can be counted as a jam. That is, if the frame rendering time is greater than 16ms, it is counted as a jam.

Stuttering rate: The percentage of the freeze time in the total test time. Assume that in a test, the application runs for 120 seconds, and the frame rate is lower than 16 FPS 4 times, and the total freeze time is 8 seconds, then: Number of freezes: 4 times. Freeze rate: 8/120*100%=6.7%. In the above gfxinfo, there is also freeze rate data information.

cpu/memory

  Taking solopi as an example, cpu refers to the CPU usage percentage of the process where the top-level Activity of the application is located and the global CPU usage percentage. For a single-process application, this data indicates the CPU usage of the application; for a multi-process application, this data indicates the CPU usage of the top-level UI process. When a process switch occurs, Soloπ can automatically switch to the new process data.

import subprocess
 
def get_memory_info(pid):
    try:
        # Run adb shell command to read /proc/<pid>/statm
        command = f"adb shell cat /proc/{pid}/statm"
        result = subprocess.check_output(command, shell=True)
        statm_data = result.decode('utf-8').strip().split()
        
        # Parse statm data
        size, resident, shared, text, lib, data, dt = map(int, statm_data)
        
        memory_info = {
            'size': size,         # total program size (pages)
            'resident': resident, # resident set size (pages)
            'shared': shared,     # shared pages (pages)
            'text': text,         # text (code) size (pages)
            'lib': lib,           # library (unused since Linux 2.6; always 0)
            'data': data,         # data + stack (pages)
            'dt': dt              # dirty pages (unused since Linux 2.6; always 0)
        }
        
        return memory_info
    except subprocess.CalledProcessError as e:
        print(f"Error executing command: {e}")
        return None
    except Exception as e:
        print(f"Error: {e}")
        return None
 
# Replace with your application's PID
pid = '12345'
memory_info = get_memory_info(pid)
if memory_info:
    print(f"Memory info for PID {pid}:")
    print(f"Size: {memory_info['size']} pages")
    print(f"Resident set size (RSS): {memory_info['resident']} pages")
    print(f"Shared pages: {memory_info['shared']} pages")
    print(f"Text (code) size: {memory_info['text']} pages")
    print(f"Data + stack size: {memory_info['data']} pages")

Response time

  Take solopi as an example.Contains response time and refresh time data for application clicks. It belongs to application dimension data. The time from the user click to the first time the system issues an interface update is the response time, and the time to the system stops refreshing the interface is the refresh time. The logic for counting response time is screen recording and framing. The solopi official website introduces that it implements automatic identification of start frames and end frames to count response time. In actual use of solopi, you will find that the counted page response time is not correct. For example, from the time the user clicks start to the time he clicks on the application page, there is a human hand speed operation time in between, and this part is counted in the response time. Therefore, if you want to calculate the response time of the prepared page, it is more accurate to only record the screen and manually identify the start and end frames of the page. The specific steps for recording the screen and framing are as follows:

The above is an understanding of the meaning of common performance indicators in mobile applications, as well as detailed descriptions on how to collect them. Regarding crashes, etc., they will be introduced in detail in subsequent blogs.