The Execution Process of an RPC Call

Title: The Execution Process of an RPC Call

Introduction

Just like MySQL slow queries, RPC service calls can also have slow requests. This article will help you understand the execution process of an RPC call, enabling you to pinpoint issues quickly.

We will use Pigeon 2.10.8 as an example; its basic framework is shown below:

Overall, Pigeon consists of a Client side and a Server side.

Client Side

The client side includes: client proxy, client filters, connection pool, business thread pool, and NettyClient.

Client Proxy

RPC aims to let you invoke remote functions as if they were local functions, so the caller must be shielded from the underlying details. Pigeon implements this using the proxy pattern.

Client Filters

The concrete implementation of the proxy pattern is linked together with a chain‑of‑responsibility pattern, giving the framework high extensibility. Current filters cover modules such as service monitoring, routing, fault injection, authentication, degradation, and invocation.

Connection Pool

First, clarify that a connection pool is created on the caller side for I/O operations, whereas a thread pool is created on the server side for business‑logic processing.

Pigeon allows a client to establish multiple connections to a single server machine; these connections are maintained in the client’s ChannelPool.

Thread Pool

Responsible for notifying (waking up) the business thread after receiving data returned by the server.

Netty Client

The network communication between Pigeon clients and servers is delegated to Netty (see https://netty.io/index.html). Netty is an event‑driven network I/O framework built on the Reactor model, and it comprises a Boss (the MainReactor in the Reactor model), Worker (the SubReactor), and a channel‑based Pipeline.

Reactor model: (more on the Reactor model… http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf)

Boss

On the client side, it initiates connection requests. On the server side, it accepts incoming connection requests from clients. After a connection is established, it hands the connection over to a Worker for maintenance.

Worker

Polls connections (I/O multiplexing) for incoming data and reads/writes the data to the appropriate Channel.

Pipeline

Processes the data flowing through a Channel. In Pigeon, the pipeline mainly handles serialization, deserialization, integrity checking, compression, etc.

Server Side

The server side includes: server filters, business thread pool, and NettyServer.

Server Filters

Corresponding to the client filters, a request must pass through modules such as service monitoring, authentication, and rate limiting before reaching the business code.

Thread Pool

Separates business logic from I/O operations. Once the data is ready, the business code runs in a business thread. In Pigeon, to prevent slow requests from affecting normal ones, qualifying slow requests are isolated into a SlowRequestPooling.

Netty Server

Works similarly to NettyClient.

---

The Execution Process of a Remote Service Call

The basic architecture and modules of Pigeon have been introduced above. Below is a step‑by‑step explanation of how a remote service call is executed.

Assume the client and server have already established a connection, and the client invokes a remote service. Following the diagram, the execution path in Pigeon is:

1. When the client calls a remote service method, the actual call goes to the invoke method of an InvocationHandler (using JDK dynamic proxies). In Pigeon, the InvocationHandler implementation is ServiceInvocationProxy, so any method declared in the interface ultimately ends up in ServiceInvocationProxy.invoke.

2. ServiceInvocationProxy.invoke triggers the client filters. The request sequentially passes through monitoring, routing, degradation, gateway, authentication, etc., and finally reaches RemoteCallInvokeFilter.

3. Inside RemoteCallInvokeFilter, the Client.write method is called. Its logic obtains a connection from the connection pool (ChannelPool)—the default connection‑acquisition timeout is 2000 ms—and writes the data into the Channel.

4. Before the data is sent to the server, it traverses the Channel’s Pipeline (serialization, compression, etc.) to reduce the amount of data transmitted over the network.

5. The data is then sent to the server. Because Netty sends messages asynchronously, for synchronous calls Pigeon makes the business thread await until a response arrives or a timeout occurs.

6. When the server receives the client’s message, it first passes the inbound data through its own Pipeline (deserialization, decompression, etc.) before reaching NettyServerHandler. Pigeon also implements service isolation on the server side:

   • Default isolation mechanism (statistics and isolation are at the method level):
     • If the number of timeouts exceeds 300 or the timeout rate exceeds 5 %, subsequent matching requests are routed to a slow thread pool.
     • If the number of timeouts is below 300 and the timeout rate is below 5 %, subsequent matching requests go to a shared thread pool.
   • Method‑level rate limiting (non‑Rhino) is enabled by default, limiting a single method to no more than 380 threads (dynamically adjustable).
   • Support for custom business‑specific thread pools is provided.

7. After selecting the appropriate thread pool and obtaining a thread, the request enters the server filters. It passes through monitoring, traffic recording, authentication, generic invocation, gateway, etc., and finally reaches BusinessProcessFilter.

8. BusinessProcessFilter uses the service information and parameters sent by the client to invoke the corresponding business service via reflection and obtains the processing result. WriteResponseProcessFilter then writes the result back into the Channel.

9. The response travels through the server’s Pipeline (serialization, compression, etc.) and is sent back to the client.

10. Upon receipt, the client’s Pipeline (deserialization, decompression, etc.) processes the message, which then reaches NettyClientHandler and is handed over to the ResponseThreadPoolProcessor.

11. ResponseThreadPoolProcessor notifies the previously awaiting business thread and passes the result to it.

12. The business thread, after receiving the signal, returns the result to the original method call.

---

Originally written by Li Wei (李唯_) and published in Chinese on 后端技术栈全书 (Full-Stack Backend Engineering). Translated and adapted for DriftSeas with permission.