Build a gRPC RTD service

This topic covers implementing a gRPC-based real-time data integration service. All new containers should be built for gRPC.

gRPC integration build guide

Complete the following steps to set up the proto files and generate the language-specific bindings your container requires. This uses protoc, the Protocol Buffers compiler, which converts .proto schema files into language-specific code. Installation instructions are available at grpc.io/docs/protoc-installation.

1. Download the official OpenRTB An open industry standard for communication between buyers and sellers of online advertising in real-time bidding auctions. It's published by the IAB. 2.6 Protocol Buffers specification from the IAB Tech Lab's openrtb2.x repository.

2. Clone the ARTF reference implementation repository, which contains the Makefile, build scripts, and proto files referenced in the steps below.

3. Generate the language-specific OpenRTB and gRPC bindings:

   1. Install make and protoc v27 or later. This is required for edition = "2023" support. The version in most base package managers is too old and errors on --experimental_editions.

   2. Install the language plugins. For example, for Go:

      go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
      go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

   3. Open the Makefile and choose the languages under LANGUAGES.

   4. Run make bindings. Plain make builds the reference agent, not the bindings.

4. Alternatively, skip generation and use the pre-generated Go bindings included in the repository under pkg/pb/ (pkg/pb/artf, pkg/pb/openrtb).

Service contract

Your container implements one gRPC service with one RPC, both defined in the proto files you generated bindings from above. That single RPC is where all of your logic lives.

Your container also exposes an HTTP metrics endpoint A URL which is configured to interact with a server in a specific way. on a separate port, SERVER_PORT. That is covered in Building a Docker container, along with the full list of environment variables Index sets for you.

Latency

Two separate numbers govern your response time, and they do different things. Getting them the wrong way round is the most common source of confusion here.

tmax is the hard deadline for a single request. It is fixed at 5ms, at both extension points, and it does not vary per request. A response that arrives after tmax is not applied to that auction.

Your 95th percentile response time is what determines whether you keep receiving full traffic. The threshold depends on which extension point the request came from:

Cross the p95 threshold for your extension point and traffic to your container is automatically throttled, and Index may lower your QPS Queries Per Second (QPS). The number of bid requests a DSP processes per second. Also known as impressions per second. accordingly.

Note: At the Publisher Request extension point this gives you real margin. A response between 5ms and 30ms misses tmax and is discarded for that auction, but it does not by itself put you over the p95 bar. At the DSP Bid Response extension point there is no margin, because the p95 bar and tmax are both 5ms. If you are implementing bid shading, that is the bar you are working to. See Implementing bid shading.

Calls use keepalive connections to minimize network overhead.

Response codes

Standard gRPC status codes should be returned.

Circuit breaker logic

If Index receives any status code other than OK, or more than 5% of requests have exceeded an internal timeout value of 30ms, Index's circuit breaker triggers and requests to your container are throttled. Requests that time out also trigger this logic.

Circuit breaker logic protects both Index and partner systems from cascade failures. Once error rates return to nominal levels, measured over a rolling window, circuit breakers reopen and regular traffic volume resumes automatically. No action is required from your side during recovery.

Note: The 30ms internal timeout applies to both gRPC and HTTP integrations. It is the outer timeout after which Index treats a request as failed, which is a different thing from tmax. See Latency above for how the two relate.

gRPC-supported lifecycle extension points and mutations

The following table summarizes all supported lifecycle extension points, the mutations available at each, and their paths. For full schema details and examples, see the Mutation reference.

What your container receives

At the Publisher Request extension point, your GetMutations handler receives an RTBRequest whose bid_request field holds the full privacy-safe OpenRTB payload. For the fields included in that payload, see the Publisher request payload reference. Index calls GetMutations in real time during auction processing, once per extension point.

Two other RTBRequest fields matter for every call:

• tmax is the maximum number of milliseconds Index waits for your response, fixed at 5ms. Treat it as a hard deadline. See Latency above.

• applicable_intents is the list of mutation intents Index accepts on that call. Filter your mutations to only these. A mutation whose intent is not in applicable_intents is silently discarded.

Note: The bid_response field is empty at the Publisher Request extension point. It is only populated when lifecycle is LIFECYCLE_DSP_BID_RESPONSE. Note that device.ip, device.geo.lat, device.geo.lon, and device.geo.zip may be anonymized, masked, or withheld depending on the request user's privacy settings. See the Publisher request payload reference for the full list of modifications.

The following is an example of a complete RTBRequest at the Publisher Request extension point, in protojson format:

{
   "id": "2545D87B17C8853B",
   "lifecycle": "LIFECYCLE_PUBLISHER_BID_REQUEST",
   "tmax": 5,
   "applicable_intents": [
      "ACTIVATE_SEGMENTS", "ACTIVATE_DEALS", "SUPPRESS_DEALS",
      "ADJUST_DEAL_FLOOR", "ADD_CIDS"
   ],
   "bid_request": {
      "id": "2545D87B17C8853B",
      "imp": [{
         "id": "1",
         "banner": { "w": 300, "h": 250, "pos": 1, "format": [{ "w": 300, "h": 250 }] },
         "bidfloor": 0.5,
         "ext": { "gpid": "/1234567/home/mpu/atf", "sid": "abc123" }
      }],
      "site": {
         "id": "12345", "domain": "example.com",
         "page": "https://example.com/section/",
         "publisher": { "id": "555555", "domain": "example.com", "name": "Publisher Name" }
      },
      "device": {
         "connectiontype": 2, "devicetype": 2, "dnt": 0,
         "ip": "2001:db8::", "language": "en", "lmt": 0,
         "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/114.0.0.0"
      },
      "regs": { "coppa": 0 }
   }
}

Note: lifecycle, id, tmax, and bid_request are required on every RTBRequest and must be set on every call. In particular, bid_request is required even at LIFECYCLE_DSP_BID_RESPONSE, where you operate on bid_response. Omitting it fails to marshal. Echo id back unchanged in RTBResponse.id.

What your container returns

Your GetMutations handler returns an RTBResponse containing the following fields:

Implementation pattern

Structure your handler as follows:

1. Respect tmax. Set a deadline on any inner work using context.WithDeadline.

2. Read signals from bid_request, and from bid_response on the post-bid path.

3. Filter by applicable_intents. Only build mutation types that are eligible for this call.

4. Build mutations per the schema (intent, path, op, value). Use one mutation per entity.

5. Return the response with the same id, your mutations, and metadata.

Next: Implement whichever mutations you need. See Mutation reference for the schema of all six, and Implementing bid shading for additional depth if bid shading is one of them. Then continue to Building a Docker container to package your container.