Preserve Source Data

Mapped WDM fields let [R]DP clients use shared values such as position and speed. Keeping the source data alongside them gives other applications access to the complete input, including custom attributes, precision, and message structure. The retained input also supports richer mappings later.

Describe the retained data

Each entry in externalRefs identifies a source record and can carry a copy of its content, called the payload. Including the payload lets consumers read it without fetching it separately.

The source format, its version or profile, and a media type identifying the kind of content (such as application/json) help consumers choose a reader. Original bytes preserve the received message; extracted values or text rebuilt from them may omit details or change its structure.

sourceSystem and externalId identify the original record so consumers can match it to the source. capturedAt records when it was captured or copied; provenance.updatedAt holds the source update time.

Pack the payload

ExternalReference.payload is a protobuf Any, a container for a message and its type identifier. Protocol Buffers (protobuf) defines WDM’s schema and data encoding. Packing encodes the message into this container so clients that support its type can decode it.

The reference’s typeUrl must match the packed type. A known protobuf source message can be packed directly; google.protobuf.Struct holds JSON-like fields for generic records or text with format information.

This example shows one source reference in ProtoJSON, protobuf’s JSON representation:

{
  "typeUrl": "type.googleapis.com/google.protobuf.Struct",
  "externalId": "track-7",
  "sourceSystem": "example-source",
  "payload": {
    "@type": "type.googleapis.com/google.protobuf.Struct",
    "value": {
      "@type": "com.example.mapping.v1.SourceRecord",
      "value": {
        "format": "example-json",
        "mediaType": "application/json",
        "representation": "decoded_record",
        "record": {
          "trackId": "track-7",
          "extension": {
            "sensorMode": "training"
          }
        }
      }
    }
  },
  "capturedAt": "2026-10-01T12:00:02Z",
  "label": "Decoded source record"
}

The outer payload.@type tells protobuf to decode a google.protobuf.Struct; its value contains the Struct’s fields. The inner @type, com.example.mapping.v1.SourceRecord, is an application-defined label for the example’s field structure. Clients interpret that label using an agreed definition.

details exposes custom attributes that clients can read and filter, while externalRefs retains the source representation. Keeping both lets consumers use selected values without unpacking the original and still recover the retained input. See Modeling with WDM for this Struct convention.

Store binary data

google.protobuf.BytesValue wraps bytes in a protobuf message that can be packed into Any. The SDK’s Any helper encodes the wrapper into Any.value, allowing consumers to recover the original bytes. In Go:

payload, err := anypb.New(wrapperspb.Bytes(originalBytes))
if err != nil {
    return err
}
// payload.TypeUrl is type.googleapis.com/google.protobuf.BytesValue.

The imports are google.golang.org/protobuf/types/known/anypb and google.golang.org/protobuf/types/known/wrapperspb. In ProtoJSON, the protobuf library represents bytes as base64, a text encoding for binary data. The bytes 00 01 02 7f 80 ff appear as:

{
  "typeUrl": "type.googleapis.com/google.protobuf.BytesValue",
  "externalId": "binary-record-7",
  "sourceSystem": "example-binary-source",
  "payload": {
    "@type": "type.googleapis.com/google.protobuf.BytesValue",
    "value": "AAECf4D/"
  },
  "capturedAt": "2026-10-01T12:00:02Z",
  "label": "Six original binary bytes"
}

See Preserve a Binary Message for a track Object, task Action, and attachment example.

A Struct can hold text and its encoding, as in the CoT example. BytesValue preserves the exact received bytes, including their original encoding.

Store large payloads separately

Large source messages can be stored separately to keep WDM records small. An attachment stores the file’s location and metadata in WDM; consumers fetch its contents from that location. See the binary example. For messages containing several records, a record key or item path identifies the part each mapping used.

Consumers need access to the stored file for as long as they use the reference. If data is partial or unavailable, metadata can state what remains and why. A digest is a short value calculated from file contents. Comparing it and the file’s byte length with the original helps detect changes.

See Map External Data to WDM for source attribution and expiry.