Sink

Pipelines that push data from [R]DP to external systems are common and usually involve Sinks. From a data model standpoint, a Sink is simply a Transformer that has no outputs; otherwise it looks identical to a Transformer and largely functions in the same way.

Sometimes Pipelines need to sink data to terminal data stores inside [R]DP, usually for later querying by a UI or external service. In that case, even though it’s moving data around inside [R]DP, the process is still a Sink.

Examples of Internal Sinks include:

Sometimes Pipelines need to send data outside of [R]DP, so the outputs aren’t natively known to the Transformation Engine. External Sinks must usually be configured with the endpoint of the target system and credentials to publish to it. Note that Sinks in the Transformation Engine are primarily push-based, rather than pull-based. For pull-based data egress, [R]DP exposes a full suite of REST APIs and native Kafka/S3 interfaces.

Examples of External Sinks include: