Source

Pipelines that ingest data from external sources are common in [R]DP, so the concept of a Source was created to encapsulate their functionality. From a data model standpoint, a Source is simply a Transformer that has no inputs; otherwise it looks identical to a Transformer and largely functions in the same way.

Ingestor Sources are used to pull data from external systems. Typically, Ingestor Sources must be configured with the endpoint of the source system, credentials to access it, and details about the query to issue or data to pull from it. The output of an Ingestor Source is often (but not always) a streaming Dataset, which enables the ingest process to push data into [R]DP as soon as it receives it. Downstream Transformers are then responsible for taking those ingested messages and aggregating them appropriately.

Examples of Ingestor Sources in [R]DP include:

Generator Sources push data into [R]DP, but instead of connecting to an external source, they generate data via internal proceses. This is helpful for development and testing since Generator Sources can be easily swapped out for Ingestor Sources once a connection to a real upstream system can be established. Generator Sources also commonly offer higher fidelity over the volume and speed of data ingestion, making them suitable for load testing.

Examples of Generator Sources in [R]DP include: