5 Pulsar IO Connectors

Messaging systems are much more useful when you can easily use them to move data into and out of other external systems such as databases, local and distributed filesystems, or other messaging systems. Consider the scenario where you want to ingest log data from external sources such as applications, platforms, and cloud-based services and publish it to a search engine for analysis. This could easily be accomplished with a pair of Pulsar IO connectors; the first would be a Pulsar source that collects the application logs, and the second would be a Pulsar Sink that writes the formatted records to Elastic Search.

Pulsar provides a collection of pre-built connectors that can be used to interact with external systems, such as Apache Cassandra, ElasticSearch, and HDFS just to name a few. The Pulsar IO framework is also extensible which allows you to develop your own connectors to support new or legacy systems as needed.

5.1      What are Pulsar IO Connectors?

5.1.1      Sink Connectors

5.1.2      Source Connectors

5.1.3      PushSource Connectors

5.2      Developing Pulsar IO Connectors

5.2.1      Developing a Sink Connector

5.2.2      Developing a PushSource Connector

5.3      Testing Pulsar IO Connectors

5.3.1      Unit Testing

5.3.2      Integration Testing

5.3.3      Packaging Pulsar IO Connectors

5.4      Deploying Pulsar IO Connectors

5.4.1      Creating and Deleting Connectors

5.4.2      Debugging Deployed Connectors

5.5      Pulsar’s Built-In Connectors

5.5.1      Launching the MongoDB Cluster

5.5.2      Link the Pulsar and MongoDB Containers

5.5.3      Configure and Create the MongoDB Sink

5.6      Administering Pulsar IO Connectors

5.6.1      Listing Connectors

5.6.2      Monitoring Connectors

5.7      Summary