4 Pulsar IO Connectors

 

This chapter covers

  • Introduction to the Pulsar IO framework
  • Configuring, deploying, and monitoring Pulsar IO connectors
  • Writing your own Pulsar IO connector in Java

Messaging systems are much more useful when you can easily use them to move data into and out of other external systems such as databases, local and distributed filesystems, or other messaging systems. Consider the scenario from the previous example where we are tasked with developing a real-time leaderboard that will track player scores from around the world and display them back in the game itself, it is quite possible that these computed scores will need to be stored in a time-series database which would require a Pulsar Sink connector to accomplish the task.

Pulsar provides a collection of pre-built connectors that can be used to interact with external systems, such as Apache Cassandra, ElasticSearch, and HDFS just to name a few. The Pulsar IO framework is also extensible which allows you to develop your own connectors to support new or legacy systems as needed.

4.1      What are Pulsar IO Connectors?

4.1.1   Sources and Sinks

4.1.2   Programming Model

4.1.3   Pulsar’s Built-In Connectors

4.1.4   Using the Built-In Connectors

4.2      Developing Pulsar IO Connectors

4.2.1   Developing a Source Connector

4.2.2   Developing a Sink Connector

4.2.3   Packaging and Deploying Pulsar IO Connectors

4.3      Deploying Pulsar IO Connectors

4.4      Administering Pulsar IO Connectors

4.4.1   Creating and Deleting Connectors

4.4.2   Listing Connectors

4.4.3   Monitoring Connectors

4.5      Summary