Source Connectors

Introduction

What is Source Connectors? Source connectors ingest data into Kafka from databases, files, SaaS tools, queues, and CDC logs.

Understanding the topic

What happens — Source Connectors:

• Define connector plugin and config.
• Source: external system → Kafka.
• Sink: Kafka → external system.
• Schema Registry validates event contracts.

Visual explanation

Pipeline view:

Source system
  ↓ Kafka Connect
Topic + Schema Registry
  ↓
Sink connector / consumer
  ↓
Data warehouse/search/app

Step-by-step explanation

1. Deploy Connect — Connect worker cluster.
2. Configure — Connector JSON properties.
3. Run tasks — Parallel ingest/egress.
4. Monitor — Task status and errors.

Informative example

Example:

kafka-topics --bootstrap-server localhost:9092 --create --topic source-connectors --partitions 6 --replication-factor 3
kafka-console-producer --bootstrap-server localhost:9092 --topic source-connectors
kafka-console-consumer --bootstrap-server localhost:9092 --topic source-connectors --from-beginning
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group source-connectors-service

Execution workflow

Best practices

• Register schemas and enforce compatibility before production deploys.
• Use connectors for commodity integration instead of writing one-off sync jobs.
• Document schema owners and evolution rules.
• Monitor connector task failures and dead-letter records.

Common mistakes

• Treating schema-less events as flexible when they are actually fragile.
• Ignoring connector offset and retry behavior.
• Changing field meaning without changing schema or docs.

Summary

Source Connectors — Treat schemas like APIs: version them, test compatibility in CI, and assign ownership.