Listen to this Post
Teams using Kafka often rely on Confluent’s Schema Registry for managing schemas at scale. While it excels with Avro, Protobuf users face challenges like:
– Manually shared `.proto` files
– Custom CI scripts for validation
– Lack of centralized governance
The Buf Schema Registry (BSR) fills these gaps and integrates seamlessly with Confluent, offering:
✅ Centralized API management for Protobuf
✅ Governance, automation, and workflow tools
✅ Auto-generated docs and SDKs
Key Differences
Confluent Schema Registry (CSR)
- Designed for Avro
- Basic Protobuf support
- Manual plugin management
- Minimal policy enforcement
Buf Schema Registry (BSR)
☑ Protobuf-native
☑ Hosted + verified plugins
☑ Enforces breaking change rules
☑ Auto-generates docs & SDKs
☑ Works with Kafka via Confluent SR API
🔗 Learn more:
You Should Know: Protobuf & Kafka Workflow
1. Setting Up Buf CLI
Install the Buf CLI to manage Protobuf schemas:
Linux/macOS curl -sSL https://buf.build/install.sh | sh Verify installation buf --version
2. Validating Protobuf Schemas
Use `buf lint` to enforce best practices:
buf lint --path ./proto
3. Breaking Change Detection
Detect backward-incompatible changes:
buf breaking --against "https://github.com/your-repo.gitbranch=main"
4. Generating Code & Docs
Automatically generate Go/Python/Java code:
buf generate
Generate API documentation:
buf build --as-file-descriptor-set -o - | protoc --descriptor_set_in=/dev/stdin --doc_out=./docs
5. Integrating with Kafka (Confluent)
Use BSR with Kafka via Confluent’s REST Proxy:
Register a Protobuf schema
curl -X POST -H "Content-Type: application/json" \
-d '{"schema": "'$(buf build --as-file-descriptor-set -o - | base64)'"}' \
http://confluent-schema-registry:8081/subjects/proto-topic/versions
6. Kafka Protobuf Producer (Python Example)
from confluent_kafka import Producer
from google.protobuf.json_format import MessageToDict
import your_proto_pb2 as proto
p = Producer({'bootstrap.servers': 'kafka-broker:9092'})
message = proto.YourMessage(name="test", id=123)
p.produce('proto-topic', MessageToDict(message).encode('utf-8'))
p.flush()
7. Kafka Protobuf Consumer (Go Example)
package main
import (
"github.com/confluentinc/confluent-kafka-go/kafka"
"google.golang.org/protobuf/proto"
)
func main() {
c, _ := kafka.NewConsumer(&kafka.ConfigMap{
"bootstrap.servers": "kafka-broker:9092",
"group.id": "proto-group",
})
c.Subscribe("proto-topic", nil)
for {
msg, _ := c.ReadMessage(-1)
var decoded proto.Message
proto.Unmarshal(msg.Value, &decoded)
}
}
What Undercode Say
- BSR is a game-changer for Protobuf-heavy Kafka pipelines.
- Confluent remains best for Avro, but BSR enhances Protobuf governance.
- Use `buf lint` and `buf breaking` for CI/CD enforcement.
- Auto-generated SDKs reduce boilerplate code.
Linux/IT Commands for Protobuf & Kafka
Check Kafka topics kafka-topics --list --bootstrap-server localhost:9092 Consume Protobuf messages in Kafka kafka-console-consumer --topic proto-topic --from-beginning \ --bootstrap-server localhost:9092 \ --property print.key=true Validate Protobuf with protoc protoc --validate_out="lang=go:./output" .proto Monitor Kafka schema registry curl http://localhost:8081/config
Expected Output:
A streamlined Protobuf + Kafka workflow with BSR, reducing manual overhead and improving API governance.
🔗 Further Reading:
References:
Reported By: Nikkisiapno Confluent – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
