Use of Kafka
Status
Accepted
Context
- System needs to be able to handle high volume of parcels.
- System needs to scale easily as the number of parcels grows.
- System needs to be fault-tolerant.
- System needs to be upgradable without downtime.
Decision
Use of event based architecture looks like no-brainer.
- It allows easy scaling as the system grows.
- It allows partial upgrades as long as events are persistent
- It is possible to plan introducing breaking changes without downtime.
- Kafka is a stable, well established and proven solution for event based architecture (Netflix for reference)
Consequences
- Needs maintenance and monitoring
- Suboptimal configuration of Kafka can lead to performance issues
- Breaking changes rollouts needs to be carefully planed in advance in production environment
| Compatibility Type | Changes allowed | Check against which schemas | Upgrade first |
|---|---|---|---|
| BACKWARD | * Delete fields * Add optional fields |
Last version | Consumers |
| BACKWARD_TRANSITIVE | * Delete fields * Add optional fields |
All previous versions | Consumers |
| FORWARD | * Add fields * Delete optional fields |
Last version | Producers |
| FORWARD_TRANSITIVE | * Add fields * Delete optional fields |
All previous versions | Producers |
| FULL | * Add optional fields * Delete optional fields |
Last version | Any order |
| FULL_TRANSITIVE | * Add optional fields * Delete optional fields |
All previous versions | Any order |
| NONE | * All changes are accepted | Compatibility checking disabled | Depends |
Source: https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html