Building a Microservices Architecture with Spring Boot and Kafka

During my final year at GUC, I led a team of 12 developers to build a Medium replica using a microservices architecture. This project taught me valuable lessons about distributed systems, team collaboration, and scalable architecture design.

The Challenge

Building a content platform like Medium requires handling multiple concerns:

• User authentication and management
• Article creation and publishing
• Media storage and serving
• Real-time notifications
• Search functionality

A monolithic approach would create tight coupling and make scaling difficult. We needed an architecture that could:

• Scale components independently
• Enable parallel development across team members
• Maintain reliability even if some services fail
• Support future feature additions without major refactoring

Architecture Design

We split the application into 5 main microservices:

1. Users Service

Handles user registration, authentication, and profile management. Built with Spring Security and JWT tokens for secure authentication.

2. Product Service

Manages articles including creation, editing, and publishing. Includes versioning and draft functionality to support the writing workflow.

3. Media Service

Handles file uploads, image processing, and CDN integration. Uses AWS S3 for reliable cloud storage.

4. Service Registry

Implements Netflix Eureka for service discovery, allowing services to find each other dynamically without hardcoded URLs.

5. API Gateway

Single entry point for all client requests. Handles routing, authentication, rate limiting, and load balancing across service instances.

Apache Kafka Integration

The key to our microservices communication was Apache Kafka. Instead of direct service-to-service HTTP calls, we used event-driven architecture.

When an article is published, the Article Service publishes an event to Kafka. Other services subscribe to these events and react accordingly. For example:

• Notification Service sends emails to followers
• Search Service indexes the new article
• Analytics Service records the publication

Benefits of Event-Driven Architecture

Loose Coupling: Services don't need to know about each other. They just publish and consume events.

Asynchronous Processing: Heavy operations like sending notifications happen in the background without blocking the main request.

Scalability: We can add more consumers to handle high event volumes during peak traffic.

Reliability: Kafka persists events, so no data is lost if a service is temporarily down. Events are processed when the service recovers.

Challenges We Faced

1. Distributed Transactions

When creating an article required operations across multiple services, maintaining consistency was tricky. We implemented the Saga pattern with compensating transactions to handle failures gracefully.

2. Service Discovery Issues

Initially, services couldn't find each other after restarts. We solved this by properly configuring Eureka health checks and implementing retry logic with exponential backoff.

3. Debugging Complexity

Tracing requests across multiple services was challenging. We added correlation IDs to track requests end-to-end and implemented centralized logging.

4. Data Consistency

With each service having its own database, ensuring data consistency across services required careful design. We used eventual consistency where appropriate and implemented data synchronization strategies.

Lessons Learned

Start Simple: We initially over-engineered with too many microservices. Consolidating some services improved performance and reduced complexity.

Invest in Monitoring: Without proper logging and monitoring, debugging distributed systems is nearly impossible. We integrated ELK stack (Elasticsearch, Logstash, Kibana) for centralized logging.

API Versioning: Breaking changes in one service affected others. Implementing proper API versioning from the start would have saved significant time.

Team Communication: With 12 developers working on different services, clear API contracts and regular sync meetings were crucial for success.

Database per Service: While this pattern provides independence, it also introduces complexity in querying across services. Plan your data access patterns carefully.

Performance Optimizations

We implemented several optimizations:

• Caching: Redis for frequently accessed data
• Load Balancing: Nginx for distributing traffic
• Connection Pooling: HikariCP for database connections
• Async Processing: CompletableFuture for non-blocking operations

Security Considerations

Security in microservices requires special attention:

• JWT tokens for authentication
• Service-to-service authentication
• API Gateway as security perimeter
• Rate limiting to prevent abuse
• Input validation at every service boundary

Key Takeaways

Building this microservices architecture taught me that:

• Microservices aren't always the answer - use them when you need independent scaling
• Event-driven architecture with Kafka provides excellent decoupling
• DevOps and monitoring are as important as the code itself
• Team organization should mirror service boundaries (Conway's Law)
• Start with a monolith and split when needed, rather than starting with microservices

Tech Stack Summary

• Backend: Java 11, Spring Boot 2.5, Spring Cloud
• Messaging: Apache Kafka 2.8
• Discovery: Netflix Eureka
• Gateway: Spring Cloud Gateway
• Database: PostgreSQL (per service)
• Caching: Redis
• Authentication: JWT with Spring Security
• Monitoring: ELK Stack, Prometheus
• Deployment: Docker, Kubernetes

Results

The project successfully demonstrated:

• Independent service deployment and scaling
• Fault isolation (one service failure doesn't bring down the system)
• Parallel development by multiple teams
• Scalability under load testing
• Production-ready architecture patterns

---

Want to see the code? Check out the GitHub repository

Questions or suggestions? Feel free to reach out through the contact form!