Distributed Systems Fundamentals.

A distributed system is a collection of independent computers that appear to users as a single coherent system. The main characteristics include:

• Multiple nodes: Components run on separate machines connected by a network
• Concurrency: Multiple processes execute simultaneously across different nodes
• Lack of global clock: No single point of time reference across all nodes
• Independent failures: Components can fail independently without affecting the entire system
• Message passing: Communication happens through network messages rather than shared memory

Examples include web services like Google Search, Netflix streaming, or banking systems where multiple servers work together to serve user requests.

The CAP theorem states that in a distributed system, you can only guarantee two out of three properties:

• Consistency (C): All nodes see the same data simultaneously
• Availability (A): System remains operational and responds to requests
• Partition Tolerance (P): System continues operating despite network failures

Implications: Since network partitions are inevitable in distributed systems, you must choose between consistency and availability:

• CP systems (like MongoDB): Sacrifice availability during partitions to maintain consistency
• AP systems (like Cassandra): Sacrifice consistency to remain available during partitions

Real-world systems often implement "eventual consistency" to balance these trade-offs.

Vertical Scaling (Scale Up):

• Adding more power (CPU, RAM, storage) to existing machines
• Simpler to implement and manage
• Limited by hardware constraints
• Single point of failure
• Example: Upgrading server from 8GB to 32GB RAM

Horizontal Scaling (Scale Out):

• Adding more machines to the system
• Better fault tolerance and theoretically unlimited scaling
• More complex to implement and coordinate
• Requires load distribution mechanisms
• Example: Adding more web servers behind a load balancer

Most modern distributed systems prefer horizontal scaling for better resilience and cost-effectiveness.

Synchronous Communication:

• Sender waits for receiver's response before continuing
• Blocking operation with immediate feedback
• Simpler error handling and debugging
• Can create cascading failures and reduce availability
• Example: HTTP request-response, RPC calls

Asynchronous Communication:

• Sender doesn't wait for response and continues processing
• Non-blocking operation with eventual notification
• Better performance and fault isolation
• More complex error handling and state management
• Example: Message queues, event-driven architectures

# Synchronous
response = api_call()  # Blocks until response
process(response)

# Asynchronous  
api_call_async(callback=process)  # Non-blocking
continue_other_work()

Load balancing distributes incoming requests across multiple servers to prevent overload and ensure high availability.

Main Types:

1. Round Robin: Requests distributed sequentially to each server
2. Least Connections: Routes to server with fewest active connections
3. Weighted Round Robin: Assigns weights based on server capacity
4. IP Hash: Routes based on client IP hash for session persistence
5. Least Response Time: Routes to server with fastest response

Layers:

• Layer 4 (Transport): Routes based on IP and port
• Layer 7 (Application): Routes based on content (HTTP headers, URLs)

Load balancers can be hardware-based, software-based, or cloud-managed services like AWS ALB.

The CAP Theorem, proposed by Eric Brewer, states that in any distributed system, you can guarantee at most two of the following three properties simultaneously:

• Consistency (C): All nodes see the same data at the same time. Every read receives the most recent write or an error.
• Availability (A): The system remains operational and responsive, meaning every request receives a response (success or failure).
• Partition Tolerance (P): The system continues to operate despite network failures that prevent communication between nodes.

The theorem essentially says you must choose between CP (Consistency + Partition Tolerance) or AP (Availability + Partition Tolerance) when network partitions occur, as CA (Consistency + Availability) is impossible in a distributed system with network partitions.

It's impossible because of the fundamental nature of network partitions in distributed systems. When a network partition occurs (nodes cannot communicate), you face a choice:

1. Choose Consistency: Stop serving requests on partitioned nodes to maintain data consistency. This sacrifices availability.
2. Choose Availability: Continue serving requests on all nodes, but risk serving stale data, sacrificing consistency.

You cannot have both because:

• To maintain consistency during a partition, you must either reject requests (losing availability) or wait for network healing (also losing availability)
• To maintain availability, you must serve requests with potentially stale data (losing consistency)

Partition tolerance is mandatory in distributed systems because network failures are inevitable.