Observability is the ability to understand the internal state of a system by examining its external outputs. It's about being able to ask arbitrary questions about your system's behavior and get answers from the data you collect.
Monitoring is the practice of collecting, aggregating, and acting on metrics and logs from your systems. It typically involves predefined dashboards and alerts for known issues.
Key differences:

• Scope: Monitoring focuses on known problems; observability helps discover unknown issues
• Approach: Monitoring is reactive; observability is more exploratory
• Data: Monitoring uses predetermined metrics; observability requires rich, high-cardinality data
• Questions: Monitoring answers "Is the system working?"; observability answers "Why isn't it working?"

The three pillars of observability are:

1. Metrics: Quantitative data about system performance (CPU usage, response time, error rates)
2. Logs: Discrete events that happened in the system with context and details
3. Traces: Records of requests as they flow through multiple services in distributed systems
   These pillars work together to provide comprehensive visibility:

• Metrics show you WHAT is happening
• Logs tell you WHY it's happening
• Traces show you WHERE it's happening across services

Telemetry data is automatically collected information about the behavior, performance, and health of systems, applications, and infrastructure. It includes metrics, logs, traces, and events generated by software and hardware components.
Importance:

• Proactive issue detection: Identify problems before users experience them
• Performance optimization: Understand bottlenecks and optimization opportunities
• Capacity planning: Make informed decisions about scaling
• Debugging: Troubleshoot issues in production environments
• Business insights: Understand user behavior and system usage patterns
• Compliance: Meet regulatory requirements for system monitoring
  Types of telemetry:
• Performance metrics (latency, throughput)
• Error rates and exceptions
• Resource utilization (CPU, memory, disk)
• Business metrics (user actions, revenue)
• Security events and audit logs

Standard log levels in order of severity:
TRACE: Finest level of detail, typically for debugging specific code paths

TRACE: Entering method calculateTax() with amount=100.50

DEBUG: Detailed information for debugging, not for production

DEBUG: Database query: SELECT * FROM users WHERE id = 123

INFO: General information about application flow

INFO: User 123 successfully logged in

WARN: Indicates a potential issue that doesn't stop the application

WARN: API rate limit approaching: 950/1000 requests used

ERROR: Error conditions that don't stop the application

ERROR: Failed to send email notification: SMTP timeout

FATAL: Severe errors that may cause the application to terminate

FATAL: Cannot connect to database, shutting down

Best practices:

• Use INFO for business events and milestones
• Use WARN for recoverable errors and degraded states
• Use ERROR for actual failures that need attention
• Use DEBUG/TRACE only in development (performance impact)
• Be consistent across your organization
• Include context (user ID, request ID, session ID)