Observability & Monitoring.

Observability is the ability to understand the internal state of a system by examining its external outputs. It's about being able to ask arbitrary questions about your system's behavior and get answers from the data you collect.
Monitoring is the practice of collecting, aggregating, and acting on metrics and logs from your systems. It typically involves predefined dashboards and alerts for known issues.
Key differences:

• Scope: Monitoring focuses on known problems; observability helps discover unknown issues
• Approach: Monitoring is reactive; observability is more exploratory
• Data: Monitoring uses predetermined metrics; observability requires rich, high-cardinality data
• Questions: Monitoring answers "Is the system working?"; observability answers "Why isn't it working?"

The three pillars of observability are:

1. Metrics: Quantitative data about system performance (CPU usage, response time, error rates)
2. Logs: Discrete events that happened in the system with context and details
3. Traces: Records of requests as they flow through multiple services in distributed systems
   These pillars work together to provide comprehensive visibility:

• Metrics show you WHAT is happening
• Logs tell you WHY it's happening
• Traces show you WHERE it's happening across services

Telemetry data is automatically collected information about the behavior, performance, and health of systems, applications, and infrastructure. It includes metrics, logs, traces, and events generated by software and hardware components.
Importance:

• Proactive issue detection: Identify problems before users experience them
• Performance optimization: Understand bottlenecks and optimization opportunities
• Capacity planning: Make informed decisions about scaling
• Debugging: Troubleshoot issues in production environments
• Business insights: Understand user behavior and system usage patterns
• Compliance: Meet regulatory requirements for system monitoring
  Types of telemetry:
• Performance metrics (latency, throughput)
• Error rates and exceptions
• Resource utilization (CPU, memory, disk)
• Business metrics (user actions, revenue)
• Security events and audit logs

Standard log levels in order of severity:
TRACE: Finest level of detail, typically for debugging specific code paths

TRACE: Entering method calculateTax() with amount=100.50

DEBUG: Detailed information for debugging, not for production

DEBUG: Database query: SELECT * FROM users WHERE id = 123

INFO: General information about application flow

INFO: User 123 successfully logged in

WARN: Indicates a potential issue that doesn't stop the application

WARN: API rate limit approaching: 950/1000 requests used

ERROR: Error conditions that don't stop the application

ERROR: Failed to send email notification: SMTP timeout

FATAL: Severe errors that may cause the application to terminate

FATAL: Cannot connect to database, shutting down

Best practices:

• Use INFO for business events and milestones
• Use WARN for recoverable errors and degraded states
• Use ERROR for actual failures that need attention
• Use DEBUG/TRACE only in development (performance impact)
• Be consistent across your organization
• Include context (user ID, request ID, session ID)

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. Key differences from traditional monitoring systems:

• Pull-based model: Prometheus scrapes metrics from targets rather than receiving pushed data
• Time-series database: Built-in TSDB optimized for metric storage
• Multi-dimensional data model: Metrics identified by metric name and key-value pairs (labels)
• Powerful query language: PromQL for flexible data analysis
• Service discovery: Automatic target discovery from various sources
• No external dependencies: Self-contained system

Traditional systems often use push-based models, require external databases, and have less flexible data models.

Prometheus architecture consists of several key components:

• Prometheus Server: Core component that scrapes and stores metrics, serves queries
• Client Libraries: For instrumenting application code to expose metrics
• Pushgateway: For short-lived jobs that can't be scraped directly
• Exporters: Proxy metrics from third-party systems (node_exporter, blackbox_exporter)
• Alertmanager: Handles alerts sent by Prometheus, manages routing, grouping, and notifications
• Service Discovery: Mechanisms to automatically discover targets
• Grafana: Common visualization layer (though separate project)

Prometheus supports four core metric types:

1. Counter: Cumulative metric that only increases (or resets to zero). Used for requests served, errors occurred, etc.
2. Gauge: Metric that can go up and down. Used for temperature, memory usage, concurrent requests
3. Histogram: Samples observations and counts them in configurable buckets. Also provides sum and count
4. Summary: Similar to histogram but calculates configurable quantiles over a sliding time window

Example:

# Counter
http_requests_total{method="GET",status="200"} 1234

# Gauge  
memory_usage_bytes 8589934592

# Histogram
http_request_duration_seconds_bucket{le="0.1"} 100
http_request_duration_seconds_bucket{le="0.5"} 150

Multiple installation methods:

Binary Installation:

# Download and extract
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
cd prometheus-*

# Create user and directories
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

# Copy binaries and config
sudo cp prometheus promtool /usr/local/bin/
sudo cp -r consoles/ console_libraries/ /etc/prometheus/
sudo cp prometheus.yml /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus/

Package Manager:

# Ubuntu/Debian
sudo apt update && sudo apt install prometheus

# RHEL/CentOS
sudo yum install prometheus

Docker:

docker run -p 9090:9090 prom/prometheus

Kibana is a data visualization and exploration tool that serves as the frontend interface for the Elastic Stack (formerly ELK Stack). It's designed to work with Elasticsearch as its primary data source and provides a web-based interface for searching, viewing, and interacting with data stored in Elasticsearch indices.
In the Elastic Stack architecture:

• Elasticsearch stores and indexes the data
• Logstash processes and transforms data before sending to Elasticsearch
• Beats are lightweight data shippers that collect data
• Kibana visualizes and explores the data
  Kibana allows users to create dashboards, visualizations, and perform real-time data analysis without needing to write complex queries directly against Elasticsearch.

Kibana consists of several key components:
Core Applications:

• Discover: Interactive data exploration and search interface
• Visualize: Create charts, graphs, and other visual representations
• Dashboard: Combine multiple visualizations into unified views
• Canvas: Create custom, pixel-perfect presentations
• Maps: Geospatial data visualization and analysis
  Management Tools:
• Dev Tools: Console for direct Elasticsearch API interaction
• Stack Management: Configure index patterns, saved objects, and system settings
• Stack Monitoring: Monitor Elastic Stack health and performance
• Machine Learning: Anomaly detection and forecasting
• Security: User authentication and role-based access control
• Alerting: Create and manage alerts based on data conditions

Kibana system requirements include:

• RAM: Minimum 1GB, recommended 2GB or more
• CPU: Multi-core processor recommended
• Disk Space: Varies based on usage, typically 200MB for installation

Software Requirements:

• Node.js: Built-in (comes with Kibana installation)
• Operating System: Linux, macOS, or Windows
• Browser: Modern browsers (Chrome, Firefox, Safari, Edge)

Network Requirements:

• Network connectivity to Elasticsearch cluster
• Default port 5601 for Kibana web interface
• HTTPS configuration recommended for production

Elasticsearch Compatibility:

• Kibana version must match Elasticsearch major version
• Minor version differences are typically supported

Jaeger is an open-source, end-to-end distributed tracing system originally developed by Uber. It helps monitor and troubleshoot complex microservices architectures by tracking requests as they flow through multiple services.

Problems it solves:

• Performance bottlenecks: Identifies slow services in request chains
• Error tracking: Pinpoints where failures occur in distributed systems
• Dependency analysis: Maps service interactions and dependencies
• Root cause analysis: Helps debug issues across multiple services
• Service optimization: Provides insights for performance improvements

Jaeger follows the OpenTracing standard and is now part of the Cloud Native Computing Foundation (CNCF).

Distributed tracing tracks requests as they travel through multiple services in a distributed system. It creates a complete picture of how a request is processed across different components.

Key components:

• Trace: Complete journey of a request through the system
• Span: Individual unit of work (e.g., HTTP request, database call)
• SpanContext: Carries trace information between services
• Tags: Key-value pairs that add metadata to spans
• Logs: Timestamped events within spans
• Baggage: Cross-service propagated key-value data

Example flow:

User Request → Service A → Service B → Database
     |            |          |          |
   Trace ID    Span 1     Span 2     Span 3

These are the three pillars of observability, each serving different purposes:

Logging:

• Records discrete events with timestamps
• Good for debugging specific issues
• High volume, text-based
• Example: "User 123 failed login at 2023-10-15 14:30:00"

Metrics:

• Numerical measurements over time
• Good for alerting and dashboards
• Low volume, aggregated data
• Example: CPU usage: 75%, Request rate: 100/sec

Tracing:

• Shows request flow across services
• Good for understanding system behavior
• Medium volume, structured data
• Example: Request path: Frontend → API → Database (150ms total)

Integration: Modern observability uses all three together - traces provide context for logs and metrics.

OpenTelemetry (OTel) is an open-source observability framework that provides a unified approach to collecting, processing, and exporting telemetry data. It addresses the three pillars of observability:

• Traces: Track requests as they flow through distributed systems, showing the path and timing of operations
• Metrics: Numerical measurements collected over time intervals (counters, gauges, histograms)
• Logs: Discrete event records with timestamps and contextual information

OpenTelemetry provides APIs, SDKs, and tools to instrument applications and infrastructure, making it easier to understand system behavior and performance.

OpenTelemetry is the successor and merger of two previous projects:

• OpenTracing: Focused primarily on distributed tracing with vendor-neutral APIs
• OpenCensus: Provided both tracing and metrics collection capabilities

Key differences:

• OpenTelemetry combines both projects' capabilities into a single, comprehensive framework
• Provides unified APIs for traces, metrics, and logs (the three pillars)
• Better standardization and broader vendor support
• More mature ecosystem with extensive instrumentation libraries
• Backward compatibility with both OpenTracing and OpenCensus

OpenTracing and OpenCensus are now in maintenance mode, with new development focused on OpenTelemetry.

The OpenTelemetry ecosystem consists of several key components:

• APIs: Language-specific interfaces for creating telemetry data
• SDKs: Implementations of the APIs with configuration capabilities
• Instrumentation Libraries: Pre-built integrations for popular frameworks and libraries
• Collector: Vendor-agnostic service for receiving, processing, and exporting telemetry data
• Exporters: Components that send data to various backends (Jaeger, Prometheus, etc.)
• Resources: Metadata describing the entity producing telemetry (service name, version, etc.)
• Semantic Conventions: Standard attribute names and meanings for consistent data representation