# AWS infrastructure audit
sao run --cloud aws --terraform-state s3://bucket/prod.tfstate

# Yandex Cloud audit
sao run --cloud yandex --terraform-state s3://bucket/yc-prod.tfstate

# On-premise (local state)
sao run --cloud onprem --terraform-state ./terraform.tfstate

# Docker image SBOM
sao extract sbom --type docker --target nginx:1.25-alpine

# Git repository dependencies
sao extract sbom --type git --target ./my-app

# OS packages
sao extract sbom --type os --target ubuntu:22.04

# CI/CD build artifacts
sao extract sbom --type build --target ./build-manifest.json

# Row-Level Security enforced at database level
class TenantMiddleware:
    async def __call__(self, request, call_next):
        tenant_id = extract_tenant(request)
        # All queries automatically filtered by tenant
        set_tenant_context(tenant_id)
        return await call_next(request)

// WebSocket connection for live security events
const ws = new WebSocket('wss://api.example.com/ws/alerts');
ws.onmessage = (event) => {
    const alert = JSON.parse(event.data);
    // Severity: critical, high, medium, low
    if (alert.severity === 'critical') {
        showNotification(alert);
    }
};

# Comprehensive vulnerability tracking
class Vulnerability(Base):
    cve_id: str              # CVE-2024-XXXXX
    severity: SeverityEnum   # CRITICAL, HIGH, MEDIUM, LOW
    cvss_score: float        # 0.0 - 10.0
    affected_assets: List[Asset]
    remediation_status: RemediationStatus
    due_date: datetime
    sla_breach: bool         # Computed from severity + age

# Every state-changing operation logged
@audit_log
async def create_finding(finding: FindingCreate):
    # Automatically captures:
    # - User ID, tenant ID
    # - Timestamp, IP address
    # - Before/after state
    # - Operation type
    return await finding_service.create(finding)

Wazuh Architecture:

Management Layer:
  Wazuh Manager Cluster (HA):
    - 3 manager nodes (active-active)
    - Load balancing for agent connections
    - Shared configuration via cluster sync
    - Deployed on AWS EC2 (m5.xlarge)

Data Storage Layer:
  OpenSearch Cluster:
    - 5 data nodes (m5.2xlarge)
    - 2 master nodes (m5.xlarge)
    - S3 for snapshot backups
    - 90-day hot retention
    - 1-year cold storage (S3 Glacier)

Agent Layer:
  200+ Wazuh Agents:
    - EKS worker nodes (100+)
    - EC2 instances (80+)
    - RDS/Aurora monitoring (indirect via CloudWatch)
    - Container-based agents (DaemonSet)

Integration Layer:
  AWS Services:
    - CloudTrail → S3 → Wazuh ingestion
    - VPC Flow Logs → S3 → Wazuh processing
    - GuardDuty → EventBridge → Wazuh
    - Security Hub → aggregation
    - Config → compliance data
    - ALB/WAF logs → S3 → analysis

Deployment:
  Platform: AWS EC2 (Auto Scaling Group)
  Instance Type: m5.xlarge (4 vCPU, 16 GB RAM)
  Count: 3 nodes (active-active cluster)
  OS: Ubuntu 22.04 LTS (CIS hardened)

Configuration:
  Cluster Communication:
    - Wazuh cluster protocol (port 1516)
    - Shared configuration synchronization
    - Automatic failover
    - Load balanced agent connections

  Agent Communication:
    - Port 1514: Agent data ingestion
    - Port 1515: Agent enrollment
    - TLS encryption enforced
    - Certificate-based authentication

  API:
    - RESTful API (port 55000)
    - JWT token authentication
    - Integration with automation tools
    - Rate limiting enabled

Storage:
  - 500 GB EBS gp3 for local buffer
  - S3 for long-term archive
  - Daily snapshots to S3

Cluster Design:
  Data Nodes:
    - Count: 5 nodes
    - Instance: m5.2xlarge (8 vCPU, 32 GB RAM)
    - Storage: 2 TB EBS gp3 per node
    - Purpose: Index and search operations

  Master Nodes:
    - Count: 2 nodes (quorum)
    - Instance: m5.xlarge (4 vCPU, 16 GB RAM)
    - Purpose: Cluster state management

Index Management:
  Hot Tier (0-30 days):
    - SSD storage (gp3)
    - High IOPS for real-time queries
    - Daily index rotation
    - Replica count: 1

  Warm Tier (31-90 days):
    - SSD storage (gp3)
    - Reduced replica count
    - Force merge for optimization

  Cold Tier (91-365 days):
    - S3 storage via snapshots
    - Searchable snapshots
    - Minimal compute cost

Security:
  - TLS 1.3 for all connections
  - OpenSearch Security plugin
  - Role-based access control (RBAC)
  - Audit logging enabled
  - VPC private subnet deployment
  - Security group restrictions

Deployment Method: DaemonSet
  Purpose: One agent per node
  Resource Limits:
    CPU: 200m request, 500m limit
    Memory: 256Mi request, 512Mi limit

Container Configuration:
  Image: wazuh/wazuh-agent:4.8.0
  Security Context:
    - Privileged: true (for host monitoring)
    - hostPID: true
    - hostNetwork: true

  Volumes:
    - /var/log → container logs
    - /var/ossec → agent data
    - /etc/os-release → OS detection
    - /var/run/docker.sock → container monitoring

Monitoring Capabilities:
  - Container lifecycle events
  - Kubernetes audit logs
  - Pod security violations
  - Node system logs
  - File integrity monitoring
  - Rootkit detection

Installation:
  Method: Ansible playbook automation
  OS Support:
    - Amazon Linux 2 / 2023
    - Ubuntu 20.04 / 22.04
    - CentOS 7 / 8

  Enrollment:
    - Automated via API
    - Certificate-based authentication
    - Group assignment by tags

Configuration Profile by Role:
  Web Servers:
    - Apache/Nginx log monitoring
    - Web attack detection
    - SSL/TLS monitoring

  Database Servers:
    - PostgreSQL audit logs
    - Failed authentication attempts
    - Privilege escalation detection

  Application Servers:
    - Application log parsing
    - API abuse detection
    - Performance metrics

Payment-Specific Threats:
  PAN Data Access Detection:
    - Regex patterns for credit card numbers
    - Unauthorized database queries
    - File access to cardholder data
    - Network transmission of sensitive data
    - Alert severity: CRITICAL

  Transaction Anomalies:
    - Unusual transaction amounts
    - Rapid transaction frequency
    - Geographic anomalies
    - Velocity checks (same card, multiple locations)
    - ML-based anomaly detection

Authentication & Access:
  Brute Force Detection:
    - Failed SSH attempts (5+ in 1 min)
    - Failed API authentication (10+ in 5 min)
    - Account lockout monitoring
    - Distributed brute force detection

  Privilege Escalation:
    - Sudo usage monitoring
    - IAM permission changes
    - Role assumption tracking
    - Unauthorized service account usage

Web Attacks:
  OWASP Top 10 Detection:
    - SQL injection attempts
    - Cross-Site Scripting (XSS)
    - Command injection
    - Path traversal
    - Insecure deserialization
    - XML External Entities (XXE)

  API Abuse:
    - Rate limiting violations
    - Invalid API token usage
    - Unusual API endpoints
    - Parameter tampering

Data Exfiltration:
  Indicators:
    - Large data transfers (>100MB)
    - Unusual outbound connections
    - Database dumps
    - SSH/SCP file transfers
    - S3 bucket data access anomalies

AWS CloudTrail:
  High-Risk Events:
    - IAM policy changes
    - Security group modifications
    - S3 bucket policy changes
    - Root account usage
    - KMS key deletion attempts
    - CloudTrail logging disabled

  Compliance Events:
    - Encryption disabled on resources
    - Public access enabled
    - Unencrypted snapshots
    - Cross-region resource access

GuardDuty Findings:
  - Malware detection
  - Cryptocurrency mining
  - Backdoor detection
  - Unusual API calls
  - Compromised credentials
  - Data exfiltration attempts

VPC Flow Logs:
  - Port scanning detection
  - DDoS indicators
  - Unusual traffic patterns
  - Blocked connection attempts
  - Internal lateral movement

Monitored Files (10,000+):

System Files:
  Linux:
    - /etc/passwd, /etc/shadow
    - /etc/ssh/sshd_config
    - /etc/sudoers
    - /boot/grub/grub.cfg
    - Systemd service files

  Frequency: Real-time
  Actions: Alert + snapshot

Configuration Files:
  Application:
    - Nginx/Apache configs
    - Application .env files
    - Database configuration
    - SSL certificates

  Kubernetes:
    - Pod manifests
    - ConfigMaps
    - Secrets (metadata only)
    - Service definitions

  Frequency: Real-time
  Actions: Alert + backup + change review

Code Directories:
  - /var/www/html
  - /opt/applications
  - Container image layers

  Frequency: Scheduled (daily)
  Actions: Alert on unauthorized changes

Logs & Audit:
  - /var/log/*
  - Application log directories
  - Audit logs

  Frequency: Real-time
  Actions: Detect log tampering

FIM Capabilities:
  - Real-time change detection
  - File checksum (SHA256)
  - File attributes (permissions, owner)
  - Who-data (who made the change)
  - Baseline comparison
  - Automated restoration (critical files)

Vulnerability Detection:
  Methods:
    - Agent-based scanning
    - Package manager integration (apt, yum)
    - CVE database correlation
    - OVAL definitions

  Scan Frequency:
    - Critical systems: Daily
    - Production servers: Daily
    - Non-production: Weekly
    - Containers: On image push

Risk-Based Prioritization:
  Scoring:
    - CVSS base score
    - Exploitability (EPSS)
    - Asset criticality
    - Network exposure
    - Data sensitivity

  SLA by Severity:
    - Critical: 24 hours
    - High: 7 days
    - Medium: 30 days
    - Low: 90 days

Integration:
  - Jira ticket creation
  - Slack notifications
  - Email alerts to teams
  - Dashboard for management
  - Monthly vulnerability reports

Remediation Tracking:
  - Patch deployment via Ansible
  - Verification scanning
  - Exception management
  - Compliance reporting

PCI DSS Dashboard (150+ Checks):

Requirement 1 & 2: Network & Configuration:
  Checks:
    - Firewall rules in place
    - Default passwords changed
    - Unnecessary services disabled
    - Configuration standards enforced

Requirement 3 & 4: Data Protection:
  Checks:
    - Encryption at rest enabled
    - TLS version compliance (1.2+)
    - Key rotation schedules
    - Sensitive data masking

Requirement 5 & 6: Malware & Development:
  Checks:
    - Anti-malware running
    - Malware signature updates
    - Secure coding practices
    - Change management process

Requirement 7 & 8: Access Control:
  Checks:
    - Least privilege enforcement
    - User access reviews
    - MFA enabled
    - Password complexity

Requirement 10: Logging & Monitoring:
  Checks:
    - Log collection enabled
    - Log retention (1 year)
    - Clock synchronization (NTP)
    - Log integrity protection
    - Audit trail completeness

Requirement 11: Security Testing:
  Checks:
    - Vulnerability scans completed
    - Penetration test schedule
    - IDS/IPS operational
    - File integrity monitoring

Automated Evidence Collection:
  - Daily compliance snapshots
  - Configuration backups
  - Change logs
  - Access reports
  - Exception documentation
  - Quarterly audit packages

Active Response:
  Automated Actions:
    IP Blocking:
      - Trigger: 5+ failed SSH attempts
      - Action: iptables block for 30 minutes
      - Scope: Source IP

    Account Lockout:
      - Trigger: 10+ failed logins
      - Action: Disable account
      - Notification: Security team + manager

    Container Quarantine:
      - Trigger: Malware detected in container
      - Action: Kill pod, taint node
      - Notification: DevOps + Security

    Process Kill:
      - Trigger: Cryptocurrency miner detected
      - Action: Kill process, block binary
      - Forensics: Memory dump

Incident Management:
  PagerDuty Integration:
    - Critical alerts → immediate page
    - High severity → notification
    - Escalation after 15 minutes
    - 24/7 SOC coverage

  Workflow:
    1. Alert triggered in Wazuh
    2. PagerDuty incident created
    3. On-call engineer notified
    4. Investigation in OpenSearch
    5. Response action (manual/automated)
    6. Incident documentation
    7. Post-incident review

Playbooks:
  - Brute force attack response
  - Malware infection containment
  - Data breach procedure
  - DDoS mitigation
  - Insider threat investigation
  - Compromised credentials

Alert Channels:
  Email:
    - Daily summary reports
    - Critical alerts (immediate)
    - Weekly compliance reports

  Slack:
    - Real-time alerts (high+)
    - Compliance violations
    - System health issues

  PagerDuty:
    - Critical security events
    - System outages
    - Escalation after 15 min

Custom Dashboards:
  Security Operations Center (SOC):
    - Real-time event stream
    - Alert count by severity
    - Top attacked assets
    - Geographic threat map
    - MITRE ATT&CK mapping

  Executive Dashboard:
    - Security posture score
    - Compliance status (%)
    - Incident trends
    - Vulnerability metrics
    - Cost of incidents

  Compliance Dashboard:
    - PCI DSS requirement status
    - Failed compliance checks
    - Remediation progress
    - Audit readiness score

module "cde_vpc" {
  source = "./modules/secure-vpc"
  
  name               = "cde-payment"
  cidr               = "10.0.0.0/16"
  azs                = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  
  # Security controls
  enable_nat_gateway = true
  single_nat_gateway = false
  
  tags = {
    Environment = "production"
    Compliance  = "pci-dss"
    DataClass   = "cardholder"
  }
}

Organization Structure (15+ Accounts):

Management OU:
  - management: Root account, Control Tower, Organizations
  - logging: Centralized logging (CloudTrail, Config, Flow Logs)
  - security: Security Hub, GuardDuty findings aggregation
  - audit: Read-only audit access for compliance

Infrastructure OU:
  - network: Transit Gateway, shared networking
  - shared-services: DNS, Active Directory, central repositories

Workloads OU:
  Production:
    - prod-cde: PCI DSS Cardholder Data Environment
    - prod-non-cde: Non-CDE production workloads

  Non-Production:
    - staging: Pre-production testing environment
    - dev: Development environment
    - sandbox: Experimentation and POCs

Security OU:
  - security-tooling: Security tools and scanning
  - incident-response: IR automation and forensics

Transit Gateway (TGW) Hub:
  Purpose: Central routing for all VPCs
  Regions:
    - Primary: eu-west-1 (Ireland)
    - DR: eu-west-2 (London)

  Routing:
    - Centralized egress via NAT Gateways
    - Inter-VPC communication controls
    - On-premise connectivity (future VPN/Direct Connect)
    - Route table segmentation for CDE isolation

VPC Design per Account:
  Subnets:
    - Public: NAT GW, ALB, bastion (jump hosts)
    - Private: Application tier, EKS nodes
    - Data: Databases, ElastiCache, MSK
    - Management: Systems Manager endpoints

  CIDR Strategy:
    - Non-overlapping ranges across all accounts
    - /16 for production, /20 for non-production
    - Reserved ranges for future expansion

Network Firewall:
  - Centralized in network account
  - Deep packet inspection
  - Intrusion prevention (IPS)
  - Domain filtering for egress
  - Threat intelligence integration

Multi-Layer Protection:
  1. NACLs: Subnet-level stateless filtering
  2. Security Groups: Instance-level stateful filtering
  3. WAF: Application layer protection (ALB/CloudFront)
  4. Shield Standard: DDoS protection (all accounts)
  5. VPC Flow Logs: Network traffic analysis

Repository Structure:
  terraform-live/
    ├── management/
    ├── production/
    │   ├── eu-west-1/
    │   │   ├── vpc/
    │   │   ├── eks/
    │   │   ├── rds/
    │   │   └── security/
    │   └── eu-west-2/
    ├── staging/
    └── modules/
        ├── vpc/
        ├── eks/
        ├── rds-aurora/
        └── security-baseline/

Terraform Stack:
  - 1000+ AWS resources managed
  - Terragrunt for DRY configuration
  - Remote state: S3 + DynamoDB locking
  - State encryption with KMS
  - Module versioning and testing

Security Scanning:
  - Checkov: Compliance and security checks
  - tfsec: Terraform security scanning
  - Terrascan: Policy as code enforcement
  - Automated in CI/CD before apply
  - Drift detection and remediation

Pipeline Stages:
  1. Validate:
     - terraform validate
     - terraform fmt check
     - Module version verification

  2. Security Scan:
     - Checkov (CIS, PCI DSS policies)
     - tfsec (AWS security best practices)
     - Secret detection (gitleaks)

  3. Plan:
     - terraform plan
     - Cost estimation (Infracost)
     - Plan review and approval

  4. Apply:
     - Manual approval gate
     - terraform apply
     - Drift detection scheduling

Landing Zone Features:
  Account Factory:
    - Automated account provisioning
    - Baseline security configuration
    - IAM Identity Center (SSO) integration
    - CloudTrail and Config enabled by default

  Guardrails (SCPs):
    Mandatory:
      - Deny disabling CloudTrail
      - Deny modifying Config rules
      - Deny root user access keys
      - Enforce MFA for root user

    Strongly Recommended:
      - Deny leaving organization
      - Deny disabling EBS encryption
      - Deny public S3 buckets
      - Enforce encrypted volumes

    Custom (PCI DSS):
      - Deny non-approved regions
      - Enforce KMS encryption
      - Restrict instance types
      - Deny IMDSv1 (require IMDSv2)

Account Baseline:
  - VPC with private subnets
  - NAT Gateway for outbound
  - VPC endpoints for AWS services
  - CloudWatch log groups
  - SNS topics for alerts
  - Systems Manager access

Production EKS Clusters:
  Primary (eu-west-1):
    - 3 Availability Zones
    - Managed node groups (on-demand)
    - Spot instances for batch jobs
    - Fargate for serverless workloads

  DR (eu-west-2):
    - Pilot-light configuration
    - Minimal capacity (cost-optimized)
    - Automated scale-up on failover

Node Configuration:
  - Instance types: m5.xlarge, r5.xlarge
  - Auto Scaling: Cluster Autoscaler
  - OS: Amazon Linux 2
  - Container runtime: containerd
  - IRSA for pod-level IAM permissions

Control Plane:
  - Control plane logging to CloudWatch
  - Private endpoint (VPC-only access)
  - Kubernetes version: 1.27+
  - Encryption: KMS for secrets

Deployment Strategy:
  - ArgoCD deployed in EKS
  - Git as single source of truth
  - 80+ microservices managed
  - Application-per-repo pattern
  - Automated sync (with approval for prod)

Progressive Delivery (Argo Rollouts):
  Strategies:
    - Blue-Green deployments
    - Canary releases (10% → 50% → 100%)
    - Automated rollback on metrics

  Analysis:
    - Prometheus metrics integration
    - Success rate, latency, error rate
    - Automated promotion or rollback

Prometheus Architecture:
  - Thanos for long-term storage (S3)
  - Multi-cluster monitoring
  - 7-day local retention
  - 1-year Thanos retention
  - AlertManager for notifications

Grafana Dashboards:
  Infrastructure:
    - EKS cluster health
    - Node and pod metrics
    - Network performance
    - Storage utilization

  Application:
    - Service-level metrics
    - Payment processing metrics
    - API response times
    - Error rates and SLIs

  Security:
    - GuardDuty findings
    - WAF blocked requests
    - Failed authentication attempts
    - Compliance posture

  Cost:
    - Per-service costs (Kubecost)
    - AWS Cost Explorer integration
    - Budget vs actual tracking

OpenSearch (ELK):
  - Centralized log aggregation
  - 30-day retention in hot tier
  - 1-year retention in S3 (cold tier)
  - Vector for log collection
  - Kibana for visualization

Loki + Promtail:
  - Kubernetes-native logging
  - Label-based log queries
  - Grafana integration
  - Lower storage costs vs ELK
  - Real-time log streaming

Log Sources:
  - Application logs (stdout/stderr)
  - AWS CloudTrail (API calls)
  - VPC Flow Logs (network traffic)
  - EKS control plane logs
  - Load balancer access logs
  - WAF logs

Configuration:
  - Centralized user management
  - Azure AD integration (SAML)
  - MFA enforcement
  - Permission sets per role:
    * Admin: Full access (break-glass only)
    * DevOps: Infrastructure management
    * Developer: Application deployment
    * ReadOnly: Audit and compliance
    * Security: Security tools access

  Access Patterns:
    - Time-limited sessions (8 hours)
    - JIT access for production
    - Approval workflow for sensitive accounts
    - Audit logging of all access

HashiCorp Vault:
  Deployment:
    - HA cluster on EKS
    - Auto-unseal with AWS KMS
    - Consul storage backend
    - Cross-region replication

  Use Cases:
    - Database credentials (dynamic)
    - API keys and tokens
    - TLS certificates (PKI engine)
    - Encryption as a service

  Authentication:
    - Kubernetes auth for pods
    - AWS IAM for services
    - OIDC for users

AWS Secrets Manager:
  - RDS password rotation
  - Cross-account secret sharing
  - Lambda rotation functions
  - Backup to Vault for redundancy

Amazon Aurora PostgreSQL:
  Configuration:
    - Multi-AZ deployment
    - Read replicas (3x)
    - Cross-region read replica (DR)
    - Performance Insights enabled

  Security:
    - Encryption at rest (KMS)
    - Encryption in transit (TLS 1.3)
    - IAM database authentication
    - Private subnet deployment
    - Security group restrictions

  Backup:
    - Automated daily snapshots
    - 35-day retention
    - Cross-region snapshot copy
    - Point-in-time recovery (PITR)

MongoDB Atlas:
  - Managed service (AWS VPC peering)
  - Replica set configuration
  - Automated backups
  - Performance monitoring

ElastiCache Redis:
  - Cluster mode enabled
  - Multi-AZ automatic failover
  - Encryption in-transit and at-rest
  - Session storage and caching

Multi-Region Strategy:
  Primary: eu-west-1 (Ireland)
  DR: eu-west-2 (London)

  Approach: Pilot Light
    - Network infrastructure pre-deployed
    - EKS cluster in standby (minimal nodes)
    - Database read replica in DR region
    - S3 cross-region replication
    - Route 53 health checks and failover

Automation:
  - Lambda-based failover orchestration
  - Automated DNS cutover (Route 53)
  - EKS cluster scale-up automation
  - Database promotion scripts
  - Runbooks in Confluence

Testing:
  - Quarterly DR drills
  - Documented runbooks
  - Automated validation scripts
  - RTO: 4 hours
  - RPO: 15 minutes

Backup Strategy:
  - Velero for EKS (daily)
  - RDS automated snapshots
  - S3 versioning enabled
  - Configuration backups in Git
  - 3-2-1 backup rule adherence

On-Premise Infrastructure (Colocation):
  Purpose: Core trading engine and cold wallet storage
  Resources:
    - 50 physical servers
    - VMware ESXi virtualization platform
    - Ceph distributed storage (200TB)
    - OPNsense firewall cluster

Hetzner Cloud:
  Purpose: Additional compute and redundancy
  Resources:
    - Dedicated servers
    - Automated provisioning via Ansible
    - Load balancing tier

Google Cloud Platform:
  Purpose: Public-facing services and analytics
  Resources:
    - GKE (Google Kubernetes Engine)
    - Cloud SQL for relational data
    - Cloud Armor for DDoS protection
    - Global load balancing

Production Clusters:
  GKE (Google Cloud):
    - Public-facing trading interface
    - API gateway services
    - Real-time market data feeds
    - User authentication services

  K3s (On-Premise):
    - Core trading engine
    - Order matching engine
    - Wallet management services
    - Blockchain node management

  Management:
    - Rancher for centralized cluster management
    - Unified monitoring and logging
    - Cross-cluster service mesh

Nexus Registry:
  - Private container registry
  - Vulnerability scanning integration
  - Image signing and verification
  - Access control and audit logging

Security Measures:
  - Network policies for pod-to-pod communication
  - RBAC with least privilege access
  - Secret management with encrypted storage
  - Regular security scanning and updates

Architecture:
  Pools:
    - Hot data pool (SSD): Trading data, active wallets
    - Cold data pool (HDD): Historical data, backups
    - Metadata pool: File system metadata

  Replication:
    - 3x replication for critical data
    - 2x replication for warm data
    - Erasure coding for cold storage

  Performance:
    - IOPS optimization for trading engine
    - Low-latency access for hot wallets
    - Bandwidth optimization for blockchain sync

Other Storage Solutions:
  - Linstor for Kubernetes persistent volumes
  - PortWorx for database workloads
  - MinIO for object storage (S3 compatible)
  - NFS for shared application data

Supported Blockchains:
  - Bitcoin (BTC): Full node + pruned nodes
  - Ethereum (ETH): Geth full nodes
  - Litecoin (LTC): Full node
  - Other altcoins: Selective node deployment

Node Management:
  - Automated synchronization monitoring
  - Health checks and auto-healing
  - Version management and updates
  - Performance optimization

Hot Wallets (Online):
  Location: Kubernetes pods with strict security
  Purpose: Active trading and withdrawals
  Security:
    - Multi-signature requirements
    - Rate limiting on withdrawals
    - Real-time monitoring and alerts
    - Encrypted keys with HSM integration

Cold Wallets (Offline):
  Location: Air-gapped servers in colocation
  Purpose: Long-term storage of customer funds
  Security:
    - Hardware security modules (HSM)
    - Physical security controls
    - Multi-party authorization
    - Regular security audits

Warm Wallets (Semi-Online):
  Purpose: Balance between hot and cold
  Process: Automated cold-to-warm-to-hot transfers

Pipeline Architecture:
  - Jenkins master on Kubernetes
  - Dynamic agent provisioning
  - Parallel job execution
  - Docker-in-Docker builds

Stages:
  1. Code checkout and validation
  2. Unit and integration tests
  3. Security scanning:
     - Trivy for vulnerabilities
     - SonarQube for code quality
  4. Container image build and push
  5. Helm chart packaging
  6. Deployment to staging
  7. Automated testing
  8. Production deployment (manual approval)

GitLab Integration:
  - Self-hosted GitLab instance
  - Git repository management
  - Code review and merge requests
  - 100+ Helm charts for deployments

OPNsense Firewall:
  - High-availability cluster
  - Intrusion Detection System (IDS)
  - Intrusion Prevention System (IPS)
  - VPN for secure remote access
  - Traffic analysis and logging

Network Segmentation:
  - Isolated trading network
  - Separate blockchain node network
  - DMZ for public-facing services
  - Management network isolation
  - Strict firewall rules between segments

DDoS Protection:
  - Cloud Armor (GCP) for public endpoints
  - Rate limiting at multiple layers
  - Traffic scrubbing and filtering
  - Automated incident response

Security Measures:
  - Two-factor authentication (2FA) mandatory
  - IP whitelisting for API access
  - API rate limiting per user/IP
  - Session management and timeout
  - Encrypted communication (TLS 1.3)
  - Regular penetration testing
  - Bug bounty program

Infrastructure Monitoring (Zabbix):
  - Server hardware metrics
  - Network device monitoring
  - Service availability checks
  - Capacity planning metrics
  - Alerting and escalation

Application Monitoring (Prometheus/Grafana):
  - Trading engine performance
  - Order processing latency
  - Wallet transaction metrics
  - API response times
  - Custom business metrics

Log Aggregation (ELK Stack):
  - Centralized logging
  - Security event correlation
  - Audit trail for compliance
  - Real-time log analysis
  - Long-term log retention

Distributed Tracing (Jaeger):
  - Request flow visualization
  - Performance bottleneck identification
  - Dependency mapping

Infrastructure:
  - WebRTC signaling servers on Kubernetes
  - TURN/STUN servers for NAT traversal
  - Media servers for group calls
  - Load balancing for 1000+ concurrent users

Features:
  - Peer-to-peer video/audio calls
  - Screen sharing capabilities
  - Recording and playback
  - Integration with trading platform

Performance:
  - Low latency (<100ms)
  - Adaptive bitrate streaming
  - Network resilience
  - Quality monitoring