CloudWatch outage stalled delivery slot service due to Docker's blocking log mode

In February a delivery slot service on four EC2 hosts stopped responding for 50 minutes: degraded CloudWatch Logs slowed log writes, and the default blocking awslogs driver left every request thread stuck writing to stdout. The team switched hosts to non-blocking mode with an 8 MB buffer and plans to move the service to a cluster that writes logs to local files.

• The outage lasted 50 minutes and all four hosts failed health checks
• Blocking awslogs mode stalls stdout writes once the 64 KB pipe fills
• Hosts now run non-blocking mode with an 8 MB buffer
• Health check no longer logs; staging blocks the CloudWatch route twice a year