Breakpoint

Payment Service Cascade Failure

Orders are failing intermittently during checkout. The payment service is returning 500 errors, but only for some users. Customer support is overwhelmed with complaints about failed purchases.

System Overview

The e-commerce platform has an Order Service that calls a Payment Gateway Service, which in turn calls a third-party payment processor. A Circuit Breaker sits between the Payment Gateway and the external processor. There is also a Notification Service that sends order confirmation emails.

Logs


[2024-03-15 09:00:12] INFO: Order #ORD-4821 created for user 'alice@example.com'
[2024-03-15 09:00:12] INFO: Calling payment-gateway for Order #ORD-4821, amount: $149.99
[2024-03-15 09:00:13] ERROR: Payment failed for Order #ORD-4821: upstream_error
[2024-03-15 09:00:13] INFO: Order #ORD-4821 status set to FAILED
[2024-03-15 09:01:30] INFO: Order #ORD-4822 created for user 'bob@example.com'
[2024-03-15 09:01:30] INFO: Calling payment-gateway for Order #ORD-4822, amount: $29.99
[2024-03-15 09:01:30] INFO: Payment successful for Order #ORD-4822
[2024-03-15 09:01:30] INFO: Order #ORD-4822 status set to CONFIRMED
[2024-03-15 09:02:45] INFO: Order #ORD-4823 created for user 'charlie@example.com'
[2024-03-15 09:02:45] INFO: Calling payment-gateway for Order #ORD-4823, amount: $599.00
[2024-03-15 09:02:46] ERROR: Payment failed for Order #ORD-4823: upstream_error
[2024-03-15 09:03:10] INFO: Order #ORD-4824 created for user 'diana@example.com'
[2024-03-15 09:03:10] INFO: Calling payment-gateway for Order #ORD-4824, amount: $75.50
[2024-03-15 09:03:11] ERROR: Payment failed for Order #ORD-4824: upstream_error

Diagnosis & Plan

Based on the logs, what do you think the root cause is, and what are your proposed next steps?