DEV@cloud builds not scheduling
Incident Report for CloudBees
Postmortem

The service that allocates jobs to the build farm experienced an unexplained fault (authentication failures to another service that was operating correctly). Once this service was restarted, builds were then allocated correctly.

The recovery period was longer than normal due to the monitoring system not alerting support via the correct channel (i.e. page).

We've increased the alert level so that we are paged more quickly to resolve this issue (builds queueing and not being allocated) if it occurs again.

Posted Mar 16, 2016 - 01:58 UTC

Resolved
All builds are proceeding as normal.
Posted Mar 16, 2016 - 01:54 UTC
Monitoring
We have restarted the faulty component and builds are now proceeding. Throughput will be reduced while we catch up and replace dead build servers.
Posted Mar 15, 2016 - 22:56 UTC
Investigating
Builds that are scheduled to run on mansion provided executors are not currently allocating out.

Investigations are underway.
Posted Mar 15, 2016 - 22:41 UTC