tag:www.cloudbeesstatus.com,2005:/historyCloudBees Status - Incident History2024-03-28T15:48:45ZCloudBeestag:www.cloudbeesstatus.com,2005:Incident/175020552023-06-07T16:11:19Z2023-06-07T16:11:19ZIssues with authentication<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>16:11</var> UTC</small><br><strong>Resolved</strong> - We have verified our fix is working.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>15:49</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>15:49</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>14:32</var> UTC</small><br><strong>Update</strong> - We have identified the root cause of the issue and working towards a fix. Logins are currently working.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>14:18</var> UTC</small><br><strong>Investigating</strong> - We are receiving errors affecting logins. We are currently investigating with our provider.</p>tag:www.cloudbeesstatus.com,2005:Incident/175020212023-06-07T14:44:44Z2023-06-07T14:44:45ZCan't log in to the CloudBees Feature Management web interface.<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>14:44</var> UTC</small><br><strong>Resolved</strong> - A configuration setting caused specific traffic to be rejected by our firewall, causing issues with authentication. The problem has been corrected and the incident is now resolved.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>14:34</var> UTC</small><br><strong>Update</strong> - Logins to the web interface are working again. We are continuing to investigate.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>14:16</var> UTC</small><br><strong>Investigating</strong> - We are aware that there are issues logging in to the CloudBees Feature Management web interface. We are investigating.</p>tag:www.cloudbeesstatus.com,2005:Incident/164777612023-03-13T16:52:00Z2023-03-13T16:52:01ZAuthentication latency and timeouts<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>16:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>16:45</var> UTC</small><br><strong>Monitoring</strong> - Our provider has implemented a fix and we are monitoring.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>16:10</var> UTC</small><br><strong>Investigating</strong> - We are seeing extreme latency and timeouts on our authentication service that is causing login failures. We are currently investigating this with our provider.</p>tag:www.cloudbeesstatus.com,2005:Incident/142230692022-12-03T04:30:00Z2022-12-03T04:59:05ZProblem updating configurations<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>04:30</var> UTC</small><br><strong>Resolved</strong> - Earlier today, for a few hours, we experienced an infrastructure problem which effected flag updates<br />This problem is now fixed, and all flags should be updated with the latest configurations</p>tag:www.cloudbeesstatus.com,2005:Incident/132636732022-11-17T14:43:25Z2022-11-17T14:43:25ZUnable to login to CloudBees Feature Management<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>14:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>11:28</var> UTC</small><br><strong>Monitoring</strong> - We have found the cause of the connectivity problems and applied a fix. The web interface and API are fully functional again and we will continue to monitor.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>10:34</var> UTC</small><br><strong>Identified</strong> - We have identified some connectivity (DNS) errors in our infrastructure and are working on a fix for this.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>09:59</var> UTC</small><br><strong>Update</strong> - On further investigation, some API requests are failing. We are continuing to investigate the cause and identify a solution!</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>09:24</var> UTC</small><br><strong>Update</strong> - The CloudBees Feature Management API is operating normally.<br />Whilst the web interface is unavailable, you can continue to make updates to Flags using the API: https://docs.cloudbees.com/docs/cloudbees-feature-management-rest-api/latest/introduction</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>09:20</var> UTC</small><br><strong>Investigating</strong> - We are aware of a problem logging into CloudBees Feature Management and are investigating the cause.<br />Flag use by SDK and delivery of your targeting is unaffected.</p>tag:www.cloudbeesstatus.com,2005:Incident/132708462022-11-17T14:43:14Z2022-11-17T14:43:15ZCloudBees Feature Management impression recording suspended<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>14:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>14:42</var> UTC</small><br><strong>Monitoring</strong> - We have identified the source of the problems that required taking analytics offline temporarily.<br />Analytics functionality is now restored.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>11:32</var> UTC</small><br><strong>Investigating</strong> - Following the earlier incident with the CloudBees Feature Management web interface, the impression analytics capability is temporarily offline whilst we investigate infrastructure problems.</p>tag:www.cloudbeesstatus.com,2005:Incident/128463012022-11-11T14:52:31Z2022-11-11T14:52:31ZFailures on User Login<p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>14:52</var> UTC</small><br><strong>Resolved</strong> - All systems healthy again, and this incident is now resolved.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>14:39</var> UTC</small><br><strong>Monitoring</strong> - We have identified the source of the issue and resolved it. User login is back to stable behavior, and currently monitoring before closing the incident.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>13:53</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>13:48</var> UTC</small><br><strong>Investigating</strong> - We have noticed sporadic failures in user login to our website app.cloudbees.com and Zendesk. Other apps such as app.rollout.io do not seem affected, but we're determining the extent of this issue.</p>tag:www.cloudbeesstatus.com,2005:Incident/110899832022-09-09T19:29:57Z2022-09-09T19:29:57ZCloudBees Feature Management<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>19:29</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>16:06</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>15:47</var> UTC</small><br><strong>Investigating</strong> - CloudBees Feature Management functionality is degraded due to an AWS outage. Affected are:<br /> - Delivery of flags configuration updates<br /> - GitHub Configuration-as-Code<br /> - Subscription management<br /><br />We, and Amazon are investigating.</p>tag:www.cloudbeesstatus.com,2005:Incident/110477012022-09-03T18:37:42Z2022-09-03T18:37:43ZScheduled authentication maintenance<p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>18:37</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>16:56</var> UTC</small><br><strong>Scheduled</strong> - Authentication may be unavailable during this window.</p>tag:www.cloudbeesstatus.com,2005:Incident/106543452022-07-20T19:19:29Z2022-07-20T19:19:29ZEmail-verification page having issues with re-send email button<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>19:19</var> UTC</small><br><strong>Resolved</strong> - Fix for the root cause has been rolled out to production. Email verification is working for all users again.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>16:54</var> UTC</small><br><strong>Investigating</strong> - We've found out that re-sending the email via the button in email-verify page is continuously throwing an error, impacting new users signing up who click it.<br /><br />We've identified the root cause and rolling out a fix.</p>tag:www.cloudbeesstatus.com,2005:Incident/103663392022-06-21T07:25:46Z2022-06-21T07:25:46ZCloudflare Service Issues<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>07:25</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>07:22</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and CloudFlare is monitoring the results.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>07:01</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>06:59</var> UTC</small><br><strong>Investigating</strong> - Cloudflare is investigating wide-spread issues with their services and/or network.<br /><br />Users may experience errors or timeouts reaching Cloudflare’s network or services.</p>tag:www.cloudbeesstatus.com,2005:Incident/100728412022-05-25T15:30:14Z2022-05-25T15:30:15ZCloudBees Feature Management Dashboard is not loading<p><small>May <var data-var='date'>25</var>, <var data-var='time'>15:30</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>15:14</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.cloudbeesstatus.com,2005:Incident/100295532022-05-19T15:48:54Z2022-05-19T15:48:54ZFlag impression metrics/analytics may stop counting for a brief time period<p><small>May <var data-var='date'>19</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>15:08</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>15:07</var> UTC</small><br><strong>Investigating</strong> - We are in the process of running a short production infrastructure test that may result in flag impression metrics/analytics not updating for a short time period. We do not expect this to last for more than 15 to 30 minutes. During this time all other services will function as normal. We apologize in advance for any inconvenience caused.</p>tag:www.cloudbeesstatus.com,2005:Incident/97593342022-04-11T15:06:12Z2022-04-11T15:06:13ZFeature flag configurations not propagating to SDK clients<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>15:06</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>14:59</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>14:15</var> UTC</small><br><strong>Identified</strong> - New feature flag configuration changes are not being propagated to SDK clients.</p>tag:www.cloudbeesstatus.com,2005:Incident/94870062022-03-08T09:24:41Z2022-03-08T09:24:41ZAdvisor GCP Migration<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>09:24</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>08:09</var> UTC</small><br><strong>Investigating</strong> - Advisor is under maintenance.</p>tag:www.cloudbeesstatus.com,2005:Incident/89236882021-12-24T05:35:17Z2021-12-24T05:35:18ZDevOptics intermittent issues<p><small>Dec <var data-var='date'>24</var>, <var data-var='time'>05:35</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>24</var>, <var data-var='time'>03:43</var> UTC</small><br><strong>Monitoring</strong> - We performed percussive maintenance on some of our infrastructure, and are now monitoring the system.</p><p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>22:50</var> UTC</small><br><strong>Investigating</strong> - We are investigating intermittent issues with the web interface for the DevOptics platform.</p>tag:www.cloudbeesstatus.com,2005:Incident/88552192021-12-15T16:19:12Z2021-12-15T16:19:14ZMajor provider outage<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>16:19</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>16:13</var> UTC</small><br><strong>Monitoring</strong> - Now monitoring major providers status</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>16:05</var> UTC</small><br><strong>Identified</strong> - Issue caused by major providers outage</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>15:57</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>15:47</var> UTC</small><br><strong>Investigating</strong> - There has been a provider outage and we're investigating right now. Currently our authentication service is down, affecting the web interface for most of our products.</p>tag:www.cloudbeesstatus.com,2005:Incident/87822772021-12-07T22:02:17Z2021-12-07T22:02:18ZFeature Management partially down<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>22:02</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Update</strong> - Some Feature Management services are beginning to return to normal. We are continuing to work on this.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>19:54</var> UTC</small><br><strong>Update</strong> - Some Feature Management services are beginning to return to normal. We are continuing to work on this.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>16:35</var> UTC</small><br><strong>Identified</strong> - Some Feature Management services are beginning to return to normal. Working to restore the rest.</p>tag:www.cloudbeesstatus.com,2005:Incident/86667972021-11-23T01:23:46Z2021-11-23T01:23:49ZCloudBees CI Update Center timing out<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>01:23</var> UTC</small><br><strong>Resolved</strong> - A fix for the underlying issue has been released and the system is now performing normally.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>00:15</var> UTC</small><br><strong>Identified</strong> - The root cause has been identified - a fix is being developed.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>23:32</var> UTC</small><br><strong>Investigating</strong> - The CloudBees CI update center (jenkins-updates.cloudbees.com) is timing out due to increased load.<br /><br />Engineers are working to resolve the issue.</p>tag:www.cloudbeesstatus.com,2005:Incident/81030792021-09-28T16:18:06Z2021-09-28T16:18:06ZAuth0 outage<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>16:18</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>16:10</var> UTC</small><br><strong>Update</strong> - EU is recovering and error rates are less than 1%. US-1 is still experiencing partial outage. Our team is working on mitigation actions and estimated time for resolution. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>16:09</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>16:04</var> UTC</small><br><strong>Investigating</strong> - Investigating elevated login error rate</p>tag:www.cloudbeesstatus.com,2005:Incident/80951542021-09-24T06:30:00Z2021-09-27T17:47:12ZSS0 Provider outage<p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>06:30</var> UTC</small><br><strong>Resolved</strong> - Our SSO provider identified issues and took action to mitigate them.</p>tag:www.cloudbeesstatus.com,2005:Incident/78665382021-08-29T07:26:18Z2021-08-29T07:26:18Zwww.cloudbees.com to GCP<p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>07:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>06:33</var> UTC</small><br><strong>Investigating</strong> - www.cloudbees.com is under maintenance.</p>tag:www.cloudbeesstatus.com,2005:Incident/76246512021-07-28T14:34:20Z2021-07-28T14:34:21ZFeature flags are not rendered properly in the UI<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>14:34</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>14:20</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>14:19</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>13:58</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.cloudbeesstatus.com,2005:Incident/71714482021-06-05T07:16:08Z2021-06-05T07:16:09ZFeature Management web interface offline<p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>07:16</var> UTC</small><br><strong>Resolved</strong> - We have resolved the underlying issue with Feature Management and restored all services.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>04:57</var> UTC</small><br><strong>Update</strong> - Engineering has implemented a fix for the Feature Management control interface (app.rollout.io)<br /><br />Impression analytics is offline and we continue to troubleshoot this service.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>03:58</var> UTC</small><br><strong>Update</strong> - Engineering has identified the root cause and is continuing to work on resolving the issue.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>02:27</var> UTC</small><br><strong>Identified</strong> - We have identified an issue with the Feature Management web interface and some associated services.<br /><br />Feature Flag delivery (to consumers) is fully operational.<br /><br />Engineers are working to correct the issue.</p>tag:www.cloudbeesstatus.com,2005:Incident/70259322021-05-18T01:35:20Z2021-05-18T01:35:21ZCloudBees Update Center and ancillary services offline<p><small>May <var data-var='date'>18</var>, <var data-var='time'>01:35</var> UTC</small><br><strong>Resolved</strong> - All services have been restored.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>00:09</var> UTC</small><br><strong>Identified</strong> - We are tracking an availability incident with our CloudBees Update Center and several ancillary services.<br /><br />Engineers are restoring service now.</p>