<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Maikiwi Global Network Status - Incident history</title>
    <link>https://maikiwi-network.instatus.com</link>
    <description>Maikiwi Global Network</description>
    <pubDate>Tue, 7 May 2024 08:21:18 +0000</pubDate>
    
<item>
  <title>Asia-Pacific Regional Outage</title>
  <description>
    Type: Incident
    Duration: 3 hours and 58 minutes

    Affected Components: Event Aggregator (Ingress), Event Digest (Ingress), Notification (Egress), Seattle, WA, USA (us-west3), Route Propagation, Singapore (ap4), Hong Kong (ap2), Tokyo, Japan (ap1), Sydney, Australia (ap3), Osaka, Japan (ap5)
    May 7, 10:19:00 GMT+0 - Resolved - All outages have been restored. Please contact us as soon as possible if you deployed any custom workload with us in Osaka, otherwise, all managed services in Osaka will be seamlessly redirected to Tokyo. May 7, 10:04:35 GMT+0 - Identified - We identified a repeatedly tripped internal failsafe that has been adding bogus routes to Aria&#039;s global routing table. Expect to receive inconsistent routes in the meantime as we patch Aria on-the-fly.  May 7, 09:45:47 GMT+0 - Monitoring - We have brought all locations online except for Osaka which will be subjected to an extended maintenance period. We are also enabling a small portion of customers to interface with Caesar again to contain potential cascading failures. May 7, 09:17:42 GMT+0 - Monitoring - We rolled back a recent update to internal critical infrastructure in Hong Kong and Singapore and are proactively monitoring the result. Caesar Monitoring will be disabled globally in this moment and time. May 7, 08:38:28 GMT+0 - Monitoring - We implemented a fix for Tokyo and are incrementally routing in traffic again.  May 7, 08:34:25 GMT+0 - Identified - We have identified an additional outage in Singapore. May 7, 08:30:47 GMT+0 - Identified - We are commencing a cold restart of all services in Japan and removing routes destined for Tokyo in Hong Kong.  May 7, 08:25:19 GMT+0 - Investigating - We are currently investigating a simultaneous regional outage. May 7, 08:21:18 GMT+0 - Investigating - We are currently investigating this incident. 
  </description>
  <content:encoded>
    <![CDATA[<p><strong>Type:</strong> Incident</p>
    <p><strong>Duration:</strong> 3 hours and 58 minutes</p>
    <p><strong>Affected Components:</strong> , , , , , , , , , </p>
    <p><small>May <var data-var='date'> 7</var>, <var data-var='time'>10:19:00</var> GMT+0</small><br><strong>Resolved</strong> -
  All outages have been restored. Please contact us as soon as possible if you deployed any custom workload with us in Osaka, otherwise, all managed services in Osaka will be seamlessly redirected to Tokyo..</p>
<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>10:04:35</var> GMT+0</small><br><strong>Identified</strong> -
  We identified a repeatedly tripped internal failsafe that has been adding bogus routes to Aria&#039;s global routing table. Expect to receive inconsistent routes in the meantime as we patch Aria on-the-fly. .</p>
<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>09:45:47</var> GMT+0</small><br><strong>Monitoring</strong> -
  We have brought all locations online except for Osaka which will be subjected to an extended maintenance period. We are also enabling a small portion of customers to interface with Caesar again to contain potential cascading failures..</p>
<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>09:17:42</var> GMT+0</small><br><strong>Monitoring</strong> -
  We rolled back a recent update to internal critical infrastructure in Hong Kong and Singapore and are proactively monitoring the result. Caesar Monitoring will be disabled globally in this moment and time..</p>
<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>08:38:28</var> GMT+0</small><br><strong>Monitoring</strong> -
  We implemented a fix for Tokyo and are incrementally routing in traffic again. .</p>
<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>08:34:25</var> GMT+0</small><br><strong>Identified</strong> -
  We have identified an additional outage in Singapore..</p>
<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>08:30:47</var> GMT+0</small><br><strong>Identified</strong> -
  We are commencing a cold restart of all services in Japan and removing routes destined for Tokyo in Hong Kong. .</p>
<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>08:25:19</var> GMT+0</small><br><strong>Investigating</strong> -
  We are currently investigating a simultaneous regional outage..</p>
<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>08:21:18</var> GMT+0</small><br><strong>Investigating</strong> -
  We are currently investigating this incident..</p>
]]>
  </content:encoded>
  <pubDate>Tue, 7 May 2024 08:21:18 +0000</pubDate>
  <link>https://maikiwi-network.instatus.com/incident/clvw4fyim562815b8n2nkiay1iv</link>
  <guid>https://maikiwi-network.instatus.com/incident/clvw4fyim562815b8n2nkiay1iv</guid>
</item>

<item>
  <title>Aegis Migration (mainland China)  </title>
  <description>
    Type: Maintenance
    Duration: 5 hours

    Affected Components: Volumetric Protection, Rate Limiter, Seattle, WA, USA (us-west3), Route Propagation, Fremont, CA, USA (us-west2), Singapore (ap4), Sydney, Australia (ap3)
    May 3, 22:00:00 GMT+0 - Completed - Maintenance has completed successfully May 3, 17:00:01 GMT+0 - Identified - Maintenance is now in progress May 3, 17:00:00 GMT+0 - Identified - We are decommissioning all hosted Aegis instances in mainland China. Instead, potentially malicious traffic destined for mainland China will soon be routed to our APAC (Singapore, Australia) or US-West Hubs (Seattle, Fremont). We expect this migration to take no more than 5 hours, and during this period, we will occasionally pause all route propagation of Aria in the above-mentioned POPs. Please contact your customer success manager if your business requires uninterrupted route propagation. 
  </description>
  <content:encoded>
    <![CDATA[<p><strong>Type:</strong> Maintenance</p>
    <p><strong>Duration:</strong> 5 hours</p>
    <p><strong>Affected Components:</strong> , , , , , , </p>
    <p><small>May <var data-var='date'> 3</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Completed</strong> -
  Maintenance has completed successfully.</p>
<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>17:00:01</var> GMT+0</small><br><strong>Identified</strong> -
  Maintenance is now in progress.</p>
<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>17:00:00</var> GMT+0</small><br><strong>Identified</strong> -
  We are decommissioning all hosted Aegis instances in mainland China. Instead, potentially malicious traffic destined for mainland China will soon be routed to our APAC (Singapore, Australia) or US-West Hubs (Seattle, Fremont). We expect this migration to take no more than 5 hours, and during this period, we will occasionally pause all route propagation of Aria in the above-mentioned POPs. Please contact your customer success manager if your business requires uninterrupted route propagation..</p>
]]>
  </content:encoded>
  <pubDate>Fri, 3 May 2024 17:00:00 +0000</pubDate>
  <link>https://maikiwi-network.instatus.com/maintenance/clvnjoe3i138674znc63fku7a9</link>
  <guid>https://maikiwi-network.instatus.com/maintenance/clvnjoe3i138674znc63fku7a9</guid>
</item>

<item>
  <title>SSL/TLS Certificate Issuing</title>
  <description>
    Type: Incident
    Duration: 12 hours and 32 minutes

    Affected Components: Certificate Authority (Internal), Event Digest (Ingress), Certificate Issuance
    Mar 4, 02:34:50 GMT+0 - Resolved - This incident has been resolved. Mar 3, 14:49:54 GMT+0 - Monitoring - We have deployed a revision to our CA and a public ACME client as a fallback in case further issues occur. Since exponential backoff was observed for many endpoints, this incident will be continually monitored until the revised build has successfully reissued all active certificates. Mar 3, 14:39:17 GMT+0 - Identified - We are continuing to work on a fix for this incident. Mar 3, 14:07:13 GMT+0 - Investigating - We are investigating a backlog of certificate signing requests that are failing internal challenges. This will impact all customer endpoints and Caesar sessions configured with 1-day expiration certificates.  Mar 3, 14:02:41 GMT+0 - Investigating - We are currently investigating this incident. 
  </description>
  <content:encoded>
    <![CDATA[<p><strong>Type:</strong> Incident</p>
    <p><strong>Duration:</strong> 12 hours and 32 minutes</p>
    <p><strong>Affected Components:</strong> , , </p>
    <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>02:34:50</var> GMT+0</small><br><strong>Resolved</strong> -
  This incident has been resolved..</p>
<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>14:49:54</var> GMT+0</small><br><strong>Monitoring</strong> -
  We have deployed a revision to our CA and a public ACME client as a fallback in case further issues occur. Since exponential backoff was observed for many endpoints, this incident will be continually monitored until the revised build has successfully reissued all active certificates..</p>
<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>14:39:17</var> GMT+0</small><br><strong>Identified</strong> -
  We are continuing to work on a fix for this incident..</p>
<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>14:07:13</var> GMT+0</small><br><strong>Investigating</strong> -
  We are investigating a backlog of certificate signing requests that are failing internal challenges. This will impact all customer endpoints and Caesar sessions configured with 1-day expiration certificates. .</p>
<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>14:02:41</var> GMT+0</small><br><strong>Investigating</strong> -
  We are currently investigating this incident..</p>
]]>
  </content:encoded>
  <pubDate>Sun, 3 Mar 2024 14:02:41 +0000</pubDate>
  <link>https://maikiwi-network.instatus.com/incident/cltbl0lh7210419bdohjgxvqgil</link>
  <guid>https://maikiwi-network.instatus.com/incident/cltbl0lh7210419bdohjgxvqgil</guid>
</item>

<item>
  <title>Spikes in Traffic Latency</title>
  <description>
    Type: Incident
    Duration: 29 minutes

    Affected Components: Fremont, CA, USA (us-west2), Tokyo, Japan (ap1)
    Mar 1, 16:54:07 GMT+0 - Resolved - This incident has been resolved. Mar 1, 16:24:45 GMT+0 - Monitoring - We have implemented a fix for Inconsistent QoS Categorization. Customers who are affected will receive prorated bandwidth credit. 
  </description>
  <content:encoded>
    <![CDATA[<p><strong>Type:</strong> Incident</p>
    <p><strong>Duration:</strong> 29 minutes</p>
    <p><strong>Affected Components:</strong> , </p>
    <p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>16:54:07</var> GMT+0</small><br><strong>Resolved</strong> -
  This incident has been resolved..</p>
<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>16:24:45</var> GMT+0</small><br><strong>Monitoring</strong> -
  We have implemented a fix for Inconsistent QoS Categorization. Customers who are affected will receive prorated bandwidth credit..</p>
]]>
  </content:encoded>
  <pubDate>Fri, 1 Mar 2024 16:24:45 +0000</pubDate>
  <link>https://maikiwi-network.instatus.com/incident/clt8v7l5y2326bhn52ct552ie</link>
  <guid>https://maikiwi-network.instatus.com/incident/clt8v7l5y2326bhn52ct552ie</guid>
</item>

  </channel>
  </rss>