New app deploys and new container starts may fail

Incident Report for Meteor Cloud

Resolved

We have not observed any further issues relating to this morning's Amazon S3 outage, so we are tentatively marking this as resolved. Please file a support ticket if you observe any further strange behavior, either relating to this outage or today's unrelated infrastructure upgrade!

Posted Feb 28, 2017 - 20:10 EST

Monitoring

Amazon's outage seems to have mostly subsided. Deployments and container starts are now operational. We are monitoring to ensure that everything continues to work well. Please contact support if you see any further issues.

Posted Feb 28, 2017 - 17:32 EST

Identified

Amazon believes they will have improved the situation within another hour, according to the AWS status page.

Posted Feb 28, 2017 - 16:03 EST

Update

Deploys in the EU cluster appear to fail because our image builder depends on base images stored in Docker Hub, which is also affected by the issue (http://status.docker.com/pages/incident/533c6539221ae15e3f000031/58b5b8c850b734490e0001f2). Starting new containers for existing apps does appear to work in EU.

Posted Feb 28, 2017 - 13:37 EST

Investigating

Due to ongoing issues in Amazon's S3 system, Galaxy operations that need to read to or write from S3 may fail now. This includes deploying new app versions and starting new containers. Existing containers are unlikely to be affected. We are investigating whether this affects our EU cluster or just our US cluster.

Posted Feb 28, 2017 - 13:15 EST