Monthly Archives: October 2016

Amazon Aurora: A production Horror story

We moved to Amazon Aurora to save costs, lower replication time and to help our database performance.  We planned to shut down read only $slaves during the evening. During the day we would add these read only slaves back.

Before Aurora “Cluster”

With vanilla mysql master-slave setup the code had to know which servers were read only slaves. Historically, we would specify a list of read only servers:

 ['hostnames'][] => '',
 ['hostnames'][] => '',
 ['hostnames'][] => '',
 ['hostnames'][] => '',

To then create a connection to a reader we would pick a read instance at random.

// Create connection
$rand = rand(0,count($hostnames['hostnames']) - 1); 
$conn = new mysqli($servername, $username, $password, $hostnames['hostnames][$rand]);

The problem 1: nothing expands and contracts

The specified servers for reading do not expand and contract.  If load dramatically increases, you would need to add new servers through the RDS console then add the host strings to your $hostnames. Due to size, adding servers typically would take us 2-4 hours.

On the other hand, reducing the load would required us to carefully drop machines and push code to remove the connection strings.

Solution, Amazon Aurora: Read only cluster

We were excited to see amazon aurora offered a read only connection string.  Instead of specifying the read db database strings:

 ['hostnames'][] => '',
 ['hostnames'][] => '',
 ['hostnames'][] => '',
 ['hostnames'][] => '',

We could now just use one read only connection string

Amazon Aurora Clusters Production

As you add Aurora slaves, they are added to the pool of machines for that one read only connection string.  You can click for details to see your one connection string.

Under cluster details you will get a connection string, sweet we now have it

 ['hostnames'][] => ''

Amazon Aurora: Let’s start dropping databases

Well. It was late at night load was light. Let’s see what happens if we nuke a reader. What could go wrong? We deleted a machine:


With the machine deleting, we thought since it was a ‘cluster’ amazon would just redirect the reads to another machine.  We continued to high-five.

Shit just got real.

Ok.  PageDuty page comes in.  Sweating. Second page comes inApparently, amazon continues to send read requests to a bad slave.


 Our site starts to timeout.

Screen Shot 2016-10-25 at 11.54.31 AM

Soon within 5 minutes, the site stabilized.

Thankfully, things cleared after 10 minutes.  So **warning** amazon aurora clusters are not meant to act like a cluster.

Purchasing a SIM card in ITALY

While using your American phone in Italy is simple, it is expensive. Unreliable Wi-Fi can make the reality of keeping in touch via a service like WhatsApp more frustrating than it should be.

Purchasing a SIM in Italy is a great way to avoid overage charges from your cell phone carrier. It will ensure you don’t get stuck with a bill and you can use up to 4G of data (10/2016).

If you anticipate using your phone data, consider traveling with a mobile phone fitted with a Italian SIM card.

What you will need
2) 20 Euros for 4G (10/2016) + Finding a TIM store
3) Your Phone
4) Your passport

I went with a SIM card from a company TIM for 20Euros with 4G. After around 20 texts, I was unable to text. HOWEVER, I was able to keep my 4G of data. With that I was able to load slack and whatsapp.

1) Find a TIM store
2) They will require your passport.
3) Demand they put it in the phone WHEN YOU ARE IN THE STORE. Don’t walk far from this store for 60 minutes.
4) After 30 minutes it should register. After 30 minutes restart your phone. IF it still does not work, go back to the store.

SIM registration is required by the Italian government prior to service activation. This can be done using your passport when purchasing a SIM card. More information on the regulatory requirements can be found here at

Below is a picture of what you will be given.  Ask them to TAPE YOUR CURRENT sim card to this card.

Slack for iOS Upload-4