Where did all of your pictures go? They’re not where they used to be. Did you notice? You may not have – many people didn’t – and that was the idea. But your Instagram photos have indeed moved. All of them. Yes, every last one of your Instagram photos was relocated during the spring from Amazon Web Services’ public Elastic Compute Cloud (EC2) over to Facebook’s data centers (DCs), Instagram recently revealed.

We talk a lot around here about all of the many, many, many reasons organizations should move their data up into the cloud, but every once in a while there’s a case in which there is a good reason to go the other way. This is one of those cases.

Back in 2012 Facebook founder and CEO Mark Zuckerberg intimated that Instagram would eventually take advantage of Facebook’s incredible infrastructure. “We will try to help Instagram continue to grow by using Facebook’s strong engineering team and infrastructure,” Zuckerberg said at the time.

Many took that to mean that Instagram data would eventually be pulled from Amazon’s DCs and placed in Facebook’s, and they were right. It only made sense, of course. Facebook didn’t want to continue rolling along as a major Amazon cloud customer. So it secretly migrated all of your photos this past spring. Despite the fact that the engineers on the project ran into a number of unanticipated issues and had to think on the fly and develop smart workarounds, they managed to make it all happen while the service’s 200 million users continued using it.

“The users are still in the same car they were in at the beginning of the journey,” Instagram founder Mike Krieger told Wired, “but we’ve swapped out every single part without them noticing.”

The whole thing even has a cutesy project name: “Instagration.” And it was quite the undertaking. Facebook has of course moved many other properties it’s acquired over the years to its own facilities from wherever they were originally stored. Most of them, however, were much smaller, and the teams involved temporarily shut them down before moving them.

Instagram is not small. Instagram does not have a few users. Instagram could not be shut down. Instead, Instagram has a huge amount of data and a couple hundred million users who expect to be able to access it 24/7. There could be no disruption. Facebook won’t reveal precisely how many virtual machines were required to run Instagram on Amazon, but it did confirm that the number was in “the thousands.”

Why Did Facebook and Instagram Do It?

The move allows Instagram to more effectively take advantage of the impressive array of computing tools that have assisted Facebook in constructing the enormous social media empire it has become. Facebook’s engineers, meanwhile, got to create a blueprint for absorbing future acquired applications into its own data center infrastructure.

It’s not just Instagram and Facebook that benefits from this move, either. They’ve given the entire tech world an example of how to move their own data from cloud servers to internal DCs without suffering from a service interruption should they ever need to do so. They couldn’t have done it without Amazon’s Virtual Private Cloud service.

The project got underway around April 2013 when Facebook Vice President of Engineering Jay Parikh said the company had plans to move its $1 billion 2012 acquisition to its own facilities. Nearly an entire year was spent doing prep work, with the actual move taking place over the course of a month. There were eight engineers taking care of the Instagram infrastructure back in ‘13. Today there are 20.

How Did Facebook and Instagram Do It?

The migration seemed simple enough at first: set up a secure connection between Amazon’s Elastic Compute Cloud (EC2) and a Facebook data center and migrate services across the gap piece by piece,” Instagram engineers Rick Branson, Pedro Cahauati and Nick Shortway wrote in a blog post.

The biggest issue was that Facebook’s private IP space was in conflict with Amazon’s. In order to make the move possible, a single private computer network spanning the Facebook data center and the Instagram operation on Amazon’s cloud servers had to be built. To do that, it was necessary to first move Instagram to another part of the Amazon cloud. So Instagram wasn’t moved once, but twice.

Amazon’s Virtual Private Cloud (VPC) allowed the team to make a logical network covering both Amazon and Facebook data centers. Doing so put all control of the internet addresses being used to run Instagram into the hands of Facebook.

“Amazon’s VPC offered the addressing flexibility necessary to avoid conflicts with Facebook’s private network,” explained the engineers.

So that fixed the problem, right? Yes, it fixed that particular problem. Things were even trickier than that, though. In order to relocate Instagram from EC2 to VPC, they had to build that common network across the two environments, which Amazon doesn’t have a solution for. Facebook built its own instead, and named it “Neti.” VPC wasn’t a thing yet when Instagram was founded in 2010, but startups of today can avoid the extra step by just building on VPC from the beginning.

Krieger said that if he “was starting a new startup or service from scratch today, [he] would totally just start on VPC.”

In any case, when the time finally came to shift all of the software and data around, Krieger and company used another tool called Chef, which essentially lets users write automated “recipes” for loading and configuring anything digital onto a large number of machines. The recipes were created during the Amazon to Amazon move and followed during the Amazon to Facebook move.

The Result

The entire migration was finished at the end of April, around which time Instagram experienced an outage, but the company swears it was unrelated. Instagram is now on dedicated Facebook servers, and no one experienced any problems that would have alerted them to the fact that all of this was happening. In fact, things are more efficient now: Instagram uses one server for every three it used at Amazon and data fetching times have been reduced by a whopping 80 percent.

The picture-sharing service now has an easier and more efficient connection to Facebook DCs the world over. Instagram doesn’t share data with Facebook itself, though, which means users won’t see targeted ads on either service based upon their actions taken on the other.

“One of the things we had to do was really silo the information, which ends up being important for privacy and other reasons,” Krieger says.

With the move complete, the engineers involved have learned a few things: don’t be afraid to try crazy ideas, because they just might end up working; reuse the same processes and concepts whenever possible to avoid unnecessary complexities; and plan for as few changes as possible to support the new facilities.

Image Source: Slash Gear