Why You Should Build Your Test Environment in the Cloud (+ How To Copy Your Production Data)

One of the favorite use cases for the public cloud is using it as test environments for the stateful production workloads. On-premises testing may be challenged by problems like limited budget and time. Public cloud can give you what on-premises environments cannot – almost immediate and low-cost deployment of the test workloads and provisioning of additional CPU, RAM, and IOPS. You can create a replica of your production workloads in the cloud and perform tests, fixes, and evaluation much faster than in a traditional environment.

Using cloud enables you to perform any type of test you want. Typical test environments are built for:

Performance testing – checks how the system behaves and performs in overall. It examines speed, stability, responsiveness, reliability, and usage of resources to help with finding bottlenecks and limitations of your software and architecture.

Load testing – checks how the system works under heavy load of concurrent users over a period of time. It helps you to determine how many users can be handled by your system and enables you to measure system response over heavy traffic.

Stress testing – a form of very intense and beyond the normal operational capacity testing of a given system. It helps to determine upper limits of the tested system under extreme loads and check how it recovers when going back to normal usage.

Functional testing – this is a process of verification against specification and system requirements.

Compatibility testing – helps to test application under different operation systems

Browser performance testing – verifies how application supports and performs in various browser types

Latency testing – to measure the latency between action and response

Performance, load and stress testing are usually performed using simulators to create peak loads. The most demanding and creating on-premises test environments for such tests is often enormously expensive. Public cloud can be a good alternative which is both, affordable and scalable. In the cloud, you can easily create large enough environment to perform a test, but not only. You can easily scale it out and up to determine how your application scales with the environment growth without buying extra hardware, and when your tests are over, you just switch off the environment and stop paying. When you need to test again, you just pull the trigger to start the test environment up again. Needless to mention, that you can automate such deployment for example with AWS CloudFormation.

Moving your test environment to the cloud is just a first step, and there is another one to be taken.

Transfer your data

The test environment is nothing without data. One can have the biggest possible environment with plenty of resources deployed automatically in the cloud, but this environment is useless without data to be used for testing purposes. This means that data has to be transferred to the cloud somehow. Of course, assuming that all the information for tests has already been anonymized, pseudonymized or whatever you need to do with it to prepare for being used in test environments. Let’s say it has been already done and all you need to do yet is to transfer data to the cloud. Depending on the data types and data amount there are a couple of possibilities how to do it.

AWS Database Migration Service

AWS DMS can migrate your data to and from most popular databases such as MySQL, PostgreSQL, Oracle, Microsoft SQL Server, MongoDB, MariaDB, Amazon Redshift, Amazon Aurora, Amazon DynamoDB and Amazon S3. DMS enables you to migrate data not only between the same database engines but also between different ones. That means you are able to migrate Oracle to MySQL for instance. The only condition is that one of your databases has to be on an AWS service. You cannot migrate from an on-premises database to another on-premises database.

One of the benefits of using DMS is that you can migrate your databases to AWS with virtually no downtime. All data changes to the source database that occur during the migration are replicated to the target database. Source database remains fully operational for the whole migration process. After the process completes, the target database remains synchronized with the source for as long as you choose, which allows you to switch to the target seamlessly at a selected time.

In overall, to migrate data with DMS, you have to:

• Provision a replication server
• Choose source and destination databases
• Create migration tasks

Replication server runs on an Amazon EC2 instance. It provides high-availability and failover support with Multi-AZ deployment. Replication server is used by AWS DMS to connect to the source database, read the source data, format source data and load it into the target database. When creating a replication server, you should consider what EC2 instance class and storage should it use. Smaller instances are often sufficient for testing purposes and small migrations. If you migrate a large number of tables or want to run multiple replication tasks, you should choose larger class instances. Regarding a disk, most of the migration processes happen in memory. However, large transactions may require buffering on disk. Cached operations and log files are also written to disk. Depending on the chosen instance class, replication server comes with either 50 GB or 100 GB of data storage which should be enough for any replication.

Three migration tasks are available:

• Migrate existing data to simply migrate the data from source to a target database. Use it when you can afford an outage long enough to copy data.
• Migrate existing data and replicate ongoing changes to perform a full data load including changes on the source. Captured changes are applied after the full load to the target is complete.
• Replicate data changes only, which allows you to capture and replicate only changes to the database, starting from a given point of time.

For any reasons, you may not want to use DMS to replicate data, for instance, when essential import/export tools are more efficient to load bulk data. However, you can still use DMS to replicate changes. The third type of the migration tasks enables you to start migration using native tools and at the very same time start to capture changes on the source database with DMS. Once replication is completed, you just sync changes.

For every case of database migration from on-premises to the cloud, you need a connectivity between them. VPN or AWS Direct Connect between your on-premises environment can be used or, if you don’t have or don’t want such connectivity, you can use the Internet for migration. In such case a public replication instance in a VPC is involved. This VPC, except replication instance, has to have an Internet Gateway and the target database.

Other options for transferring data

Except for the AWS Database Migration Service, there are other options to transfer data to the cloud. Especially if they are objects like files and not databases. What are those options?

• Sending files over Internet directly to the S3 buckets, which may be not very efficient, but may be suitable for small amount of files to be sent
• Leveraging the AWS Storage Gateway, which can automate transferring data to and from the cloud. Storage Gateway enables you to connect cloud resources as a file storage, block storage or a virtual tape library.

The above can be used for the relatively small amount of data. But what about petabyte or exabyte scale? Are there any options to transfer an enormous amount of data to the AWS? Sure. They usually are not used to transfer data to the test environments, but eventually, who knows. Those options are AWS Snowball/Snowball Edge and AWS Snowmobile.

AWS Snowball and Snowball Edge are petabyte-scale data transfer solution that uses secure appliances to transfer data to the cloud. It is simple, fast, secure and much cheaper than the high-speed Internet. The difference between a Snowball and Snowball Edge is that Edge has a compute power for select AWS capabilities. To order one, you just have to create a new data transfer job in the AWS console, wait for the device, transfer data and send it back to move your data to the S3. The whole process can be tracked via SNS, text messaging or directly in the console.

Alternatively, if you still have too much data to be transferred to use Snowball, you can order the AWS Snowmobile which is a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck. With Snowmobile you can transfer up to 100 petabytes of data. It should be enough to send your whole datacenter to the cloud.

Is it worth to use the cloud for test environments?

Moving to the cloud may give you a lot of benefits. It can automate and ease of deploying repeatable test environments. You can deploy one within minutes and remove it just after the tests are done. It saves time and money. You do not have to provide a big hardware to test your application against enormously heavy load which also saves money. So it looks like it is worth to move with the tests to the cloud or at least consider such movements. But remember to transfer some test data before you transfer your test systems. Testing applications without real data may not be what you want to do.