Lift’n’shift to Snowflake

You have chosen the Snowflake Data Cloud as your future platform and now you want to deliver the migration project on time and with as little risk as possible. To achieve this goal, you must dedicate a significant role to automation.

The most common approach to take full advantage of the Snowflake architecture and its capabilities is to lift ‘n’ shift the data from the source to the cloud and run data integration and transformation processes there. If there is one task that can be fully automated, it is moving data from legacy systems to Snowflake. However, it requires some time to prepare.

Extracting data from legacy sources is addressed with many challenges such as:

• Limited and short time frames for retrieving and extracting data because operational systems are performance sensitive.
• On-premises applications generate enormous amounts of data on a daily or even hourly basis.
• Each data source has its own unique characteristics, e.g., IBM DB2 Mainframe, SAP Hana, Microsoft SQL Server.
• Security-related challenges to access the data, e.g., VPN is quite common.

antFarm is designed with efficiency in mind. It solves many challenges of reading data sources quickly and generating files optimized for the fastest loading of data. Data is imported into the target in batches (bulk loads), supported by parallel execution and scalable architecture.

The entire process of data movement is fully automated. While preparing the data load, antFarm automatically:

• Retrieves the list of tables and their definition from the data source catalogue.
• Creates a metadata repository where the application source definitions, table lists, optimization rules, such as partitions, etc, are stored.
• Creates creation scripts for target tables according to the source table definitions.
• Converts data types based on the source and target databases, if required.
• Generates SQL queries to lift the data.

Since all processing is defined using standard SQL syntax, antFarm is very easy to use. It is a completely open solution, so it can be integrated into any data integration tool, and you can run any kind of custom SQL on source or target..

antFarm has become an indispensable tool in Snowflake data migration projects as there many use cases where it can help, for example:

• Implement an extract/staging area of the data warehouse with lift ‘n’ shift of an initial data set and daily increments from the legacy sources.
• Replicate data from the legacy data warehouse to reduce on-premises infrastructure costs and deliver compelling performance for complex analytic queries or to analyse data in near real-time without negatively impacting transactional application performance.
• Import data from CSV files into the Snowflake Data Cloud.
• Use it as a backbone for Snowflake POC projects as you can move data from 100+ on-premises tables to the target destination in a single day.

The last option is especially popular because decisions about your future data platform are non-trivial. A data platform is like the foundation of a house. Once laid down and you start building on top of it, it’s hard to change things without much loss. Therefore, you must choose wisely. Before you make a decision, you should experience a potential data platform firsthand, on your sets of data. And since Snowflake comes with great benefits (e.g. secure & governed access to your data, virtually unlimited scale, seamless access across clouds and near-zero maintenance) we’re sure you will not be disappointed by the results. Furthermore, we’re sure you’ll be blown away by the results you’ll get.